CN113761103A

CN113761103A - Batch data processing method and device and electronic equipment

Info

Publication number: CN113761103A
Application number: CN202011596568.6A
Authority: CN
Inventors: 殷宏磊; 赵敏; 李明; 詹仙园; 郑宇�; 霍雨森
Original assignee: Jingdong City Beijing Digital Technology Co Ltd
Current assignee: Jingdong City Beijing Digital Technology Co Ltd
Priority date: 2020-12-29
Filing date: 2020-12-29
Publication date: 2021-12-07

Abstract

The application discloses a batch data processing method and device and electronic equipment, wherein the method comprises the following steps: acquiring batch metadata to be processed; analyzing the batch metadata to determine the service scene to which the batch metadata belongs, the acquisition time of each metadata and the data type; determining a first data processing mode according to the service scene, the acquisition time of each metadata and the data type; the batch metadata is processed based on a first data processing mode. Therefore, the batch metadata are processed according to the service scenes to which the batch metadata belong, the acquisition time of each metadata and the data type, so that the batch metadata are processed aiming at different service scenes, the reliability of data processing is improved, manual processing links are reduced, and the labor cost is reduced.

Description

Batch data processing method and device and electronic equipment

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for processing batch data, an electronic device, a storage medium, and a computer program product.

Background

With the development of big data and artificial intelligence, the relationship between the big data and the artificial intelligence is more and more intimate, and the big data and the artificial intelligence can not be supported by mass data when falling to the ground. Large data in the energy industry domain refers to the large amount of structured, semi-structured, and unstructured data obtained from various heterogeneous sources. This data is generally considered to have valuable information hidden in it, but requires a great deal of effort and resources to acquire and process.

In the field of energy industry, the method has high requirements on the authenticity and reliability of application scene production data sets. Therefore, how to reliably produce a data set meeting the requirements when acquiring original data under different application scenes is a key point for improving the intelligent construction of energy industrial control at present. However, in the related art, batch data processing for different service scenarios is lacked, so that the reliability of batch data processing is low.

Disclosure of Invention

The application provides a batch data processing method and device and electronic equipment.

According to a first aspect of the present application, there is provided a method for processing batch data, including:

acquiring batch metadata to be processed;

analyzing the batch metadata to determine the service scene to which the batch metadata belongs, the acquisition time of each metadata and the data type;

determining a first data processing mode according to the service scene, the acquisition time of each metadata and the data type;

processing the batch metadata based on the first data processing mode.

According to a second aspect of the present application, there is provided an apparatus for processing batch data, comprising:

the system comprises a first acquisition module, a second acquisition module and a processing module, wherein the first acquisition module is used for acquiring batch metadata to be processed;

the first determining module is used for analyzing the batch metadata to determine the service scene to which the batch metadata belongs, the acquisition time of each metadata and the data type;

the second determining module is used for determining a first data processing mode according to the service scene, the acquisition time of each metadata and the data type;

and the first processing module is used for processing the batch metadata based on the first data processing mode.

According to a third aspect of the present application, there is provided an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the batch data processing method of the embodiment of the above aspect.

According to a fourth aspect of the present application, there is provided a non-transitory computer-readable storage medium storing thereon a computer program for causing a computer to execute the method for processing batch data according to the embodiment of the above-described aspect.

According to a fifth aspect of the present application, there is provided a computer program product, which when executed by a processor, implements the method for processing batch data according to the embodiment of the above-mentioned aspect.

According to the technical scheme, the batch metadata are processed according to the service scenes to which the batch metadata belong, the acquisition time of each metadata and the data type, so that the batch metadata are processed aiming at different service scenes, the reliability of data processing is improved, manual processing links are reduced, and the labor cost is reduced.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present application, nor do they limit the scope of the present application. Other features of the present application will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:

fig. 1 is a schematic flowchart of a batch data processing method according to an embodiment of the present disclosure;

fig. 2A is a schematic diagram of data before format normalization processing according to an embodiment of the present application;

fig. 2B is a schematic diagram of data after format normalization processing according to an embodiment of the present application

Fig. 3 is a schematic diagram of configuration information Data _ bound.csv according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram of configuration information Data _ config.csv according to an embodiment of the present application;

FIG. 5 is a schematic diagram illustrating comparison between before and after data padding according to an embodiment of the present application;

fig. 6 is a schematic flowchart of data processing based on a first data processing mode according to an embodiment of the present application;

FIG. 7 is a diagram of a data system according to an embodiment of the present application;

fig. 8 is a schematic flowchart of another data processing based on a first data processing mode according to an embodiment of the present application;

fig. 9 is a schematic flowchart of an index search performed on a data processing result according to an embodiment of the present application;

FIG. 10 is a schematic diagram of a batch data processing pipeline provided by an embodiment of the present application;

FIG. 11 is a diagram illustrating a final data processing result according to an embodiment of the present disclosure;

FIG. 12 is a schematic structural diagram of an apparatus for processing batch data according to an embodiment of the present disclosure;

fig. 13 is a block diagram of an electronic device for implementing a batch data processing method according to an embodiment of the present application.

Detailed Description

The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

The following describes a batch data processing method, a batch data processing device, and an electronic device according to an embodiment of the present application with reference to the drawings.

Fig. 1 is a schematic flow chart of a batch data processing method according to an embodiment of the present disclosure.

It should be noted that the execution main body of the batch data processing method in the embodiment of the present application may be an electronic device, specifically, the electronic device may be, but is not limited to, a server and a terminal, and the terminal may be, but is not limited to, a personal computer, a smart phone, an IPAD, and the like.

The embodiment of the present application is exemplified by a method for processing batch data being configured in a device for processing batch data, and the device can be applied to an electronic device, so that the electronic device can execute the method for processing batch data.

As shown in fig. 1, the method for processing batch data includes the following steps:

s101, acquiring batch metadata to be processed.

In the embodiment of the present application, any original batch data that needs to be processed in any business scenario may be referred to as to-be-processed batch metadata. The service scenarios may include, but are not limited to, industrial complex systems, such as models for combustion optimization, coal pulverizer optimization, cold end optimization, soot blowing optimization, and the like in thermal power generation, and also include multiple service scenarios such as digital twin technology application in complex industrial systems.

Wherein the number of metadata may be plural, which may be structured data, semi-structured data, and/or unstructured data.

Specifically, if the batch metadata needs to be processed, the batch metadata to be processed is acquired. Wherein the batch metadata may be collected by a sensor.

S102, analyzing the batch metadata to determine the service scene to which the batch metadata belongs, the acquisition time of each metadata and the data type.

It should be noted that data types of the batch metadata in different service scenarios may be different, for example, the data types in an industrial environment monitoring scenario may include temperature data, humidity data, and the like, the data types in an industrial electricity statistics scenario may include voltage data, current data, and the like, and the data types in an equipment operation state monitoring scenario may include operation time, operation efficiency, and the like.

Specifically, after the batch metadata to be processed is acquired, the batch metadata may be analyzed through an analysis method in the related art, such as statistics, combination, analysis, and the like, so as to determine a service scenario to which the batch metadata belongs, an acquisition time of each metadata, and a data type. Wherein the determination of the parsing may be related to a service scenario.

It should be noted that, in the embodiment of the present application, the manner of analyzing the batch metadata may also be other manners in the related art, and it is only necessary to intelligently determine the service scenario to which the batch metadata belongs, the acquisition time of each metadata, and the data type, which is not limited in the embodiment of the present application.

S103, determining a first data processing mode according to the service scene, the acquisition time of each metadata and the data type.

The first data processing mode may refer to a mode for processing the batch metadata, which may include a processing manner (e.g., merging, statistics, filtering, etc.) and a processing time.

Specifically, after the service scene to which the batch metadata belongs, the acquisition time and the data type of each metadata are determined, a first data processing mode is determined according to the service scene, the acquisition time and the data type of each metadata.

Specifically, the service scene, the acquisition time of each metadata, and the first data processing mode corresponding to the data type can be obtained by querying a preset data diagram, a data table, or a data text, and the like, wherein the data diagram, the data table, or the data text can record the corresponding relationship between the service scene and the data processing mode, between the acquisition time of the data and the data processing mode, and between the data type and the data processing mode.

S104, processing the batch metadata based on the first data processing mode

Specifically, after the first data processing mode is determined, the batch data is processed based on the first data processing mode.

For example, for temperature monitoring data collected in the same service scene or environment, partial data can be merged according to the position relationship of monitoring points, a specific merging mode can be related to the service scene, if the service scene needs to be predicated based on data by using a model, the merging mode can be selected according to the type of data input by the model, so as to merge the data according to the merging mode; if the business scene only requires pure statistical data for later application, the metadata can be counted according to the corresponding acquisition time and can be stored for subsequent calling.

By executing the steps S101 to S104, when the raw data in different application scenarios or service scenarios is acquired, the raw data can be processed for different application scenarios or service scenarios, so as to quickly and accurately generate a data set that meets the actual requirements of the service scenarios (for example, required by a relevant algorithm model).

After the batch metadata is processed, the processed batch metadata may be called according to the requirement of the business scenario, for example, the processed batch metadata may be input into a corresponding algorithm model to perform model prediction.

According to the batch data processing method, the batch metadata are processed according to the service scenes to which the batch metadata belong, the acquisition time of each metadata and the data type, so that the batch data are processed aiming at different service scenes, the reliability of data processing is improved, manual processing links are reduced, and the labor cost is reduced.

The first data processing mode in step S103 may be an execution order of the data processing rule and each processing rule, or may be a processing time of the data processing rule and each processing rule.

In an embodiment of the application, when the first data processing mode is a data processing rule and an execution sequence between the data processing rules, the determining the first data mode in step S103 may include: and determining the current data processing rule and the execution sequence among the processing rules.

According to the embodiment of the application, multiple data processing rules with wide coverage range and execution sequences among the processing rules can be configured and stored in advance, and calling can be performed when the data processing rules are used subsequently.

It should be noted that the number of the determined data processing rules may be one or more, and when the number is one, only the current data processing rule is determined; when the number is multiple, the current data processing rule and the execution sequence among the processing rules need to be determined.

Specifically, after the service scenario to which the batch metadata belongs, the collection time of each metadata, and the data type are determined, the current data processing rule and the execution sequence among the processing rules may be selected from the multiple data processing rules according to the service scenario, the collection time of each metadata, and the data type. Wherein the selection basis may be specifically determined based on actual requirements. Then, the step S104 can be executed to process the batch metadata based on the current data processing rule and the execution sequence among the processing rules, so as to implement the processing of the batch data.

For example, assuming that the service scenario is a boiler temperature monitoring scenario and each monitored temperature data needs to be input into an algorithm model for model prediction to determine the operating condition of the boiler, two processing rules, namely "data format normalization" and "multiple data merging", may be selected, and an execution sequence between "data format normalization" and "multiple data merging", such as "data format normalization" in a restrictive manner, may be further selected and then "multiple data merging" is executed. And then, carrying out format normalization on each monitored temperature data, and then combining a plurality of data, thereby realizing the processing of each monitored temperature data.

Therefore, the batch metadata is processed based on the current data processing rule and the execution sequence among the processing rules, and the accuracy of data processing is improved.

The method comprises the following steps of analyzing and summarizing specific problems occurring in the data acquisition process of a plurality of sensors: the data collected by the sensor is stored in the data storage, and noise data caused by sensor failure, data transmission errors and the like exist. The noise data includes shutdown data that fails to properly display an industrial scene, or abnormal, missing, and invalid data that occurs under normal operation. In the embodiment of the present application, a data processing rule may be configured for the noise data.

That is, in one embodiment of the present application, the data processing rules may include at least one of the following rules: format normalization, abnormal data elimination based on boundary limit values, data elimination based on specified time intervals, merging of multiple monitoring point data based on homogeneous data, abnormal data elimination based on service scenes and missing data filling based on known data.

The format normalization can be used for converting different types of data formats into a unified data format, the abnormal data elimination based on the boundary limit value can be used for eliminating abnormal data exceeding the boundary limit value (the boundary upper limit and the boundary lower limit), the data elimination based on the specified time interval can be used for eliminating repeated data in the specified time interval, the combination of a plurality of monitoring point data based on the same type of data can be used for combining a plurality of monitoring point data with similar meanings, the abnormal data elimination based on the service scene can be used for eliminating abnormal data corresponding to the service scene, and the missing data filling based on the known data can be used for filling different missing types of data on the basis of the existing data.

Specifically, after the service scenario to which the batch metadata belongs, the collection time of each metadata, and the data type are determined, the current data processing rule and the execution sequence among the processing rules may be selected from the multiple data processing rules according to the service scenario, the collection time of each metadata, and the data type. The data processing rules can include at least one of format normalization, abnormal data elimination based on boundary limit values, data elimination based on specified time intervals, merging of multiple monitoring point data based on homogeneous data, abnormal data elimination based on service scenes and missing data filling based on known data.

Then, the step S104 is executed, that is, the batch metadata is processed based on the current data processing rule and the execution sequence among the processing rules, and the step S104 is described below based on the data processing rules:

specifically, when the data processing rule includes format normalization, normalization processing may be performed on sensor data (metadata) in different formats, such as text data, image data, or formats of single sensor measurement point data, for example, effective feature extraction may be performed on the text data and the image data according to requirements of a business scene, and the text data and the image data may be converted into single sensor data. It should be noted that different data processing modes can be set according to different data types, such as: and converting the data information in the picture into uniform standard data by using an image digital identification technology.

In the format normalization, the same type of data may be subjected to format normalization, for example, time data and weight data are unified into the same format.

For example, if statistics are taken of fly ash combustibles and slag combustibles in the coal mill plant, the comparison of metadata before and after the format normalization process can be as shown in fig. 2A and 2B, with reference to fig. 2A, before the format normalization process, the metadata includes time data, load data, fly ash combustibles data and slag combustibles data, and at this time, the time data can be normalized to the same format, the load data can be normalized to the same format, and the fly ash combustibles and slag combustibles data can be normalized to the same format. Referring to fig. 2B, the format of time data after format normalization may be "xxx x yearly xxx x monthly xxx", the format of the related value (value) may be "xx ×: xxx", and the format of the related quality (quality) data may be "xx.

It should be noted that, in the embodiment of the present application, a data processing configuration file may also be determined according to a plurality of pieces of batch metadata (monitoring point data), where the following data processing rules: abnormal data elimination based on boundary limit values, data elimination based on specified time intervals, merging of multiple monitoring point data based on homogeneous data, abnormal data elimination based on service scenes and missing data filling based on known data can be executed through a data processing configuration file.

That is, when the Data processing rule includes the abnormal Data elimination based on the boundary limit value, the abnormal Data exceeding the boundary limit value may be eliminated or deleted according to the upper and lower boundaries in the configuration information Data _ bound.

When the data processing rule includes data elimination based on a specified time interval, the single sensors can be merged according to the specified time interval, repeated data is eliminated, and only the data of the same sensor at the same time point is reserved, so that the data of the single sensors in multiple time periods can be merged into one file. And resampling operation can be performed on the data according to the configuration file.

When the data processing rule includes merging of a plurality of monitoring point data based on the same kind of data, wherein the plurality of monitoring point data may be measured by a plurality of single sensors, in order to better utilize the data, when a plurality of sensor monitoring point data belonging to the same kind of data (data having similar meaning) exist in the data, merging operation is performed on the plurality of sensor monitoring point data to merge a plurality of single sensor data files into a single multi-sensor data file, thereby reducing data dimensionality and reducing influence caused by single sensor abnormality. Wherein multiple sensor data may be combined over time, while multiple sensor monitor point data are combined to form one combined point based on groupings between sensors (e.g., grouping based on sensor type). Furthermore, when a plurality of monitoring points are combined, 2 monitoring point data can be combined and averaged, 3 monitoring point data can be taken as a median, and the data of more than 3 monitoring points are combined and averaged after the maximum and minimum values need to be removed. The Data processing rule can be realized by configuration information Data _ config.csv in a Data configuration file, an example of the Data _ config.csv can be shown in fig. 4, the Data _ config.csv also comprises a Data characteristic dimension (kks) contained in a generated characteristic, and the rule merges a plurality of monitoring point Data representing the same information to generate new characteristic point Data.

When the data processing rule includes abnormal data elimination based on a service scene, setting a related judgment rule of invalid data or abnormal data according to service knowledge (such as common sense or common working condition) under different service scenes, for example, when the current of a main motor of a coal mill is greater than 200A (amperes) at the starting time of the coal mill, deleting the whole main motor current data; in a treatment scene related to the boiler efficiency, when the boiler efficiency exceeds 95%, deleting the whole boiler efficiency data; when the negative pressure of the hearth is beyond +/-300, the hearth negative pressure data are deleted wholly under abnormal working conditions. In addition, the abnormal condition of the multi-sensor data can be judged according to different data requirements. Such as: and if the coal feeding amount of each coal mill is less than 1 and the total coal feeding amount is less than 1, the data belong to invalid data and the whole data are deleted. The data processing rule may be implemented by configuration information Shut _ down.

When the data processing rule includes missing data padding based on known data, padding operations are performed on data of different types of missing on the basis of existing data. For example, data missing caused by sensor damage or abnormality, data missing caused by abnormality in the process of data transmission and storage, and the like, and filling processing is performed. The data processing rule may be implemented by configuration information Mu1ti _ point.

For example, as can be seen from the comparison between before and after data padding in fig. 5, the left diagram has a phenomenon of data missing, and the data missing in the left diagram is padded by the data padding, so that the data shown in the right diagram after the padding is obtained.

Based on the above data processing rules, in order to reliably eliminate the noise data to further improve the accuracy of data processing, the data processing rules may include all the above rules, and at this time, the execution flow or execution sequence of the data processing rules may be:

first, format unification.

And secondly, removing abnormal data based on the boundary limit value.

And thirdly, removing data based on the specified time interval.

And fourthly, merging a plurality of monitoring point data based on the same type of data.

And fifthly, eliminating abnormal data based on the service scene.

And sixthly, filling missing data based on the known data.

When data of a sensor monitoring point is abnormal or missing due to damage or deterioration of a sensor, processing (namely, rejecting) is performed in a third step and a fifth step to reject all missing or abnormal data, and meanwhile, padding under the condition of data missing is performed through a padding algorithm in a sixth step.

Therefore, various data processing rules are configured, data processing modes are enriched, noise data can be eliminated reliably, and accuracy of data processing is further improved.

As above, the specific first data processing mode is described, and how the batch metadata is processed based on the specific data processing mode in the above-described step S104.

In the step S104, after the data processing, in order to further improve the accuracy and reliability of the data processing, the processed data may be checked or subjected to deviation detection, and based on this, the following embodiments are proposed in the embodiments of the present application:

it should be noted that, in the embodiment of the present application, a data threshold may be preset, so as to determine whether there is deviation data in the data processed by the data processing mode based on the data threshold. The data threshold value can be determined by combining the distribution condition of the data of the sensor monitoring points, the service knowledge required by the service scene and the application requirement of the data.

In an embodiment of the present application, as shown in fig. 6, the step S104 may include the following steps S601 to S603:

s601, deviation detection is carried out on the first output data processed in the first data processing mode, so as to determine whether deviation data exists in the first output data.

Specifically, after the first data processing mode is determined, the batch metadata may be first input into the first data processing mode, so that the first data processing mode performs processing, and then the first output data is output after the first data processing mode is processed, at this time, whether the first output data exceeds a data threshold value may be detected, and if the first output data exceeds the data threshold value, the first output data has deviation data; if the first output data does not exceed the data threshold, the first output data has no deviation data.

It is to be understood that deviation data may refer to the absolute value of the difference between the first output data and the data threshold.

And S602, under the condition that deviation data exists in the first output data, determining a second data processing mode according to the deviation data, the service scene, the acquisition time of each metadata and the data type.

Specifically, when the deviation data exists in the first output data, the second data processing mode is determined according to the deviation data, the service scene determined in the step S102, the acquisition time of each metadata, and the data type.

It should be noted that the second data processing mode may include at least one of data culling, homogeneous data re-acquisition, and data padding. When the second data processing mode includes a plurality of processing rules, an execution order among the plurality of processing rules may also be determined.

And S603, processing the batch metadata based on the second data processing mode.

Specifically, after the second processing mode is determined, the batch metadata is processed based on the second data processing mode.

Specifically, in the case where the second processing mode includes data culling, data re-acquisition of homogeneous data, and data re-filling, if a certain first output data exceeds a data threshold, the first output data may be culled, and thereafter, similar data may be re-acquired to re-supplement the acquired similar data.

For example, if the first output data is the data in fig. 7, and the data amount (total data number in the figure) of the monitor point at the monitor point (point) w.unit1.amo2manco is much smaller than the data amount of other measure points as shown in fig. 7, the data (corresponding maximum value, minimum value, mean value, standard deviation, total data number, non-duplicate data, low quartile and high quartile) has deviation data, at this time, the data can be deleted, and similar data can be obtained again to supplement the obtained similar data again; the monitor points w.unit1.had11ct229 and w.unit1.had11ct224 are the same kind of monitor points as the two sensor monitor points (w.unit1. had1ct217 and w.unit1.had13ct216), but the monitor points w.unit1.had11ct229 and w.unit1.had11ct224 are different from the monitor points w.unit1. had1ct217 and w.unit1.had13ct216, indicating that the two monitor points have a problem, i.e. deviation data, at which time the data of the two monitor points can be deleted and similar data can be re-acquired to re-supplement the acquired similar data.

Therefore, when the first output data processed in the first data processing mode has deviation, the batch metadata is reprocessed according to the deviation data, the service scene, the acquisition time and the data type of each metadata, so that the deviation of data processing is avoided, and the accuracy and the reliability of data processing are further improved.

In another embodiment of the present application, as shown in fig. 7, the step S104 may include the following steps S801 to S804:

s801, sequentially processing the batch metadata based on each processing rule and execution sequence in the first data processing mode to obtain input data and output data corresponding to each processing rule.

It is understood that the processing rules and the execution sequence in the first data processing mode have been described above, and are not described herein again to avoid redundancy. The foregoing has described a specific embodiment of sequentially processing the batch metadata based on each processing rule and execution sequence in the first data processing mode, and for avoiding redundancy, no further description is given here.

The input data and the output data corresponding to the processing rule may be referred to as data before processing and data after processing, respectively.

S802, deviation detection is carried out on the second output data processed in the first data processing mode, so that whether deviation data exists in the second output data or not is determined.

Specifically, after determining each processing rule and the execution sequence in the first data processing mode, the batch metadata may be first input into each processing rule according to the execution sequence, and then each processing rule outputs second output data after processing, at this time, it may be detected whether the second output data exceeds a data threshold, and if the second output data exceeds the data threshold, the second output data has deviation data; if the second output data does not exceed the data threshold, then the second output data has no deviation data.

And S803, under the condition that deviation data exists in the second output data, determining a third data processing mode according to the deviation data, the service scene, the acquisition time and the data type of each metadata.

Specifically, when the deviation data exists in the first output data, the third data processing mode is determined according to the deviation data, the service scene determined in the step S102, the acquisition time of each metadata, and the data type.

The third data processing mode may include each processing rule and an execution order among the processing rules, and the same processing rule as the first data processing mode may exist in the third data processing mode.

S804, when the first N processing rules in the third data processing mode are the same as the first N processing rules in the first data processing mode and the execution sequence is the same, taking the output data corresponding to the nth processing rule in the first data processing mode as the input data of the (N + 1) th processing rule in the third data processing mode, so as to complete the processing procedure of the batch metadata in the third data processing mode, where N is a positive integer.

It should be noted that, if the first N processing rules in the third data processing mode are the same as the first N processing rules in the first data processing mode and the execution sequence is the same, since both data processing modes process the same batch metadata, the output data corresponding to the first N processing rules in the third data processing mode is the same as the output data corresponding to the first N processing rules in the first data processing mode. If the first N processing rules in the third data processing mode are the same as the first N processing rules in the first data processing mode but are executed in a different order, the output data corresponding to the first N processing rules in the third data processing mode may be different from the output data corresponding to the first N processing rules in the first data processing mode.

Specifically, after the third data processing mode is determined, that is, each processing rule and the execution order are determined, each processing rule in the third data processing mode may be compared with each processing rule and the execution data in the first data processing mode, and in the case that the first N processing rules in the third data processing mode are the same as the first N processing rules in the first data processing mode and the execution order is the same, in order to avoid processing the same batch metadata based on the same processing rule again, the obtained output data corresponding to the nth processing rule in the first data processing mode may be used as the input data of the (N + 1) th processing rule in the third data processing mode, so as to complete the processing process of the batch metadata in the third data processing mode.

For example, if 5 processing rules (rule G1, rule G2, rule G3, rule G4, rule G5) are included in the third data processing mode, and the execution order of the 5 processing rules is: rule G1 → rule G2 → rule G3 → rule G4 → rule G5, the first data processing mode includes 4 processing rules (rule G1, rule G2, rule G3, rule G6), and the execution order of the 4 processing rules is: rule G1 → rule G2 → rule G3 → rule G6, then the first 3 processing rules in the third data processing mode are the same as the first 3 processing rules in the first data processing mode and the execution sequence is the same (all are the rules G1, G2 and G3 are executed in sequence), then the output data Y3 corresponding to the 3 rd processing rule G3 in the acquired first data processing mode is used as the input data of the 4 th processing rule G4 in the third data processing mode, that is, the input data Y3 is processed based on the 4 th processing rule G4, the output data Y4 is output after processing, the output data Y4 is processed based on the fifth processing rule G5, and finally processed data is obtained, thereby completing the processing process of the batch metadata by the third data processing mode.

Further, in step S804, before the output data N corresponding to the nth processing rule in the first data processing mode is used as the input data of the (N + 1) th processing rule in the third data processing mode, the method may further include: and taking the identifiers of the previous N processing rules and the corresponding execution sequence as indexes, and retrieving and acquiring output data corresponding to the Nth processing rule from the database.

It should be noted that the input data and the output data corresponding to each processing rule acquired in step S801 may be stored in the database according to the identifier of the corresponding processing rule and the corresponding execution sequence for subsequent invocation.

The identifier of the processing rule may be any identifier that can be used as an index of the processing rule, and may be the content of a data file or a configuration file of the processing rule, such as keywords such as "culling", "filling", and "exception", or may be an intermediate result (input data and output data in the processing process) of the processing rule.

Specifically, in the case that the first N processing rules in the third data processing mode are the same as the first N processing rules in the first data processing mode and the execution order is the same, the identifiers of the first N processing rules may be obtained, and the output data corresponding to the nth processing rule may be retrieved and obtained from the database with the identifiers of the first N processing rules and the corresponding execution order as indexes. During retrieval, reverse-order searching can be performed in the database to search the output data corresponding to the nth processing rule.

Specifically, as shown in fig. 9, first, a file index of all processing rules may be obtained, where the index may be a data file index or a configuration file index, or an intermediate result index, where the data file index and the configuration file index are expressed by performing index operation on contents therein, and the intermediate result index is indexed by inputting the data file index and a configuration file in a corresponding flow; then, carrying out reverse order searching, wherein in the reverse order searching process, starting from the last step of data processing, searching the index of the current processing intermediate result in a processing file storage list; if the search is carried out, the search result can be directly returned as a result; if not, the previous step in the data processing flow is entered to perform the same search processing.

Therefore, the output results with the same function in the previous processing process are used, so that the repeated processing of the same data can be avoided, the processing time of the data is reduced, the processing resources are saved, and the data processing efficiency is improved.

After the batch metadata is processed based on the third data processing mode, deviation detection may be performed on the processed data to further determine whether there is a deviation in the processed data.

That is, in an embodiment of the present application, after the step S804, the method may further include: performing deviation detection on the third output data processed by the third data processing mode to determine whether deviation data exists in the third output data; and deleting the input data and the output data corresponding to each processing rule in the database when the third output data does not have deviation data.

Specifically, after the processing process of the batch metadata in the third data processing mode is completed, the processed third output data can be obtained, and whether the third output data exceeds the data threshold can be detected, if the third output data exceeds the data threshold, deviation data exists in the third output data; if the third output data does not exceed the data threshold, the third output data has no deviation data. And under the condition that the third output data does not have deviation data, deleting the input data and the output data corresponding to each processing rule in the database, so that after the same metadata is processed, deleting the intermediate results in the original processing processes.

That is, if the same batch of metadata is processed in N rounds, all new intermediate processing results in the previous N-1 rounds of processing can be stored for subsequent invocation.

In the embodiment of the application, after the batch metadata is processed based on the data processing mode each time, deviation detection can be carried out on the processed data, so that corresponding processing can be carried out according to the result of the deviation detection, and the authenticity and the effectiveness of data processing are greatly improved.

The following describes a specific implementation of an embodiment of the present application with reference to fig. 10:

as shown in fig. 10, the sensor a, the sensor B, and the sensor C collect monitoring point data or metadata, acquire collected batch metadata, perform uniform format check and processing on the batch metadata, and then process the batch metadata. Meanwhile, a Data processing configuration file can be determined and acquired according to the sensor, so that a Data processing process is realized based on the Data processing configuration file, for example, abnormal Data exceeding the upper and lower limits of the boundary is removed or deleted through the upper and lower boundaries in the configuration information Data _ bound.csv; merging the single monitoring point Data list into a multi-point Data set through configuration information Data _ config.csv; deleting shutdown data through configuration information Shut _ down.csv; abnormal data is deleted and missing data is filled through the configuration information Mu1ti _ point.

After the data processing is finished, deviation detection is carried out on the processed output data, if the output data has deviation, the data processing does not pass due to disqualification, and then the next data processing is continued until the data processing passes; if the output data has no deviation, the data is passed through, that is, the data is processed, and when the data passes through, the processed data is collected to form an algorithm data set for subsequent use, wherein the final data set can be seen as shown in fig. 11.

In summary, the embodiment of the application provides a set of data processing flow for the whole process of converting the data of the sensor measuring points into the required data set, and meanwhile, a set of data processing assembly line for optimizing time cost and labor cost is established on the basis of the whole flow. The data processing results are counted and stored in a mode of coding files and operation modes, human participation in the data processing process is reduced, and the uniformity of configuration files and the data processing results is ensured; meanwhile, the reusability of the intermediate result of data processing is utilized, the time cost of data repeated processing is reduced, and the data processing efficiency is improved.

An embodiment of the present application further provides a device for processing batch data, and fig. 12 is a schematic structural diagram of the device for processing batch data provided in the embodiment of the present application.

As shown in fig. 12, the apparatus 1200 for processing batch data includes: a first obtaining module 1210, a first determining module 1220, a second determining module 1230, and a third determining module 1240.

The first obtaining module 1210 is configured to obtain batch metadata to be processed; the first determining module 1220 is configured to analyze the batch metadata to determine a service scenario to which the batch metadata belongs, acquisition time of each piece of metadata, and a data type; a second determining module 1230, configured to determine the first data processing mode according to the service scenario, the acquisition time of each metadata, and the data type; the first processing module 1240 is configured to process the batch metadata based on the first data processing mode.

In an embodiment of the present application, the second determining module 1230 may include: the first determining unit is used for determining the current data processing rule and the execution sequence among the processing rules.

In one embodiment of the application, the data processing rules include at least one of the following rules: format normalization, abnormal data elimination based on boundary limit values, data elimination based on specified time intervals, merging of multiple monitoring point data based on homogeneous data, abnormal data elimination based on service scenes and missing data filling based on known data.

In an embodiment of the present application, the first processing module 1240 may include: the second determining unit is used for carrying out deviation detection on the first output data processed in the first data processing mode so as to determine whether deviation data exists in the first output data; the third determining unit is used for determining a second data processing mode according to the deviation data, the service scene, the acquisition time of each metadata and the data type under the condition that the deviation data exists in the first output data; and the first processing unit is used for processing the batch metadata based on the second data processing mode.

In an embodiment of the present application, the first processing module 1240 may include: the first acquisition unit is used for sequentially processing the batch metadata based on each processing rule and execution sequence in the first data processing mode so as to acquire input data and output data corresponding to each processing rule; the fourth determining unit is used for carrying out deviation detection on the second output data processed in the first data processing mode so as to determine whether deviation data exists in the second output data; a fifth determining unit, configured to determine a third data processing mode according to the deviation data, the service scenario, the acquisition time of each metadata, and the data type when the deviation data exists in the second output data; and the second processing unit is used for taking the output data corresponding to the Nth processing rule in the first data processing mode as the input data of the (N + 1) th processing rule in the third data processing mode to finish the processing process of the batch metadata in the third data processing mode under the condition that the first N processing rules in the third data processing mode are the same as the first N processing rules in the first data processing mode and the execution sequence is the same, wherein N is a positive integer.

In an embodiment of the present application, the first processing module 1240 may further include: and the second acquisition unit is used for retrieving and acquiring the output data corresponding to the Nth processing rule from the database by taking the identifiers of the previous N processing rules and the corresponding execution sequence as indexes.

In an embodiment of the present application, the first processing module 1240 may further include: a sixth determining unit, configured to perform deviation detection on the third output data processed in the third data processing mode to determine whether deviation data exists in the third output data; and the first deleting unit is used for deleting the input data and the output data corresponding to each processing rule in the database under the condition that the third output data does not have abnormal data.

It should be noted that, for other specific embodiments of the batch data processing apparatus in the embodiment of the present application, reference may be made to the specific embodiment of the foregoing batch data processing method, and details are not described here again to avoid redundancy.

The batch data processing device provided by the embodiment of the application processes the batch metadata according to the service scenes to which the batch metadata belong, the acquisition time of each metadata and the data type, so that the batch metadata processing is realized for different service scenes, the reliability of data processing is improved, and the device is beneficial to reducing manual processing links and reducing labor cost.

According to an embodiment of the application, the application further provides an electronic device, a readable storage medium and a computer program product of the batch data processing method. This will be explained with reference to fig. 13.

Fig. 13 is a block diagram of an electronic device of a batch data processing method according to an embodiment of the present application.

As shown in fig. 13, electronic device 1300 may include: a memory 1310 and at least one processor 1320, a bus 1330 connecting the various components, including the memory 1310 and the processor 1320.

The memory 1310 has stored thereon a computer program that, when executed by the processor 1320, implements the batch data processing method of the embodiments of the present application.

Bus 1330 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Electronic device 1300 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by electronic device 1300 and includes both volatile and nonvolatile media, removable and non-removable media.

Memory 1310 may include computer system readable media in the form of volatile Memory, such as Random Access Memory (RAM) 1340 and/or cache Memory 1350. The electronic device 1300 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, the storage system 1360 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 13, and commonly referred to as a "hard drive"). Although not shown in FIG. 13, a disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a Compact disk Read Only Memory (CD-ROM), a Digital versatile disk Read Only Memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 1330 through one or more data media interfaces. Memory 1310 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 1380 having a set (at least one) of program modules 1370 may be stored, for example, in the memory 1310, such program modules 1370 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may include an implementation of a network environment. The program modules 1370 generally perform the functions and/or methodologies of the described embodiments of the invention.

The electronic device 1300 may also communicate with one or more external devices 1390 (e.g., keyboard, pointing device, display 1391, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 1300, and/or with any devices (e.g., network card, modem, etc.) that enable the electronic device 1300 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 1392. Moreover, the electronic device 1300 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public Network, such as the internet) via the Network adapter 1393. As shown, the network adapter 1393 communicates with the other modules of the electronic device 1300 via a bus 1330. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The processor 1320 executes various functional applications and data processing, for example, implementing the methods mentioned in the foregoing embodiments, by executing programs stored in the memory 1310.

In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.

The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.

It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.

In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.

Claims

1.A method for processing batch data is characterized by comprising the following steps:

acquiring batch metadata to be processed;

processing the batch metadata based on the first data processing mode.

2. The method of claim 1, wherein said determining a first data processing mode comprises:

and determining the current data processing rule and the execution sequence among the processing rules.

3. The method of claim 2, wherein the data processing rules include at least one of the following rules: format normalization, abnormal data elimination based on boundary limit values, data elimination based on specified time intervals, merging of multiple monitoring point data based on homogeneous data, abnormal data elimination based on service scenes and missing data filling based on known data.

4. The method of any of claims 1-3, wherein said processing the batch of metadata based on the first data processing schema comprises:

performing deviation detection on the first output data processed by the first data processing mode to determine whether deviation data exists in the first output data;

under the condition that deviation data exist in the first output data, determining a second data processing mode according to the deviation data, the service scene, the acquisition time of each metadata and the data type;

processing the batch metadata based on the second data processing mode.

5. The method of any of claims 1-3, wherein said processing the batch of metadata based on the first data processing schema comprises:

processing the batch metadata in sequence based on each processing rule and execution sequence in the first data processing mode to obtain input data and output data corresponding to each processing rule;

performing deviation detection on the second output data processed by the first data processing mode to determine whether deviation data exists in the second output data;

under the condition that deviation data exist in the second output data, determining a third data processing mode according to the deviation data, the service scene, the acquisition time of each metadata and the data type;

and when the first N processing rules in the third data processing mode are the same as the first N processing rules in the first data processing mode and the execution sequence is the same, taking output data corresponding to the nth processing rule in the first data processing mode as input data of the (N + 1) th processing rule in the third data processing mode to complete the processing process of the batch metadata in the third data processing mode, wherein N is a positive integer.

6. The method of claim 5, wherein prior to said taking the output data N corresponding to the Nth processing rule in the first data processing mode as input data for the (N + 1) th processing rule in the third data processing mode, further comprising:

and taking the identifications of the first N processing rules and the corresponding execution sequence as indexes, and retrieving and acquiring the output data corresponding to the Nth processing rule from a database.

7. The method of claim 5, wherein after said completing the processing of the batch of metadata by the third data processing schema, further comprising:

performing deviation detection on the third output data processed by the third data processing mode to determine whether deviation data exists in the third output data;

and deleting the input data and the output data corresponding to each processing rule in the database when the third output data does not have deviation data.

8. An apparatus for processing batch data, comprising:

9. The apparatus of claim 8, wherein the second determining module comprises:

the first determining unit is used for determining the current data processing rule and the execution sequence among the processing rules.

10. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of processing batch data of any one of claims 1-7.

11. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of processing batch data of any one of claims 1-7.

12. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the method of processing batch data of any one of claims 1-7.