CN115408390A - A data processing method, device and electronic equipment - Google Patents

Abstract

The application provides a data processing method and device and electronic equipment. The method and the device adopt a Hash digest algorithm to determine the corresponding Hash identification and storage path based on the variable identification, the target period and the target time interval of the target variable corresponding to the target data, thereby creating a corresponding target file under the corresponding path and writing the target data in, and realizing data storage. According to the data storage method and device, data under different variables, different time periods and different periods can be independently stored in different files respectively according to a Hash algorithm, so that the characteristics that high-frequency data collection and query are carried out at equal time intervals and specific time periods are fully utilized, the utilization rate of a storage space can be effectively improved, follow-up data query with a higher effective data ratio can be conveniently carried out according to information such as variables, periods and time periods, and the device data storage requirement under a large-data-volume scene can be effectively met compared with a traditional data storage mode.

Description

A data processing method, device and electronic equipment

technical field

The present application relates to data processing technology, in particular to a data processing method, device and electronic equipment.

Background technique

For all kinds of industrial equipment with real-time monitoring requirements, the data generated and recorded during operation are of great value for status monitoring, online diagnosis and offline analysis.

Currently commonly used data storage methods include relational databases (such as MySQL), non-relational databases (such as MongoDB), time series databases (such as TSDB), text files (such as CSV files), and so on.

However, with the development of industrial control technology, the number of monitored variables and the frequency of data sampling are increasing continuously during the operation of industrial equipment, which poses a severe challenge to the storage of massive data, such as for the sampling period of milliseconds or shorter For high-frequency data, a single variable can generate tens of thousands to millions of pieces of data every day, and the magnitude of data generated every year can reach hundreds of millions.

The above-mentioned traditional data storage methods cannot meet the production practice requirements in large data volume scenarios in terms of storage efficiency and access speed.

Contents of the invention

Embodiments of the present application provide a data processing method, device, and electronic equipment to solve the problem of difficulty in storing high-frequency variables in existing data storage solutions.

In the first aspect, the embodiment of the present application provides a data processing method, including:

Obtain the target data of the target variable in the target time period and target period;

Determine the hash identifier and storage path of the target file through the target hash algorithm at least according to the above target time period, the above target period, and the variable identification of the above target variable;

The above-mentioned target file is created according to the above-mentioned hash identifier under the above-mentioned storage path, and the above-mentioned target data is written into the above-mentioned target file.

In a possible implementation, the target data of the above-mentioned target variables in the target time period and target period include:

The target sampling data of the above-mentioned target variable in the target time period and target sampling period, and/or, the target statistical data of the above-mentioned target variable in the target time period and target statistical period;

Wherein, the above-mentioned target statistical period is greater than the above-mentioned target sampling period, and the above-mentioned target statistical data is data obtained by performing statistics on sampling data or other statistical data based on the above-mentioned target statistical period.

In a possible implementation, at least according to the above-mentioned target time period, the above-mentioned target cycle, and the variable identification of the above-mentioned target variable, the hash identification and storage path of the target file are determined through the target hash algorithm, including:

According to the data type of the above-mentioned target data, the above-mentioned target time period, the above-mentioned target cycle, and the variable identification of the above-mentioned target variable, the hash identification and storage path of the target file are determined through the target hash algorithm.

In a possible implementation, at least according to the above-mentioned target time period, the above-mentioned target cycle, and the variable identification of the above-mentioned target variable, the hash identification and storage path of the target file are determined through the target hash algorithm, including:

At least according to the above-mentioned target period, the above-mentioned target period, and the variable identification of the above-mentioned target variable, determine the hash value matching the above-mentioned target data through the target hash algorithm;

Select a character with a preset number of digits from the above hash value as the hash identifier of the above target file;

Divide the characters identified by the above hash, query the storage location matching the division result in the preset multi-level file directory, and determine the storage path of the above target file.

In a possible implementation manner, the above-mentioned writing of the above-mentioned target data into the above-mentioned target file includes:

Writing the data points in the above target data into the above target file sequentially in binary format according to time sequence, wherein the binary storage digits of the data points in the above target file are fixed digits;

If there is a faulty data point with missing data, the above-mentioned faulty data point is occupied by the above-mentioned fixed number of digits based on an interpolation algorithm or a preset special value.

In a possible implementation, the above method also includes:

When a query instruction for data query of the above-mentioned target variable is received, according to the target query period, target query period, and variable identification carried in the above-mentioned query instruction, determine the number of one or more files to be queried through the above-mentioned target hash algorithm Hash ID and storage path;

Query the one or more files to be queried based on the hash identifier and the storage path, and obtain the data to be queried of the target variable in the target query period and the target query period.

In a possible implementation, the above-mentioned target query period, target query period, and variable identification carried in the above-mentioned query instruction are used to determine the hash ID and storage path of one or more files to be queried through the target hash algorithm, including:

One or more actual query periods are determined according to the above target query period carried in the above query instruction based on the period division rule matching the above target query period;

Based on the above-mentioned one or more actual query periods, the target query cycle carried in the above-mentioned query instructions, and the variable identification, the hash identification and storage path of one or more files to be queried are determined through the target hash algorithm; wherein, each actual query The period matches the target period of a file to be queried.

In a possible implementation, the above-mentioned target file and the above-mentioned file to be queried also include file header information for data self-description, and the above-mentioned file header information includes variable identification, target time period, target period, encryption algorithm type, compression At least one of the algorithm types; verification information for data verification of the above-mentioned target files is stored in the external device;

After the above-mentioned one or more files to be queried are queried based on the above-mentioned hash identifier and the above-mentioned storage path, the above-mentioned method also includes:

Obtaining the above verification information in the above external device, and judging whether the above verification information matches the above file header information in the above file to be queried;

If it matches, it is determined that there is no data abnormality in the above-mentioned file to be queried, and the data to be queried under the above-mentioned target query period and target query period of the above-mentioned target variable is obtained;

If they do not match, it is determined that there is a data exception in the above-mentioned file to be queried, and an error message is returned.

In a second aspect, the embodiment of the present application provides a data processing device, including:

The data acquisition unit is used to acquire the target data of the target variable in the target time period and target period;

A hash determination unit, configured to determine the hash identifier and storage path of the target file through a target hash algorithm at least according to the above target time period, the above target cycle, and the variable identification of the above target variable;

A file creation unit, configured to create the above-mentioned target file according to the above-mentioned hash identifier under the above-mentioned storage path, and write the above-mentioned target data into the above-mentioned target file.

In a third aspect, the embodiment of the present application further provides an electronic device, the electronic device includes: a processor and a machine-readable storage medium;

The machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

The above-mentioned processor is used to execute machine-executable instructions to realize the steps of the methods disclosed above.

It can be seen from the above technical solutions that this embodiment uses the hash digest algorithm to determine the corresponding hash identifier and storage path based on the variable identifier, target cycle, and target time period of the target variable corresponding to the target data, thereby creating Corresponding target file and write target data to realize data storage. In this embodiment, the data of different variables, different time periods, and different periods can be stored independently in different files according to the hash algorithm, thus making full use of the high-frequency data collection and query at equal time intervals and for specific time periods The characteristics of storage can effectively improve the utilization rate of storage space, and facilitate subsequent data query with higher effective data ratio based on variables, periods, time periods and other information. Compared with traditional data storage methods, it can effectively meet the needs of devices in large data volume scenarios. Data storage requirements.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

Fig. 1 is the flow chart of the method provided by the embodiment of the present application;

FIG. 2 is a schematic diagram of a directory storage structure provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of the data file structure provided by the embodiment of the present application;

Figure 4 is a schematic diagram of data storage performance comparison;

Figure 5 is a schematic diagram of data query performance comparison;

Figure 6 is a structural diagram of the device provided by the embodiment of the present application;

FIG. 7 is a structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present application as recited in the appended claims.

The terminology used in this application is for the purpose of describing particular embodiments only, and is not intended to limit the application. As used in this application and the appended claims, the singular forms "a", "the", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise.

In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the following describes the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings Further detailed instructions.

In the field of industrial control, it is necessary to collect data on various variables of industrial equipment during the operation process for real-time status monitoring, online diagnosis, offline analysis and other purposes. The industrial equipment here includes wind turbines, chemical reactors, transformers, and other equipment that need to monitor or collect operating data. The variables to be collected can be any industrial production control or equipment detection such as wind speed, speed, oil temperature, hydraulic pressure, and power. Variables required in the process.

Usually, data whose sampling period is at the millisecond level or shorter is called high-frequency data, and variables that require high-frequency data collection are called high-frequency variables. Take wind power generators as an example: the number of high-frequency variable points with 20ms resolution that a single wind turbine needs to collect can reach more than 1,000, and the amount of data that needs to be stored every day is 50/second*86400 seconds/day*1000 points /unit=4.32 billion/day·unit, if the storage period of high-frequency data is 180 days, the amount of data that a single wind turbine needs to store in a storage period will reach 4.32 billion/day*180 days=777.6 billion. Taking 4 bytes of data (single-precision floating-point number) as an example, the total storage capacity will reach 777.6 billion*4=3.11 trillion bytes, or 3.11TB.

However, the current commonly used data processing methods have some defects in the above-mentioned large data storage scenarios. For example, the storage structure of relational databases is difficult to support the effective storage and access of trillions of data, and each record needs to store ID and time. Standardized extra fields cause the total storage capacity after adding indexes to far exceed the capacity of the data itself; non-relational databases need to use memory as the main cache medium, and when the amount of data (such as TB level) exceeds the available memory (such as GB level), the performance drops sharply , and there is also the problem that the introduction of additional fields leads to a low proportion of valid data; while the general optimization of time-series databases for time-series data can greatly improve performance, but it also fails to take advantage of the characteristics of high-frequency data and business needs to customize, store Efficiency and access performance are still difficult to meet business needs, and the requirements for hardware devices are high; although text files have the advantage of being readable by users, the cost is that storage efficiency and access performance are far inferior to the first three types of databases, and can only be used for small data Data exchange under high volume conditions, etc.

In this regard, an embodiment of the present application provides a data processing method to solve the above problem. Referring to FIG. 1 , FIG. 1 is a flowchart of a method provided by an embodiment of the present application. The sampling period of the target variable involved in the process may be at millisecond level, or at any higher or lower sampling level, which is not limited in this embodiment.

In this embodiment, the data involved in the flow of the above data processing method may come from any industrial production equipment that requires data collection and storage, including wind power generators, chemical reactors, etc., or other equipment with data storage requirements; The above obtained data may be stored in any electronic device such as a personal computer, a server, an embedded industrial control device, a mobile phone, etc., which is not limited in this embodiment.

As shown in Figure 1, the process may include the following steps:

In step 101, the target data of the target variable in the target time period and target period are acquired.

In this embodiment, in order to conduct independent query based on variables, time periods, and intervals (i.e., sampling periods) for high-frequency data, and data query and collection are generally equal time intervals for data storage, it is necessary to first determine the acquisition The variable, time period, and period information of the target data, that is, the target data of the target variable in the target period and target period are obtained to generate subsequent target files for data storage.

Wherein, the above-mentioned target time period can be the time range corresponding to the above-mentioned target data, that is, from the time corresponding to the first data point in the above-mentioned target data to the time corresponding to the last data point, such as 10:00 on a certain day of a certain year ~11:00, 12:05:00~12:05:59, etc.; the above-mentioned target period can be the corresponding time interval between two adjacent data points in the target data, and the time interval in the same group of target data can be It is a fixed value, such as 20 milliseconds, 1 minute, 1 hour, etc.

As a preferred embodiment, the target data of the above-mentioned target variable in the target time period and target period may specifically be: the target sampling data of the above-mentioned target variable in the target period and target sampling period, and/or, the above-mentioned target variable in the target Target stats under time period and target statistical period.

Among them, the sampling data can be the original data directly obtained by sampling data according to a certain sampling period; the statistical data can be the data obtained by statistically collecting sampling data or other statistical data based on the statistical period, and the above-mentioned target statistical period is greater than the above-mentioned target The sampling period.

For example, if data sampling is performed with a target sampling period of 20ms in a wind turbine, the target sampling data with a target sampling period of 20ms can be obtained; on this basis, statistics can be continued to obtain statistical data, for example, data can be sampled from the above target Select continuous 3000 data points, determine the maximum value of every adjacent 50 data points to obtain 60 data points, which can be used as the target statistical data with a target statistical period of 20ms*50=1s; further, the same It is reasonable to continue to perform maximum statistics on these 60 data points to further obtain target statistical data with a target statistical period of 1s*60=1min, etc.; wherein, the above-mentioned maximum value can be replaced by any statistical value in the same way, such as the minimum value, average value, peak-to-valley difference, standard deviation, final value, etc., which are not limited in this embodiment.

Since the query of high-frequency data is usually performed for a specified time interval, for example, there is no need to access the millisecond-level raw sampling data when querying the coarse-grained statistical data at the minute level, and there is no need to access the minute-level sampling data when querying the coarse-grained statistical data at the hour level. Statistical data, etc. Therefore, synchronously storing statistical data in different cycles during data storage is conducive to improving the efficiency of subsequent data queries, reducing hardware requirements, and optimizing user experience.

Step 102: Determine the hash identifier and storage path of the target file through a target hash algorithm at least according to the target time period, the target period, and the variable identifier of the target variable.

In this embodiment, after the above-mentioned target data is obtained, at least based on the time period, period, and variable identification of the target data, a preset hash algorithm needs to be used to generate a hash identification corresponding to the target data for subsequent use based on the target data. The hash identifier determines the storage path and the name of the storage file, etc.

Specifically, the hash value that matches the above target data can be determined through the target hash algorithm, and then a character with a preset number of digits is selected from the hash value as the hash identifier of the above target file for subsequent hashing The identified characters are divided, and the storage location matching the division result is queried in the preset multi-level file directory to determine the storage path of the above-mentioned target file; the relevant content of the multi-level file directory will be given in detail in conjunction with the relevant content in Figure 2 in the following text, I won't go into details here.

For example, when the target time period of a certain target data is 2020-08-14 17:00:00～2020-08-14 18:00:00, the target period is 50ms, and the variable ID is 1008, when the preset hash algorithm used When the MD5 algorithm is used, the method of calculating the above-mentioned hash identifier can be: MD5("1008#2020081417#50ms")=CBC0F1300ECA444B8C30764463E368D9.

Among them, "1008," "2020081417" and "50ms" respectively correspond to the variable identification, target time period, and target period of the above-mentioned target variable, and the specific format, separator, and arrangement order of the above-mentioned target variable can be adjusted arbitrarily. This is not limited; and, the hash calculation result can be directly determined as the above-mentioned hash identifier, and the specified number of digits (such as taking the first ten digits: CBC0F1300E) can also be selected as the above-mentioned hash identifier. Not limited. The above preset hash algorithm can be any hash digest algorithm such as MD5, SHA-1, SHA2, RIPEMD-160, etc., to ensure that the hash algorithm used when generating the hash identifier is consistent with the hash algorithm used when querying the stored data files. It is sufficient that the hash algorithm is the same target hash algorithm.

In this embodiment, after the above-mentioned hash identifier is determined, the storage path of the target file to be generated can be determined based on the hash identifier; as an optional embodiment, it can be pre-based on a certain number of hash characters ( 0, 1, 2...F) Combine to generate several levels of file directory storage structure, and then determine the directory location matching the above hash identifier in the multi-level file directory as the storage path.

As an optional embodiment, the above-mentioned directory storage structure of the multi-level file directory can be implemented with reference to the content shown in FIG. 2 . The root directory shown in the figure may be a designated root directory for uniformly storing all data files, such as C:\data. Each subdirectory under this root directory can be made of several hash characters, for example, when adopting 2 hash characters, then can contain 16*16=256 first-level subdirectories under the root directory, followed by 00, 01, 02...FF. On this basis, you can set the first-level subdirectory to not directly store files, but further set several second-level subdirectories. For example, when the second-level subdirectory is also composed of two hash characters, each first-level subdirectory Similarly, the directory contains 256 second-level subdirectories, and the root directory contains a total of 256*256=65536 second-level subdirectories. Subsequent generated data files are all stored in the second-level subdirectories. Among them, this embodiment does not limit the total number of levels of subdirectories and the number of subdirectories of each level, and usually two levels of subdirectories are set, and each level of subdirectories is composed of two hash characters to meet the data requirements under normal circumstances. Capacity requirements, if the requirements cannot be met, it can still continue to expand to the third level or above subdirectories.

As a specific example, based on the above-mentioned directory storage structure in Figure 2, it is assumed that the variable identifier is 0001, the target period is 2021-09-10 10:00:00.000～10:59:59.999, and the target period is 20ms. Data, and its corresponding hash identifier is: HASH("0001#2021091010#20ms")="006E8D78A940," then it can be determined that its first-level subdirectory is "00" (1st to 2nd digits of the hash identifier). The second-level subdirectory under the first-level subdirectory is "6E" (the 3rd to 4th digits of the hash identifier), so the storage path of the target file to be generated corresponding to the target data can be C:\data\00 \6E; Assume that for the target statistical data whose variable ID is 0002, the target period is the whole year of 2021, and the target period is 1 hour, the corresponding hash ID is: HASH("0002#2021#1h")="009D1129E3BF," Then its storage path can be C:\data\00\9D similarly.

As a preferred embodiment, when the above target data is target statistical data determined based on statistics, the basis for determining the hash identifier also includes the data type of the target data. The data type corresponds to the specific statistical value type of the target statistical data. For example, when the data type of the target statistical data corresponds to the maximum value, the data type is determined to be 01; when it corresponds to the minimum value, the data type is determined to be 02, etc. , add this data type to the calculation basis of the above-mentioned hash identifier, so as to store the statistical data of different statistical value types under the same variable, the same time period, and the same statistical cycle in different data files.

As another preferred embodiment, when the above-mentioned target data is the target statistical data determined based on statistics, the statistical data of different statistical value types under the same variable, the same time period, and the same statistical cycle can also be stored in the same data file; for example , the maximum value, minimum value, and average value data can be stored in the same data file. The 1st, 4th, 7th... 3*n-2 data points in this file correspond to the maximum value, and the 2nd, 5th, and The 8...3*n-1 data points correspond to the minimum value, the 3rd, 6th, 9th...3*n data points correspond to the average value, etc. Among them, data of different statistical value types can be stored in different byte lengths, but the data of the same statistical value type must be guaranteed to have the same byte length to ensure that subsequent data queries can be based on the specified byte length. Time data for positioning.

Based on the above-mentioned directory storage structure and storage path determination method, it can ensure that a large number of data files are statistically evenly distributed to each subdirectory. Too many data files lead to a decrease in file access efficiency.

It should be noted that the above-mentioned same root directory can only store data under the same variable from the same device, or can store data under different variables from different devices, which is not limited in this embodiment; Only a specific variable is stored in the root directory, and the data of different variables are divided into different root directories, that is, the data of different variables are stored separately from each other, which can effectively reduce the consumption of computing resources in subsequent data queries.

And, by using the same variable in different target time periods, the total amount of data that needs to be retrieved in the subsequent data query can be greatly reduced to improve query efficiency; similarly, the data of the same variable in different target periods are stored independently, that is, for Different time granularities such as seconds, minutes, hours, and days are preprocessed. Although the total amount of storage space occupied will be slightly increased compared with the solution of only storing original data, it can greatly reduce the geometric size of the data that needs to be searched in the query phase and obtain data. Performance improvements brought about by locality.

Step 103: Create the above-mentioned target file according to the above-mentioned hash identifier under the above-mentioned storage path, and write the above-mentioned target data into the above-mentioned target file.

In this embodiment, based on the foregoing steps 101 and 102, the above-mentioned target data, hash identifier, and storage path have been determined, so a target file can be created at least based on the hash identifier at the above-mentioned storage path for storing the above-mentioned target data. The file name of the target file can be equal to the above-mentioned hash identifier, or can be a character corresponding to the preset number of digits in the above-mentioned hash identifier; in addition to the above-mentioned hash identifier, the file name can also contain additional fields added based on preset rules etc. This embodiment does not limit this.

As an optional embodiment, writing the above-mentioned target data into the above-mentioned target file may be specifically writing data points in the above-mentioned target data into the above-mentioned target file sequentially in binary format according to time sequence.

Among them, the target file can be a binary file, such as a file in bin (binary: binary) format; the binary storage digits of the data points in the above target file is a fixed number of digits, if there is a faulty data point with missing data, then based on interpolation Algorithms or preset special values occupy the above-mentioned faulty data points as the above-mentioned fixed number of digits to ensure that data points can be located based on byte length in subsequent data queries. For example, interpolation is used based on adjacent non-missing data points The algorithm calculates the replacement value used for the placeholder at the fault point, or fills the place with 0, etc.

As an optional embodiment, the structure of the above target file can be implemented with reference to the schematic diagram of the data file structure shown in FIG. 3 . As shown in Figure 3, each data file contains a total of N data points arranged in chronological order, each data point is stored in binary, and its storage length can range from 1/8 byte (1 bit) to 8 Bytes (double-precision floating-point numbers) are configured as needed; the above N data points in the same data file can be written at one time or divided into multiple writes, which is not limited in this embodiment.

As a preferred embodiment, when a query instruction for data query of the above-mentioned target variable is received, according to the target query time period, target query period, and variable identification carried in the above-mentioned query instruction, a target hash algorithm is used to determine a The hash identifiers and storage paths of one or more files to be queried, query the one or more files to be queried based on the hash identifiers and the storage paths, and obtain the to-be-queried values of the target variable in the target query period and the target query period data.

In this embodiment, the data stored in the target variable may be queried based on the received query instruction. Wherein, the target query time period is the time range in which the corresponding data of the target variable needs to be queried and obtained, and the target query period is the sampling or statistics cycle of the corresponding data of the target variable to be queried and obtained.

Further, as a preferred embodiment, the above-mentioned data query process is specifically based on the period division rule matching the above-mentioned target query period, and one or more actual query periods are determined according to the above-mentioned target query period carried in the above-mentioned query instruction, and then Based on the above-mentioned one or more actual query periods, the target query cycle carried in the above-mentioned query instruction, and the variable identifier, the hash identifier and storage path of one or more files to be queried are determined through a target hash algorithm. Wherein, each actual query time period matches a target time period of a file to be queried.

In this embodiment, since the target query period carried in the query instruction may not be exactly the same as the aforementioned target period, it is necessary to first convert the target query period into one or more actual queries based on the preset period division rule time period.

For example, assuming that based on the preset period division rule, when the target period of the target variable is 20ms, there are 50*60*60=180000 data points stored in the corresponding data file, that is, the data in each data file corresponds to 1 Hours, so the target time period of the above target variables under this target cycle is 10:00～11:00, 11:00～12:00 and other whole hour division time periods. At this time, if the target query time period is 10:30 to 11:30 and other non-hourly time periods, there will be a problem that there is no target time period that is the same as the target query time period. At this time, based on the preset period division rule (for example, when the target period is 20ms, the corresponding period is divided in whole hours), it can be determined that the actual query period matching the target query period 10:30-11:30 is 10: 30～11:00, and, 11:00～11:30, there are two actual query time periods in total; they are used to determine the files to be queried that match the two actual query time periods based on these two actual query time periods. Hash ID and storage path, that is, two data storage files with a target period of 20 ms and a target period of 10:00-11:00 and 11:00-12:00 respectively for the target variable; optionally, the above actual query The time period may also be 10:00-11:00 and 11:00-12:00, etc., which are not limited in this embodiment.

Similarly, in the above period division rule, the period corresponding to the target period of 1 minute can be divided into whole days or whole months, and the period corresponding to the target period of 1 hour can be divided into units of whole month or year, etc. and, it may also be that the above target query time period is the same as the target time period, or that the above target query time period is completely contained by a certain target time period, etc., the file to be queried at this time may be a certain file instead of multiple files.

As an optional embodiment, the above-mentioned file may also contain file header information for data self-description or verification, and the above-mentioned file header information may include variable identification, target period, target period, byte length, data length One or more of , encryption algorithm type, and compression algorithm type. The above file header information will also be stored as verification information in at least one external device independent of the device. In the subsequent data query process, the file header information used for self-description in the data file obtained by the judgment query can be compared with the external Whether the verification information stored in the device matches, so as to determine whether there is any data anomaly in the data file obtained by query, which has the effect of redundancy verification.

Specifically, after the above-mentioned one or more files to be queried are queried based on the hash identifier and storage path, the above-mentioned verification information in the above-mentioned external device can also be obtained, and the above-mentioned verification information and the above-mentioned file header in the above-mentioned to-be-queried file can be determined. Whether the information matches, if it matches, it is determined that there is no data abnormality in the above-mentioned file to be queried, and the data to be queried under the above-mentioned target query period and query cycle of the above-mentioned target variable is obtained, if it does not match, it is determined that there is a data anomaly in the above-mentioned file to be queried, Return an error message.

Among them, the above-mentioned encryption algorithm type and compression algorithm type are mainly used to encrypt and/or compress the above-mentioned data files after the data is written, so as to further improve data security and protect data privacy. If encryption and/or compression operations are used in the data storage process, the corresponding encryption algorithm type and/or compression algorithm type should be included in the file header information, so that in the subsequent data query process, after determining the file to be queried, use the same The same encryption algorithm type and/or compression algorithm type in the storage process decrypts and/or decompresses the file to be queried, and reads the corresponding query data from it.

So far, the process shown in Figure 1 is completed.

It can be seen from the process shown in Figure 1 that this embodiment uses the hash digest algorithm to determine the corresponding hash identifier and storage path based on the variable identifier, target cycle, and target time period of the target variable corresponding to the target data, so that the corresponding path Create the corresponding target file and write the target data to realize data storage. In this embodiment, the data of different variables, different time periods, and different periods can be stored independently in different files according to the hash algorithm, thus making full use of the high-frequency data collection and query at equal time intervals and for specific time periods The characteristics of storage can effectively improve the utilization rate of storage space, and facilitate subsequent data query with higher effective data ratio based on variables, periods, time periods and other information. Compared with traditional data storage methods, it can effectively meet the needs of devices in large data volume scenarios. Data storage requirements.

For the data files stored and obtained based on the above method, the same hash digest algorithm can be used to directly determine the corresponding file name and storage address to be queried according to the query conditions of "variable identification + time period + period", which can basically reach the O(1) level The constant access speed, compared with the traditional full table traversal or field index method, can effectively improve the retrieval speed in scenarios with large amounts of data.

In addition, since the file name is determined based on the hash identifier, information such as the variable name, sampling period, and sampling period will not be directly displayed, so even if a data leak occurs, the data obtainer cannot reversely determine the content of the file based on the file name and other content. For the specific data meaning, only a series of meaningless binary codes can be obtained, which can effectively improve data security; and because the corresponding time interval and data length between adjacent data points in the file are fixed, it can avoid saving time stamps, Spacers and other content have basically reached 100% of the effective data ratio, further improving storage space utilization and query efficiency.

In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the above data processing method will be described below in conjunction with specific embodiments. As an optional embodiment, assume that there are 1020 high-frequency variables of 20 ms in a certain Inner Mongolia wind farm prototype A that need to be collected and stored. Refer to the above storage method to create a secondary agent under the specified root directory /home/agent/store The directory structure stores each data point with a fixed length of 4 bytes (single-precision floating point number). The target data to be stored includes the target sampling data with a target sampling period of 20ms, and the target statistical periods are 1 second and 1 minute respectively. , 1 hour, 1 day target statistical data, and the target statistical data contains five statistical values including maximum value, minimum value, average value, peak-to-valley difference, and standard deviation. The target hash algorithm uses the MD5 algorithm.

Exemplarily, the system for performing data storage can run under the configuration of Intel J1900 low-power CPU, 2G memory, and 1TB mechanical hard disk. The actual test shows that when the historical curve query of the stored data is performed through the front-end interface under this condition It can display smoothly without obvious lag, and all queries including data file decompression can be completed within 3 seconds. The average memory usage is about 800MB, the average CPU utilization rate is less than 10%, and the amount of uncompressed data in a single day is about 800MB. 18GB, the daily data volume after compression is about 3GB, which realizes a data storage method with higher storage space utilization, higher access efficiency, and lower hardware requirements, which is better than the aforementioned traditional database storage method.

As a specific example, suppose it is necessary to store the raw data of the average wind speed, the target variable is 1012, the target time period is 2021-09-10 10:00:00 (that is, 10:00-11:00), and the target period is 20ms, the hash calculation process is: MD5("1012#2021091010#20ms")=A3D35596E3A9B99D99D1EF89D0493AC2, take the complete calculation result as the hash identifier, and create " A3D35596E3A9B99D99D1EF89D0493AC2.bin" file as the target file.

In this file, 500 pieces of 20ms original sampling data are written in batches every 10 seconds. The byte length is fixed at 4 bytes of single-precision floating point numbers, and 2000 bytes are written each time. If the data is missing, fill in -1.0 or Special values such as 0 are used as indicators of missing data.

Among them, the above-mentioned real-time data is cached for a period of time and then written in batches instead of continuously, which can effectively solve the problem that the number of simultaneously opened files exceeds the limit caused by the number of written data files being proportional to the number of measurement points. For example, if data is written in real time for all variables, at least 1020 files need to be processed at the same time, resulting in a huge hardware load; if it is set to cache for a period of time and write in batches, each variable can be processed sequentially within one cache cycle The corresponding data reduces the number of files that need to be processed at the same time and reduces the computing power load. At the same time, by setting the cache time before writing, a reasonable balance can be achieved between writing efficiency and failure loss. Since statistical data can be quickly recovered from high-frequency data, only the allowable loss time of high-frequency data needs to be considered. The failure loss caused by batch writing can be eliminated by saving the original data package file, so that the lost data cannot be recovered after a failure occurs.

In this embodiment, the above-mentioned target file can be set so that it does not need to be compressed during the data writing process, and can be compressed as a whole after all the data writing is completed, so as to avoid the repeated process of compression-decompression-compression during the data writing process , to improve data writing efficiency. If the file is in an uncompressed state when accessing the target file, it can be read directly; if it has been compressed, it will be decompressed to a preset temporary directory and read, such as a memory-based file system, etc., after a preset timeout period. Delete the corresponding decompressed files in the above temporary directory to free up storage space.

And, it is possible to control the total amount of data points in a single data file by reasonably controlling the period span of each type of data file, and balance the contradiction between batch storage to improve storage efficiency and fast decompression to improve access efficiency, so as to ensure that after data files are compressed, The time consumption of query decompression does not significantly affect the user experience; in actual use, the above-mentioned time generally needs to be controlled at the millisecond level, that is, the number of data points in a single data file can be controlled at about 100,000.

As another specific example, suppose it is necessary to store the minute-level statistical data of the average wind speed, the target variable is 1012, the target period is 2021-09-10 (that is, 2021-09-10 to 2021-09-11), the target The period is 1min, then the hash calculation process is: MD5("1012#202109#1m")=7C9BA434540CC928A9A9F0F7D1DD4603, take the complete calculation result as the hash identifier, and create it under the storage path /home/agent/store/7C/9B/ The "7C9BA434540CC928A9A9F0F7D1DD4603.bin" file is used as the target file.

In this file, 300 statistical values with a statistical interval of 1 minute are written in batches every hour, where the statistical values are written in a fixed order of maximum value, minimum value, average value, peak-to-valley difference, and standard deviation, and each type The statistical value contains a total of 60 data points, which correspond to the statistical data within the hour. Assuming that the byte length of all statistical values is fixed at 4 bytes of single-precision floating-point numbers, 1200 bytes are written each time.

Based on the above data storage process, as a specific embodiment, suppose that the stored raw data of the average wind speed needs to be queried, the variable ID of the variable to be queried is 1012, and the target query period is 2021-09-10 10:55～ A total of 10 minutes of high-frequency raw sampling data between 11:05, based on the preset time division rules, the corresponding actual query time period should be 2021-09-10 10:00:00 (that is, 10:00～11:00 ) and 2021-09-10 11:00:00 (that is, 11:00～12:00), and the target query period is 20ms, then the hash identifier is determined based on the above two actual query periods: MD5(“1012#2021091010# 20ms")=A3D35596E3A9B99D99D1EF89D0493AC2, MD5("1012#2021091011#20ms")=DD6A61096E6E118F2B73BE7491E4D89B.

Based on the above hash ID, go to /home/agent/store/A3/D3 to get the file "A3D35596E3A9B99D99D1EF89D0493AC2.bin," decompress and open the file, calculate the data offset as 55*60*50*4=660000 bytes, Continuously read 5*60*50*4=60000 bytes from the 660001th byte and return a total of 15000 floating point numbers; go to /home/agent/store/DD/6A to get the file "DD6A61096E6E118F2B73BE7491E4D89B.bin," decompress and Open the file without data offset, read 5*60*50*4=60,000 bytes continuously from the first byte and return 15,000 floating-point numbers, combine the above 30,000 floating-point numbers to get 2021-09-10 10 minutes of high-frequency original sampling data between 10:55 and 11:05 are returned to the user side.

Wherein, the above-mentioned data offset is determined based on the byte length and the corresponding relationship between the target query time period and the actual query time period, and is used to indicate the starting position of the data to be queried to be returned in the file to be queried; when the above-mentioned file contains a file header information, the data offset needs to be increased by the same number of bytes as the length of the file header information.

As another specific example, assume that the stored minute-level sampling data of the average wind speed needs to be queried, and specifically the average data of the aforementioned five statistical values, the variable ID of the variable to be queried is 1012, and the target query The time period is 2021-09-10 10:00-12:00, a total of two hours, and the target query cycle is 1 minute. Based on the preset time division rules, the corresponding actual query time period should be 2021-09 (that is, September to October months), the hash calculation process is: MD5("1012#202109#1m")=7C9BA434540CC928A9A9F0F7D1DD4603.

Based on the above hash ID, go to /home/agent/store/7C/9B to obtain the file "7C9BA434540CC928A9A9F0F7D1DD4603.bin," decompress and open the file, and calculate the data offset as (1440*9+60*10)*4= 54240 bytes, read the third floating-point number from every 5 floating-point numbers starting from the 54241th byte, read 120 floating-point numbers continuously, the above 120 are 2021-09-10 10:00～12: Between 00 and 2 hours, the target query period is 1min, and the average statistical data is returned to the user side.

Compared with the traditional database, the above-mentioned data processing method provided by this embodiment can refer to the data storage performance comparison schematic diagram shown in Figure 4 for its data storage performance advantages, and can refer to the data query performance shown in Figure 5 for the data query performance advantages Compare the schematic diagrams; the actual test shows that in the high-frequency data scenario, the storage space of the above method is about 1/6-1/8 of the traditional database, and the query speed is about 100-1000 times that of the traditional database, which effectively reduces the industrial cost. The high-frequency data storage space requirement of the device greatly improves the query efficiency, thereby reducing the requirements for hardware storage space and computing speed, and finally achieves the optimal economic benefit under the condition of meeting business needs.

So far, the introduction of the process of the above method has been completed in conjunction with specific embodiments.

The method provided by the embodiment of the present application is described above, and the device provided by the embodiment of the present application is described below:

Referring to FIG. 6, FIG. 6 is a structural diagram of a device provided by an embodiment of the present application. As shown in Figure 6, the device may include:

A data acquisition unit 601, configured to acquire the target data of the target variable in the target period and target period;

A hash determining unit 602, configured to determine the hash identifier and storage path of the target file through a target hash algorithm at least according to the above target time period, the above target period, and the variable identification of the above target variable;

The file creation unit 603 is configured to create the above-mentioned target file according to the above-mentioned hash identifier under the above-mentioned storage path, and write the above-mentioned target data into the above-mentioned target file.

In a possible implementation, in the data acquisition unit 601, the acquisition of the target data of the target variable in the target time period and target period is specifically used for:

Obtain the target sampling data of the above target variable in the target period and target sampling period, and/or obtain the target statistical data of the above target variable in the target period and target statistical period;

Wherein, the above-mentioned target statistical period is greater than the above-mentioned target sampling period, and the above-mentioned target statistical data is data obtained by performing statistics on sampling data or other statistical data based on the above-mentioned target statistical period.

In a possible implementation, in the hash determination unit 602, at least according to the target time period, the target period, and the variable identification of the target variable, when determining the hash ID and storage path of the target file through the target hash algorithm Specifically for:

According to the data type of the above-mentioned target data, the above-mentioned target time period, the above-mentioned target cycle, and the variable identification of the above-mentioned target variable, the hash identification and storage path of the target file are determined through the target hash algorithm.

In a possible implementation, in the hash determination unit 602, at least according to the target time period, the target period, and the variable identification of the target variable, when determining the hash ID and storage path of the target file through the target hash algorithm Specifically for:

At least according to the above-mentioned target period, the above-mentioned target period, and the variable identification of the above-mentioned target variable, determine the hash value matching the above-mentioned target data through the target hash algorithm;

Select a character with a preset number of digits from the above hash value as the hash identifier of the above target file;

Divide the characters identified by the above hash, query the storage location matching the division result in the preset multi-level file directory, and determine the storage path of the above target file.

In a possible implementation manner, in the above-mentioned file creation unit 603, the above-mentioned writing of the above-mentioned target data into the above-mentioned target file is specifically used for:

The data points in the above target data are sequentially written into the above target file in binary format according to the time sequence, wherein the binary storage digits of the data points in the above target file are fixed digits;

If there is a faulty data point with missing data, the above-mentioned faulty data point is occupied by the above-mentioned fixed number of digits based on an interpolation algorithm or a preset special value.

In a possible implementation, the file creation unit 603 is further configured to:

When a query instruction for data query of the above-mentioned target variable is received, according to the target query period, target query period, and variable identification carried in the above-mentioned query instruction, determine the number of one or more files to be queried through the above-mentioned target hash algorithm Hash ID and storage path;

Query the one or more files to be queried based on the hash identifier and the storage path, and obtain the data to be queried of the target variable in the target query period and the target query cycle.

In a possible implementation, in the above-mentioned file creation unit 603, according to the target query time period, target query period, and variable identification carried in the above-mentioned query command, the hash of one or more files to be queried is determined through the target hash algorithm. Specifically used when identifying and storing paths:

One or more actual query periods are determined according to the above target query period carried in the above query instruction based on the period division rule matching the above target query period;

Based on the above-mentioned one or more actual query periods, the target query cycle carried in the above-mentioned query instructions, and the variable identification, the hash identification and storage path of one or more files to be queried are determined through the target hash algorithm; wherein, each actual query The period matches the target period of a file to be queried.

In a possible implementation, the above-mentioned target file and the above-mentioned file to be queried also include file header information for data self-description, and the above-mentioned file header information includes variable identification, target time period, target period, byte length, encryption At least one of algorithm type and compression algorithm type; verification information for data verification of the above-mentioned target files is stored in the external device;

In the above-mentioned file creation unit 603, after the above-mentioned one or more files to be queried are queried based on the above-mentioned storage path and the above-mentioned hash identifier, it is also specifically used for:

Obtaining the above verification information in the above external device, and judging whether the above verification information matches the above file header information in the above file to be queried;

If it matches, it is determined that there is no data abnormality in the above-mentioned file to be queried, and the data to be queried under the above-mentioned target query period and query cycle of the above-mentioned target variable is obtained;

If they do not match, it is determined that there is a data exception in the above-mentioned file to be queried, and an error message is returned.

So far, the structural description of the device shown in FIG. 6 is completed.

The embodiment of the present application also provides the hardware structure of the device shown in FIG. 6 . Referring to FIG. 7, FIG. 7 is a structural diagram of an electronic device provided by an embodiment of the present application. As shown in FIG. 7, the hardware structure may include: a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions, In order to realize the method disclosed in the above example of the present application.

Based on the same application concept as the above-mentioned method, the embodiment of the present application also provides a machine-readable storage medium. The above-mentioned machine-readable storage medium stores a number of computer instructions. When the above-mentioned computer instructions are executed by a processor, the above-mentioned Methods exposed by the example.

Exemplarily, the above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, and so on. For example, the machine-readable storage medium can be: RAM (Radom Access Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid-state hard disk, any type of storage disk (such as CD, DVD, etc.), or similar storage media, or a combination of them.

The systems, devices, modules, or units described in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementing device is a computer, which may take the form of a personal computer, laptop computer, cellular phone, camera phone, smart phone, personal digital assistant, media player, navigation device, e-mail device, game control device, etc. desktops, tablets, wearables, or any combination of these.

For the convenience of description, when describing the above devices, functions are divided into various units and described separately. Of course, when implementing the present application, the functions of each unit can be implemented in one or more pieces of software and/or hardware.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

Moreover, these computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, The instruction means implements the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable equipment to produce computer-implemented processing, so that the information executed on the computer or other programmable equipment The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

The above descriptions are only examples of the present application, and are not intended to limit the present application. For those skilled in the art, various modifications and changes may occur in this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims (10)

1. A method of data processing, the method comprising:

acquiring target data of a target variable in a target time period and a target period;

determining a hash identifier and a storage path of a target file through a target hash algorithm at least according to the target time interval, the target period and the variable identifier of the target variable;

and creating the target file according to the hash identification under the storage path, and writing the target data into the target file.

2. The method of claim 1, wherein the target variable comprises target data at a target time period and a target period, comprising:

target sampling data of the target variable in a target time period and a target sampling period, and/or target statistical data of the target variable in the target time period and the target statistical period;

the target statistical period is greater than the target sampling period, and the target statistical data is data obtained by counting sampling data or other statistical data based on the target statistical period.

3. The method of claim 2, wherein determining the hash identifier and the storage path of the target file by a target hash algorithm according to at least the target time interval, the target period, and the variable identifier of the target variable comprises:

and determining the hash identification and the storage path of the target file through a target hash algorithm according to the data type of the target data, the target time interval, the target period and the variable identification of the target variable.

4. The method of claim 1, wherein determining the hash identifier and the storage path of the target file by a target hash algorithm according to at least the target time interval, the target period, and the variable identifier of the target variable comprises:

determining a hash value matched with the target data through a target hash algorithm at least according to the target time interval, the target period and the variable identification of the target variable;

selecting characters with preset digits from the hash value as hash marks of the target file;

dividing the characters of the hash marks, inquiring a storage position matched with a division result in a preset multistage file directory, and determining a storage path of the target file.

5. The method of claim 1, wherein writing the target data to the target file comprises:

sequentially writing data points in the target data into the target file in a binary format according to a time sequence, wherein the binary storage bit number of the data points in the target file is a fixed bit number;

and if the fault data points with missing data exist, occupying the fault data points as the fixed number based on an interpolation algorithm or a preset special value.

6. The method according to any one of claims 1-5, further comprising:

when a query instruction for performing data query on the target variable is received, determining hash identifications and storage paths of one or more files to be queried through a target hash algorithm according to a target query time interval, a target query cycle and a variable identification carried in the query instruction;

and inquiring the one or more files to be inquired based on the hash identification and the storage path, and acquiring the data to be inquired of the target variable in the target inquiry time period and the target inquiry period.

7. The method according to claim 6, wherein the determining the hash identifier and the storage path of one or more files to be queried by a target hash algorithm according to the target query time interval, the target query cycle, and the variable identifier carried in the query instruction comprises:

determining one or more actual query time periods according to the target query time period carried in the query instruction based on a time period division rule matched with the target query cycle;

determining hash identifications and storage paths of one or more files to be queried through a target hash algorithm based on the one or more actual query time intervals, the target query cycle and the variable identifications carried in the query instruction; wherein each actual query time interval is matched with a target time interval of a file to be queried.

8. The method according to claim 6, wherein the target file and the file to be queried further include file header information for performing data self-description, and the file header information includes at least one of a variable identifier, a target time period, a target period, a byte length, an encryption algorithm type, and a compression algorithm type; the external equipment stores verification information used for performing data verification on the target file;

after the one or more files to be queried are queried based on the hash identifier and the storage path, the method further includes:

acquiring the verification information in the external equipment, and judging whether the verification information is matched with the file header information in the file to be inquired;

if so, determining that the file to be queried has no data abnormality, and acquiring the data to be queried of the target variable in the target query time interval and the target query cycle;

if not, determining that the file to be queried has data exception, and returning an error prompt.

9. A data processing apparatus, characterized in that the apparatus comprises:

the data acquisition unit is used for acquiring target data of the target variable in a target time period and a target period;

a hash determining unit, configured to determine, according to at least the target time interval, the target period, and the variable identifier of the target variable, a hash identifier and a storage path of the target file through a target hash algorithm;

and the file creating unit is used for creating the target file according to the hash identifier under the storage path and writing the target data into the target file.

10. An electronic device, comprising: a processor and a machine-readable storage medium;

the machine-readable storage medium stores machine-executable instructions executable by the processor;

the processor is configured to execute machine executable instructions to perform the method steps of any of claims 1-8.

Images