CN116501709B

CN116501709B - IEC61850 data service function-based data storage method and device

Info

Publication number: CN116501709B
Application number: CN202310745045.0A
Authority: CN
Inventors: 黎波; 苏则驰; 杨岩; 赵忠; 吴栋频; 李玉杰
Original assignee: Shenzhen Shuanghe Electric Co ltd
Current assignee: Shenzhen Shuanghe Electric Co ltd
Priority date: 2023-06-25
Filing date: 2023-06-25
Publication date: 2023-09-05
Anticipated expiration: 2043-06-25
Also published as: CN116501709A

Abstract

The invention discloses a data storage method and equipment based on IEC61850 data service function, wherein the data storage method comprises a storage process, a read-write process and a retrieval process; in the storage process, the variable list and the variable information thereof are directly stored in a disk file without being put into a memory; the variable list and the variable information thereof are read from the disk file into the memory when being accessed, and are released from the memory after being accessed, thereby solving the problem that the IEC61850 data service function reads long-time span historical data; in addition, the invention improves the data reading, writing and searching speed by the consistency of the basic block size of the memory allocation in the application and the block size of the file system and the data compression function.

Description

IEC61850 data service function-based data storage method and device

Technical Field

The invention relates to the technical field of data storage, in particular to a power system in the industrial field using IEC61850 standard, and specifically relates to a data storage method and device based on an IEC61850 data service function.

Background

The IEC61850 standard is widely used in digital stations (or intelligent stations) of power systems in China, and all intelligent devices in the digital stations uniformly execute the standard. The data service function in the IEC61850 standard allows the master station system to call the history data of the intelligent equipment through the data service. The core of the data service function is a data control block, which is a defined structure body and is a runtime object, and only the program runtime is constructed, and the data control block is resident in the memory. The data is added to the data control block and is also placed in memory, and the standard does not define and specify the disk storage of the data. In practical application, the data is stored in the memory as history data, and the more the data is, the larger the occupied memory is. When the storage time span of the history data is required to be large to a certain extent by the practical application, the memory resources of the equipment are exhausted, and finally the equipment is crashed.

Disclosure of Invention

The following presents a simplified summary of embodiments of the application in order to provide a basic understanding of some aspects of the application. It should be understood that the following summary is not an exhaustive overview of the application. It is not intended to identify key or critical elements of the application or to delineate the scope of the application. Its purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is discussed later.

According to one aspect of the present application, there is provided a data storage method based on an IEC61850 data service function, including a storage process, a read-write process, and a retrieval process;

in the storage process, the variable list and the variable information thereof are directly stored in a disk file without being put into a memory; the variable list and the variable information thereof are read from the disk file into the memory when being accessed, and are released from the memory after being accessed;

in the read-write process, the disk file is directly read and written through IO;

in the searching process, first data are rapidly positioned, item information and all variable information of the first data are read, the largest and smallest index numbers of the variable information are recorded until all item information of the first data is processed, then the first data are positioned to the next data, the operation is repeated (the item information and all variable information of the first data are read, the largest index numbers of the variable information are recorded, the smallest index numbers of the variable information are recorded, and the maximum index numbers of the variable information are recorded, the smallest index numbers of the variable information are recorded, and the variable information of all the data are obtained, and then the data between the smallest index numbers of the variable information and the largest index numbers of the variable information are read into a memory at one time, and then the item information and the variable information read into the memory are processed; the block serial number where the variable information is located is the index number blockidx, the block is the size of a file system file page, and the variable information writes a file by taking the block as a basic unit.

Further, the searching process reads a plurality of pieces of data according to the initial ID of the data entry, or searches all the data entries in the reading time period according to the time period, and specifically comprises the following steps:

step Q1: rapidly positioning a first piece of data, which specifically comprises: when initializing, reading index data from an index file to a memory to form an index list variable, and reading data information from a data information file to the memory to form a data information list variable; obtaining first index data from index list variables of the memory, calculating an interval position of the first index data to be searched according to the searched starting ID or the difference value between the starting time and the first index data, and jumping the interval position to obtain a file offset position of data information; positioning the data information list variable designated position of the memory according to the file offset position of the data information to obtain the data information of the first piece of data;

step Q2: acquiring item information and all variable information of the piece of data;

step Q3: recording the maximum and minimum values of index numbers blockidx in the variable information obtained in the step Q2;

step Q4: positioning the next piece of data, repeating the step Q2 and the step Q3 until all the data items to be retrieved are circularly processed, and finally obtaining two values: maximum and minimum index numbers blockidx;

Step Q5: and shifting to the shifting position of the data file according to the minimum index number blockidxS, wherein the shifting position is calculated according to the formula: minimum index number blockidxS block; block is the file system file page size dimension;

step Q6: directly reading data from the minimum index number blockidxS to the maximum index number blockidxM into a memory through one IO operation;

step Q7: processing the first piece of data in the memory obtained in the step Q6, and processing item information to obtain item variable information;

step Q8: converting index numbers blockidx and blockOffset in the entry variable information into memory addresses, and converting the memory addresses into a formula: (index number blockidx-minimum index number blockidxS) block+blockoffset; the block offset refers to the offset of the variable information in the block;

step Q9: taking the data on the memory address as the data of the item variable information;

step Q10: repeating the steps Q7-Q8, and circularly processing all item variable information and data items;

step Q11: and after the data entry is read, releasing the data read from the data file after the subsequent communication processing is completed, thereby releasing the memory.

The above steps can be seen that the data is retrieved and read, the first piece of data is searched, and because the index data is equally spaced, the index of the first piece of data can be hit by one jump in the memory, and then the entry information and the variable information of the first piece of data are found according to the file offset position of the index. All variables in the search range are traversed in the memory to obtain the minimum and maximum data block index numbers blockidx, and the processing process can be completed quickly. These two values are obtained to determine the location of the data in the data file and its read range, which covers the data, just the data that is needed for retrieval, not more or less. The target position is hit once, a proper amount of data is read, the file is read through direct IO operation, and the large block of data is read once, which are all guarantees of quick data retrieval.

Further, the data storage method further includes a file definition process including definitions of the data index file, the data information file, and the data file;

wherein, the data index file is defined as follows:

1) The data index file is generated according to logic devices, each logic device corresponds to one index file, and the file name is provided with a logic device number;

2) The data structure stored in the data index file is marked as Q, and each time one piece of data is generated, an index data block of the data structure Q is generated and stored in the index file;

the data information file is defined as follows:

1) The data information file is generated according to logic equipment or a time period; the data information file name is generated according to the logic device and is provided with a logic device number; generating according to the time period, and carrying time information; if the data information is generated only according to the logic device, one logic device corresponds to one data information file; if the data information file is generated according to the time period, one logic device corresponds to a plurality of data information files;

2) The data structure of the data information file storage comprises a data structure A defined by the IEC61850 standard and a self-defined data structure B, wherein the self-defined data structure B is formed by expanding a pointer list_of_var pointing to a variable list in the data structure A defined by the IEC61850 standard and storing the pointer list_of_var into the information file;

3) Each time a piece of data is generated, a data block of a data structure a is written into the data information file.

4) All variables in the variable list are then processed according to the list_of_var loop in data structure a.

5) Obtaining the position of the variable in list_of_var and the length of variable information after processing one variable; the information forms a data block of a data structure B, and is written in sequence behind the data block of the data structure A;

the data file is defined as follows:

1) The data file is generated according to the logic device and according to the time period, and the name of the data file is provided with the logic device number and the data starting time;

2) Reading the data file is dependent on information in the data information file;

the logical relationship of the data index file, the data information file, and the data file is defined as follows:

1) An index data in the index file represents a specific data entry, and its file offset position represents the position of the data information of the entry stored in the data information file;

2) Storing information of each piece of data in the data information file in sequence;

3) The data structure A in the data information file stores basic information of data; the data structure A is followed by a plurality of data structures B, and the data structure B stores the variable data length and the block index and the block offset position of the variable data in the data file; the data of the first data structure B is the information of the first variable, the data of the second data structure B is the information of the second variable, and so on;

4) The same variable on different data entries may point to the same data;

further, the storing process further includes: setting the basic block size of memory allocation and the block size of a file system to be consistent; when a data file is written, a memory buffer area with the size of N block is allocated, and when the data of the memory buffer area is fully written, the data of the whole memory buffer area is written into a disk at one time; and the length N of each write data is equal to the block, so that the alignment of data and the pages of the disk file is ensured, and the write performance is improved.

Further, in the read-write process, the disk is directly read and written through IO; specifically, the beginning block and the last block are found according to the queried data, the number N of related blocks is calculated, a memory buffer area with the size of N block is allocated, the data with the size of N block are directly read at one time, the memory is directly operated by the follow-up operation, frequent disk IO is avoided, and the overall performance is greatly improved.

Further, the data storage method further comprises a data self-checking process, when the device is restarted, checking the data related files (all data index files, data information files and data files in the disk), removing the destroyed part of data, and reserving the intact data, and specifically comprises the following steps:

Step A1: opening a data index file and a data information file;

step A2: traversing all index data in the data index file;

step A3: reading data in the data information file according to the file offset position of the index data;

step A4: judging whether the item ID in the index data and the item ID in the data information are consistent; if not, jumping to the step A8;

step A5: reading all variable information in the data information file;

step A6: judging whether the position of the variable index is abnormal, if so, jumping to the step A8; wherein, the index position is negative or abnormal when the index position is a preset value;

step A7: judging whether the offset position of the variable information is abnormal, if so, jumping to the step A8; the offset position is negative or larger than the size of the data file, and is abnormal;

step A8: deleting the data corresponding to the data index and the information and data of all the data behind the data index, and then exiting from the self-checking;

step A9: and processing index data of the next piece of data until all index data are processed, and exiting the self-checking.

Furthermore, the data storage method also comprises a data initialization process, when the equipment is restarted, data is initialized, historical data information data is read from the file and added into an IEC61850 data control block for the master station system to call, and the method specifically comprises the following steps:

Step B1: opening a data index file and a data information file;

step B2: the index data is put into the idxlist memory variable, and the information data is put into the info list memory variable, so that the subsequent use is facilitated without repeating the operation file;

step B3: traversing all index data;

step B4: reading data information data according to the file offset position of the index data;

step B5: reading all variable information;

step B6: adding the data information into an IEC61850 data control block;

step B7: the next piece of index data is processed until all index data are processed.

Further, the read-write process further includes a step of generating a data entry, specifically including:

step C1: the method comprises the steps of distributing a write buffer area in the first running process, wherein the size of the buffer area is N, N is an integer greater than 0, and block is the file page size of a file system;

step C2: judging whether the time of the data item is a new period;

step C3: if the time period is a new time period, closing the original data file, and creating a new data file according to a rule;

step C4: otherwise, continuing to use the original data file;

step C5: judging whether the data quantity exceeds a set value; if yes, deleting the old data; the deletion strategy is to delete the oldest data file, deduce the data item involved in deletion, and delete the data index file, the data information file and all corresponding item information in the IEC61850 data control block; if not, continuing to execute the next step C6;

Step C6: generating a new index block according to the new data entry information, putting the new index block into an idxlist memory variable generated during data initialization, and writing the whole idxlist data into an index file according to a certain strategy regularly to prevent a large amount of data loss caused by restarting the equipment;

step C7: generating item information and compressing data:

the first piece of data is not entangled with the historical data, the length of the current data file is read, if the length is 0, the index number blockidx value of all variables is 0, and the current index number blockidx value is 0; if the index value is not 0, dividing the file length by the block to obtain the number of blocks of the existing data of the file, wherein the current index value is the value of the block idx, namely the current data starts from a new block; all variable information of a variable list in a data item is copied to a write buffer area in sequence, and the offset positions of the copies of the variable information are recorded as blockOffset; the data information of the data is put into a temporary memory tmpinflist list;

if the current variable value is not the same as the previous value, the variable is set as the index numbers blockidx and blockOffset of the previous value according to the new-to-old sequence, the index numbers blockidx and blockOffset of the same variable of the previous data are searched from a tmpinflist memory temporary list, the values are found in a write buffer, and whether the current variable value is the same as the previous value is judged; if the difference is not the same, continuing to find the next piece of data; if the searched previous value of the index number blockidx and the deviation from the current index number blockidx are somewhat larger, or the first index number which is not found yet is found, the index number blockidx of the variable is set as the current index number blockidx, the data of the index number blockidx is copied to the tail part of a write buffer area, and the deviation of the position of the data and the head part of the current block is calculated as the blockOffset; all variables are treated according to the same logic; the data compression function can be realized by the processing of the step;

When the variable information is fully written with one block, adding 1 to the current index number blockidx, and calculating the blockOffset of the next variable from 0;

finally, the generated item information is put into an Infolist memory variable generated during data initialization, and the whole Infolist data is written into an information file regularly according to a certain strategy, so that a large amount of data loss caused by restarting the equipment is prevented;

the finally generated item information is put into a tmpinflist temporary list for subsequent value comparison. Defining the upper limit of the number of the tmpinflist temporary list storage items according to a certain strategy, and deleting the oldest data items if the number exceeds the upper limit;

step C8: the new data item information is obtained in the last step, variable data is copied to a write buffer zone, when the write buffer zone is full, a write-once disk function is called, and the whole write buffer zone data is directly written into a disk without a kernel; the data of the next variable is written from the head of the writing buffer, tmpinflist is emptied, but the value of the current index number blockidx is not affected; and large data is written once, the data size is integral multiple of a file page, the alignment of the data and the continuous writing are ensured, and the reading and writing efficiency is greatly improved.

Step C9: the IEC61850 data control block also adds a piece of data entry information.

As can be seen from the above steps, when the data stores the file, if the values of the same variable in the data of different items are the same, only the offset pointer points to the same position, and new data is not generated, so that the purpose of data compression is achieved.

According to another aspect of the present application, there is provided a data storage device based on an IEC61850 data service function, which performs the above-described data storage method based on the IEC61850 data service function.

By adopting the scheme, the variable list and the variable information thereof are not put into the memory, but put into the disk file, and are read from the file into the memory when accessed and released from the memory after accessed, thereby solving the problem of reading long-time span historical data by the IEC61850 data service function; the basic block size distributed by the internal memory in the application is consistent with the block size of the file system, so that the data reading and writing speed is improved; in addition, for a specific scene of the power system, since many data are unchanged in a certain time period (unchanged in the data precision range and negligible in the outside of the precision range), the same data only need to actually access one data, so that the purpose of data compression is achieved, and the read-write performance is further improved.

Drawings

The invention may be better understood by referring to the following description in conjunction with the accompanying drawings in which like or similar reference numerals are used to indicate like or similar elements throughout the several views. The accompanying drawings, which are included to provide a further illustration of the preferred embodiments of the invention and together with a further understanding of the principles and advantages of the invention, are incorporated in and constitute a part of this specification. In the drawings:

FIG. 1 is a flow chart of data self-checking according to an embodiment of the present invention;

FIG. 2 is a flow chart of data initialization according to an embodiment of the present invention;

FIG. 3 is a flow chart of generating data entries according to an embodiment of the present invention;

FIG. 4 is a flow chart of data retrieval and reading according to an embodiment of the present invention;

FIG. 5 is a diagram illustrating a memory structure of a data control block according to an embodiment of the present invention;

FIG. 6 is a logical relationship between three types of files according to an embodiment of the present invention.

Detailed Description

Embodiments of the present invention will be described below with reference to the accompanying drawings. Elements and features described in one drawing or embodiment of the invention may be combined with elements and features shown in one or more other drawings or embodiments. It should be noted that the illustration and description of components and processes known to those skilled in the art, which are not relevant to the present invention, have been omitted in the drawings and description for the sake of clarity.

The embodiment of the invention provides a data storage method based on an IEC61850 data service function, which has the magnetic disk storage capacity, the data compression capacity, the data quick read-write magnetic disk capacity, the data quick retrieval capacity, the data retrieval capacity according to an item ID, the data retrieval capacity according to a time period, the data self-checking function and the data self-maintenance function of IEC61850 data.

The data storage method designs three types of files: a data index file, a data information file and a data file.

Data index file:

the data index file is generated according to logic devices, each logic device corresponds to one index file, and the file name is provided with a logic device number. (logical devices are virtual devices defined by the IEC61850 standard, which are a collection of related models, data sets, and functions, a physical device is typically composed of multiple logical devices

The data index file stores the following data structure:

struct log_idx

{

unsigned int entry _id;// item id

CharDate [9 ]// entry time

IntfileOffset;// offset of entry information in data information file

}

entry_id, entry id: each time a piece of data is generated, a unique ID is generated.

date, entry time: each time a piece of data is generated, a corresponding time is taken.

Offset of entry information in the data information file: every time a piece of data is generated, there is entry information written into another file: data file information. Here its offset position in the data file information is recorded, facilitating fast retrieval.

Each time a piece of data is generated, an index data block of a struct log_idx structure is generated and stored in an index file. From the above, it is known that the byte length of the index data block structure is fixed, and the data size is not large, which is the key of quick retrieval.

Data information file:

the data information file is generated by the logic device and may be generated again in time periods if desired. The data information file name is provided with a logic device number; if also generated in time slots, time information is also provided. If the data information is generated only according to the logic device, one logic device corresponds to one data information file; if it is also generated in time slots, one logical device corresponds to a plurality of data information files.

The data structure of the data information file storage is as follows:

a) Structure 1:

typedef struct mvlJournalEntry

{

DBL_LNK l; /* for user, not used by MVL */

ST_INT entry_id_len; /* Octet string ID, size 1-8 */

ST_UCHAR entry_id[8];

APP_REF orig_app;

MMS_BTOD occur_time; /* occurrence time */

ST_INT16 entry_form_tag; /* entry form tag */

union

{

struct /* entry form is DATA */

{

ST_BOOLEAN event_pres; /* event present */

OBJECT_NAME evcon_name; /* event condition name */

ST_INT16 cur_state; /* current state */

ST_BOOLEAN list_of_var_pres; /* list of variables present */

ST_INT num_of_var; /* number of variables */

VAR_INFO *list_of_var; /* ptr to array */

} data;

ST_CHAR *annotation; /* pointer to annotation */

}ef;

} MVL_JOURNAL_ENTRY;

this is a structure defined by the IEC61850 standard, and the structure will not be described in terms of reference to known general standards.

b) Structure 2:

struct write_var_info

{

index of the variable Intvar_idx;// var in varlist

int datalen;// var variable information length

Int index number blockidx;// data block index

Offset of int blockOffset;// var variable information in block data block

};

This is a custom structure that is developed for list_of_var in structure 1 and stored into an information file (list_of_var is a pointer to a list of variables). Wherein:

var_idx refers to which variable in the list of list_of_var variables the variable is.

datalen refers to the data length of the variable.

Index number blockidx, wherein the block is the size of a file system file page, variable information is written into a file by taking the block as a basic unit, the data of the variable information is in which block, and the block serial number is the index number blockidx. The index number blockidx is a data block index, the data is stored in a disk and is artificially divided into a plurality of blocks (blocks), and each block is provided with an index number corresponding to the block idx. The first data block in the file has a blockidx of 0 and the last block has a blockidx of N. When retrieving data according to the condition, a piece of continuous data in the file is required to be retrieved, and the piece of continuous data may relate to a plurality of continuous blocks, and the blocks have corresponding indexes (such as 7, 8, 9 and 10), for example, a certain retrieval, the minimum blockidx is 7, and the maximum blockidx is 10. According to the two information, the position of the read data is rapidly positioned; and how much data is read at one time, and the data is read for several times, so that IO operation times are reduced as much as possible, and the reading performance is improved

The block offset refers to the offset of the data of the variable in the block.

Every time a piece of data is generated, a data block with a structure 1 is written into the data information file.

All variables in the variable list are then processed according to the list_of_var loop in structure 1.

Each time a variable is processed, the location of the variable in list of var is known, along with the length of the variable information. If the variable information is the same as the existing identical variable information, pointing to the existing data position through index numbers blockidx and blockOffset; if the data are not the same, copying the data to the back of the last data of the current block, setting the index number blockidx as the index number of the current block, and setting the blockOffset as the offset of the current writing position. This information constitutes a data block of structure 2, which is written sequentially after the data block of structure 1.

Data file:

the data volume of the data file is relatively large, so that the data file is generated according to logic equipment and also generated according to time periods, and the file is prevented from being excessively large. Thus, the name of the data file carries the logical device number, as well as the data start time.

From the above description of the design of the data information file it does not have any self-describing capabilities. Its reading must depend on the information in the data information file.

The logical relationship of these three files is shown in FIG. 6:

one index data in the index file represents one specific data entry. Its file offset location represents the location where the data information for the entry is stored in the data information file.

The data information file stores the information of each piece of data in sequence.

The "structure 1" in the data information file is the mvlJourn entry structure, which stores the basic information of the data. The structure 1 is followed by a plurality of structures 2, namely a write_var_info structure, which stores the data length of the variable and the block index and the block offset position of the data in the data file. The first "structure 2" data is variable 1 information, the second "structure 2" data is variable 2 information, and so on.

It can be seen that the same variable on different data entries can point to the same data.

The data storage method of the embodiment of the invention comprises a data self-checking process, and when the equipment is suddenly powered off or restarted, the data file is possibly damaged, so that the data related file is necessarily checked when the equipment is restarted and the program runs for the first time, the damaged part of the data is removed, and the intact data is reserved. The data self-checking process is shown in fig. 1, and comprises the following steps:

Step 1: the data index file and the data information file are opened first.

Step 2: traversing all index data in the data index file.

Step 3: and reading the data in the data information file according to the file offset position of the index data.

Step 4: it is determined whether the entry ID in the index data and the entry ID in the data information agree.

Step 5: and reading all variable information in the data information.

Step 6: and judging whether the position of the variable index is abnormal. An index position is negative or a large value is abnormal. Wherein, the large value is described as follows: the blockidx value is checked for validity in order to deal with data corruption that may be caused by a sudden power-down restart of the device. The fact that blockidx exceeds a large value is not effective is not specifically limited, but is empirically determined by a developer at the time of actual development. In practical development, the data amount of a file can be estimated according to specific situations, such as about 1 gigabyte of a file. Then, each data block (block) is set to be 4K or 128K, and the large value is preset by a developer according to actual conditions. However, when the maximum capacity of the file is determined, and the block size is determined, a certain margin is considered, so that the effective maximum value of the blockidx can be determined.

Step 7: and judging whether the offset position of the variable information is abnormal. The offset position is negative or larger than the size of the data file, and is abnormal.

Step 8: if the item IDs in the step 4 are inconsistent or one of the item IDs in the step 6 and the step 7 is judged to be abnormal, the step is entered, the data corresponding to the data index and the information and the data of all the data behind the data index are deleted, and then the self-checking is exited.

Step 9: otherwise, step 8 is skipped, the index data of the next piece of data is processed until all the index data are processed and the self-checking is stopped. Each piece of data stored is recorded, and corresponding information, such as a serial number, a generation time, an index number blockidx value corresponding to the data stored in the file, and the like, is additionally generated, and the information is called index data.

The data storage method of the embodiment of the invention comprises a data initialization process, IEC61850 data service, wherein the data information is firstly acquired in two steps, and then whether the data of certain data items are acquired is determined according to the need. Therefore, when the equipment is restarted, data is initialized, historical data information data is read from the file and added into an IEC61850 data control block for the master station system to call. The data information data is usually not too large, and the normal operation of the equipment is not affected when the data information data is completely read into the memory. As shown in fig. 2, the data initialization process includes the steps of:

Step 1: opening a data index file and a data information file;

step 2: the index data is put into the idxlist memory variable, and the information data is put into the info list memory variable, so that the subsequent use is facilitated without repeating the operation file;

step 3: traversing all index data;

step 4: reading data information data according to the file offset position of the index data;

step 5: reading all variable information;

step 6: adding the data information into an IEC61850 data control block;

step 7: the next piece of index data is processed until all index data are processed.

The data storage method of the embodiment of the invention comprises a data entry generation process, wherein the data entry is generated by other modules outside the program and is generated according to the service demand period. After the data entry is generated, the function related to the invention is called for processing, and as shown in fig. 3, the process of generating the data entry specifically comprises the following steps:

step 1: the method comprises the steps of distributing a write buffer area in the first running process, wherein the size of the buffer area is N, N is an integer greater than 0, and block is the file page size of a file system;

step 2: it is determined whether the time of the data entry is a new period. This is handled according to a data period partitioning strategy, such as if the partitioning strategy is by day, then the new data and the last piece of data are not the same day, then the new period;

Step 3: if the time period is a new time period, the original data file is closed, and a new data file is created according to the rule.

Step 4: otherwise, the original data file is used continuously.

Step 5: and judging whether the data quantity exceeds a set value.

Step 6: if so, the old data is deleted. The deletion policy is to delete the oldest data file, then deduce the data entry involved in the deletion, then delete the data index file, the data information file, and all the corresponding entry information in the IEC61850 data control block. Therefore, a plurality of data can be deleted once, so that not only can the data in the data file be prevented from being moved, but also the data in the index file and the information file can be prevented from being moved frequently, and the system performance is improved.

Step 7: if not, step 5 is skipped.

Step 8: and generating a new index block according to the new data entry information, putting the new index block into an idxlist memory variable generated during data initialization, and writing the whole idxlist data into an index file according to a certain strategy regularly to prevent a large amount of data loss caused by restarting the equipment.

Step 9: generating item information:

a) The first piece of data is not entangled with the historical data, the length of the current data file is read, if the length is 0, the index number blockidx value of all variables is 0, and the current index number blockidx value is 0; if the index value is not 0, dividing the file length by the block to obtain the number of blocks of the existing data of the file, wherein the current index value is the value of the block idx, namely the current data starts from a new block. All the variable information of the variable list in the data entry is copied to the write buffer in sequence, and the offset position of the copies is recorded as the blockOffset. The data information of the data is put into a temporary list of the memory tmpinflist.

b) If the current variable value is not the same as the previous value, the variable is set as the index numbers blockidx and blockOffset of the previous value according to the new-to-old sequence, the index numbers blockidx and blockOffset of the same variable of the previous data are searched from a tmpinflist memory temporary list, the values are found in a write buffer, and whether the current variable value is the same as the previous value is judged; if not, the next piece of data is continued to be found. If the found previous value of the index number blockidx and the deviation from the current index number blockidx are somewhat larger, or the first index number which is not found yet is found, the index number blockidx of the variable is set as the current index number blockidx, the data of the index number is copied to the tail part of the write buffer, and the deviation of the position of the index number blockidx and the current block head is calculated as the blockOffset. All variables are treated as the same logic. This processing realizes the data compression function.

c) When the variable information is written with one block, the current index number blockidx is added with 1, and the blockOffset of the next variable is calculated from 0.

d) And finally, the generated item information is put into an Infolist memory variable generated during data initialization, and the whole Infolist data is written into an information file according to a certain strategy at regular intervals, so that a large amount of data loss caused by restarting the equipment is prevented.

e) The finally generated item information is put into a tmpinflist temporary list for subsequent value comparison. Defining the upper limit of the number of the tmpinflist temporary list storage items according to a certain strategy, and deleting the oldest data items if the number exceeds the upper limit;

step 10: and when the write buffer is full, a write-once disk function is called, and the data of the whole write buffer is directly written into a disk without passing through a kernel. The data of the next variable is then written from the write buffer header, tmpinflist is emptied, but does not affect the value of the current index number blockidx. The large block of data is written once, the data size is integral multiple of the file page, the alignment of the data and the continuous writing are ensured, and the reading and writing efficiency is greatly improved;

step 11: the IEC61850 data control block also adds a piece of data entry information.

The data storage method of the embodiment of the invention comprises a data retrieval and reading process, wherein the data retrieval can read a plurality of pieces of data according to the initial ID of the data item; all data entries within the read time period may also be retrieved by time period. As shown in fig. 4, the data retrieval reading process includes the steps of:

Step 1: the first piece of data is located quickly. The first index data is obtained from the idxlist (the idxlist reads the index data from the index file to the index list variable name of the memory when initializing), and then the interval position of the first index data to be searched can be conveniently calculated according to the searching initial ID or the difference value between the initial time and the first index data, and the file offset position of the data information can be obtained by jumping the interval. And positioning to the designated position of the info list according to the file offset position of the data information to obtain the data information of the first piece of data. (Infolist is the variable name of the data information list that reads the data information from the data information file to the memory when initializing;

step 2: acquiring item information and all variable information;

step 3: recording the maximum and minimum values of index numbers blockidx in the variable information;

step 4: and (3) circularly processing all the data items to be retrieved to finally obtain two values: maximum and minimum index numbers blockidx;

step 5: and shifting to the shifting position of the data file according to the minimum index number blockidxS, wherein the shifting position is calculated according to the formula: minimum index number blockidxS. Block is the file system file page size dimension;

Step 6: reading data from the minimum index number blockidxS to the maximum index number blockidxM to a memory through direct IO operation once;

step 7: processing the first piece of data, processing the item information, and obtaining item variable information;

step 8: converting index numbers blockidx and blockOffset in variable information into memory addresses, and converting the memory addresses into a formula: (index number blockidx-minimum index number blockidxS) block+blockoffset;

step 9: taking the data on the memory address as the data of the variable;

step 10: circularly processing all variables and data entries;

step 11: and after the data entry is read, releasing the data read from the data file after the subsequent communication processing is completed, thereby releasing the memory.

The above steps can be seen that the data is retrieved and read, the first piece of data is searched, and because the index data is equally spaced, the index of the first piece of data can be hit by one jump in the memory, and then the entry information and the variable information of the first piece of data are found according to the file offset position of the index.

All variables in the search range are traversed in the memory to obtain the minimum and maximum data block index numbers blockidx, and the processing process can be completed quickly. These two values are obtained to determine the location of the data in the data file and its read range, which covers the data, just the data that is needed for retrieval, not more or less. The target position is hit once, a proper amount of data is read, the file is read through direct IO operation, and the large block of data is read once, which are all guarantees of quick data retrieval.

The embodiment of the application also provides a data storage device based on the IEC61850 data service function, which executes the data storage method based on the IEC61850 data service function.

As shown in FIG. 5, the IEC61850 data control block is shown in the data structure in the memory, when a new data entry is generated, an entry information is added in the data information list, and a new variable list and the data of the corresponding variable are generated. So the memory growth is relatively fast. In practice, however, the variable list and its variable information are only used when accessed, and are not used at all other times. In order to solve the problem of too fast memory growth, the application comprises the following steps: the variable list and the variable information thereof are not put into the memory, but are put into the disk file. They are read from the file into memory only when they are accessed and released after access.

In order to ensure the performance of the system and ensure the high-efficiency reading and writing of data, the scheme of the application is designed as follows:

1. and directly IO read-write the disk. Direct IO read-write may be referred to in technical terms as zero copy. Typical file reads and writes to disk through the kernel, which may involve copying data from user space to kernel space and from kernel space to disk and vice versa. Such an operation has two more memory copies and two context switches, which has a relatively large impact on system performance. The disk is directly read and written, the kernel layer is skipped, the user layer is directly connected with the hardware layer, and redundant memory copy and context switching are avoided. Furthermore, the usual file read-write, which is affected by the whole environment, is not optimally suited for our application by the operating system managing the read-write buffer with a general policy. Because our application behavior can be estimated, data is generated periodically, the data length is determined, and self-management buffering in the application is more suitable for our application. Therefore, the disk is directly read and written, and the application manages the buffer, so that redundant memory copy and context switching are reduced, and the system performance can be improved to the greatest extent.

2. The size of the basic block allocated by the internal memory of the application is consistent with the block size of the file system. When writing data files, memory buffer areas with the size of N block are allocated, and only if the data of the buffer areas are fully written, the data of the whole buffer areas can be directly written into the disk once. The length N of each writing data is equal to the block, so that the alignment of data and the pages of the disk file is ensured, and the writing performance can be improved. Because the operation of the equipment is stable, and the unexpected restart phenomenon is less, N can be slightly larger under the condition of not affecting the whole memory of the system. Therefore, the disk can be written at low frequency, and the writing performance is greatly improved. And each time large data is written, the continuity of the disk address is ensured, which has great benefit to the reading and writing performance.

3. When reading data, firstly finding out the beginning block and the last block according to the queried data, calculating the number N of related blocks, distributing a memory buffer area with the size of N block, directly reading the data with the size of N block at one time, directly operating a memory by subsequent operation, avoiding frequent disk IO and greatly improving the overall performance. And the read data length N is equal to the block each time, so that the alignment of data and disk file pages is ensured, and the read performance can be improved. Of course, if the value of N is too large, the data may be read in several times.

4. And (5) data compression. The data compression algorithm is specific to the specific scene of application, has no generality, but is effective for our scene. In our application scenario, especially in power systems, many data are constant over a period of time. By data invariant, it is meant that the data is invariant within the accuracy of the data, and variations outside the accuracy range are negligible, such as 110.233KV and 110.232KV, which can be considered identical if we are concerned only with the last two decimal places. Even though there is a change in the data, they have a high probability of varying slightly above and below a mean line, so looking forward, or the same data can be found. The same data can be accessed only by one data, so that the purpose of data compression is achieved. The data is compressed, and the read-write performance is greatly improved.

Through the scheme, the problem that the IEC61850 data service function reads long-time span historical data is solved, and meanwhile, the data reading and writing speed and the data searching speed are improved.

In the above embodiments and examples, reference numerals of numerical composition are used to denote the individual steps and/or units. It should be understood by those of ordinary skill in the art that these reference numerals are merely for ease of description and illustration and are not meant to be a sequential or any other limitation thereof.

Furthermore, the methods of the present invention are not limited to being performed in the time sequence described in the specification, but may be performed in other time sequences, in parallel or independently. Therefore, the order of execution of the methods described in the present specification does not limit the technical scope of the present invention.

While the invention has been disclosed in the context of specific embodiments, it should be understood that all embodiments and examples described above are illustrative rather than limiting. Various modifications, improvements, or equivalents of the invention may occur to persons skilled in the art and are within the spirit and scope of the following claims. Such modifications, improvements, or equivalents are intended to be included within the scope of this invention.

Claims

1. A data storage method based on IEC61850 data service function is characterized in that: the method comprises a storage process, a read-write process and a retrieval process;

in the storage process, the variable list and the variable information thereof are directly stored in the disk file and are not put in the memory, the variable list and the variable information thereof are read from the disk file into the memory when being accessed, and the variable list and the variable information thereof are released from the memory after being accessed;

in the searching process, first data are rapidly positioned, item information and all variable information of the first data are read, the largest and smallest blockidx in the variable information are recorded until all item information of the first data is processed, then the first data are positioned to the next data, the operation is repeated until the largest blockidx and the smallest blockidx of the variable information of all the data are obtained, the data between the smallest blockidx and the largest blockidx are read into a memory at one time, and then the item information and the variable information read into the memory are processed; the block serial number where the variable information is located is the block idx of the variable information, the block is the size of a file system file page, and the variable information writes a file by taking the block as a basic unit;

the data storage method further comprises a data initialization process, when the equipment is restarted, data are initialized, historical data information data are read from the file and added into an IEC61850 data control block for the master station system to call, and the method specifically comprises the following steps:

step B1: opening a data index file and a data information file;

Step B3: traversing all index data;

step B5: reading all variable information;

step B6: adding the data information into an IEC61850 data control block;

step B7: processing the next index data until all index data are processed;

the read-write process further includes a step of generating a data entry, specifically including:

step C2: judging whether the time of the data item is a new period;

step C4: otherwise, continuing to use the original data file;

step C7: generating item information and compressing data:

the first piece of data is not entangled with the historical data, the length of the current data file is read, if the length is 0, the blockidx value of all variables is 0, and the current blockidx value is 0; if the current block is not 0, dividing the file length by the block to obtain the number of blocks of the existing data of the file, wherein the current block idx value is a numerical value, namely the current data starts from a new block; all variable information of a variable list in a data item is copied to a write buffer area in sequence, and the offset positions of the copies of the variable information are recorded as blockOffset; the data information of the data is put into a temporary memory tmpinflist list;

if the current variable value is not the same as the previous value, the previous variable value is set to be the same as the previous value by searching the previous variable value from the tmpinflist memory temporary list according to the new-to-old sequence; if the difference is not the same, continuing to find the next piece of data; if the searched previous value blockidx and the deviation from the current blockidx are somewhat larger, or the first block of the same value is found but not found yet, the blockidx of the variable is set as the current blockidx, the data of the variable is copied to the tail part of a write buffer area, and the deviation between the position of the variable and the head part of the current block is calculated to set the blockOffset of the variable; all variables are treated according to the same logic; the data compression function can be realized by the processing of the step;

When the variable information is written with one block, adding 1 to the current block idx, and calculating the block offset of the next variable from 0;

finally, the generated item information is put into a tmpinflist temporary list for subsequent value comparison; defining the upper limit of the number of the tmpinflist temporary list storage items according to a certain strategy, and deleting the oldest data items if the number exceeds the upper limit;

step C8: the new data item information is obtained in the last step, variable data is copied to a write buffer zone, when the write buffer zone is full, a write-once disk function is called, and the whole write buffer zone data is directly written into a disk without a kernel; the data of the next variable is written from the head of the writing buffer, tmpinflist is emptied, but the value of the current blockidx is not affected; the large block of data is written once, the data size is integral multiple of the file page, the alignment of the data and the continuous writing are ensured, and the reading and writing efficiency is greatly improved;

2. The data storage method of claim 1, wherein: the data storage method further includes a file definition process including definitions of a data index file, a data information file, and a data file;

wherein, the data index file is defined as follows:

the data information file is defined as follows:

3) Writing a data block of a data structure A into the data information file when generating one piece of data;

4) Then process all the variables in the variable list according to list_of_var loop in data structure a;

the data file is defined as follows:

4) The same variable on different data entries points to the same data.

3. The data storage method of claim 1, wherein: the stored procedure further comprises: setting the basic block size of memory allocation and the block size of a file system to be consistent; when a data file is written, a memory buffer area with the size of N block is allocated, and when the data of the memory buffer area is fully written, the data of the whole memory buffer area is written into a disk at one time; and the length N of each write data is equal to the block, so that the alignment of data and the pages of the disk file is ensured, and the write performance is improved.

4. The data storage method of claim 1, wherein: in the read-write process, the disk is directly read and written through IO; specifically, the beginning block and the last block are found according to the queried data, the number N of related blocks is calculated, a memory buffer area with the size of N block is allocated, the data with the size of N block are directly read at one time, the memory is directly operated by the follow-up operation, and frequent disk IO is avoided.

5. The data storage method of claim 1, wherein: the searching process reads a plurality of pieces of data according to the initial ID of the data item, or searches all the data items in the reading time period according to the time period, and specifically comprises the following steps:

step Q2: acquiring item information and all variable information;

step Q3: recording the maximum and minimum values of the blockidx in the variable information;

step Q4: and (3) circularly processing all the data items to be retrieved to finally obtain two values: maximum and minimum blockidx;

step Q5: and according to the offset position of the minimum blockidx to the data file, an offset position calculation formula is as follows: minimum blockidx; block is the file system file page size dimension;

Step Q6: reading data from the minimum blockidx to the maximum blockidx to a memory through direct IO operation once;

step Q7: processing the first piece of data, processing the item information, and obtaining item variable information;

step Q8: converting the blockidx and the blockOffset in the variable information into memory addresses, and converting the formulas: (blockidx-minimum blockidx) block+blockoffset; the block offset refers to the offset of the variable information in the block;

step Q9: taking the data on the memory address as the data of the variable;

step Q10: circularly processing all variables and data entries;

6. The data storage method of claim 1, wherein: the data storage method also comprises a data self-checking process, when the equipment is restarted, the data related file is checked, the damaged part of data is removed, and the intact data is reserved, and the method specifically comprises the following steps:

step A1: opening a data index file and a data information file;

step A2: traversing all index data in the data index file;

step A5: reading all variable information in the data information file;

step A6: judging whether the position of the variable index is abnormal, if so, jumping to the step A8; wherein, the index position is abnormal when it is negative or is a large value;

step A9: and processing the index data of the next piece of data until all the index data are processed, and exiting the self-checking.

7. A data storage device based on IEC61850 data service function, characterized in that: a data storage method based on IEC61850 data service functions according to any of the preceding claims 1-6.