CN106708653A - Mixed tax administration data security protecting method based on erasure code and multi-copy - Google Patents

Abstract

The invention discloses a hybrid tax big data security protection method based on erasure codes and multiple copies. When the tax data in the tax data distributed storage system is normal, the multi-copy and erasure code storage mode storage process of the tax data is started. When the tax data in the tax data distributed storage system fails, start the tax data fault-tolerant processing process. The present invention uses the characteristics of tax data at different times to store in different modes, distributes erasure coding tasks to different nodes, adopts the mode of first copy and then erasure code, and comprehensively improves the security and data repair performance of the entire tax data. It improves the overall coding performance of the system and ensures the security of data before erasure coding is completed.

Description

A hybrid tax big data security protection method based on erasure code and multiple copies

technical field

The invention relates to the technical field of computer data management, in particular to a security protection method for hybrid tax big data based on erasure codes and multiple copies.

Background technique

With the economic globalization and the continuous in-depth development of my country's economy, the number of taxpayers in my country has grown rapidly and the types of taxes have become more abundant. Facing the increasingly large tax data, distributed storage is a mainstream storage solution with high cost performance and scalability. For tax data, its data security in a distributed storage environment is a key point worth studying. A distributed storage system contains a large number of nodes, and node failure or external intrusion may lead to incomplete data. In order to avoid data loss, a fault tolerance method based on redundant data is usually adopted. There are two main types of redundant fault tolerance: one is multi-copy fault tolerance, which replicates redundant data for fault tolerance; the other is erasure code fault tolerance, which is generated by encoding Redundant data for fault tolerance.

The currently widely used fault tolerance method is multi-copy fault tolerance based on replication: copy the original data into c copies, and then distribute the c data copies to c different storage nodes, so that when any c-1 nodes fail, every There is at least 1 copy of the data. Multi-copy fault tolerance has the advantages of simple implementation, less computing overhead, and good data access performance. However, multi-copy fault tolerance also has a very prominent disadvantage: high storage overhead. For tax data, which is huge in itself and has been growing at a high speed, multi-copy fault tolerance based on replication is not applicable.

With the explosive growth of data, erasure code fault tolerance has become a research hotspot in recent years because it can provide the same or even higher data reliability with much lower storage overhead. The fault-tolerant strategy of erasure code is: divide a piece of data into c data blocks, and then encode the c data blocks into n (n>c) coded blocks and distribute them to n different disks, so that when a node fails, as long as the The data still has c encoding blocks, and the original data can be decoded. Compared with the widely used three-copy fault-tolerant scheme, RS erasure coding can reduce storage space consumption by 53% and double the fault-tolerant capacity at the same time. However, the disadvantage of erasure codes is that the performance of data reconstruction is low, especially in distributed storage. Since data reconstruction requires the cooperation of multiple nodes, it will inevitably lead to a large amount of network resource consumption and computing resource consumption. For distributed data such as tax data, this will become a key bottleneck in the performance of the entire system.

Contents of the invention

In view of this, in order to solve the above-mentioned problems in the prior art, the present invention proposes a hybrid tax big data security protection method based on erasure codes and multiple copies.

The present invention solves the above problems by the following technical means:

A hybrid tax big data security protection method based on erasure codes and multiple copies. When the tax data in the tax data distributed storage system is normal, the multi-copy and erasure code storage method storage process of tax data is started;

When the tax data in the tax data distributed storage system fails, start the tax data fault-tolerant processing process;

The storage process of the multi-copy and erasure code storage method includes the following steps:

Step S11, dividing the tax data into historical data and recent data according to time, the recent data includes a plurality of different recent data packets;

Step S12, storing the recent data in the multi-copy storage module according to the multi-copy storage method, and storing the historical data in the erasure code storage module according to the erasure code storage method;

Step S13, when a recent data packet is marked as completed, then transfer the recent data packet to the erasure code storage module to form historical data;

The tax data fault-tolerant processing flow includes the following steps:

Step S21, according to the data management node of the multi-copy storage module, it is judged whether the expired tax data is stored in the multi-copy storage module or in the erasure code storage module;

Step S22, if the invalid tax data is stored in the multi-copy storage module, send a test message to the relevant multi-copy storage node according to the record of the multi-copy management node, select the copy corresponding to the invalid tax data according to the feedback delay of the test message, and Restoring copies into valid tax data;

Step S23, if the invalid tax data is stored in the erasure code storage module, it is necessary to further search for the records of the erasure code management node, send a test message to the relevant erasure code storage node, and then select the corresponding data according to the feedback delay of the test message. Encoding blocks, after obtaining a sufficient number of encoding blocks, the tax data can be restored;

The tax data distributed storage system is used to provide storage and fault-tolerant services for tax data;

The tax data is the data input by the client of the tax data distributed storage system;

The historical data is the data before the time division point of the tax data distributed storage system, which is stored in the erasure code storage module;

The recent data is the data after the time division point of the tax data distributed storage system, and is stored in the multi-copy storage module;

The multi-copy storage module is used to store and process recent data, including a multi-copy storage module data management node and at least one multi-copy storage node;

The multi-copy management node is used to manage the replication, distribution and storage of data in the multi-copy storage module, and record data information;

The multi-copy storage node is used to store recent data;

The erasure code storage module is used to store and process historical data, including an erasure code management node and at least one erasure code storage node;

The erasure code management node is used to manage the encoding, distribution and storage of data in the erasure code storage module, and record the data information;

The erasure code storage node is used to store historical data;

The multi-copy storage method is used to read, store, record and restore recent data through the tax data distributed storage system;

The erasure code storage method is used to dump, read, record and restore historical data through the tax data distributed storage system;

The coded block is the coded block formed after the recent data to be dumped is sub-packaged and coded, and stored in the erasure code storage node, and used to restore and restore the tax data in the fault-tolerant processing of the tax data.

Further, the erasure code storage method described in step S12 includes the following steps:

Step S1221, the erasure code management node judges whether the frequency of external access to the erasure code storage module is lower than the access frequency threshold, thereby judging whether the current erasure code storage module is in an idle state, and if so, activates all erasure codes storage node;

Step S1222, judge each activated erasure code storage node as follows: whether the storage load of the erasure code storage node exceeds the storage full load threshold, and whether the network load of the erasure code storage node exceeds the network full load threshold, if If they are not exceeded, then request the data to be dumped from the multi-copy management node;

Step S1223, after encoding the data to be dumped, distribute and save it in the erasure code storage node, and record the distribution information in the erasure code management node;

Step S1224, after confirming that the data transfer is successful, delete all the transferred tax data and its copies in the multi-copy storage module;

The data to be dumped is a certain copy data of the data to be dumped in the record of the multi-copy management node. The selection principle of the copy data must conform to the load balance. The copy data is used for distribution after encoding and stored in the erasure code storage node;

The distribution information is record information that multiple coded blocks are distributed to multiple erasure code storage nodes, and is used to guide multiple coded blocks to restore and restore tax data.

Further, in the multi-copy storage method described in step S12, the processing flow of writing recent data includes the following steps:

Step S1231, when the client sends a recent data request to write, the multi-copy management node responds;

Step S1232, copying the written tax data to form a copy, and storing the written tax data and its copies separately in different multi-copy storage nodes;

Step S1233, recording the storage information of the written tax data in the multi-copy management node.

Further, in the multi-copy storage method described in step S12, the processing flow of reading recent data includes the following steps:

Step S1241, when the client sends a recent data read request, the multi-copy management node responds and sends a test message to the relevant multi-copy storage node according to the record and requests for calculation load;

Step S1242, comprehensively select the corresponding multi-copy storage node by testing the delay of message feedback and the computing load of the related multi-copy storage node;

Step S1243, according to the distribution of the multi-copy management node, the tax data in the corresponding multi-copy storage node is directly sent to the client;

The client is a client of the distributed storage system and is used for writing and reading tax data.

Further, in step S1232, the storage method of the written tax data and its copies is to physically isolate different copies of the same tax data, and select different cabinets or computer rooms for storage.

Further, the process of forming the coding block includes the following steps:

Step 61, the recent data to be dumped is subpackaged into C data blocks;

Step 62, encoding C data blocks into N encoding blocks, the number of N is greater than C;

Step 63, N coded blocks are distributed to N different erasure code storage modules;

Step 64: Record the distribution information of the N coded blocks in the erasure code management node where the coded blocks are located.

The present invention utilizes the characteristics of tax data at different times to store in different modes, and comprehensively improves the security and data restoration performance of the entire tax data. Because the access frequency of tax data has the characteristics of periodic changes, the access frequency of recent data is the highest, while the access frequency of historical data is relatively low. Using erasure codes on data with low access frequency can improve storage space utilization, and using multiple copies on data with high access frequency improves data repair performance.

Secondly, the present invention distributes erasure correction coding tasks to different nodes, fully considers the load conditions of the nodes when selecting nodes, and shares the calculation load and network transmission load among multiple nodes to improve the overall coding performance of the system.

Thirdly, the present invention adopts the mode of copy first and then erasure code, which can ensure the security of data before erasure code is completed, and makes up for the situation of data loss in coding that is easy to encounter when simply using erasure code for error tolerance.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

Fig. 1 is a working flow diagram of a hybrid tax big data security protection method based on erasure codes and multiple copies in the present invention;

Fig. 2 is a schematic structural diagram of a multi-copy storage module of the present invention;

Fig. 3 is a schematic structural diagram of an erasure code storage module of the present invention.

detailed description

In order to make the above objects, features and advantages of the present invention more comprehensible, the technical solutions of the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be pointed out that the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all those skilled in the art can obtain without creative work. Other embodiments all belong to the protection scope of the present invention.

As shown in Figure 1, a hybrid tax big data security protection method based on erasure codes and multiple copies, when the tax data in the tax data distributed storage system is normal, the multi-copy and erasure code storage mode storage of tax data is started process;

When the tax data in the tax data distributed storage system fails, start the tax data fault-tolerant processing process;

The storage process of the multi-copy and erasure code storage method includes the following steps:

Step S11, dividing the tax data into historical data and recent data according to time, the recent data includes a plurality of different recent data packets;

Step S12, storing the recent data in the multi-copy storage module according to the multi-copy storage method, and storing the historical data in the erasure code storage module according to the erasure code storage method;

Step S13, when a recent data packet is marked as completed, then transfer the recent data packet to the erasure code storage module to form historical data;

The tax data fault-tolerant processing flow includes the following steps:

Step S21, according to the data management node of the multi-copy storage module, it is judged whether the expired tax data is stored in the multi-copy storage module or in the erasure code storage module;

Step S22, if the invalid tax data is stored in the multi-copy storage module, send a test message to the relevant multi-copy storage node according to the record of the multi-copy management node, select the copy corresponding to the invalid tax data according to the feedback delay of the test message, and Restoring copies into valid tax data;

Step S23, if the invalid tax data is stored in the erasure code storage module, it is necessary to further search for the records of the erasure code management node, send a test message to the relevant erasure code storage node, and then select the corresponding data according to the feedback delay of the test message. Encoding blocks, after obtaining a sufficient number of encoding blocks, the tax data can be restored;

The tax data distributed storage system is used to provide storage and fault-tolerant services for tax data;

The tax data is the data input by the client of the tax data distributed storage system;

The historical data is the data before the time division point of the tax data distributed storage system, which is stored in the erasure code storage module;

The recent data is the data after the time division point of the tax data distributed storage system, and is stored in the multi-copy storage module.

As shown in Figure 2, the multi-copy storage module is used to store and process recent data, including a multi-copy storage module data management node and at least one multi-copy storage node;

The multi-copy management node is used to manage the replication, distribution and storage of data in the multi-copy storage module, and record data information;

The multi-copy storage nodes are used to store recent data.

As shown in Figure 3, the erasure code storage module is used to store and process historical data, including an erasure code management node and at least one erasure code storage node;

The erasure code management node is used to manage the encoding, distribution and storage of data in the erasure code storage module, and record the data information;

The erasure code storage node is used to store historical data.

The multi-copy storage method is used to read, store, record and restore recent data through the tax data distributed storage system;

The erasure code storage method is used to dump, read, record and restore historical data through the tax data distributed storage system;

The coded block is the coded block formed after the recent data to be dumped is sub-packaged and coded, and stored in the erasure code storage node, and used to restore and restore the tax data in the fault-tolerant processing of the tax data.

The erasure code storage method described in step S12 includes the following steps:

Step S1221, the erasure code management node judges whether the frequency of external access to the erasure code storage module is lower than the access frequency threshold, thereby judging whether the current erasure code storage module is in an idle state, and if so, activates all erasure codes storage node;

Step S1222, judge each activated erasure code storage node as follows: whether the storage load of the erasure code storage node exceeds the storage full load threshold, and whether the network load of the erasure code storage node exceeds the network full load threshold, if If they are not exceeded, then request the data to be dumped from the multi-copy management node;

Step S1223, after encoding the data to be dumped, distribute and save it in the erasure code storage node, and record the distribution information in the erasure code management node;

Step S1224, after confirming that the data transfer is successful, delete all the transferred tax data and its copies in the multi-copy storage module;

The data to be dumped is a certain copy data of the data to be dumped in the record of the multi-copy management node. The selection principle of the copy data must conform to the load balance. The copy data is used for distribution after encoding and stored in the erasure code storage node;

The distribution information is record information that multiple coded blocks are distributed to multiple erasure code storage nodes, and is used to guide multiple coded blocks to restore and restore tax data.

In the multi-copy storage method described in step S12, the processing flow of writing recent data includes the following steps:

Step S1231, when the client sends a recent data request to write, the multi-copy management node responds;

Step S1232, copying the written tax data to form a copy, and storing the written tax data and its copies separately in different multi-copy storage nodes;

Step S1233, recording the storage information of the written tax data in the multi-copy management node.

In the multi-copy storage method described in step S12, the processing flow of reading recent data includes the following steps:

Step S1241, when the client sends a recent data read request, the multi-copy management node responds and sends a test message to the relevant multi-copy storage node according to the record and requests for calculation load;

Step S1242, comprehensively select the corresponding multi-copy storage node by testing the delay of message feedback and the computing load of the related multi-copy storage node;

Step S1243, according to the distribution of the multi-copy management node, the tax data in the corresponding multi-copy storage node is directly sent to the client;

The client is a client of the distributed storage system and is used for writing and reading tax data.

In step S1232, the written tax data and its copies are stored by physically separating different copies of the same tax data and selecting different cabinets or computer rooms for storage.

The process of forming the coding block includes the following steps:

Step 61, the recent data to be dumped is subpackaged into C data blocks;

Step 62, encoding C data blocks into N encoding blocks, the number of N is greater than C;

Step 63, N coded blocks are distributed to N different erasure code storage modules;

Step 64: Record the distribution information of the N coded blocks in the erasure code management node where the coded blocks are located.

The present invention utilizes the characteristics of tax data at different times to store in different modes, and comprehensively improves the security and data restoration performance of the entire tax data. Because the access frequency of tax data has the characteristics of periodic changes, the access frequency of recent data is the highest, while the access frequency of historical data is relatively low. Using erasure codes on data with low access frequency can improve storage space utilization, and using multiple copies on data with high access frequency improves data repair performance.

Secondly, the present invention distributes erasure correction coding tasks to different nodes, fully considers the load conditions of the nodes when selecting nodes, and shares the calculation load and network transmission load among multiple nodes to improve the overall coding performance of the system.

Thirdly, the present invention adopts the mode of copy first and then erasure code, which can ensure the security of data before erasure code is completed, and makes up for the situation of data loss in coding that is easy to encounter when simply using erasure code for error tolerance.

The above-mentioned embodiments only express several implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but should not be construed as limiting the patent scope of the present invention. It should be pointed out that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention should be based on the appended claims.

Claims (6)

1. a kind of mixing tax big data method for security protection based on correcting and eleting codes with many copies, it is characterised in that when tax number According to the tax data of distributed memory system it is normal when, many copies and the correcting and eleting codes storage mode for starting tax data store stream Journey；

When the tax data of tax data distributed memory system fails, start tax data fault-tolerant processing flow；

Many copies comprise the following steps with correcting and eleting codes storage mode Stored Procedure：

Step S11, tax data is temporally divided into historical data and Recent data, and the Recent data includes multiple different Recent data bag；

Step S12, the Recent data is stored in many copy memory modules according to many copy storage modes, by the history number Stored in correcting and eleting codes memory module according to according to correcting and eleting codes storage mode；

Step S13, when Recent data coating is labeled as completion status, then dumps to correcting and eleting codes storage by the Recent data bag Module is so as to history of forming data；

The tax data fault-tolerant processing flow comprises the following steps：

Step S21, according to many copy memory module data management nodes, judges that failure tax data storage stores mould in many copies In block or in correcting and eleting codes memory module；

Step S22, if the tax data storage of failure is in many copy memory modules, according to many replica management nodes record to Related many copy memory nodes send test packet, according to test packet feedback delay selection pair corresponding with failure tax data This, and copy is reverted into effective tax data；

Step S23, if the tax data of failure is stored in correcting and eleting codes memory module, need to further search for correcting and eleting codes management node Record, to related correcting and eleting codes memory node send test packet, then selected successively accordingly according to test packet feedback delay Encoding block, after obtaining sufficient amount encoding block, you can reduction recovers tax data.

2. a kind of mixing tax big data method for security protection based on correcting and eleting codes with many copies as claimed in claim 1, its It is characterised by, the correcting and eleting codes storage mode described in step S12 is comprised the following steps：

Whether step S1221, judge the outside visiting frequency to correcting and eleting codes memory module less than access by correcting and eleting codes management node Frequency threshold, so as to judge that whether current correcting and eleting codes memory module, in idle condition, if it is activates whole correcting and eleting codes and deposits Storage node；

Step S1222, makes the following judgment to the correcting and eleting codes memory node that each is activated：The correcting and eleting codes memory node is deposited Whether storage load exceedes storage loading thresholds, and whether the offered load of the correcting and eleting codes memory node is fully loaded with threshold more than network Value, if be no more than, unloading data is treated to the request of many replica management nodes；

Step S1223, by after after unloading data encoding, distributing and be stored in correcting and eleting codes memory node, and will distribute information record In correcting and eleting codes management node；

Step S1224 is complete by the tax data and its copy of unloading in many copy memory modules after confirming data conversion storage success Delete in portion.

3. a kind of mixing tax big data method for security protection based on correcting and eleting codes with many copies as claimed in claim 1, its It is characterised by, in many copy storage modes described in step S12, the handling process for writing Recent data comprises the following steps：

Step S1231, when client sends Recent data request write-in, many replica management nodes are responded；

Step S1232, the tax data to writing carries out duplication and forms copy, and the tax data and its copy of write-in are separated It is stored in different many copy memory nodes；

Step S1233, the storage information of the tax data that will be write is recorded in many replica management nodes.

4. a kind of mixing tax big data method for security protection based on correcting and eleting codes with many copies as claimed in claim 1, its It is characterised by, in many copy storage modes described in step S12, the handling process for reading Recent data comprises the following steps：

Step S1241, when client sends Recent data read requests, many replica management nodes are responded and according to record Test packet is sent to related many copy memory nodes and ask computational load；

Step S1242, the time delay and the computational load of related many copy memory nodes fed back by test packet are come comprehensive selection Corresponding many copy memory nodes；

Step S1243, the tax data that the distribution according to many replica management nodes allows in corresponding many copy memory nodes is direct It is sent in client.

5. a kind of mixing tax big data method for security protection based on correcting and eleting codes with many copies as claimed in claim 3, its It is characterised by, in step S1232, the tax data of write-in and its location mode of copy are that the difference of same tax data is secondary Originally it is physically separated, selects different rack or computer room storages.

6. a kind of mixing tax big data method for security protection based on correcting and eleting codes with many copies as claimed in claim 1, its It is characterised by, the process that the encoding block is formed is comprised the following steps：

Step 61, treats that the Recent data of unloading is packetized into C data block；

Step 62, N number of encoding block is encoded into by C data block, and the number of the N is more than C；

Step 63, N number of encoding block is distributed to N number of different correcting and eleting codes memory module；

Step 64, information record is distributed in the correcting and eleting codes management node where encoding block by N number of encoding block.