CN111866038A - Distributed storage dynamic defense system and method based on heterogeneous multiple copies - Google Patents
Distributed storage dynamic defense system and method based on heterogeneous multiple copies Download PDFInfo
- Publication number
- CN111866038A CN111866038A CN201910339982.XA CN201910339982A CN111866038A CN 111866038 A CN111866038 A CN 111866038A CN 201910339982 A CN201910339982 A CN 201910339982A CN 111866038 A CN111866038 A CN 111866038A
- Authority
- CN
- China
- Prior art keywords
- heterogeneous
- distributed storage
- storage system
- copy
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 234
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000007123 defense Effects 0.000 title claims abstract description 37
- 230000005540 biological transmission Effects 0.000 claims abstract description 3
- 238000004140 cleaning Methods 0.000 claims description 9
- 230000001360 synchronised effect Effects 0.000 claims description 9
- 230000006798 recombination Effects 0.000 claims description 5
- 238000005215 recombination Methods 0.000 claims description 5
- 238000010926 purge Methods 0.000 claims description 4
- 238000013500 data storage Methods 0.000 claims description 3
- 235000008113 selfheal Nutrition 0.000 claims 1
- 230000008260 defense mechanism Effects 0.000 abstract 1
- 230000007246 mechanism Effects 0.000 description 22
- 238000013508 migration Methods 0.000 description 18
- 230000005012 migration Effects 0.000 description 18
- 238000007726 management method Methods 0.000 description 15
- 238000005516 engineering process Methods 0.000 description 10
- 238000013461 design Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 9
- 238000010276 construction Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000011084 recovery Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000035876 healing Effects 0.000 description 2
- 238000002347 injection Methods 0.000 description 2
- 239000007924 injection Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000008439 repair process Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 241000700605 Viruses Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000011449 brick Substances 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012938 design process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- ZXQYGBMAQZUVMI-GCMPRSNUSA-N gamma-cyhalothrin Chemical compound CC1(C)[C@@H](\C=C(/Cl)C(F)(F)F)[C@H]1C(=O)O[C@H](C#N)C1=CC=CC(OC=2C=CC=CC=2)=C1 ZXQYGBMAQZUVMI-GCMPRSNUSA-N 0.000 description 1
- 238000002513 implantation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000003999 initiator Substances 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000007711 solidification Methods 0.000 description 1
- 230000008023 solidification Effects 0.000 description 1
- 239000000243 solution Substances 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a distributed storage dynamic defense system based on heterogeneous multi-copy, which comprises a user application layer, a heterogeneous multi-copy storage system control scheduler, a control channel, a data channel, a multi-element heterogeneous distributed storage system and a heterogeneous component pool, wherein the user application layer is used for sending an instruction to the heterogeneous multi-copy storage system control scheduler; the heterogeneous multi-copy storage system control scheduler is used for receiving instructions of a user application layer and uniformly scheduling and controlling; the control channel is used for issuing a control command to the multi-element heterogeneous distributed storage system and realizing interface calling; the data channel is used for data transmission with the multi-element heterogeneous distributed storage system; the multi-element heterogeneous distributed storage system is used for receiving the control command and realizing multi-element storage of the distributed system; the heterogeneous component pool is used for storing data resources. The distributed storage method and the distributed storage system realize distributed storage, improve a defense mechanism, and solve the computation pressure and the disk write bottleneck of the heterogeneous multi-copy distributed storage system.
Description
Technical Field
The invention relates to the field of distributed storage, in particular to a distributed storage dynamic defense system and a distributed storage dynamic defense method based on heterogeneous multiple copies.
Background
The existing distributed storage systems such as GlusterFS, Ceph and MooseFS mainly focus on the storage mode of fusing and sharing storage media with data spanning multiple nodes. The storage system software has a data consistency guarantee mechanism for multiple copies or multiple backups of the stored data, but the security of the stored data does not have a sufficient guarantee mechanism. Therefore, when the single distributed storage system with a single technical source is attacked by a hacker and Hook API injection attack of a Rootkit hidden back door is carried out, read-write data of a user can be intercepted, written into a nested Trojan program and the like, and the purpose of tampering and stealing the user data is achieved. In addition, a single distributed storage system always has advantages and disadvantages in some aspects in the aspects of metadata management, data consistency management, large file write IO storage efficiency, small file write IO efficiency, data redundancy mechanism design and the like, so that a short board always exists in the aspects of security, storage performance, storage utilization rate and the like, and the existing system cannot well meet the requirements of users on high-standard use scenes.
The invention patent with application number 201410206795.1 discloses a method and a device for managing heterogeneous copies in a distributed storage system, wherein the method comprises the following steps: acquiring a write request parameter for storing a copy of a file; acquiring the position information of each storage server in the distributed storage system from the metadata server according to the write request parameters; converting the copies of the files according to a preassigned format according to the write request parameters to obtain a plurality of heterogeneous copies of the files with different formats; and respectively storing the converted heterogeneous copies of the files with different formats on a specified storage server according to the position information of the storage server acquired from the metadata server. The embodiment of the invention can respectively store a plurality of different heterogeneous copies in a plurality of storage servers, and when the heterogeneous copies of the file are read, the corresponding heterogeneous copies can be read according to the requirement, so that the working efficiency of processing data by a user can be effectively improved. The comparison patent mainly manages the metadata servers, acquires the position information of each storage server in the distributed storage system from the metadata servers, and converts the copies of the files according to the request parameters and the pre-specified format to obtain a plurality of heterogeneous copies of the files with different formats. The method mainly comprises the steps that a typical distributed storage system commonly used in the prior art is used as a basis based on the advantages and the characteristics of each distributed storage system, the framework mode of the original distributed storage system is kept unchanged, and a heterogeneous distributed storage system with a security defense strategy based on dynamic heterogeneous redundancy is constructed. The comparison patent mainly comprises the steps of converting copies of files according to a pre-specified format through a write request parameter to obtain heterogeneous copies of the files with different formats, and storing the heterogeneous copies of the files with different formats, which are obtained through conversion, on a specified storage server respectively according to the position information of the storage server obtained from a metadata server. According to the method, an optimized heterogeneous storage executive body is selected to be constructed to meet the business requirements according to the specific redundancy quantity requirements of user business data, data reading and writing are maintained by each distributed storage system, a data consistency check value and the data are managed by a heterogeneous multi-copy distributed storage system on a certain distributed storage system, and the principle of not repeatedly manufacturing wheels is followed.
The invention patent with application number 201510893938.5 discloses a data migration method of a heterogeneous distributed storage system, which adopts an MVC mode and specifically comprises the following steps: 1) initiating a migration task by the View module; 2) the Controller module receives a migration command; 3) the Controller module starts data migration; 4) and the View module automatically inquires the migration progress to the Controller module at set time intervals. Compared with the prior art, the method has the advantages of realizing migration visualization, managing the migration process, not interrupting the online migration of the service, being simple to use, completing the migration process without professional personnel, ensuring no data loss and the like. Compared with the prior art, the method has the advantages that the MVC mode is adopted, the migration task is initiated based on the View module, the migration command is received based on the Controller module, the Controller module starts data migration, and the View module automatically inquires the original log data in the migration progress server from the Controller module at set time intervals to perform compression processing. The patent focuses on managing the stored data in a copy-based mode, and fully applies a consistency management mode, an availability management mode and a fault tolerance management mode of the distributed storage system. Instead of independently performing the MVC pattern self-development program for data compression and backup. Compared with the prior art, the method has the advantages that the migration visualization is realized, the migration process can be managed, the online migration of the service is not interrupted, the use is simple, the migration process can be completed without professional personnel, and the data are not lost. The patent mainly checks the consistency of data through a copy redundancy technology, an erasure code redundancy technology, a stripe striping redundancy technology and the like which are typically applied, and checks copy data in each isomer in a timing and real-time mode in the following mode, so that the purpose of automatically curing and recovering the data attacked by malicious tampering is achieved.
Scientific publishing agency discloses 'network space mimicry defense introduction', and the known technology does not describe in detail how to realize a dynamic defense security protection mechanism for the storage system and the stored data by constructing a heterogeneous distributed storage system, does not consider multi-dimensional detection, verification and automatic recovery of tampered data, focuses on the judgment of network defense in the hopping process, and focuses on how to select a heterogeneous executive. In the known technology, a multi-mode decision model is mainly adopted, and the decision control is influenced based on the input of the decision parameters and the control parameters of a multi-mode decision and negative feedback controller. The method not only considers the parameter-based control mode of the decision strategy technology, but also introduces a self-learning mechanism based on machine learning, is a brand-new more dynamic and active strategy mode, and achieves the purpose of preventing the data security damage generated by unknown attacks in advance by continuously learning and updating the dynamic defense security knowledge base.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a distributed storage dynamic defense system and a distributed storage dynamic defense method based on heterogeneous multi-copy.
The invention provides a distributed storage dynamic defense system based on heterogeneous multi-copy, which comprises a user application layer, a heterogeneous multi-copy storage system control scheduler, a control channel, a data channel, a multi-element heterogeneous distributed storage system and a heterogeneous component pool, wherein:
the user application layer is used for sending instructions to the heterogeneous multi-copy storage system control scheduler;
the heterogeneous multi-copy storage system control scheduler is used for receiving an instruction of a user application layer and performing unified scheduling control on the multi-element heterogeneous distributed storage system;
the control channel is used for issuing a control command to the multi-element heterogeneous distributed storage system and providing interface calling of the control command to the user application layer;
the data channel is used for data transmission with the multi-element heterogeneous distributed storage system;
the multi-element heterogeneous distributed storage system is used for receiving the control command and realizing multi-element storage of the distributed system;
the heterogeneous component pool is used for storing data resources.
Preferably, the control command includes a scheduling command, an arbitration command, a flush command, or a synchronization command, wherein:
the scheduling command includes: when a task needs to be executed, selecting a proper executive body in the heterogeneous component pool according to a predefined method to execute the task;
The arbitration command includes: after the executive body executes the task, judging an execution result according to a predefined arbitration strategy;
the purge command includes: the executive body can periodically or aperiodically execute pre-cleaning or initialization of different levels, or reconfiguration and recombination operations;
the synchronization command includes: the state or scenario of the waiting executives and the online executives is synchronized.
Preferably, the multi-element heterogeneous distributed storage system comprises a plurality of distributed storage systems, and the distributed storage systems are synchronized and self-healed.
Preferably, the multi-element heterogeneous distributed storage system provides a uniform scheduling interface outwards.
Preferably, the heterogeneous component pool includes a plurality of executables, each corresponding to one of the distributed storage systems.
Preferably, the executables include a GlusterFS executables, a CephFS executables, or a MooseFS executables.
The invention also provides a distributed storage dynamic defense method based on the heterogeneous multi-copy, which comprises the following steps:
an information acquisition step: the user application layer interacts with an interface provided by the heterogeneous multi-copy storage system control scheduler to obtain the function of the multi-element heterogeneous distributed storage system;
And an instruction control step: the heterogeneous multi-copy storage system control scheduler receives the issuing of a control channel control command according to the requirements of a user application layer; reading and writing the storage files of the heterogeneous component pool through a data channel;
a data storage step: and the multi-element heterogeneous distributed storage system realizes the distributed storage of data according to the control command.
Preferably, the control command includes a scheduling command, an arbitration command, a flush command, or a synchronization command, wherein:
the scheduling command includes: when a task needs to be executed, selecting a proper executive body in the heterogeneous component pool according to a predefined method to execute the task;
the arbitration command includes: after the executive body executes the task, judging an execution result according to a predefined arbitration strategy;
the purge command includes: the executive body can periodically or aperiodically execute pre-cleaning or initialization of different levels, or reconfiguration and recombination operations;
the synchronization command includes: the state or scenario of the waiting executives and the online executives is synchronized.
Preferably, the multi-element heterogeneous distributed storage system comprises a plurality of distributed storage systems, the distributed storage systems are synchronized and self-healed, the multi-element heterogeneous distributed storage system provides a uniform scheduling interface outwards, the heterogeneous component pool comprises a plurality of executors, and each executor corresponds to one distributed storage system.
Preferably, the executables include a GlusterFS executables, a CephFS executables, or a MooseFS executables.
Compared with the prior art, the invention has the following beneficial effects:
1. the distributed storage system has the advantages that the safety of the distributed storage system is greatly improved, the distributed storage system has general adaptability of data value protection, the distributed storage system can dynamically defend malicious tampering and virus Trojan injection by hackers based on a single technical source, the probability that the hackers cannot normally detect and diagnose the hacks is prevented, the technical threshold and the technical complexity of malicious tampering attack are improved, and the safety is improved by about 3 times;
2. the invention realizes the automatic healing and recovery of the data copy attacked by the backdoor, prevents the possibility of data stealing and leakage, and protects and improves the self security and the write-in data security of the single distributed storage system in the view of a dynamic defense method;
3. the distributed storage dynamic defense method based on the heterogeneous multi-copy can maximally cross the performance bottleneck of the storage system, maximally integrates the self physical computing resources and the disk medium storage resources of each physical node by using the distributed storage dynamic defense method based on the heterogeneous multi-copy, and respectively writes the same data source into different heterogeneous executors;
4. The invention solves the computation pressure and disk write bottleneck of the heterogeneous multi-copy distributed storage system. From the analysis of actual experimental results, it is very obvious to the great improvement of the storage performance that a single storage executive builds a 3-copy heterogeneous multi-copy distributed storage system again.
5. The invention can automatically configure a reasonable storage scheme according to the service requirement. To address the short-board disadvantage of a certain memory technology in a certain aspect of function or performance in a common memory scheme design. According to the invention, mechanisms such as a core dynamic heterogeneous redundancy strategy and a multi-mode decision algorithm are added in a heterogeneous multi-copy distributed storage system, each single distributed storage system is managed and controlled from a management plane and a data plane, detection is carried out according to service requirements, and then preferential combination is carried out, so that the advantages and the characteristics of each heterogeneous storage execution body are absorbed;
6. the invention can construct the distributed storage system which is most suitable for the storage requirement of the service system, and the best service is adapted and applied without being influenced by the disadvantage short board of a single distributed storage system.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a heterogeneous storage system model architecture diagram of a heterogeneous multi-copy based distributed storage dynamics defense system;
FIG. 2 is a diagram of a heterogeneous multi-copy controller schematic for a heterogeneous multi-copy based distributed storage dynamics defense system;
FIG. 3 is a heterogeneous multi-copy data flow control diagram of a distributed storage dynamic defense method based on heterogeneous multi-copy.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
As shown in fig. 1 to fig. 3, the present invention provides a distributed storage dynamic defense system and method based on heterogeneous multiple copies. As shown in fig. 1, the distributed storage dynamic defense system based on heterogeneous multi-copy provided by the present invention includes a user application layer, a control scheduler of the heterogeneous multi-copy storage system, a control channel, a data channel, a multi-heterogeneous distributed storage system, and a heterogeneous component pool, wherein:
The user application layer is an actual use entity which has a demand on the storage system, is an initiator of the instruction system, and can be a super-fusion virtualization platform or a set of management information system.
The control scheduler is constructed on a diversified distributed storage system to perform uniform heterogeneous fusion on the diversified distributed storage system. And a multi-copy data redundancy storage scheme of the heterogeneous storage system is realized through a copy redundancy technology, the constitution of the copy is identified and judged by using a multi-mode judgment strategy, and the data consistency of the heterogeneous copy is ensured through unified management and scheduling.
The control channel mainly issues a control command to the heterogeneous storage executive pool, for example, controls the issuing of a decision-making strategy, triggers the cleaning operation of the attack, and performs the synchronous operation of the state and the scene. And simultaneously, providing interface calling of a control command for an upper application system, and triggering a non-cooperative control prevention permeator to implement cooperative and consistent homomorphic attack on a space-time dimension by using a possible synchronization mechanism.
The data channel is a channel for transmitting data with the heterogeneous storage executive body pool, and completes file control, reading and writing of copy data, data storage solidification, management of heterogeneous storage pool node data, migration maintenance of heterogeneous storage pool copies, data consistency check and the like.
The multi-element distributed storage system is a component part forming a heterogeneous storage execution body pool and is a set of cooperative work of each independent distributed storage system. It uses the storage space on each node in the storage cluster through the network and forms these scattered storage resources into a virtual storage device or storage domain. The heterogeneous distributed storage system has multiple algorithms and protocols to guarantee the consistency and redundancy of data and guarantee the high availability and the transverse and longitudinal expansion capabilities of the distributed storage system.
The heterogeneous construction pool is composed of storage systems with different system structures and different physical storage structures, and data resources of the application system are stored through the heterogeneous storage systems.
Heterogeneous multi-copy controller system diagram referring to fig. 2, a heterogeneous multi-copy controller is composed of two planes: a control plane and a data plane. The control plane is mainly used for executing the multi-mode arbitration strategy, scheduling and constructing heterogeneous multi-copy, registering the storage executive body and managing the data consistency; the data plane is mainly used for providing data repair and synchronization between the multi-copy pool and the heterogeneous copies and providing a direct data access channel for a user service system client through a service unit and a uniform data interface.
On a control plane, a user service system client, a heterogeneous storage executive registration service and a multi-mode judgment strategy scheduler are used for controlling the construction of heterogeneous copies by inputting user requirement parameters, multi-mode judgment strategy parameters, copy quantity and the like to a multi-copy scheduling construction controller; the multi-copy scheduling construction controller sends a control command to the multi-copy pool, the multi-copy pool organizes a plurality of heterogeneous executives according to a scheduling mechanism, constructs a heterogeneous executor meeting specified input parameters, and periodically or temporarily triggers the heterogeneous executives to complete consistency repair and self-healing of the data copies, so that the control plane can uniformly manage and control the heterogeneous multi-copy storage executives. The heterogeneous multi-copy pool, the GlusterFS executor, the CephFS executor, the MooseFS executor, the construction pool and the like work on a data plane, respond to a control command issued by the control plane and provide service data management service. And the copy pool constructed by the execution body selects the primary copy A to provide data service for the outside through the arbitration of an application layer, and when the primary copy A cannot work normally, the secondary copy B/C is activated to be authorized to become a new primary copy.
For further explaining the flow of the whole storage system, the following takes constructing an executive with 3 copies, and each set of distributed storage system maintains 1 copy as an example, to discuss the working principle of the heterogeneous controller, interact with the multiple distributed storage systems by the heterogeneous controller, complete organic cooperation and coordination of the multiple heterogeneous storage systems, and jointly realize each function of the heterogeneous distributed storage systems. The flow of distributing 3 copy files from the application layer to 3 specific examples of storage executors is shown in fig. 3. And the application layer system issues a File1 File and stores the 3-copy data into different distributed storage systems. The heterogeneous multi-copy controller interactively acquires application layer requirements and instructions for an upper application layer and an application layer, realizes a specific storage function for each set of distributed storage system for lower control and scheduling, and a GlusterFS executive, a CephFS executive and a MooseFS executive respectively create 1-copy distributed storage volume. Specifically, a 1-copy distributed volume of GlusterFS consists of 1 or more Brick build pools, a 1-copy distributed volume of CephFS consists of 1 or more data object storage device build pools, and a 1-copy distributed volume of MooseFS consists of 1 or more Chunk build pools.
By constructing the heterogeneous multi-copy distributed storage system, the invention can greatly reduce the probability of hacker attack on the distributed storage system based on a single technical source, and reduce the risk of damage to user data and the distributed storage system. According to a multi-mode decision-making strategy adopted by the heterogeneous multi-copy distributed storage system and input factors such as decision parameters, control parameters and the like, the method can detect each heterogeneous storage system executor in real time or in a timing manner; the tampered user data can be automatically cured and restored, and the safety problem that the distributed storage system data is maliciously tampered is solved.
The invention can further disperse the read-write pressure of the disk, decompose the disk read-write bottleneck of the heterogeneous storage executive into the executive of the single distributed storage system layer by layer, and perform data write-in operation based on copy number by each physical node of a plurality of different single distributed storage systems. The bottleneck of the read-write data performance of the distributed storage system is solved, and the storage performance of the heterogeneous distributed storage system is improved.
The invention can detect the performance of different using modes of the distributed storage system according to the requirements of service scenes. The requirements of the service scene comprise the requirement on the number of data redundant storage copies, the requirement on the read-write storage performance of large files, the requirement on the read-write storage performance of small files, the requirement on data security and the like. The system tests each distributed storage system by probing the service requirements and constructs a heterogeneous storage executive body according to optimal adaptation, thereby meeting the index requirements of users to the maximum extent. The method can intelligently select the optimal distributed storage scheme for the service system and provide the most suitable demand scheme of the service scene.
The invention also provides a distributed storage dynamic defense method based on the heterogeneous multi-copy, which comprises the following specific steps:
firstly, an application layer interacts with an interface provided by a heterogeneous multi-copy storage system control scheduler to obtain the function of a bottom-layer distributed storage system.
Secondly, the heterogeneous multi-copy storage system control scheduler controls the heterogeneous multi-copy storage system control scheduler to issue relevant control commands such as scheduling, arbitration, cleaning, synchronization and the like through a control channel according to the requirements of a user layer; and reading and writing the storage files of the heterogeneous storage executive body pool through the data channel. Wherein:
The scheduling mechanism comprises: when a task needs to be executed, the scheduling mechanism selects a proper executive body to execute the task according to a predefined method. The executive scheduling strategy is realized based on normal distribution.
The arbitration mechanism comprises: after the executive executes the task, the system will determine whether to accept the execution result according to the predefined arbitration policy. The arbitrator of the present invention selects a decision principle based on the confidence level of the execution body.
The cleaning mechanism is as follows: the heterogeneous executive body can regularly or irregularly execute pre-cleaning or initialization of different levels, or reconfiguration and recombination operations;
the synchronization mechanism comprises: the state or scenario of the waiting executives and the online executives is synchronized.
Finally, each heterogeneous distributed storage system server receives the instruction of the system, queries a metadata server or calculates the file position through an inode and the like, completes the accurate positioning of the file in the file system, calls a Linux Virtual File System (VFS), a Linux file system and the like on a system common support layer, realizes the storage of the memory data to the physical disk, and achieves the purpose of storing the data in the heterogeneous storage pool.
In the whole design process of the heterogeneous multi-copy storage model, a heterogeneous multi-copy storage system control scheduler works between an application layer and a distributed storage system, provides interface requirements for controlling various storage systems upwards, and schedules and controls various distributed storage systems downwards in a unified manner.
The distributed storage dynamic defense method based on the heterogeneous multi-copy is designed to be applied to a dynamic redundancy defense strategy based on multi-mode decision in a heterogeneous multi-copy storage model based on the principle of the defense strategy of the heterogeneous redundancy dynamic model. Meanwhile, a negative feedback controller and a multi-mode arbitrator of the dynamic redundancy defense strategy are constructed, the strategy scheme is subjected to standard quantization, and control is performed through specific control parameters and arbitration parameters.
According to the heterogeneous characteristics of the heterogeneous multi-copy model, the method enables diversified individuation of heterogeneous storage executors to form uniform commonality through integrated scheduling. Based on different storage executors with different characteristics, such as different metadata positioning algorithms, customized data consistency algorithms, non-uniform storage special interfaces, diversified storage construction pools and the like, the method constructs a heterogeneous multi-copy storage model capable of performing unified management, unified scheduling and unified control. The storage model provides scheduling forms such as a uniform access interface and the like for a user application layer, and realizes that not only redundant backup but also cooperative work can be carried out among different storage executors.
The method realizes the decision and control mechanism generated by the scheduling controller by using a mode of inputting control parameters and arbitration parameters, and performs centralized unified scheduling, unified management and unified control on each distributed storage system executive. In order to realize a centralized scheduling control management mechanism, the method designs an arbitration mechanism scheme, a cleaning mechanism scheme, a synchronization mechanism scheme, an interface mechanism scheme and a de-cooperation mechanism scheme of a control scheduler, and a multi-scheme guarantee mechanism realizes the isomer control scheduler.
The method comprises the steps that through a distributed heterogeneous storage executive body pool and an interface of the heterogeneous executive body pool to the upper part/the lower part, the functions of the components are utilized, and meanwhile, various characteristics of consistency, safety, fault tolerance and the like of a storage system are combined. The method also realizes the command request of the control channel interface module, the data channel interface module and the scheduling controller. In addition, the heterogeneous storage executors have specific methods for new metadata management and data consistency management services that maintain consistency management and data access to multiple executors and multiple copies.
The method adopts the data security self-healing recovery function design of the heterogeneous distributed storage system. In single storage execution, a distributed file system used by a user cannot detect that copy data is tampered after being subjected to Hook API, so that a backdoor vulnerability attack is successfully implanted into the user system, and the core data of the user is stolen by a long-term latent user. Because each bottom-layer storage executive body of the heterogeneous dynamic redundancy multi-copy storage technology adopts different design frameworks and API (application programming interface) calling interfaces of different distributed systems, the difficulty of Hook attack and vulnerability backdoor implantation of hackers on different executive bodies is increased to a certain extent. Therefore, the dynamic defense method based on the heterogeneous storage executor is effectively realized, the damage of a hacker to the system operation based on a single technology, a single storage source and a single attack means is effectively avoided, and the core data resources of the user are effectively protected. And the distributed system damaged by malicious attack and the data copy thereof can be accurately judged according to the MD5 check value of the same data in each execution body, and automatic healing recovery is carried out by applying a correct copy data source.
The method adopts the storage performance probing module design of the multi-element distributed storage system. When a heterogeneous redundant distributed storage architecture scheme is constructed, the method needs to test and verify each executed index first, so that an optimal scheme of preferential configuration is provided for the construction of a heterogeneous multi-copy distributed execution body. The design basis of the preferred scheme comprises multiple dimensions such as heterogeneous executive large file read-write performance, small file read-write performance, scheme mechanism of different executors for data redundancy storage, physical disk utilization rate and the like.
The method adopts an automatic visual heterogeneous distributed storage function and a safety evaluation view design. Through the foregoing steps and design solutions, automated diagram presentation functionality is designed on the view presentation. And performing automatic chart dragging layout according to data stored in the database, and realizing page layout of a display page, customization of local charts and configuration of a data source and a data set. And using a related drawing engine and a uniform data acquisition interface to carry out integral drawing on the data source to be presented at the front end of the Web page according to the distribution of a local graph and a page layout. By means of an automatic visual interface configuration method, full graphical presentation of data view and function view display required by the heterogeneous multi-copy distributed storage system is comprehensively presented.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (10)
1. A distributed storage dynamic defense system based on heterogeneous multi-copy is characterized by comprising a user application layer, a heterogeneous multi-copy storage system control scheduler, a control channel, a data channel, a multi-element heterogeneous distributed storage system and a heterogeneous component pool, wherein:
the user application layer is used for sending instructions to the heterogeneous multi-copy storage system control scheduler;
the heterogeneous multi-copy storage system control scheduler is used for receiving an instruction of a user application layer and performing unified scheduling control on the multi-element heterogeneous distributed storage system;
the control channel is used for issuing a control command to the multi-element heterogeneous distributed storage system and providing interface calling of the control command to the user application layer;
the data channel is used for data transmission with the multi-element heterogeneous distributed storage system;
The multi-element heterogeneous distributed storage system is used for receiving the control command and realizing multi-element storage of the distributed system;
the heterogeneous component pool is used for storing data resources.
2. The heterogeneous, multi-copy based distributed storage dynamics defense system of claim 1, wherein the control commands comprise scheduling commands, arbitration commands, flush commands, or synchronization commands, wherein:
the scheduling command includes: when a task needs to be executed, selecting a proper executive body in the heterogeneous component pool according to a predefined method to execute the task;
the arbitration command includes: after the executive body executes the task, judging an execution result according to a predefined arbitration strategy;
the purge command includes: the executive body can periodically or aperiodically execute pre-cleaning or initialization of different levels, or reconfiguration and recombination operations;
the synchronization command includes: the state or scenario of the waiting executives and the online executives is synchronized.
3. The heterogeneous multi-replica based distributed storage dynamics defense system according to claim 1, wherein the multi-element heterogeneous distributed storage system comprises a plurality of distributed storage systems, and the plurality of distributed storage systems are synchronized to self-heal.
4. The heterogeneous multi-copy based distributed storage dynamics defense system according to claim 1, wherein the multi-element heterogeneous distributed storage system provides a unified scheduling interface to the outside.
5. The heterogeneous multi-copy based distributed storage dynamics defense system according to claim 3, wherein the heterogeneous component pool comprises a plurality of executables, one distributed storage system for each executable.
6. The heterogeneous multi-copy based distributed storage dynamic defense system according to claim 5, wherein the executables include a GlusterFS executables, a CephFS executables, or a MooseFS executables.
7. A method of the distributed storage dynamic defense system based on the heterogeneous multi-copy is characterized by comprising the following steps:
an information acquisition step: the user application layer interacts with an interface provided by the heterogeneous multi-copy storage system control scheduler to obtain the function of the multi-element heterogeneous distributed storage system;
and an instruction control step: the heterogeneous multi-copy storage system control scheduler receives the issuing of a control channel control command according to the requirements of a user application layer; reading and writing the storage files of the heterogeneous component pool through a data channel;
A data storage step: and the multi-element heterogeneous distributed storage system realizes the distributed storage of data according to the control command.
8. The method of claim 7, wherein the control command comprises a scheduling command, an arbitration command, a flush command, or a synchronization command, and wherein:
the scheduling command includes: when a task needs to be executed, selecting a proper executive body in the heterogeneous component pool according to a predefined method to execute the task;
the arbitration command includes: after the executive body executes the task, judging an execution result according to a predefined arbitration strategy;
the purge command includes: the executive body can periodically or aperiodically execute pre-cleaning or initialization of different levels, or reconfiguration and recombination operations;
the synchronization command includes: the state or scenario of the waiting executives and the online executives is synchronized.
9. The method for dynamic defense of distributed storage based on heterogeneous multi-copy according to claim 7, wherein the multi-element heterogeneous distributed storage system comprises a plurality of distributed storage systems, the plurality of distributed storage systems are self-healing synchronously, the multi-element heterogeneous distributed storage system provides a unified scheduling interface outwards, the heterogeneous component pool comprises a plurality of executives, and each executor corresponds to one distributed storage system.
10. The heterogeneous multi-copy based distributed storage dynamic defense method of claim 9, the executables comprising a GlusterFS executor, a CephFS executor, or a MooseFS executor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910339982.XA CN111866038A (en) | 2019-04-25 | 2019-04-25 | Distributed storage dynamic defense system and method based on heterogeneous multiple copies |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910339982.XA CN111866038A (en) | 2019-04-25 | 2019-04-25 | Distributed storage dynamic defense system and method based on heterogeneous multiple copies |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111866038A true CN111866038A (en) | 2020-10-30 |
Family
ID=72951265
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910339982.XA Pending CN111866038A (en) | 2019-04-25 | 2019-04-25 | Distributed storage dynamic defense system and method based on heterogeneous multiple copies |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111866038A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112532635A (en) * | 2020-12-01 | 2021-03-19 | 郑州昂视信息科技有限公司 | Security verification method and device of mimicry defense equipment |
| CN113282661A (en) * | 2021-05-31 | 2021-08-20 | 河南信大网御科技有限公司 | Heterogeneous execution body trusted configuration synchronization method and system |
| CN114398683A (en) * | 2022-03-24 | 2022-04-26 | 之江实验室 | Endogenous safety database storage method and device based on heterogeneous subsystem |
| CN116318945A (en) * | 2023-03-09 | 2023-06-23 | 南京航空航天大学 | Multi-target service function chain deployment method based on endophytic dynamic defense architecture |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1801923A (en) * | 2005-01-05 | 2006-07-12 | 中央电视台 | Graded memory management system |
| CN105404474A (en) * | 2015-12-07 | 2016-03-16 | 上海爱数信息技术股份有限公司 | Data migration method of heterogeneous distributed memory system |
| CN107733986A (en) * | 2017-09-15 | 2018-02-23 | 中国南方电网有限责任公司 | Support the protection of integrated deployment and monitoring operation big data support platform |
| CN107943867A (en) * | 2017-11-10 | 2018-04-20 | 中国电子科技集团公司第三十二研究所 | High-performance hierarchical storage system supporting heterogeneous storage |
| CN109101320A (en) * | 2018-08-08 | 2018-12-28 | 中科边缘智慧信息科技（苏州）有限公司 | Heterogeneous processor fusion of platforms management system |
| CN109327539A (en) * | 2018-11-15 | 2019-02-12 | 上海天玑数据技术有限公司 | A kind of distributed block storage system and its data routing method |
-
2019
- 2019-04-25 CN CN201910339982.XA patent/CN111866038A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1801923A (en) * | 2005-01-05 | 2006-07-12 | 中央电视台 | Graded memory management system |
| CN105404474A (en) * | 2015-12-07 | 2016-03-16 | 上海爱数信息技术股份有限公司 | Data migration method of heterogeneous distributed memory system |
| CN107733986A (en) * | 2017-09-15 | 2018-02-23 | 中国南方电网有限责任公司 | Support the protection of integrated deployment and monitoring operation big data support platform |
| CN107943867A (en) * | 2017-11-10 | 2018-04-20 | 中国电子科技集团公司第三十二研究所 | High-performance hierarchical storage system supporting heterogeneous storage |
| CN109101320A (en) * | 2018-08-08 | 2018-12-28 | 中科边缘智慧信息科技（苏州）有限公司 | Heterogeneous processor fusion of platforms management system |
| CN109327539A (en) * | 2018-11-15 | 2019-02-12 | 上海天玑数据技术有限公司 | A kind of distributed block storage system and its data routing method |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112532635A (en) * | 2020-12-01 | 2021-03-19 | 郑州昂视信息科技有限公司 | Security verification method and device of mimicry defense equipment |
| CN113282661A (en) * | 2021-05-31 | 2021-08-20 | 河南信大网御科技有限公司 | Heterogeneous execution body trusted configuration synchronization method and system |
| CN114398683A (en) * | 2022-03-24 | 2022-04-26 | 之江实验室 | Endogenous safety database storage method and device based on heterogeneous subsystem |
| CN114398683B (en) * | 2022-03-24 | 2022-06-10 | 之江实验室 | Endogenous safety database storage method and device based on heterogeneous subsystem |
| CN116318945A (en) * | 2023-03-09 | 2023-06-23 | 南京航空航天大学 | Multi-target service function chain deployment method based on endophytic dynamic defense architecture |
| CN116318945B (en) * | 2023-03-09 | 2023-10-20 | 南京航空航天大学 | Multi-target service function chain deployment method based on endophytic dynamic defense architecture |
| US12003528B1 (en) | 2023-03-09 | 2024-06-04 | Nanjing University Of Aeronautics And Astronautics | Endogenous dynamic defense architecture-based multi-objective service function chain deployment method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111866038A (en) | Distributed storage dynamic defense system and method based on heterogeneous multiple copies | |
| CN113169952B (en) | Container cloud management system based on block chain technology | |
| CN112840326B (en) | Test engine for automated operation management | |
| CN110096857B (en) | Authority management method, device, equipment and medium for block chain system | |
| US8229897B2 (en) | Restoring a file to its proper storage tier in an information lifecycle management environment | |
| JP2019008791A (en) | Smart contract life cycle management | |
| CN107710215A (en) | The method and apparatus of mobile computing device safety in test facilities | |
| JP5999574B2 (en) | Database management system and computer system | |
| EP3217248A1 (en) | Method and device for writing data, and system | |
| US9405484B2 (en) | System of managing remote resources | |
| US20180089039A1 (en) | Recovery of an infected and quarantined file in a primary storage controller from a secondary storage controller | |
| US20170185505A1 (en) | Systems and methods for implementing an automated parallel deployment solution | |
| US11281550B2 (en) | Disaster recovery specific configurations, management, and application | |
| CN108846053A (en) | Data copy management method, device, equipment, system and readable storage medium storing program for executing | |
| US11960369B2 (en) | Efficient creation of a secondary database system | |
| CN117296043A (en) | Method, medium, and system for lease management | |
| CN116226092B (en) | Database updating method and device | |
| US20240241981A1 (en) | Methods and systems for data synchronization, and computer-readable storage media | |
| CN114564466B (en) | Database management system and method for managing database | |
| CN114564706A (en) | User authority management method and device, electronic equipment and storage medium | |
| US11968241B1 (en) | Auto-tuning permissions using a learning mode | |
| CN107506231A (en) | A kind of method and system of VDI data protections | |
| US20080222374A1 (en) | Computer system, management computer, storage system and volume management method | |
| CN108491295A (en) | A kind of data verification method based on cloud computing | |
| CN116975158B (en) | Request processing method, apparatus, computer device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201030 |