CN111176894A - Storage layering technology for data storage and data disaster recovery - Google Patents
Storage layering technology for data storage and data disaster recovery Download PDFInfo
- Publication number
- CN111176894A CN111176894A CN201911380332.6A CN201911380332A CN111176894A CN 111176894 A CN111176894 A CN 111176894A CN 201911380332 A CN201911380332 A CN 201911380332A CN 111176894 A CN111176894 A CN 111176894A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- data block
- specific
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 113
- 238000013500 data storage Methods 0.000 title claims abstract description 34
- 238000011084 recovery Methods 0.000 title claims abstract description 25
- 238000005516 engineering process Methods 0.000 title claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 10
- 238000000034 method Methods 0.000 claims description 10
- 238000012986 modification Methods 0.000 claims description 10
- 230000004048 modification Effects 0.000 claims description 10
- 238000012217 deletion Methods 0.000 claims description 9
- 230000037430 deletion Effects 0.000 claims description 9
- 238000013508 migration Methods 0.000 claims description 5
- 230000005012 migration Effects 0.000 claims description 5
- 238000005192 partition Methods 0.000 claims description 4
- 238000002360 preparation method Methods 0.000 claims description 4
- 230000000694 effects Effects 0.000 description 5
- 238000010276 construction Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 241000700605 Viruses Species 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 235000013601 eggs Nutrition 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
Abstract
the invention relates to the technical field of computers, and discloses a storage layering technology for data storage and data disaster recovery, which ① contains a storage module for segmenting storage objects in the data storage and data disaster recovery, a merging processing module for ② determining whether segmented data in the step ① contains the same or similar data blocks, and if so, adopting a preset merging rule to reserve one data block in repeated data blocks.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a storage layering technology for data storage and data disaster recovery.
Background
The data storage and data disaster recovery are integrated IT application environments formed by large data concentration, are various IT application service providing centers, are data calculation, network and storage centers, realize the unified deployment of security strategies, the unified operation and maintenance management of IT infrastructures, business applications and data, the data storage and data disaster recovery are the key points of the IT construction of current operators and various large enterprises, the data center construction is under progress or completed by operators, finance, electric power, governments, energy, traffic, education, manufacturing industry, large enterprises, websites, e-commerce companies and the like, the examination is greatly prompted to realize the integration and centralized management of an IT system through the construction of the data center, the internal operation and management efficiency and the external service level are improved, the integral TCO of the IT construction is reduced, and higher requirements are also provided for the data security while the large data concentration is realized, how to deal with the risk caused by putting eggs in a basket becomes the key point of attention of a plurality of users, and in order to deal with the impact of various natural disasters (fire, flood, earthquake and the like) and artificial disasters (misoperation, virus and the like) on enterprise data safety, in recent years, data storage and data disaster recovery become the development trend of data protection.
In the traditional network storage system, a centralized storage server is generally adopted in the data storage process, the storage server becomes the bottleneck of the system performance, is also the focus of reliability and safety, and cannot meet the requirement of large-scale storage application, so that a storage layering technology for data storage and data disaster recovery is provided to solve the problems.
Disclosure of Invention
Technical problem to be solved
Aiming at the defects of the prior art, the invention provides a storage layering technology for data storage and data disaster recovery, which has the advantages of storage layering and the like, and solves the problems that a storage server becomes the bottleneck of system performance, is also the focus of reliability and safety and cannot meet the requirements of large-scale storage application in the process of storing data by generally adopting a centralized storage server in the conventional network storage system.
(II) technical scheme
In order to achieve the purpose of the storage layering, the invention provides the following technical scheme: a storage layering technique for data storage and data disaster recovery, comprising:
carrying out sectional processing on the storage objects in the data storage and data disaster recovery through a storage module;
determining whether segmented data in the step ① contains the same or similar data blocks through a merging processing module, if so, adopting a preset merging rule, reserving one data block in repeated data blocks, merging attribute information in other repeated data blocks into attribute information of the reserved data block, taking the data block of which the attribute information meets a hierarchical storage condition as data to be migrated, and if not, performing partition storage on a storage object in the data storage and data disaster preparation in the step ① through a storage module;
obtaining the equipment address corresponding to the hierarchical storage condition of the data to be migrated in the step ② through an address obtaining module, modifying the address pointing to the data to be migrated in the metadata corresponding to the data to be migrated in the step ② through an address modifying module according to the equipment address corresponding to the hierarchical storage condition;
receiving a data access request through an receiving module, wherein the data access request comprises the data block identifier of the data to be migrated in the step ③, inquiring and acquiring an address pointing to the data block identifier in metadata corresponding to the data block identifier through an access module according to the data request, and accessing data information in the data block corresponding to the data block identifier according to the address pointing to the data block identifier in the metadata corresponding to the data block identifier;
and fifthly, migrating the data to be migrated in the step ④ to the storage equipment corresponding to the data block identifier in the step ④ through a migration module, thereby finishing the storage and layering of the data.
Preferably, the fractionated storage conditions include a combination of one or more of the following: specific user information, specific creation time, specific access time, specific modification time, specific access frequency and a file type to which specific data block information belongs; or a specific user information range, a specific creation time range, a specific access time range, a specific modification time range, a specific access frequency range, and a file type range to which specific data block information belongs.
Preferably, the specific data block information includes one or more of the following combinations: user information, creation time, access time, modification time, access frequency of the data block, and the file type of the data block.
Preferably, the storage module is one or a combination of multiple types of fast storage devices supporting repeated data deletion, slow storage devices supporting repeated data deletion, or medium-speed storage devices supporting repeated data deletion, and the like, the performance of the fast storage devices is higher than that of the medium-speed storage devices, and the performance of the medium-speed storage devices is higher than that of the slow storage devices.
Preferably, the storage module divides the data stream stored on the storage module into data blocks by using a secure hash algorithm, an information digest algorithm, and the like, and records attribute information of each data block, where the attribute information of a data block is specifically a feature value reflecting a feature of the data block, and the attribute information is a combination of one or more feature values.
(III) advantageous effects
Compared with the prior art, the invention provides a storage layering technology for data storage and data disaster recovery, which has the following beneficial effects:
the storage layering technology for data storage and data disaster recovery achieves the effects of simple and convenient classification and storage by carrying out segmentation processing on storage objects in the data storage and data disaster recovery through a storage module, combining data blocks with the same or similar data blocks through a combining processing module to generate data to be migrated and carrying out partition storage on data blocks without the same or similar data blocks, meanwhile, obtaining an equipment address corresponding to the hierarchical storage condition corresponding to the data to be migrated through an address obtaining module, modifying a pointed address in metadata of the data to be migrated through an address modifying module, and finally carrying out precise migration and storage on the data to be migrated through a receiving module, an accessing module and a migration module, achieves the effects of efficient and precise storage and layering, and avoids natural disasters (fire disasters, disasters and the like during the data storage process of a centralized storage server, Flood, earthquake and the like) and artificial disasters (misoperation, virus and the like) to cause the conditions of slow data storage speed and data loss, thereby improving the practicability of the storage layering technology and facilitating the popularization of the storage layering technology of data storage and data disaster recovery.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
A storage layering technique for data storage and data disaster recovery, comprising:
carrying out sectional processing on the storage objects in the data storage and data disaster recovery through a storage module;
determining whether segmented data in the step ① contains the same or similar data blocks through a merging processing module, if so, adopting a preset merging rule, reserving one data block in repeated data blocks, merging attribute information in other repeated data blocks into attribute information of the reserved data block, taking the data block of which the attribute information meets a hierarchical storage condition as data to be migrated, and if not, performing partition storage on a storage object in the data storage and data disaster preparation in the step ① through a storage module;
obtaining the equipment address corresponding to the hierarchical storage condition of the data to be migrated in the step ② through an address obtaining module, modifying the address pointing to the data to be migrated in the metadata corresponding to the data to be migrated in the step ② through an address modifying module according to the equipment address corresponding to the hierarchical storage condition;
receiving a data access request through an receiving module, wherein the data access request comprises the data block identifier of the data to be migrated in the step ③, inquiring and acquiring an address pointing to the data block identifier in metadata corresponding to the data block identifier through an access module according to the data request, and accessing data information in the data block corresponding to the data block identifier according to the address pointing to the data block identifier in the metadata corresponding to the data block identifier;
and fifthly, migrating the data to be migrated in the step ④ to the storage equipment corresponding to the data block identifier in the step ④ through a migration module, thereby finishing the storage and layering of the data.
Preferably, the fractionated storage conditions include a combination of one or more of the following: specific user information, specific creation time, specific access time, specific modification time, specific access frequency and a file type to which specific data block information belongs; or a specific user information range, a specific creation time range, a specific access time range, a specific modification time range, a specific access frequency range, and a file type range to which specific data block information belongs.
Preferably, the specific data block information includes one or more of the following combinations: user information, creation time, access time, modification time, access frequency of the data block, and the file type of the data block.
Preferably, the storage module is one or a combination of multiple fast storage devices supporting repeated data deletion, slow storage devices supporting repeated data deletion, or medium-speed storage devices supporting repeated data deletion, and the like, where the performance of the fast storage devices is higher than that of the medium-speed storage devices, and the performance of the medium-speed storage devices is higher than that of the slow storage devices, for example, when the fast storage devices and the slow storage devices perform the same operation, the response time of the fast storage devices is often shorter than that of the slow storage devices.
Preferably, the storage module divides the data stream stored on the storage module into data blocks by using a secure hash algorithm, an information digest algorithm, and the like, and records attribute information of each data block, where the attribute information of a data block is specifically a feature value reflecting a feature of the data block, and the attribute information is a combination of one or more feature values.
The invention has the beneficial effects that: the storage objects in the data storage and data disaster preparation are segmented by the storage module, the data blocks with the same or similar data blocks are merged by the merging processing module to generate data to be migrated, the data blocks without the same or similar data blocks are stored in a partitioned manner, the effect of simple and convenient classification and storage is achieved, meanwhile, the equipment address corresponding to the hierarchical storage condition of the data to be migrated is obtained by the address obtaining module, the pointing address in the metadata of the data to be migrated is modified by the address modifying module, and finally, the data to be migrated is accurately migrated and stored by the receiving module, the access module and the migrating module, so that the effect of efficient and accurate storage and layering is achieved, and the situations that the data storage speed is slow and the data is lost due to natural disasters (fire, flood, earthquake and the like) and man-made disasters (misoperation, viruses and the like) in the data storage process of the centralized storage server are avoided, therefore, the practicability of the storage layering technology is improved, and the storage layering technology of data storage and data disaster recovery is convenient to popularize.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (5)
1. A storage layering technique for data storage and data disaster recovery, comprising:
carrying out sectional processing on the storage objects in the data storage and data disaster recovery through a storage module;
determining whether segmented data in the step ① contains the same or similar data blocks through a merging processing module, if so, adopting a preset merging rule, reserving one data block in repeated data blocks, merging attribute information in other repeated data blocks into attribute information of the reserved data block, taking the data block of which the attribute information meets a hierarchical storage condition as data to be migrated, and if not, performing partition storage on a storage object in the data storage and data disaster preparation in the step ① through a storage module;
obtaining the equipment address corresponding to the hierarchical storage condition of the data to be migrated in the step ② through an address obtaining module, modifying the address pointing to the data to be migrated in the metadata corresponding to the data to be migrated in the step ② through an address modifying module according to the equipment address corresponding to the hierarchical storage condition;
receiving a data access request through an receiving module, wherein the data access request comprises the data block identifier of the data to be migrated in the step ③, inquiring and acquiring an address pointing to the data block identifier in metadata corresponding to the data block identifier through an access module according to the data request, and accessing data information in the data block corresponding to the data block identifier according to the address pointing to the data block identifier in the metadata corresponding to the data block identifier;
and fifthly, migrating the data to be migrated in the step ④ to the storage equipment corresponding to the data block identifier in the step ④ through a migration module, thereby finishing the storage and layering of the data.
2. The storage tiering technique for data storage and data disaster recovery as claimed in claim 1, wherein said tiered storage conditions include a combination of one or more of: specific user information, specific creation time, specific access time, specific modification time, specific access frequency and a file type to which specific data block information belongs; or a specific user information range, a specific creation time range, a specific access time range, a specific modification time range, a specific access frequency range, and a file type range to which specific data block information belongs.
3. The storage tiering technique for data storage and data disaster recovery as claimed in claim 2, wherein said specific data block information comprises one or more of the following combinations: user information, creation time, access time, modification time, access frequency of the data block, and the file type of the data block.
4. The storage layering technology for data storage and data disaster recovery as claimed in claim 1, wherein the storage module is one or more of a fast storage device supporting duplicate data deletion, a slow storage device supporting duplicate data deletion, or a medium-speed storage device supporting duplicate data deletion, and the performance of the fast storage device is higher than that of the medium-speed storage device, and the performance of the medium-speed storage device is higher than that of the slow storage device.
5. The storage tiering technique for data storage and data disaster recovery as claimed in claim 1, wherein the storage module employs a secure hash algorithm, an information digest algorithm, etc. to divide the data stream stored on the storage module into data blocks, and records attribute information of the data blocks in each data block, the attribute information of the data blocks is specifically a feature value reflecting the feature of the data block, and the attribute information is a combination of one or more feature values.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911380332.6A CN111176894A (en) | 2019-12-27 | 2019-12-27 | Storage layering technology for data storage and data disaster recovery |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911380332.6A CN111176894A (en) | 2019-12-27 | 2019-12-27 | Storage layering technology for data storage and data disaster recovery |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111176894A true CN111176894A (en) | 2020-05-19 |
Family
ID=70624252
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911380332.6A Pending CN111176894A (en) | 2019-12-27 | 2019-12-27 | Storage layering technology for data storage and data disaster recovery |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111176894A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111930685A (en) * | 2020-09-21 | 2020-11-13 | 柏科数据技术（深圳）股份有限公司 | Element selection method, device, server and storage medium based on CEPH |
| CN112677438A (en) * | 2020-12-08 | 2021-04-20 | 上海微亿智造科技有限公司 | Segmented construction and screening method and system based on injection molding and machine debugging data characteristics |
| CN112749623A (en) * | 2020-12-08 | 2021-05-04 | 上海微亿智造科技有限公司 | Processing and feature extraction method and system for high-frequency sensor data of injection molding process |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020255A (en) * | 2012-12-21 | 2013-04-03 | 华为技术有限公司 | Hierarchical storage method and hierarchical storage device |
| CN104239164A (en) * | 2013-06-19 | 2014-12-24 | 国家电网公司 | Cloud storage based disaster recovery backup switching system |
-
2019
- 2019-12-27 CN CN201911380332.6A patent/CN111176894A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020255A (en) * | 2012-12-21 | 2013-04-03 | 华为技术有限公司 | Hierarchical storage method and hierarchical storage device |
| CN104239164A (en) * | 2013-06-19 | 2014-12-24 | 国家电网公司 | Cloud storage based disaster recovery backup switching system |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111930685A (en) * | 2020-09-21 | 2020-11-13 | 柏科数据技术（深圳）股份有限公司 | Element selection method, device, server and storage medium based on CEPH |
| CN112677438A (en) * | 2020-12-08 | 2021-04-20 | 上海微亿智造科技有限公司 | Segmented construction and screening method and system based on injection molding and machine debugging data characteristics |
| CN112749623A (en) * | 2020-12-08 | 2021-05-04 | 上海微亿智造科技有限公司 | Processing and feature extraction method and system for high-frequency sensor data of injection molding process |
| CN112677438B (en) * | 2020-12-08 | 2021-11-16 | 上海微亿智造科技有限公司 | Segmented construction and screening method and system based on injection molding and machine debugging data characteristics |
| CN112749623B (en) * | 2020-12-08 | 2023-04-07 | 上海微亿智造科技有限公司 | Processing and feature extraction method and system for high-frequency sensor data of injection molding process |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111176894A (en) | Storage layering technology for data storage and data disaster recovery | |
| CN103218175B (en) | The cloud storage platform access control system of many tenants | |
| RU2705429C1 (en) | Method and device for distributed processing of stream data | |
| US11841861B1 (en) | Constrained query execution | |
| CN103544319A (en) | Multi-tenant database sharing method and multi-tenant database as-a-service system | |
| Zhang et al. | Research on hadoop-based enterprise file cloud storage system | |
| WO2017028394A1 (en) | Example-based distributed data recovery method and apparatus | |
| US20180278634A1 (en) | Cyber Security Event Detection | |
| CN105635311A (en) | Method for synchronizing resource pool information in cloud management platform | |
| CN113448938A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN104182306A (en) | Low-cost intelligent breakdown instant switching method for cloud host | |
| Jin et al. | Distributed storage system for electric power data based on Hbase | |
| Fang | Design of smart travel management system based on cloud service | |
| Silkina et al. | Using Blockchain-based approach for building the system events logging service | |
| CN102880832B (en) | A kind of implementation method of the system of the data magnanimity management under cluster | |
| CN204906437U (en) | Big data storage application network framework | |
| CN111737655A (en) | User authority management method, system and storage medium of cloud management platform | |
| BRPI0920319B1 (en) | method for accessing magnitude data from the smart grid services database and system and device for it | |
| CN110705712A (en) | Artificial intelligence basic resource and technology open platform for third-party social service | |
| CN114879907A (en) | Data distribution determination method, device, equipment and storage medium | |
| US11204717B2 (en) | Object storage system with access control quota status check | |
| CN112866386A (en) | Data storage data construction method based on cloud computing | |
| Zhao et al. | Research and Design of Power Big Data Desensitization System Based on K-means++ Clustering Algorithm | |
| CN111147575A (en) | Data storage system based on block chain | |
| CN112732706B (en) | Hardware equipment resource information storage method, system, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200519 |