CN107977168A - A kind of data based on cloud storage disperse storage system - Google Patents
A kind of data based on cloud storage disperse storage system Download PDFInfo
- Publication number
- CN107977168A CN107977168A CN201711351926.5A CN201711351926A CN107977168A CN 107977168 A CN107977168 A CN 107977168A CN 201711351926 A CN201711351926 A CN 201711351926A CN 107977168 A CN107977168 A CN 107977168A
- Authority
- CN
- China
- Prior art keywords
- data
- cloud storage
- except
- storage device
- data message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
- G06F3/0641—De-duplication techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The invention discloses a kind of data based on cloud storage to disperse storage system, is related to technical field of data storage.The present invention includes client, data except molality block, processor, data referencing monitoring device, the first cloud storage device, the second cloud storage device, the 3rd cloud storage device；Data receive the data message of client upload except molality block and data message are carried out by data de-duplication except weight, and processor is by except the data message after weight carries out data layout and is respectively stored into the first cloud storage device, the second cloud storage device and the 3rd cloud storage device.The present invention to data by being handled to obtain except data message after weight, and obtain data referencing rate, the segment data information attribute is judged by data referencing rate, data message store so as to reduce the calculation step needed for processor using based on the data distribution strategy of duplication；Using the data layout strategy based on correcting and eleting codes so as to obtaining higher storage efficiency.
Description
Technical field
The invention belongs to technical field of data storage, disperses storage system more particularly to a kind of data based on cloud storage
System.
Background technology
With the explosive growth of data, how data-storage system effectively inquires about mass data, be write
It is processed into the research emphasis for field of data storage.Data-storage system refers to be set by the various storages of storage program and data
The system that the equipment and Processing Algorithm of standby, control unit and management information scheduling are formed.As storage data are more and more, deposit
The memory space of storage system is also increasing, and the process performance requirement to data-storage system is also higher and higher.
The mode of data storage at present is this mainly by setting a large database come dedicated storage mass data
Data storage method, although can meet the high storage capacity requirement of mass data by large database, from large database
Inquiring about, write the efficiency of a certain data significantly reduces, and sacrifices data-handling efficiency, therefore it provides a kind of be based on cloud storage
Data disperse storage system, solve the above problems.
The content of the invention
It is an object of the invention to provide a kind of data based on cloud storage to disperse storage system, passes through client, data
Except molality block, processor, data referencing monitoring device, the first cloud storage device, the second cloud storage device, the 3rd cloud storage device
Setting, solve the problems, such as in the case of existing mass data high storage capacity that data-handling efficiency is low, poor safety performance.
In order to solve the above technical problems, the present invention is achieved by the following technical solutions：
The present invention disperses storage system, including client, data except molality block, processing for a kind of data based on cloud storage
Device, data referencing monitoring device, the first cloud storage device, the second cloud storage device, the 3rd cloud storage device；Wherein, the visitor
Data message of the family end to data required storage except molality block uploads；The data receive the data of client upload except molality block
Information simultaneously carries out data message by data de-duplication except weight, the data are used for except the data after weight are believed except molality block
Breath is transferred to processor；Wherein, the processor is electrically connected with data referencing monitoring device, the data referencing monitoring device with
Data are electrically connected except molality block；The data referencing monitoring device is used to obtain the citation rate of data message automatically and is transmitted
To processor；The processor receives the data message removed after weight that data remove weight module transfer；The processor receives data
The data message citation rate of monitoring device transmission is quoted, the processor is by except the data message after weight carries out data layout and divides
Do not store to the first cloud storage device, the second cloud storage device and the 3rd cloud storage device.
Further, the data de-duplication includes the following steps：SS01：Automatically retrieval is carried out to all data to go forward side by side
Row piecemeal；SS02：Repeated data information is judged from data message using the deblocking monitoring technology based on block level automatically；
SS03：Repeated data information is deleted and retains the single copy of repeated data information, and uses direction single copy
Pointer replaces other copies repeated；SS04：Obtain except the data message after weight.
Further, the data layout includes the following steps：S1：Processor is obtained automatically by data referencing monitoring device
Access is according to the data message citation rate got except molality block when to data message except weight；S2：Processor draws data message
With rate compared with citation rate preset value, when data message citation rate is more than citation rate preset value using the number based on duplication
Data message is stored according to Distribution Strategy；S3：Base is used when data message citation rate is less than or equal to citation rate preset value
In the data layout strategy of correcting and eleting codes；S4：By data information memory to the first cloud storage device, the second cloud storage device, the 3rd
Cloud storage device.
Further, the deblocking monitoring technology based on block level uses the deblocking technology based on fixed length.
The invention has the advantages that：
1st, the present invention removes the setting of molality block by data, and data can be handled to obtain the data letter except after weight
Breath, and by data except molality block obtains the citation rate of data, judge that the segment data information is heat by the citation rate of data
Point information or general information, for the hot information being often accessed by the user, using the data distribution strategy based on duplication
Storage is carried out to data message so as to reduce the calculation step needed for processor；And it can then be used for general information and be based on entangling deleting
The data layout strategy of code is so as to obtaining higher storage efficiency.
2nd, the present invention is realized the secure storage of information, is avoided to the full extent by the setting of multiple cloud storage devices
Cloud disk stores the problem of loss of data, largely protects the information security of people；The present invention is easy and effective, is easy to make
With.
Certainly, implement any of the products of the present invention and do not necessarily require achieving all the advantages described above at the same time.
Brief description of the drawings
In order to illustrate the technical solution of the embodiments of the present invention more clearly, embodiment will be described below required
Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for ability
For the those of ordinary skill of domain, without creative efforts, it can also be obtained according to these attached drawings other attached
Figure.
Fig. 1 is the structure diagram that the data of the invention based on cloud storage disperse storage system.
Embodiment
Below in conjunction with the attached drawing in the embodiment of the present invention, the technical solution in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained all other without creative efforts
Embodiment, belongs to the scope of protection of the invention.
Refering to Figure 1, the present invention disperses storage system, including client, number for a kind of data based on cloud storage
According to except molality block, processor, data referencing monitoring device, the first cloud storage device, the second cloud storage device, the 3rd cloud storage dress
Put；Wherein, data message of the client to data required storage except molality block uploads；Data are uploaded except molality block receives client
Data message and by data de-duplication to data message carry out except weight, data except molality block be used for by except the data after weight
Information is transferred to processor；Wherein, processor is electrically connected with data referencing monitoring device, and data referencing monitoring device is removed with data
Molality block is electrically connected；Data referencing monitoring device is used to obtain the citation rate of data message automatically and is transmitted to processor；
Processor receives the data message removed after weight that data remove weight module transfer；Processor receives the transmission of data referencing monitoring device
Data message citation rate, processor is by except the data message after weight carries out data layout and is respectively stored into the first cloud storage dress
Put, the second cloud storage device and the 3rd cloud storage device.
Wherein, data de-duplication includes the following steps：SS01：Automatically retrieval is carried out to all data and carries out piecemeal；
SS02：Repeated data information is judged from data message using the deblocking monitoring technology based on block level automatically；SS03：It is right
Repeated data information is deleted and retains the single copy of repeated data information, and is replaced using the pointer for being directed toward single copy
Other copies repeated；SS04：Obtain except the data message after weight.
Wherein, data layout includes the following steps：S1：Processor obtains data by data referencing monitoring device and removes automatically
The data message citation rate that molality block is got when to data message except weight；S2：Processor is by data message citation rate with drawing
It is compared with rate preset value, when data message citation rate is more than citation rate preset value using the data distribution plan based on duplication
Slightly data message is stored；S3：Used when data message citation rate is less than or equal to citation rate preset value and be based on correcting and eleting codes
Data layout strategy；S4：By data information memory to the first cloud storage device, the second cloud storage device, the 3rd cloud storage dress
Put.
Wherein, the deblocking monitoring technology based on block level uses the deblocking technology based on fixed length.
In the description of this specification, the description of reference term " one embodiment ", " example ", " specific example " etc. means
At least one implementation of the present invention is contained in reference to the embodiment or example particular features, structures, materials, or characteristics described
In example or example.In the present specification, schematic expression of the above terms may not refer to the same embodiment or example.
Moreover, particular features, structures, materials, or characteristics described can close in any one or more embodiments or example
Suitable mode combines.
Present invention disclosed above preferred embodiment is only intended to help and illustrates the present invention.Preferred embodiment is not detailed
All details are described, are not limited the invention to the specific embodiments described.Obviously, according to the content of this specification,
It can make many modifications and variations.This specification is chosen and specifically describes these embodiments, is in order to preferably explain the present invention
Principle and practical application so that skilled artisan can be best understood by and utilize the present invention.The present invention is only
Limited by claims and its four corner and equivalent.
Claims (4)
1. a kind of data based on cloud storage disperse storage system, it is characterised in that including client, data except molality block, place
Manage device, data referencing monitoring device, the first cloud storage device, the second cloud storage device, the 3rd cloud storage device；
Wherein, data message of the client to data required storage except molality block uploads；The data are received except molality block
The data message of client upload simultaneously carries out data message except weight, the data are used for except molality block by data de-duplication
By except the data information transfer after weight to processor；
Wherein, the processor is electrically connected with data referencing monitoring device, and the data referencing monitoring device removes molality with data
Block is electrically connected；The data referencing monitoring device is used to obtain the citation rate of data message automatically and is transmitted to processor；
Wherein, the processor receives the data message removed after weight that data remove weight module transfer；The processor receives data
The data message citation rate of monitoring device transmission is quoted, the processor will carry out data layout except the data message after weight, and
It is respectively stored into the first cloud storage device, the second cloud storage device and the 3rd cloud storage device.
2. a kind of data based on cloud storage according to claim 1 disperse storage system, it is characterised in that the repetition
Data, which are deleted, to be included the following steps：
SS01：Automatically retrieval is carried out to all data and carries out piecemeal；
SS02：Repeated data information is judged from data message using the deblocking monitoring technology based on block level automatically；
SS03：Repeated data information is deleted and retains the single copy of repeated data information, and uses the single pair of direction
This pointer replaces other copies repeated；
SS04：Obtain except the data message after weight.
3. a kind of data based on cloud storage according to claim 1 disperse storage system, it is characterised in that：
The data layout includes the following steps：
S1：Processor obtains data except molality block is got when to data message except weight automatically by data referencing monitoring device
Data message citation rate；
S2：Processor compared with citation rate preset value, quotes data message citation rate when data message citation rate is more than
Data message is stored using the data distribution strategy based on duplication during rate preset value；
S3：When data message citation rate is less than or equal to citation rate preset value using the data layout strategy based on correcting and eleting codes；
S4：By data information memory to the first cloud storage device, the second cloud storage device, the 3rd cloud storage device.
4. a kind of data based on cloud storage according to claim 2 disperse storage system, it is characterised in that described to be based on
The deblocking monitoring technology of block level uses the deblocking technology based on fixed length.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711351926.5A CN107977168B (en) | 2017-12-15 | 2017-12-15 | Data dispersed storage system based on cloud storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711351926.5A CN107977168B (en) | 2017-12-15 | 2017-12-15 | Data dispersed storage system based on cloud storage |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107977168A true CN107977168A (en) | 2018-05-01 |
| CN107977168B CN107977168B (en) | 2021-01-01 |
Family
ID=62006457
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711351926.5A Active CN107977168B (en) | 2017-12-15 | 2017-12-15 | Data dispersed storage system based on cloud storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107977168B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109491591A (en) * | 2018-09-17 | 2019-03-19 | 广东工业大学 | A kind of information diffusion method suitable for cloudy storage system |
| CN110618968A (en) * | 2019-08-13 | 2019-12-27 | 数字视觉云(北京)科技发展有限公司 | Media asset storage system based on ipfs |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678158A (en) * | 2013-12-26 | 2014-03-26 | 中国科学院信息工程研究所 | Optimization method and system for data layout |
| US20140101398A1 (en) * | 2008-10-31 | 2014-04-10 | Netapp Inc. | Remote office duplication |
| CN104932841A (en) * | 2015-06-17 | 2015-09-23 | 南京邮电大学 | Saving type duplicated data deleting method in cloud storage system |
| CN106020722A (en) * | 2016-05-19 | 2016-10-12 | 浪潮(北京)电子信息产业有限公司 | Method, device and system for deduplication of repeated data of cloud storage system |
| CN107463334A (en) * | 2016-06-03 | 2017-12-12 | 三星电子株式会社 | System and method for providing expansible and contractile memory overload configuration |
-
2017
- 2017-12-15 CN CN201711351926.5A patent/CN107977168B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140101398A1 (en) * | 2008-10-31 | 2014-04-10 | Netapp Inc. | Remote office duplication |
| CN103678158A (en) * | 2013-12-26 | 2014-03-26 | 中国科学院信息工程研究所 | Optimization method and system for data layout |
| CN104932841A (en) * | 2015-06-17 | 2015-09-23 | 南京邮电大学 | Saving type duplicated data deleting method in cloud storage system |
| CN106020722A (en) * | 2016-05-19 | 2016-10-12 | 浪潮(北京)电子信息产业有限公司 | Method, device and system for deduplication of repeated data of cloud storage system |
| CN107463334A (en) * | 2016-06-03 | 2017-12-12 | 三星电子株式会社 | System and method for providing expansible and contractile memory overload configuration |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109491591A (en) * | 2018-09-17 | 2019-03-19 | 广东工业大学 | A kind of information diffusion method suitable for cloudy storage system |
| CN110618968A (en) * | 2019-08-13 | 2019-12-27 | 数字视觉云(北京)科技发展有限公司 | Media asset storage system based on ipfs |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107977168B (en) | 2021-01-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102662992B (en) | Method and device for storing and accessing massive small files | |
| US10901619B2 (en) | Selecting pages implementing leaf nodes and internal nodes of a data set index for reuse | |
| CN109783013A (en) | Configure and access the method and system of expansible object storage | |
| CN103870202B (en) | A kind of distributed storage method and system of block device | |
| CN107436725A (en) | A kind of data are write, read method, apparatus and distributed objects storage cluster | |
| CN103986694B (en) | Control method of multi-replication consistency in distributed computer data storing system | |
| CN107451486A (en) | The authority setting method and device of a kind of file system | |
| CN104346458B (en) | Date storage method and storage device | |
| CN107153644A (en) | A kind of method of data synchronization and device | |
| CN106302595A (en) | A kind of method and apparatus that server is carried out physical examination | |
| CN106886610A (en) | The file management method and device of a kind of distributed file system | |
| CN109842652A (en) | A kind of method for uploading of file, terminal, Cloud Server and computer storage medium | |
| CN102143228A (en) | Cloud storage system, cloud client and method for realizing storage area network service | |
| CN107977168A (en) | A kind of data based on cloud storage disperse storage system | |
| CN104346345A (en) | Data storage method and device | |
| CN104956340B (en) | Expansible Data duplication is deleted | |
| CN104158844A (en) | Remote real-time monitoring system | |
| CN107506438A (en) | A kind of data processing storage method and device for Internet of Things | |
| CN105204782B (en) | A kind of method and device for realizing data storage | |
| CN106980618B (en) | File storage method and system based on MongoDB distributed cluster architecture | |
| CN105068760B (en) | Date storage method, data storage device and storage device | |
| CN106951190B (en) | Data storage and access method, node and server cluster | |
| CN108363727A (en) | A kind of date storage method and device based on ZFS file system | |
| CN108304555A (en) | Distributed maps data processing method | |
| CN102780780B (en) | Method, equipment and system for data processing in cloud computing mode |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: 230000 floors 4-5, building A1, Zhongguancun collaborative innovation Zhihui Park, the intersection of Nanfeihe road and Lanzhou Road, Baohe Economic Development Zone, Hefei, Anhui Province Patentee after: Anhui Changtai Technology Co.,Ltd. Address before: 210-d16, building A3, Hefei Innovation Industrial Park, No. 800, Wangjiang West Road, high tech Zone, Hefei City, Anhui Province 230000 Patentee before: ANHUI CHANGTAI INFORMATION SECURITY SERVICE Co.,Ltd. |
|
| CP03 | Change of name, title or address |