CN103905517A - Data storage method and equipment - Google Patents
Data storage method and equipment Download PDFInfo
- Publication number
- CN103905517A CN103905517A CN201210587462.9A CN201210587462A CN103905517A CN 103905517 A CN103905517 A CN 103905517A CN 201210587462 A CN201210587462 A CN 201210587462A CN 103905517 A CN103905517 A CN 103905517A
- Authority
- CN
- China
- Prior art keywords
- data storage
- data
- demand information
- storage plan
- plan
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a data storage method. Based on basic needs and special needs of data storage, a data storage plan with the highest cost performance is determined, data storage is carried out according to the data storage plan with the highest cost performance, and the QoS needs of data storage are guaranteed. The invention further discloses data storage equipment.
Description
Technical field
The present invention relates to communication technical field, particularly a kind of date storage method, the invention still further relates to a kind of data storage device.
Background technology
At present, distributed memory system has become the infrastructure component of large market demand solution, have a large amount of software systems and application based on distributed storage exploitation and operation, and each software and application has the demand of different service quality QoSs to distributed memory system.For example, apply for distributed data digging, it need to be stored a large amount of data and carry out Distributed Calculation analysis with this, typical case's application as " great Yun " BC-PDM(parallel data mining) it operates in HDFS distributed file system, and utilizes MapReduce parallel computation framework to analyze.BC-PDM is data exchange capability between large capacity and node to the main demand of distributed memory system.And for " key-value pair " the system HBase being structured in equally on HDFS, it except capacity requirement, also needs large memory headroom demand and disk read-write IO fast to HDFS.Even, modules in HBase internal system is also different to the demand of storage system, the for example metadata information of HBase system and data message are all kept in distributed file system, and because metadata information often upgrades and reads, it need to be stored on the disk that performance is higher.
As shown in Figure 1, be the structural representation of distributed memory system, system is generally made up of from node, one or more host node and client multiple.Data are dispersed and store each into from node, and host node is in charge of the metadata information in system, are responsible for the storage of data from node, and client is by with host node with realize the read-write of data from node direct interaction.
In order to meet the demand of data storage, prior art has proposed the following solution:
(1) adjustment number of copies meets the demand of readwrite performance: as the temperature of monitor data, adjust the quantity of data trnascription according to temperature;
(2) adjust the position of data placement and meet the demand of network: as by data trnascription is placed on to Edge Server, thereby meet the requirement of the network bandwidth.
Above scheme only meets the requirement of readwrite performance and the network bandwidth by simple increase copy or change data placement position, cannot provide the data storage service that QoS ensures.
Summary of the invention
The invention provides a kind of date storage method, determine according to the primary demand of data storage and specific demand the data storage plan that cost performance is the highest, and store data with this, thus the QoS demand of guarantee data storage.
For achieving the above object, one aspect of the present invention provides a kind of method of data storage, comprising:
Receive data storage request message, obtain primary demand information and the specific demand information of in described data storage request message, carrying;
According to system resource information and at least one data storage plan of described primary demand Information generation;
Determine the comprehensive grading of described data storage plan according to described primary demand information and specific demand information;
Determine the cost performance of described data storage plan according to the comprehensive grading of described system resource information and described data storage plan;
Choose the data storage plan that cost performance is the highest, and according to the highest data storage plan storage data of described cost performance.
On the other hand, the present invention also provides a kind of data storage device, comprising:
Acquisition module, primary demand information and the specific demand information of carrying for obtaining the data storage request message that receives;
Generation module, for according to system resource information and at least one data storage plan of described primary demand Information generation; ;
Determination module, for determining the comprehensive grading of described data storage plan according to described primary demand information and specific demand information, and obtain the cost performance of described data storage plan according to the comprehensive grading of described system resource information and described data storage plan;
Memory module, for choosing the data storage plan that cost performance is the highest, and according to the highest data storage plan storage data of described cost performance.
Compared with prior art, the present invention has the following advantages:
The present invention is based on primary demand and the specific demand of data storage, determine the data storage plan that cost performance is the highest, and carry out data storage according to the highest data storage plan of cost performance, ensured the QoS demand of data storages.
Brief description of the drawings
Fig. 1 is the structural representation of distributed memory system in prior art;
A kind of date storage method schematic flow sheet that Fig. 2 provides for the embodiment of the present invention;
A kind of date storage method schematic flow sheet that Fig. 3 provides for the specific embodiment of the invention;
A kind of data storage device structural representation that Fig. 4 provides for the embodiment of the present invention.
Embodiment
Based on the deficiency of prior art scheme, the embodiment of the present invention provides a kind of date storage method, determine according to the primary demand of data storage and specific demand the data storage plan that cost performance is the highest, and store data with this, thus the QoS demand of guarantee data storage.
Below in conjunction with accompanying drawing, the embodiment of the present invention is described in detail.
Referring to Fig. 2, be a kind of date storage method that the present invention proposes, the method specifically comprises the following steps:
Due to the operating characteristics difference of various application to data, therefore receiving after data storage request message, first obtain wherein entrained primary demand information and specific information, these information all set in advance among request message before application or client transmission data storage request message.
According to the primary demand of obtaining in previous step and existing resource information table, generation can meet the data storage plan of storage demand.In this step, can generate in advance all data storage plan, more therefrom select according to primary demand information, or according in terms by terms generated data storage plan of primary demand information, can select flexibly according to actual conditions with upper type.
In this step, will according to the primary demand of data and specific demand, the data storage plan of previous step generation be assessed and be adjusted further; Specific demand wherein at least comprise reading/writing characteristics (write less mutiread, write less read, read and write balanced etc.) more, with the operative relationship of data with existing (as, first read existing data, then the data of reading to write at present; Or the concurrent data of reading existing data and writing at present), and data storage duration etc.
In order further to set forth technological thought of the present invention, existing in conjunction with concrete application scenarios, technical scheme of the present invention is described.
As shown in Figure 3, be a kind of date storage method flow process that the specific embodiment of the invention proposes, this flow process comprises the following steps:
In distributed memory system, because each node hardware resource isomerism cannot be avoided, therefore the node in system receives the own resource information that each other nodes send in advance, and according to the rule of setting, all kinds of resources is classified.For example, storage resources is divided into several classes such as high, medium and low performance, and the IO literacy of different stage is provided respectively.For the storage resources with high-speed network appliance, be high to this resource mark network attribute, for the storage resources with large capacity internal memory, be high to this resource mark memory attribute.Due to the performance data of each disk of each node incomplete same, need to be normalized each disk performance, particularly, present node is receiving after the resource information that other nodes send, and the system resource information table of its structure is as follows:
Table 1. system resource information table
< Data Identification, number of copies, < memory node, memory disk > list, storage duration >
Except essential information, above-mentioned format content also comprised data has been stored as to several copies, and each copy should store on which disk of which memory node, and the information such as time of depositing.
In this step, the initial score of each data storage plan can be expressed as follows: in Performance=A* (total IO throughput/actual demand IO throughput of memory disk in current planning)+B*(current planning, store place meshed network amount of bandwidth/actual demand network bandwidth amount) store place node computing capability/actual demand computing capability in+C*(current planning), the ratio that wherein A, B and C are these parameters, A+B+C=1.
It should be noted that; the score calculation scheme adopting for the data storage plan above example that only just the present invention proposes according to actual conditions; the factor " throughput ", " network bandwidth amount ", " computing capability " and corresponding parameter ratio thereof wherein can be carried out suitable adjustment or be changed other the factor under different application scenarioss, and such change does not affect protection scope of the present invention.
In this specific embodiment, taking the operative relationship of data with existing in reading and writing data feature, this data and system, data storage duration three as example, this step is elaborated respectively:
(1) reading and writing data feature comprises: write read less many, write that mutiread is few, read-write is balanced etc.If data are read manyly for writing less, the Performance numerical value that adopts the more plan of distribution of number of copies is improved, the pressure of as far as possible data being read is shared to multiple nodes.If data are few for writing mutiread, the Performance numerical value that adopts the more plan of distribution of number of copies is reduced, because in the time that number of copies is more, keep the overhead of copy consistency to increase.
(2) in current data writing and storage system, the operative relationship feature of data with existing comprises: current data and data with existing are that order is read, current data and data with existing are to walk abreast to read etc.Carrying out MapReduce while calculating, be parallel reading for multiple data blocks of a large file of distributed file system.For be stored in the data on HDFS as HBase, the data that newly add and old data generally present the feature that order is read.In the time that current data and data with existing are parallel reading, in order better to make, multiple disks are concurrent to be read, by data placement position and data with existing in plan of distribution not in the plan Performance of disk raising, thereby improve the concurrent efficiency of reading; When current data and data with existing are orders while reading, because current data and data with existing do not exist IO competition, data placement position and data with existing in plan of distribution are positioned to the plan Performance raising of same disk.
(3) deposit data duration feature: when time data memory in short-term, data are placed on frequent erasablely on the less data storage scheme of disk life-span impact, improve its Performance.If the resting period is longer, take efficient storage resource after for a long time for fear of it, the Performance of the data distribution plan that is stored in low speed disk and many copies is improved.
The some preferred implementations that only propose for the embodiment of the present invention for the processing method of each specific demand information above; those skilled in the art can carry out other adaptive features on this basis to be increased or adjusts, and these improve and all belong to protection scope of the present invention.
In this step, first adopt following formula to calculate the expense of each data storage plan:
The required storage resources amount * unit price of current I O ability+the provide required amount of network resources * unit price of current network bandwidth+provide current computing capability required amount of computational resources * unit price is provided Cost=.
According to R rates quoted above Cost, then calculate cost performance corresponding to each data storage plan with Performance/Cost, select cost performance the highest data storage plan.
Because the corresponding cost performance of each data storage plan is finally a numerical value; therefore as long as under the prerequisite that the cost performance numerical procedure adopting for current all data storage plan is consistent; those skilled in the art also can take other mode to calculate the cost performance of each data storage plan, and such change all belongs to protection scope of the present invention.
Meanwhile, the invention allows for a kind of data storage device, as described in Figure 4, this data storage device comprises:
Further, in concrete application scenarios, described determination module 430, specifically for:
According to the initial score of data storage plan described in described primary demand Information generation, and described initial score is adjusted according to described specific demand information.
Further, in concrete application scenarios, described determination module 430, specifically for:
According to the corresponding weight factor of each performance option in described primary demand information, and the ratio of each performance option between described data storage plan and described primary demand information, generate the initial score of described data storage plan.
Further, in concrete application scenarios,
Described primary demand information at least comprises one or more following performance options: data space, reading and writing data speed, the network bandwidth;
Described specific demand information at least comprises one or more following performance options: reading and writing data feature, the operative relationship of data and system data to be stored, time data memory.
Further, in concrete application scenarios, also comprise: processing module 450,
Described processing module 450, the node resource information sending specifically for receiving each other nodes, and the described node resource information receiving is classified according to default rule, generate described system resource information.
As can be seen here, the present invention is based on primary demand and the specific demand of data storage, determine the data storage plan that cost performance is the highest, and carry out data storage according to the highest data storage plan of cost performance, ensured the QoS demand of data storages.
Through the above description of the embodiments, those skilled in the art can be well understood to the present invention and can realize by hardware, and the mode that also can add necessary general hardware platform by software realizes.Based on such understanding, technical scheme of the present invention can embody with the form of software product, it (can be CD-ROM that this software product can be stored in a non-volatile memory medium, USB flash disk, portable hard drive etc.) in, comprise that each implements the method described in scene in order to make a computer equipment (can be personal computer, server, or the network equipment etc.) carry out the present invention in some instructions.
It will be appreciated by those skilled in the art that accompanying drawing is a schematic diagram of preferably implementing scene, the module in accompanying drawing or flow process might not be that enforcement the present invention is necessary.
It will be appreciated by those skilled in the art that the module in the device of implementing in scene can be distributed in the device of implementing scene according to implementing scene description, also can carry out respective change and be arranged in the one or more devices that are different from this enforcement scene.The module of above-mentioned enforcement scene can be merged into a module, also can further split into multiple submodules.
The invention described above sequence number, just to describing, does not represent the quality of implementing scene.
Disclosed is above only several concrete enforcement scene of the present invention, and still, the present invention is not limited thereto, and the changes that any person skilled in the art can think of all should fall into protection scope of the present invention.
Claims (10)
1. a method for data storage, is characterized in that, comprising:
Receive data storage request message, obtain primary demand information and the specific demand information of in described data storage request message, carrying;
According to system resource information and at least one data storage plan of described primary demand Information generation;
Determine the comprehensive grading of described data storage plan according to described primary demand information and specific demand information;
Determine the cost performance of described data storage plan according to the comprehensive grading of described system resource information and described data storage plan;
Choose the data storage plan that cost performance is the highest, and according to the highest data storage plan storage data of described cost performance.
2. the method for claim 1, is characterized in that, determines the comprehensive grading of described data storage plan according to described primary demand information and described specific demand information, is specially:
According to the initial score of data storage plan described in described primary demand Information generation, and described initial score is adjusted according to described specific demand information.
3. method as claimed in claim 2, is characterized in that, according to the initial score of data storage plan described in described primary demand Information generation, is specially:
According to the corresponding weight factor of each performance option in described primary demand information, and the ratio of each performance option between described data storage plan and described primary demand information, generate the initial score of described data storage plan.
4. the method as described in claim 1-3 any one, is characterized in that,
Described primary demand information at least comprises one or more following performance options: data space, reading and writing data speed, the network bandwidth;
Described specific demand information at least comprises one or more following performance options: reading and writing data feature, the operative relationship of data and system data to be stored, time data memory.
5. the method for claim 1, is characterized in that, before receiving data storage request message, also comprises:
Present node receives the node resource information that each other nodes send, and the described node resource information receiving is classified according to default rule, generates described system resource information.
6. a data storage device, is characterized in that, comprising:
Acquisition module, primary demand information and the specific demand information of carrying for obtaining the data storage request message that receives;
Generation module, for according to system resource information and at least one data storage plan of described primary demand Information generation; ;
Determination module, for determining the comprehensive grading of described data storage plan according to described primary demand information and specific demand information, and obtain the cost performance of described data storage plan according to the comprehensive grading of described system resource information and described data storage plan;
Memory module, for choosing the data storage plan that cost performance is the highest, and according to the highest data storage plan storage data of described cost performance.
7. equipment as claimed in claim 6, is characterized in that, described determination module, specifically for:
According to the initial score of data storage plan described in described primary demand Information generation, and described initial score is adjusted according to described specific demand information.
8. equipment as claimed in claim 7, is characterized in that, described determination module, specifically for:
According to the corresponding weight factor of each performance option in described primary demand information, and the ratio of each performance option between described data storage plan and described primary demand information, generate the initial score of described data storage plan.
9. the equipment as described in claim 6-8 any one, is characterized in that,
Described primary demand information at least comprises one or more following performance options: data space, reading and writing data speed, the network bandwidth;
Described specific demand information at least comprises one or more following performance options: reading and writing data feature, the operative relationship of data and system data to be stored, time data memory.
10. equipment as claimed in claim 6, is characterized in that, also comprises: processing module,
Described processing module, the node resource information sending specifically for receiving each other nodes, and the described node resource information receiving is classified according to default rule, generate described system resource information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210587462.9A CN103905517A (en) | 2012-12-28 | 2012-12-28 | Data storage method and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210587462.9A CN103905517A (en) | 2012-12-28 | 2012-12-28 | Data storage method and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103905517A true CN103905517A (en) | 2014-07-02 |
Family
ID=50996672
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210587462.9A Pending CN103905517A (en) | 2012-12-28 | 2012-12-28 | Data storage method and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103905517A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105072201A (en) * | 2015-08-28 | 2015-11-18 | 北京奇艺世纪科技有限公司 | Distributed storage system and storage quality control method and device thereof |
| CN105335450A (en) * | 2014-08-14 | 2016-02-17 | 中兴通讯股份有限公司 | Data storage processing method and apparatus |
| CN106874144A (en) * | 2016-12-30 | 2017-06-20 | 上海中信信息发展股份有限公司 | Storage backup policy evaluation method based on electronic record attribute |
| CN106897279A (en) * | 2015-12-17 | 2017-06-27 | 阿里巴巴集团控股有限公司 | For the method and apparatus of distributed document treatment |
| CN108108384A (en) * | 2016-11-25 | 2018-06-01 | 中国移动通信有限公司研究院 | A kind of date storage method and device |
| CN111414135A (en) * | 2020-03-11 | 2020-07-14 | 国家海洋环境预报中心 | IO efficiency optimization method of high-performance computer system |
| CN111723078A (en) * | 2020-06-24 | 2020-09-29 | 苏州松鼠山人工智能科技有限公司 | Data storage method and device |
| CN112446490A (en) * | 2020-11-27 | 2021-03-05 | 苏州浪潮智能科技有限公司 | Network training data set caching method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101187931A (en) * | 2007-12-12 | 2008-05-28 | 浙江大学 | Distribution type file system multi-file copy management method |
| CN101232422A (en) * | 2008-01-18 | 2008-07-30 | 北京交通大学 | Network storage system based on graticule technique |
| CN101827121A (en) * | 2010-03-12 | 2010-09-08 | 成都市华为赛门铁克科技有限公司 | Method, service end and system for creating files in RAID (Redundant Array of Independent Disk) |
| CN102170396A (en) * | 2011-05-06 | 2011-08-31 | 浙江大学 | QoS control method of cloud storage system based on differentiated service |
| WO2012106918A1 (en) * | 2011-07-22 | 2012-08-16 | 华为技术有限公司 | Content processing method, device and system |
-
2012
- 2012-12-28 CN CN201210587462.9A patent/CN103905517A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101187931A (en) * | 2007-12-12 | 2008-05-28 | 浙江大学 | Distribution type file system multi-file copy management method |
| CN101232422A (en) * | 2008-01-18 | 2008-07-30 | 北京交通大学 | Network storage system based on graticule technique |
| CN101827121A (en) * | 2010-03-12 | 2010-09-08 | 成都市华为赛门铁克科技有限公司 | Method, service end and system for creating files in RAID (Redundant Array of Independent Disk) |
| CN102170396A (en) * | 2011-05-06 | 2011-08-31 | 浙江大学 | QoS control method of cloud storage system based on differentiated service |
| WO2012106918A1 (en) * | 2011-07-22 | 2012-08-16 | 华为技术有限公司 | Content processing method, device and system |
Non-Patent Citations (1)
| Title |
|---|
| 许春聪等: "分布式文件系统存储介质评测与分析", 《计算机学报》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105335450A (en) * | 2014-08-14 | 2016-02-17 | 中兴通讯股份有限公司 | Data storage processing method and apparatus |
| CN105335450B (en) * | 2014-08-14 | 2020-06-05 | 中兴通讯股份有限公司 | Data storage processing method and device |
| CN105072201A (en) * | 2015-08-28 | 2015-11-18 | 北京奇艺世纪科技有限公司 | Distributed storage system and storage quality control method and device thereof |
| CN105072201B (en) * | 2015-08-28 | 2018-04-13 | 北京奇艺世纪科技有限公司 | A kind of distributed memory system and its storage method of quality control and device |
| CN106897279A (en) * | 2015-12-17 | 2017-06-27 | 阿里巴巴集团控股有限公司 | For the method and apparatus of distributed document treatment |
| CN108108384A (en) * | 2016-11-25 | 2018-06-01 | 中国移动通信有限公司研究院 | A kind of date storage method and device |
| CN106874144A (en) * | 2016-12-30 | 2017-06-20 | 上海中信信息发展股份有限公司 | Storage backup policy evaluation method based on electronic record attribute |
| CN111414135A (en) * | 2020-03-11 | 2020-07-14 | 国家海洋环境预报中心 | IO efficiency optimization method of high-performance computer system |
| CN111723078A (en) * | 2020-06-24 | 2020-09-29 | 苏州松鼠山人工智能科技有限公司 | Data storage method and device |
| CN112446490A (en) * | 2020-11-27 | 2021-03-05 | 苏州浪潮智能科技有限公司 | Network training data set caching method, device, equipment and storage medium |
| WO2022110861A1 (en) * | 2020-11-27 | 2022-06-02 | 苏州浪潮智能科技有限公司 | Method and apparatus for data set caching in network training, device, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103905517A (en) | Data storage method and equipment | |
| US8590050B2 (en) | Security compliant data storage management | |
| CN102332029B (en) | Hadoop-based mass classifiable small file association storage method | |
| KR101644894B1 (en) | Cloud storage service apparatus and method | |
| US9811529B1 (en) | Automatically redistributing data of multiple file systems in a distributed storage system | |
| US9860317B1 (en) | Throughput throttling for distributed file storage services with varying connection characteristics | |
| US10356150B1 (en) | Automated repartitioning of streaming data | |
| CN103905311A (en) | Flow table matching method and device and switch | |
| CN103631538A (en) | Cold and hot data identification threshold value calculation method, device and system | |
| EP2625604A2 (en) | Automatic replication of virtual machines | |
| US10684996B1 (en) | Distributing data of multiple logically independent file systems in distributed storage systems including physically partitioned disks | |
| Chen et al. | E2FS: an elastic storage system for cloud computing | |
| CN110968554A (en) | Block chain storage method, storage system and storage medium based on file chain blocks | |
| US20200042398A1 (en) | Systems and methods for predictive data protection | |
| CN103095788A (en) | Cloud resource scheduling policy based on network topology | |
| US20220179585A1 (en) | Management of Idle Time Compute Tasks in Storage Systems | |
| CN112434938A (en) | Resource capacity management method and device | |
| CN104182487A (en) | Unified storage method supporting various storage modes | |
| EP4170491A1 (en) | Resource scheduling method and apparatus, electronic device, and computer-readable storage medium | |
| Tu et al. | An optimized cluster storage method for real-time big data in Internet of Things | |
| CN103440182A (en) | Adaptive allocation method and device and adaptive replica consistency method | |
| CN101419600A (en) | Data copy mapping method and device based on object-oriented LANGUAGE | |
| CN104021088A (en) | Log storage method and device | |
| CN103618804A (en) | Performance difference-based load balancing method for distributed key value storage system | |
| Zhang et al. | Cost-effective and traffic-optimal data placement strategy for cloud-based online social networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140702 |