CN103678563A - Capacity obtaining method and system - Google Patents
Capacity obtaining method and system Download PDFInfo
- Publication number
- CN103678563A CN103678563A CN201310659397.0A CN201310659397A CN103678563A CN 103678563 A CN103678563 A CN 103678563A CN 201310659397 A CN201310659397 A CN 201310659397A CN 103678563 A CN103678563 A CN 103678563A
- Authority
- CN
- China
- Prior art keywords
- disk
- capacity
- storage block
- capacity information
- amount
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0623—Securing storage systems in relation to content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The invention provides a capacity obtaining method and system so as to solve the problem that IO consumption is too large, so that a burden is brought to the system in the capacity obtaining process. In order to solve the problem, the capacity obtaining method comprises the steps that a request for calling magnetic disk metadata is sent to a file system of a magnetic disk; the returned magnetic disk metadata are received, and the magnetic disk metadata comprise the total quantity of storage blocks in the magnetic disk and the free quantity of the storage blocks; capacity information of the magnetic disk is obtained through calculation of the total quantity of the storage blocks in the magnetic disk and the free quantity of the storage blocks. According to the capacity obtaining method and system, the magnetic disk metadata are directly called, IO consumption can be avoided, the utilization rate of the magnetic disk is kept in a normal level all the time, and burdens can not be brought to the system.
Description
Patented claim of the present invention be that Dec 27, application number in 2011 are 201110444072.1 the applying date, name is called the dividing an application of Chinese invention patent application of " capacity acquiring method in distributed file system and procurement of reserve capacity system ".
Technical field
The application relates to computer memory technical, particularly relates to a kind of capacity acquiring method and system.
Background technology
HDFS(Hadoop Distributed File System) be a distributed file system, the feature that has high fault tolerance, and the data that provide high-throughput to visit application program, are applicable to having the application program of super large data set, can solve the problem of mass data storage.
The host node of HDFS (NameNode) is communicated by letter by carrying out periodic heartbeat with memory node (DataNode), obtains the capacity information of each memory node, and then obtains the capacity information of whole cluster.And each memory node comprises a plurality of disks, so memory node adopts du command mode to obtain the space hold amount of each disk, and then obtains local space hold amount.
The specific practice that du command mode is obtained the space hold amount of each disk is, each disk that searching loop memory node comprises, and carry out following operation for each disk: from the root directory of disk storage data (for example, the catalogue of parameter d fs.data.dir configuration) start, take this root directory according to the mode of depth-first, to travel through this directory tree as root node.If run into intermediate node, this node correspondence is sub-directory, continues traversal sub-directory, until run into leaf node, what this node was corresponding is file, obtains corresponding file metadata, obtains the size information of file from file metadata.Then using after the cumulative summation of the size information of all leaf node files corresponding to certain intermediate node as the size of giving intermediate node correspondence sub-directory.Then successively add up until described directory tree has traveled through, obtained the space hold amount of All Files in whole directory tree, acquired the space hold amount of this disk, then return to du order.
In above-mentioned process, need the All Files in scanning disk; therefore conventionally can produce a large amount of IO (Input/Output; I/O) consume; this process causes disk utilization saturated; be that disk utilization has reached 100%; this has caused very large burden to system, can affect even in some cases the performance of HDFS.
Summary of the invention
The application provides a kind of capacity acquiring method and system, to solve in the capacity of obtaining IO, consumes and excessive system is made to cumbrous problem.
In order to address the above problem, the application discloses a kind of capacity acquiring method, comprising:
File system to disk sends the request of calling disk metadata;
The disk metadata that reception is returned, described disk metadata comprises the total amount of storage block and the free time of storage block amount in disk;
By the calculating to the total amount of storage block in described disk and the free time of storage block amount, obtain the capacity information of disk.
Preferably, described capacity information comprises space total amount, Spatial Residual amount and space hold amount.
Preferably, the capacity of each storage block is fixed, described by the calculating to the total amount of storage block in described disk and the free time of storage block amount, obtains the capacity information of disk, comprising:
The total amount of storage block in the capacity of described storage block and described disk is multiplied each other, and result of calculation is as the space total amount of disk;
The free time amount of storage block in the capacity of described storage block and described disk is multiplied each other, and result of calculation is as the Spatial Residual amount of disk.
Preferably, the space total amount of disk is deducted to the Spatial Residual amount of disk, the space hold amount of result of calculation disk.
Preferably, after obtaining the capacity information of disk, also comprise:
The capacity information of disk is returned to df order.
Preferably, memory node calls df order to obtain the capacity information of each disk, and the capacity information of all disks is added and as the capacity information of this memory node.
Preferably, class DF calls df order for realizing, and described memory node, by carrying out df order to obtain the space hold amount of each disk, comprising:
Memory node, by calling the getDfsUsed method of FSDataset class, finally calls the getDfsUsed method of FSVolume class;
The getDfsUsed method of described FSVolume class is realized and is called df order by class DF, to obtain the capacity information of disk.
Preferably, memory node periodically calls df order to obtain corresponding capacity information, and described capacity information is periodically transferred to host node.
Preferably, it is benchmark that memory node be take the capacity information of each disk, in corresponding disk, distributes and save data.
Accordingly, disclosed herein as well is a kind of procurement of reserve capacity system, comprising:
Send request module, for the file system to disk, send the request of calling disk metadata;
Receive meta data block, for receiving the disk metadata of returning, described disk metadata comprises the total amount of storage block and the free time of storage block amount in disk;
Calculated capacity module, for the calculating of the free time amount by the total amount of described disk storage block and storage block, obtains the capacity information of disk.
Preferably, described capacity information comprises space total amount, Spatial Residual amount and space hold amount.
Preferably, the capacity of each storage block is fixed, and described calculated capacity module, comprising:
Computer memory total amount submodule, for the total amount of the capacity of described storage block and described disk storage block is multiplied each other, result of calculation is as the space total amount of disk;
Computer memory surplus submodule, for the free time amount of the capacity of described storage block and described disk storage block is multiplied each other, result of calculation is as the Spatial Residual amount of disk.
Preferably, described calculated capacity module, also comprises:
Computer memory occupancy submodule, for deducting the space total amount of disk the Spatial Residual amount of disk, the space hold amount of result of calculation disk.
Preferably, described system also comprises:
Capacity returns to module, for the capacity information of disk is returned to df order.
Preferably, described system also comprises:
Obtain capacity module, for memory node, call df order to obtain the capacity information of each disk, and the capacity information of all disks is added and as the capacity information of this memory node.
Preferably, class DF calls df order for realizing, described in obtain capacity module and comprise:
Call method submodule, by calling the getDfsUsed method of FSDataset class, finally calls the getDfsUsed method of FSVolume class for memory node;
Call instruction submodule, calls df order for the getDfsUsed method of described FSVolume class by class DF realization, to obtain the capacity information of disk.
Preferably, described system also comprises:
Transport module, periodically calls df order to obtain corresponding capacity information for memory node, and described capacity information is periodically transferred to host node.
Distribute data module, for memory node, take the capacity information of each disk is benchmark, in corresponding disk, distributes and save data.
Compared with prior art, the application comprises following advantage:
First, the application does not need to travel through according to this logical storage structure of directory tree each file of disk, obtain the size information in the metadata of each file, after successively adding up again, obtain the space hold amount of disk, but according to the physical storage structure of disk, file system to disk sends the request of calling disk metadata, the disk metadata that reception is returned, described disk metadata comprises the total amount of storage block and the free time of storage block amount in disk, again by the calculating to the total amount of storage block in described disk and the free time of storage block amount, obtain the capacity information of disk.In this process, directly call disk metadata, can not produce IO and consume, make the utilization factor of disk remain at normal level, can not cause burden to system.
Accompanying drawing explanation
Fig. 1 is a kind of capacity acquiring method process flow diagram described in the embodiment of the present application;
Fig. 2 is distributed system architecture figure described in the embodiment of the present application;
Fig. 3 is a kind of procurement of reserve capacity system construction drawing described in the embodiment of the present application.
Embodiment
For the application's above-mentioned purpose, feature and advantage can be become apparent more, below in conjunction with the drawings and specific embodiments, the application is described in further detail.
The application provides a kind of capacity acquiring method, memory node directly calls disk metadata according to the physical arrangement of data storage when obtaining the capacity information of each disk, and then the capacity information that obtains disk can not produce IO consumption, make the utilization factor of disk remain at normal level, can not cause burden to system.
Disk can be divided into logical storage structure and physical storage structure when storage data.
Described logical storage structure be take file as storage cell, sets up a directory tree, and the host node in directory tree is master catalogue, and intermediate node is sub-directory, and leaf node is exactly file.
Described physical storage structure be take storage block as storage cell, a disk is divided into many storage blocks, the capacity of storage block is smaller, and the capacity of storage block is fixed, a common file can be stored in several storage blocks, for example storage block is 1KB, and the size of certain file is 5KB, so just needs 5 storage blocks to store this file.
With reference to Fig. 1, provided a kind of capacity acquiring method process flow diagram described in the embodiment of the present application.
In the file system of disk, there is disk metadata, the data relevant with capacity of described disk meta-data preservation are according to the storage of disk physical storage structure, so disk metadata can comprise the data such as the total amount of storage block and the free time of storage block amount in disk.
The application obtains capacity according to disk physical storage structure, therefore, when obtaining the capacity information of each disk, can directly call the disk metadata in disk file system.Therefore need to send the request of calling disk metadata to the file system of disk.
Described file system, also claims file management system, is the software mechanism of being in charge of in operating system with storage file information.File system is comprised of three parts: software relevant to file management, be managed file and implement file management desired data structure.
The file system of disk, after receiving the request of calling disk metadata, can be returned to disk metadata, now receives described disk metadata, just can receive the data such as the total amount of storage block and the free time of storage block amount in disk.
Described capacity information comprises space total amount, Spatial Residual amount and space hold amount, and the capacity of each storage block is fixed, and therefore, according to the total amount of storage block in disk metadata and the free time of storage block amount, can calculate the capacity information that obtains disk.
Concrete calculation procedure is as follows:
Step 131, multiplies each other the total amount of storage block in the capacity of described storage block and described disk, and result of calculation is as the space total amount of disk;
Step 132, multiplies each other the free time amount of storage block in the capacity of described storage block and described disk, and result of calculation is as the Spatial Residual amount of disk.
Step 133, deducts the space total amount of disk the Spatial Residual amount of disk, the space hold amount of result of calculation disk.
For example, in certain disk, the total amount of storage block is 1024, and the free time amount of storage block is 512, and the capacity of each storage block is 1KB.
The space total amount of this disk is 1KB * 1024=1024KB=1M
The Spatial Residual amount of this disk is 1KB * 512=512KB
The space hold amount of this disk is 1024KB-512KB=512KB
By above-mentioned computing method, can obtain the capacity information of this disk.
Prior art is that logical storage structure according to disk is from host node, along intermediate node, finally can traverse leaf node, from master catalogue, start finally to traverse file, and obtain file metadata, cumulative until master catalogue successively again, obtains the space occupancy volume of this disk.This process often can produce a large amount of IO (Input/Output; I/O) consume, this process causes disk utilization saturated, and disk utilization has reached 100%; this has caused very large burden to system, can affect even in some cases the performance of HDFS.
The application is according to the physical storage structure of disk, file system to disk sends the request of calling disk metadata, the disk metadata that reception is returned, described disk metadata comprises the total amount of storage block and the free time of storage block amount in disk, by the calculating to the total amount of storage block in described disk and the free time of storage block amount, obtain the capacity information of disk again.In this process, directly call disk metadata, can not produce IO and consume, make the utilization factor of disk remain at normal level, can not cause burden to system.
Preferably, after obtaining the capacity information of disk, also comprise:
The capacity information of disk is returned to df order.
Df order can perform step 11 processes to step 13 by call function, and this function finally can get the capacity information of disk, then described capacity information is returned to df order.
Preferably, memory node calls df order to obtain the capacity information of each disk, and the capacity information of all disks is added and as the capacity information of this memory node.
In the application, by df command calls function, finally get the capacity information of disk, therefore described memory node can call the capacity information that each disk is obtained in df order, then the capacity information of all disks is added and as the capacity information of this memory node.
For example, a certain memory node comprises 3 disks, by df command calls function, this function by sending request, reception of magnetic disc metadata and calculating, the capacity information that finally returns to disk 1 comprises: space total amount is 100M, Spatial Residual amount 40M and space hold amount 60M.The capacity information of disk 2 comprises: space total amount is 200M, Spatial Residual amount 120M and space hold amount 80M.The capacity information of disk 3 comprises: space total amount is 50M, Spatial Residual amount 40M and space hold amount 10M.
The capacity information of final this memory node is:
Space total amount 100M+200M+50M=350M
Spatial Residual amount 40M+120M+40M=200M
Space hold amount 60M+80M+10M=150M
Preferably, class DF calls df order for realizing, and described memory node, by carrying out df order to obtain the space hold amount of each disk, comprising:
Memory node, by calling the getDfsUsed method of FSDataset class, finally calls the getDfsUsed method of FSVolume class;
The getDfsUsed method of described FSVolume class is realized and is called df order by class DF, to obtain the capacity information of disk.
The getDfsUsed method that memory node (DataNode) calls FSDataset class is obtained the capacity information of disk, then by communicating by letter and transmit this information with the heartbeat of host node (NameNode).And the getDfsUsed method of FSDataset class is finally called the getDfsUsed method of FSVolume class.
Owing to having comprised the class that realizes of df, i.e. class DF in FSVolume.So the realization of getDfsUsed method need to be configured such that to the version by DF class.
The following situation of literary composition is set:
Wherein, object usage is the example of DF class, call the realization that " usage.getUsed () " is df order, final call function obtains space total amount and the Spatial Residual amount of disk from the management structure of the file system of each piece disk, is calculating the space hold amount that obtains disk.
Preferably, memory node periodically calls df order to obtain corresponding capacity information, and described capacity information is periodically transferred to host node.
In HDFS system, the host node of HDFS (NameNode) is communicated by letter by carrying out periodic heartbeat with memory node (DataNode), obtains the capacity information of each memory node, and then obtains the capacity information of whole cluster.
Therefore memory node can periodically call df order to obtain corresponding capacity information, and then described capacity information is periodically transferred to host node.
The application can be applied to HDFS(Hadoop Distributed File System) in distributed file system, this system can solve the problem of mass data storage.
Follow-up, host node, when storage new data, can, according to the capacity information of each memory node transmission, be determined and be stored in which memory node.
Memory node is after receiving the data of host node transmission, and the capacity information of each disk of take is benchmark, in corresponding disk, distributes and save data.
For example, the size of certain data is 200M, and the capacity information of memory node 1 is: disk A space total amount is 100M, Spatial Residual amount 40M and space hold amount 60M, and disk B space total amount is 300M, Spatial Residual amount 120M and space hold amount 180M.The capacity information of memory node 2 is: disk A space total amount is 500M, Spatial Residual amount 105M and space hold amount 395M, and disk B space total amount is 500M, Spatial Residual amount 320M and space hold amount 180M.
Therefore, host node can store data in memory node 2, and it is benchmark that memory node 2 be take the capacity information of each disk again, and the Spatial Residual amount of disk B is 320M, is greater than the big or small 200M of data, therefore can be by data allocations to storing in disk B.
With reference to Fig. 2, provided distributed system architecture figure described in the embodiment of the present application.
In described distributed system architecture, comprised following module:
Client is client, obtains the memory location of certain file from host node, then carries out write/read operation to corresponding memory node place.
NameNode is the host node in hadoop distributed file system, safeguarded the metadata information of the All Files in hadoop distributed file system, the position of the data storage of the access rights of include file, host and file etc., and these metadata informations finally all can be persisted in disk.
DataNode is the memory node in hadoop distributed file system, has stored the content of the file in hadoop distributed file system.
SecondaryNameNode is the standby host node of hadoop distributed file system, and described standby host node hadoop metadata of distributed type file system regular and that host node keeps carries out synchronously, the data of persistence on host node being merged simultaneously.
Wherein, the file in each memory node of hadoop distributed file system, is finally divided into fixed-size (for example the size of a piece is 128MB, configurable) and deposits.If block size of the big or small deficiency of file, file is stored as a piece, and its size is the actual size of file, does not understand and takes up room more.If file size is not block size alignment, last piece can not take the space of a piece, and all block size summations are the space of file actual size.
In concrete enforcement, each memory node (DataNode) can send to host node (NameNode) being stored in local piece termly.So that NameNode can perception hadoop file system in the data storage location of All Files.
Now, described file is divided into fixed-size to be deposited, and is that this is for hadoop distributed file system, for determining the data storage location of file.
And file discussed above is divided into several storage blocks, deposit, for disk, only can get capacity information.The technology that this two place is discussed is different, can not obscure and look it.
Capacity acquiring method described in the application can be applied to hadoop distributed file system, and the memory node that is specifically applied to this system is obtaining the capacity information of each disk.Memory node obtains the capacity information of each disk by calling df order, and then obtains the capacity information of this memory node, more periodically carries out heartbeat with host node and communicate by letter, and informs the capacity information of oneself.
Follow-up host node, when the deposit position of decision-making new data, can pass through the capacity information of certain memory node, comes this memory node of decision-making whether can also continue to deposit new data.
With reference to Fig. 3, provided a kind of procurement of reserve capacity system construction drawing described in the embodiment of the present application.
Accordingly, the application also provides a kind of procurement of reserve capacity system, comprises and sends request module 11, reception meta data block 12 and calculated capacity module 13, wherein
Send request module 11, for the file system to disk, send the request of calling disk metadata;
Receive meta data block 12, for receiving the disk metadata of returning, described disk metadata comprises the total amount of storage block and the free time of storage block amount in disk;
Wherein, described capacity information comprises space total amount, Spatial Residual amount and space hold amount.
Preferably, the capacity of each storage block is fixed, and described calculated capacity module 13, comprising:
Computer memory total amount submodule 131, for the total amount of the capacity of described storage block and described disk storage block is multiplied each other, result of calculation is as the space total amount of disk;
Computer memory surplus submodule 132, for the free time amount of the capacity of described storage block and described disk storage block is multiplied each other, result of calculation is as the Spatial Residual amount of disk;
Computer memory occupancy submodule 133, for deducting the space total amount of disk the Spatial Residual amount of disk, the space hold amount of result of calculation disk.
Preferably, described system also comprises:
Capacity returns to module 14, for the capacity information of disk is returned to df order.
Obtain capacity module 15, for memory node, call df order to obtain the capacity information of each disk, and the capacity information of all disks is added and as the capacity information of this memory node.
Transport module 16, periodically calls df order to obtain corresponding capacity information for memory node, and described capacity information is periodically transferred to host node.
Distribute data module 17, for memory node, take the capacity information of each disk is benchmark, in corresponding disk, distributes and save data.
Preferably, class DF calls df order for realizing, described in obtain capacity module 15 and comprise:
Call method submodule 151, by calling the getDfsUsed method of FSDataset class, finally calls the getDfsUsed method of FSVolume class for memory node;
Call instruction submodule 152, calls df order for the getDfsUsed method of described FSVolume class by class DF realization, to obtain the capacity information of disk.
For system embodiment, because it is substantially similar to embodiment of the method, so description is fairly simple, relevant part is referring to the part explanation of embodiment of the method.
Each embodiment in this instructions all adopts the mode of going forward one by one to describe, and each embodiment stresses is the difference with other embodiment, between each embodiment identical similar part mutually referring to.
The application can describe in the general context of the computer executable instructions of being carried out by computing machine, for example program module.Usually, program module comprises the routine carrying out particular task or realize particular abstract data type, program, object, assembly, data structure etc.Also can in distributed computing environment, put into practice the application, in these distributed computing environment, by the teleprocessing equipment being connected by communication network, be executed the task.In distributed computing environment, program module can be arranged in the local and remote computer-readable storage medium that comprises memory device.
Finally, also it should be noted that, in this article, relational terms such as the first and second grades is only used for an entity or operation to separate with another entity or operational zone, and not necessarily requires or imply and between these entities or operation, have the relation of any this reality or sequentially.And, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thereby the process, method, commodity or the equipment that make to comprise a series of key elements not only comprise those key elements, but also comprise other key elements of clearly not listing, or be also included as the intrinsic key element of this process, method, commodity or equipment.The in the situation that of more restrictions not, the key element being limited by statement " comprising ... ", and be not precluded within process, method, commodity or the equipment that comprises described key element and also have other identical element.
A kind of capacity acquiring method and the system that above the application are provided, be described in detail, applied specific case herein the application's principle and embodiment are set forth, the explanation of above embodiment is just for helping to understand the application's method and core concept thereof; Meanwhile, for one of ordinary skill in the art, the thought according to the application, all will change in specific embodiments and applications, and in sum, this description should not be construed as the restriction to the application.
Claims (18)
1. a capacity acquiring method, is characterized in that, comprising:
File system to disk sends the request of calling disk metadata;
The disk metadata that reception is returned, described disk metadata comprises the total amount of storage block and the free time of storage block amount in disk;
By the calculating to the total amount of storage block in described disk and the free time of storage block amount, obtain the capacity information of disk.
2. method according to claim 1, is characterized in that, described capacity information comprises space total amount, Spatial Residual amount and space hold amount.
3. method according to claim 2, is characterized in that, the capacity of each storage block is fixed, described by the calculating to the total amount of storage block in described disk and the free time of storage block amount, obtains the capacity information of disk, comprising:
The total amount of storage block in the capacity of described storage block and described disk is multiplied each other, and result of calculation is as the space total amount of disk;
The free time amount of storage block in the capacity of described storage block and described disk is multiplied each other, and result of calculation is as the Spatial Residual amount of disk.
4. method according to claim 3, is characterized in that, also comprises:
The space total amount of disk is deducted to the Spatial Residual amount of disk, the space hold amount of result of calculation disk.
5. according to the arbitrary described method of its claim 1 to 4, it is characterized in that, after obtaining the capacity information of disk, also comprise:
The capacity information of disk is returned to df order.
6. method according to claim 5, is characterized in that, also comprises:
Memory node calls df order to obtain the capacity information of each disk, and the capacity information of all disks is added and as the capacity information of this memory node.
7. method according to claim 6, is characterized in that, class DF calls df order for realizing, and described memory node, by carrying out df order to obtain the space hold amount of each disk, comprising:
Memory node, by calling the getDfsUsed method of FSDataset class, finally calls the getDfsUsed method of FSVolume class;
The getDfsUsed method of described FSVolume class is realized and is called df order by class DF, to obtain the capacity information of disk.
8. method according to claim 6, is characterized in that, also comprises:
Memory node periodically calls df order to obtain corresponding capacity information, and described capacity information is periodically transferred to host node.
9. method according to claim 6, is characterized in that, also comprises:
It is benchmark that memory node be take the capacity information of each disk, in corresponding disk, distributes and save data.
10. a procurement of reserve capacity system, is characterized in that, comprising:
Send request module, for the file system to disk, send the request of calling disk metadata;
Receive meta data block, for receiving the disk metadata of returning, described disk metadata comprises the total amount of storage block and the free time of storage block amount in disk;
Calculated capacity module, for the calculating of the free time amount by the total amount of described disk storage block and storage block, obtains the capacity information of disk.
11. systems according to claim 10, is characterized in that, described capacity information comprises space total amount, Spatial Residual amount and space hold amount.
12. systems according to claim 11, is characterized in that, the capacity of each storage block is fixed, and described calculated capacity module, comprising:
Computer memory total amount submodule, for the total amount of the capacity of described storage block and described disk storage block is multiplied each other, result of calculation is as the space total amount of disk;
Computer memory surplus submodule, for the free time amount of the capacity of described storage block and described disk storage block is multiplied each other, result of calculation is as the Spatial Residual amount of disk.
13. systems according to claim 12, is characterized in that, described calculated capacity module, also comprises:
Computer memory occupancy submodule, for deducting the space total amount of disk the Spatial Residual amount of disk, the space hold amount of result of calculation disk.
14. according to claim 10 to 12 arbitrary described systems, it is characterized in that, also comprise:
Capacity returns to module, for the capacity information of disk is returned to df order.
15. systems according to claim 14, is characterized in that, also comprise:
Obtain capacity module, for memory node, call df order to obtain the capacity information of each disk, and the capacity information of all disks is added and as the capacity information of this memory node.
16. systems according to claim 15, is characterized in that, class DF calls df order for realizing, described in obtain capacity module and comprise:
Call method submodule, by calling the getDfsUsed method of FSDataset class, finally calls the getDfsUsed method of FSVolume class for memory node;
Call instruction submodule, calls df order for the getDfsUsed method of described FSVolume class by class DF realization, to obtain the capacity information of disk.
17. systems according to claim 15, is characterized in that, also comprise:
Transport module, periodically calls df order to obtain corresponding capacity information for memory node, and described capacity information is periodically transferred to host node.
18. systems according to claim 15, is characterized in that, also comprise:
Distribute data module, for memory node, take the capacity information of each disk is benchmark, in corresponding disk, distributes and save data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310659397.0A CN103678563A (en) | 2011-12-27 | 2011-12-27 | Capacity obtaining method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310659397.0A CN103678563A (en) | 2011-12-27 | 2011-12-27 | Capacity obtaining method and system |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110444072.1A Division CN102542040B (en) | 2011-12-27 | 2011-12-27 | Capacity acquiring method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103678563A true CN103678563A (en) | 2014-03-26 |
Family
ID=50316108
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310659397.0A Pending CN103678563A (en) | 2011-12-27 | 2011-12-27 | Capacity obtaining method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103678563A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110134738A (en) * | 2019-05-21 | 2019-08-16 | 中国联合网络通信集团有限公司 | Distributed memory system resource predictor method, device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009121057A2 (en) * | 2008-03-28 | 2009-10-01 | Alibaba Group Holding Limited | File folder transmission on network |
-
2011
- 2011-12-27 CN CN201310659397.0A patent/CN103678563A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009121057A2 (en) * | 2008-03-28 | 2009-10-01 | Alibaba Group Holding Limited | File folder transmission on network |
Non-Patent Citations (4)
| Title |
|---|
| KMESG: "linux的du和df命令", 《CSDN》 * |
| 刁树民: "NTFS文件系统簇映射剖析", 《佳木斯大学学报（自然科学版）》 * |
| 李智: "对Hadoop-HDFS性能造成重大影响的杀手－shell", 《CSDN》 * |
| 武雅丽等: "利用VB 4.0获取系统信息的程序设计", 《现代电子技术》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110134738A (en) * | 2019-05-21 | 2019-08-16 | 中国联合网络通信集团有限公司 | Distributed memory system resource predictor method, device |
| CN110134738B (en) * | 2019-05-21 | 2021-09-10 | 中国联合网络通信集团有限公司 | Distributed storage system resource estimation method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104113597B (en) | The HDFS data read-write method of a kind of many Data centres | |
| CN100583832C (en) | Data management method and system | |
| KR101959153B1 (en) | System for efficient processing of transaction requests related to an account in a database | |
| CN103106152B (en) | Based on the data dispatching method of level storage medium | |
| CN107766402A (en) | A kind of building dictionary cloud source of houses big data platform | |
| CN105074702A (en) | Database system providing single-tenant and multi-tenant environments | |
| CN103399894A (en) | Distributed transaction processing method on basis of shared storage pool | |
| CN106570113B (en) | Mass vector slice data cloud storage method and system | |
| CN105683928B (en) | For the method for data cache policies, server and memory devices | |
| CN111737168A (en) | Cache system, cache processing method, device, equipment and medium | |
| CN103886109A (en) | Method and device for realizing row lock of database | |
| CN109767274B (en) | Method and system for carrying out associated storage on massive invoice data | |
| CN105354250A (en) | Data storage method and device for cloud storage | |
| CN110109868A (en) | Method, apparatus and computer program product for index file | |
| CN113515545A (en) | Data query method, device, system, electronic equipment and storage medium | |
| CN102542040B (en) | Capacity acquiring method and system | |
| CN103678562A (en) | Capacity obtaining method and file data allocation method | |
| CN101783814A (en) | Metadata storing method for mass storage system | |
| CN102724290A (en) | Method, device and system for getting target customer group | |
| CN103678563A (en) | Capacity obtaining method and system | |
| Perwej et al. | An extensive investigate the mapreduce technology | |
| CN109299059A (en) | File storage, search method, device, storage medium and server | |
| Neumann et al. | Combining query performance with data integrity in the cloud: a hybrid cloud storage framework to enhance data access on the Windows Azure platform | |
| CN112540954B (en) | Multi-level storage construction and online migration method in directory unit | |
| CN109635042B (en) | OLTP and OLAP integrated automobile financial big data system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140326 |
|
| RJ01 | Rejection of invention patent application after publication |