CN103605795A - Metadata-based file storage method and device - Google Patents
Metadata-based file storage method and device Download PDFInfo
- Publication number
- CN103605795A CN103605795A CN201310654319.1A CN201310654319A CN103605795A CN 103605795 A CN103605795 A CN 103605795A CN 201310654319 A CN201310654319 A CN 201310654319A CN 103605795 A CN103605795 A CN 103605795A
- Authority
- CN
- China
- Prior art keywords
- file
- storage
- metadata
- information
- service system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
Abstract
The invention is applicable to the field of software and provides a metadata-based file storage method and device. The method comprises the steps that an operation system sends a storage request to a service system and transmits a file corresponding to the request; the service system acquires the file to be stored and generates metadata corresponding to the file and information of the metadata, wherein the information of the metadata comprises file header information, file body information and file extension property information; the service system acquires a storage strategy for the file according to user settings; the service system sends the storage strategy, the file and the information of the metadata to a storage system, and the storage system stores the file according to the storage strategy; the service system returns the file header information to the operation system. The method and device provided by the invention have the advantage that the file is managed and stored reasonably.
Description
Technical field
The invention belongs to software field, relate in particular to a kind of file memory method and device based on metadata.
Background technology
Along with the web2.0 epoch arrive, be internet or the data of enterprises in scale and quantitatively all had epoch of the variation, particularly mass data of matter how reasonably fast, reasonably to store these data and become particularly important.
Current document storage system has multiple storage policy, as being stored in DB after file serializing, being stored in local disk system.Current storage system can meet the demands substantially at storage file, but is reasonably managing and storage file, supports all Shortcomings of a lot of storage systems in aspect such as quick-searching, file system Quick Extended.So the technical scheme of prior art cannot reasonably be managed and storage file.
Summary of the invention
The object of the embodiment of the present invention is to provide a kind of file memory method based on metadata, and it solves the rational management of prior art existence and the problem of storage file.
The embodiment of the present invention is achieved in that one side, and a kind of file memory method based on metadata is provided, and described method comprises:
Operation system sends storage resource request to service system, transmits this and asks corresponding file;
Service system is obtained file to be stored, the metadata that generation this document is corresponding and the information of metadata, and this metadata information comprises file header information, file body information and file extent attribute;
Service system arranges the storage policy that obtains described file according to user;
Service system sends to storage system by this storage policy, file and metadata information, and storage system is carried out the storage of file according to this storage policy;
Service system is to operation system backspace file header.
Optionally, described file header information comprises: file name, size, file identification, module coding, uploader, reviser.
Optionally, described file body information comprises: the actual storage locations of file, the storage class of file.
Optionally, described extended attribute information note comprises: the information of certain document, extended attribute.
Optionally, after the described method storage policy in service system is obtained this document extended attribute, also comprise:
Service system is obtained after this storage policy, judges that whether disk space is sufficient, as inadequate in disk space, this storage policy is modified as and stores under new disk or under storage medium.
On the other hand, provide a kind of file storage device based on metadata, described device comprises: operation system, service system and storage system;
Operation system, for sending storage resource request to service system, transmits this and asks corresponding metadata;
Service system, for obtaining file to be stored, generate metadata that this document is corresponding and, this metadata information comprises file header information, file body information and file extent attribute; The storage policy that obtains described file is set according to user; This storage policy, metadata and metadata information are sent to storage system;
Storage system, for carrying out the storage of file according to this storage policy;
Service system, also for to operation system backspace file header.
Optionally, described file header information comprises: file name, size, file identification, module coding, uploader, reviser.
Optionally, described file body information comprises: the actual storage locations of file, the storage class of file.
Optionally, described file extent attribute comprises: the information of certain document, extended attribute.
In embodiments of the present invention, the document storage mode of technical scheme provided by the invention based on metadata, which is supported according to different business module storage files, for disparate modules, can formulate different storage policies, support extended attribute, realize the quick-searching of fileinfo, support multiple file storage medium advantage.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of a kind of file memory method based on metadata provided by the invention;
Fig. 2 is the structural drawing of the file storage device based on metadata provided by the invention.
Embodiment
In order to make object of the present invention, technical scheme and advantage clearer, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein, only in order to explain the present invention, is not intended to limit the present invention.
The specific embodiment of the invention provides a kind of file memory method based on metadata, and the method as shown in Figure 1, comprising:
101, operation system sends storage resource request to service system, transmits this and asks corresponding file;
102, service system is obtained file to be stored, the metadata that generation this document is corresponding and the information of metadata, and this metadata information comprises file header information, file body information and file extent attribute;
103, service system arranges the storage policy that obtains described file according to user;
104, service system sends to storage system by this storage policy, metadata and metadata information, and storage system is carried out the storage of file according to this storage policy;
105, service system is to operation system backspace file header.
Adopt said method user to customize and be applicable to own storage policy, can facilitate like this user to select to be applicable to the storage policy of oneself to adapt to different users for the different demand of storing.
In operation system, the user of file system only need to call and upload, and downloads the interface method that waits operation.Service system can send request to background service system according to different host-host protocols (as http, ftp etc.).During upload operation, user only need provide the file that need to upload.Service system can be returned to according to result the metadata information of a file header, and this metadata information comprises more documentary essential informations, as the title of file, size, file identification, module coding, uploader, reviser.And caller is by being used this metadata information, can retrieve file, download, the operation such as copy, and do not need to pay close attention to the physical location of file.In addition, by using metadata, can realize file storing process, store with represent separated.Metadata information is for file, and light weight very, is kept in database, can realize fast the operations such as retrieval and inquiry.Can realize easily the backup of metadata information.User, when using fileinfo, does not need any storing process of relation and strategy completely, and the header by metadata can download to corresponding file easily.Different operation systems also can, according to the business demand of oneself, retrieve file by file extent attribute fast.
Optionally, file header information can comprise: file name, size, file identification, module coding, uploader, reviser.
Optionally, what file body information was preserved is the storage information of file, comprising: the actual storage locations of file, the storage class of file etc.
Optionally, file extent attribute be business information, comprising: the information of certain document, extended attribute; Its maximum feature is to support expansion, for different callers, can generate multi-form extended attribute information.Extended attribute information is mainly used in the retrieval of file, and the value by extended attribute can retrieve corresponding fileinfo fast.
Optionally, said method can also comprise after 103:
Service system is obtained after this storage policy, judges that whether disk space is sufficient, as inadequate in disk space, this storage policy is modified as and stores under new disk or under storage medium.
Above-mentioned storage policy is that service system is to change according to actual conditions, so its storage is more flexible, efficiency is higher, and in addition, file system is without knowing file body information, so it also can play certain privacy functions,
The present invention also provides a kind of file storage device based on metadata, and this installs as shown in Figure 2, comprising:
Operation system:
Operation system is the interface that this device is exposed to third party's caller (as operation system).Be the entrance and exit of file storage, operation system is supported the request of rest style, and operation system sends request (as http, ftp) to the service system on backstage, and receives the file metadata information of passing back from service system, is returned to third party's caller.
Service system:
Service system is the core of this device, its support moduleization storage, the metadata information of management document, the file storage operations such as user's control of authority and compression.Service system is separated operation and the operation system of file storage completely, operation system does not need to be related to the memory location of reality completely, the storage operation of service system asynchronous process file, when the storage for large file, can reduce the stand-by period so greatly.
Storage system:
Storage system is responsible for real storage operation, according to the storage policy of service system, file is left in local disk or HDFS etc.
In operation system, the user of file system only need to call and upload, and downloads the interface method that waits operation.Service system can send request to background service system according to different host-host protocols (as http, ftp etc.).During upload operation, user only need provide the file that need to upload.Service system can be returned to according to result the metadata information of a file header, and this metadata information comprises more documentary essential informations, as the title of file, size, file identification, module coding, uploader, reviser.And caller is by being used this metadata information, can retrieve file, download, the operation such as copy, and do not need to pay close attention to the physical location of file.
In service system, for each file, can produce the metadata information of three types, respectively: FileHeader---file header information, FileBody---file body information, FileExt-file extent attribute.What file header information was preserved is file presenting information, as file name, and size, file identification, module coding, uploader, reviser.What file body information was preserved is the storage information of file, as the actual storage locations of file, and the storage class of file etc.Extended attribute information recording be business information, as the information of certain document, its maximum feature is to support expansion, for different callers, can generate multi-form extended attribute information.Extended attribute information is mainly used in the retrieval of file, and the value by extended attribute can retrieve corresponding fileinfo fast.
When operation system sends storage resource request to service system, service system can first be formulated storage policy according to the module of caller, and the storage policy of different modules can be different, can realize the differentiation storage of module, can improve like this efficiency of file storage.This device can be realized the extending transversely of module, when Insufficient disk space, certain module can be moved on to and be switched under new disk or under storage medium.
All metadata informations are kept in database above, and wherein the information of file header information and file body information is fixed, once definite, just can not change again.The metadata information of extended attribute is that each operation system determines, service system can generate table dynamically.Fabulous support the demand of different business systems.
The generation of file metadata information is very fast with respect to the process of the storage of large file, both in speed, be not a level other, asynchronous completing when actual storage operation and the generation of metadata information, operation system obtains just having completed the operation of once uploading after metadata information, and the storage operation on backstage is still continuing, user does not need to be concerned about this process.
Storage system is responsible for real storage operation, according to the storage policy of service system, file is left in local disk or HDFS etc.
This device is based on metadata, to carry out the storing process of management document, by the retrieval to file metadata information, can inquire fast actual file, and supports the inquiry of any business.Improved greatly effectiveness of retrieval.
The Core System device of this device, and most crucial in service system be metadata management.By using metadata, can realize file storing process, store with represent separated.Metadata information is for file, and light weight very, is kept in database, can realize fast the operations such as retrieval and inquiry.Can realize easily the backup of metadata information.User, when using fileinfo, does not need to be concerned about any storing process and strategy completely, and the header by metadata can download to corresponding file easily.Different operation systems also can, according to the business demand of oneself, retrieve file by file extent attribute fast.
One of ordinary skill in the art will appreciate that all or part of step realizing in the various embodiments described above method is to come the hardware that instruction is relevant to complete by program, corresponding program can be stored in a computer read/write memory medium, described storage medium, as ROM/RAM, disk or CD etc.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, all any modifications of doing within the spirit and principles in the present invention, be equal to and replace and improvement etc., within all should being included in protection scope of the present invention.
Claims (9)
1. the file memory method based on metadata, is characterized in that, described method comprises:
Operation system sends storage resource request to service system, transmits this and asks corresponding file;
Service system is obtained file to be stored, the metadata that generation this document is corresponding and the information of metadata, and this metadata information comprises file header information, file body information and file extent attribute;
Service system arranges the storage policy that obtains described file according to user;
Service system sends to storage system by this storage policy, file and metadata information, and storage system is carried out the storage of file according to this storage policy;
Service system is to operation system backspace file header.
2. method according to claim 1, is characterized in that, described file header information comprises: file name, size, file identification, module coding, uploader, reviser.
3. method according to claim 1, is characterized in that, described file body information comprises: the actual storage locations of file, the storage class of file.
4. method according to claim 1, is characterized in that, described extended attribute information note comprises: the information of certain document, extended attribute.
5. method according to claim 1, is characterized in that, after the storage policy of described method in service system is obtained this document extended attribute, also comprises:
Service system is obtained after this storage policy, judges that whether disk space is sufficient, as inadequate in disk space, this storage policy is modified as and stores under new disk or under storage medium.
6. the file storage device based on metadata, is characterized in that, described device comprises: operation system, service system and storage system;
Operation system, for sending storage resource request to service system, transmits this and asks corresponding file;
Service system, for obtaining file to be stored, generate metadata that this document is corresponding and, this metadata information comprises file header information, file body information and file extent attribute; The storage policy that obtains described file is set according to user; This storage policy, metadata and metadata information are sent to storage system;
Storage system, for carrying out the storage of file according to this storage policy;
Service system, also for to operation system backspace file header.
7. device according to claim 6, is characterized in that, described file header information comprises: file name, size, file identification, module coding, uploader, reviser.
8. device according to claim 6, is characterized in that, described file body information comprises: the actual storage locations of file, the storage class of file.
9. device according to claim 6, is characterized in that, described file extent attribute comprises: the information of certain document, extended attribute.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310654319.1A CN103605795A (en) | 2013-12-05 | 2013-12-05 | Metadata-based file storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310654319.1A CN103605795A (en) | 2013-12-05 | 2013-12-05 | Metadata-based file storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103605795A true CN103605795A (en) | 2014-02-26 |
Family
ID=50124017
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310654319.1A Pending CN103605795A (en) | 2013-12-05 | 2013-12-05 | Metadata-based file storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103605795A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104281503A (en) * | 2014-09-30 | 2015-01-14 | 华为数字技术（成都）有限公司 | Data backup method and related system |
| CN110399337A (en) * | 2019-07-24 | 2019-11-01 | 江苏物联网研究发展中心 | File automating method of servicing and system based on data-driven |
| CN110928484A (en) * | 2018-09-19 | 2020-03-27 | 上海仪电（集团）有限公司中央研究院 | Hybrid cloud storage method based on software defined storage |
-
2013
- 2013-12-05 CN CN201310654319.1A patent/CN103605795A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104281503A (en) * | 2014-09-30 | 2015-01-14 | 华为数字技术（成都）有限公司 | Data backup method and related system |
| CN110928484A (en) * | 2018-09-19 | 2020-03-27 | 上海仪电（集团）有限公司中央研究院 | Hybrid cloud storage method based on software defined storage |
| CN110928484B (en) * | 2018-09-19 | 2023-12-22 | 上海仪电（集团）有限公司中央研究院 | Hybrid cloud storage method based on software defined storage |
| CN110399337A (en) * | 2019-07-24 | 2019-11-01 | 江苏物联网研究发展中心 | File automating method of servicing and system based on data-driven |
| CN110399337B (en) * | 2019-07-24 | 2023-05-12 | 江苏物联网研究发展中心 | File automation service method and system based on data driving |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11734125B2 (en) | Tiered cloud storage for different availability and performance requirements | |
| US8555018B1 (en) | Techniques for storing data | |
| KR101994021B1 (en) | File manipulation method and apparatus | |
| US20130238557A1 (en) | Managing tenant-specific data sets in a multi-tenant environment | |
| CN107436725A (en) | A kind of data are write, read method, apparatus and distributed objects storage cluster | |
| CN102111438B (en) | Method and device for parameter adjustment and distributed computation platform system | |
| US9432484B1 (en) | CIM-based data storage management system having a restful front-end | |
| US20160364407A1 (en) | Method and Device for Responding to Request, and Distributed File System | |
| US9424314B2 (en) | Method and apparatus for joining read requests | |
| EP3624398A1 (en) | Storage capacity evaluation method and apparatus based on cdn application | |
| CN110046133A (en) | A kind of metadata management method, the apparatus and system of storage file system | |
| US20130325932A1 (en) | Electronic device and method for storing distributed documents | |
| CN104331428A (en) | Storage and access method of small files and large files | |
| US8732355B1 (en) | Dynamic data prefetching | |
| CN102148870A (en) | Cloud storage system and implementation method thereof | |
| CN103067479A (en) | Network disk synchronized method and system based on file coldness and hotness | |
| CN102821111A (en) | Real-time synchronizing method for file cloud storage | |
| CN110399348A (en) | File deletes method, apparatus, system and computer readable storage medium again | |
| CN109189772A (en) | File management method and system for no file system storage medium | |
| US11210282B2 (en) | Data placement optimization in a storage system according to usage and directive metadata embedded within the data | |
| CN104079600B (en) | File memory method, device, access client and meta data server system | |
| CN102523301A (en) | Method for caching data on client in cloud storage | |
| CN103605795A (en) | Metadata-based file storage method and device | |
| CN104915376B (en) | A kind of archival compression method of file in cloud storage | |
| CN103501341A (en) | Method and device for establishing Web service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100094 Beijing city Haidian District North Road No. 68, UFIDA Software Park Applicant after: Yonyou Network Technology Co., Ltd. Address before: 100094 Beijing city Haidian District North Road No. 68, UFIDA Software Park Applicant before: UFIDA Software Co., Ltd. |
|
| COR | Change of bibliographic data | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140226 |