CN109542861A - File management method, device and system - Google Patents
File management method, device and system Download PDFInfo
- Publication number
- CN109542861A CN109542861A CN201811325312.4A CN201811325312A CN109542861A CN 109542861 A CN109542861 A CN 109542861A CN 201811325312 A CN201811325312 A CN 201811325312A CN 109542861 A CN109542861 A CN 109542861A
- Authority
- CN
- China
- Prior art keywords
- file
- metadata
- file destination
- storage
- destination
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007726 management method Methods 0.000 title claims abstract description 78
- 238000000034 method Methods 0.000 claims abstract description 20
- 238000013500 data storage Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000008520 organization Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 241000282813 Aepyceros melampus Species 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a file management method, a device and a system, wherein the method comprises the following steps: receiving a target file to be stored uploaded by a user; distributing a corresponding file identifier for the target file, wherein the file identifier is used for identifying the identity of the target file; determining a target storage system corresponding to the data type of the target file from at least two different types of storage systems included in a file storage cluster, and storing the target file into the target storage system; and storing the metadata of the target file into a metadata storage cluster, wherein the metadata of the target file comprises the file identifier and a storage path of the target file. The device includes: the file storage system comprises a file receiving unit, an identification distribution unit, a file storage unit and a metadata storage unit. The scheme can manage the data more conveniently.
Description
Technical field
The present invention relates to field of computer technology, in particular to a kind of file management method, device and system.
Background technique
Large enterprise possesses a large amount of data, these data are according to the difference of its fields by with the more of corresponding responsibility
A department or employee are managed.In enterprise's normal operation, it is frequently necessary to exchange between different departments or employee
File in order to facilitate swap file between department, employee and guarantees the safeties of data, and enterprise usually passes through self-built Dropbox platform
To carry out data exchange.By in the data storage to Dropbox platform of enterprise, the access authority by configuring department, employee realizes number
It is shared according in particular range.
For the self-built Dropbox platform of current enterprise, Dropbox platform includes certain types of data-storage system, is uploaded
Various types of data to Dropbox platform are stored in the data-storage system.
Since different types of data-storage system has the characteristics that different, such as Hadoop distributed file system
(Hadoop Distributed File System, HDFS) is suitable for storing bigger file, but retardance is relatively high, with
Machine-readable write performance is poor, and Hbase distributed memory system is suitable for the random read-write of low latency, but data analysis performance is poor.
Therefore, the self-built Dropbox platform of existing enterprise stores various types of data into same type of data-storage system,
Biggish inconvenience is caused to the management of data.
Summary of the invention
The embodiment of the invention provides a kind of file management method, device and system, can more easily to data into
Row management.
In a first aspect, the embodiment of the invention provides a kind of file management methods, comprising:
Receive the file destination to be stored that user uploads；
Corresponding file identification is distributed for the file destination, wherein the file identification is used for the target text
The identity of part is identified；
The determining and file destination from least two different types of storage systems included by file storage cluster
The corresponding target storage system of data type, and by the file destination storage into the target storage system；
The metadata of the file destination is stored into metadata storage cluster, wherein first number of the file destination
According to the store path for including the file identification and the file destination.
Optionally, before the metadata by the file destination is stored into metadata storage cluster, further
Include:
Generate the user group's table for corresponding to the file destination, user message table, storing device information table and file letter
Cease table, wherein user group's table is used to record the attribute information organized belonging to the user, and the user message table is used for
The identity information of the user is recorded, the storing device information table is used to record the attribute information of the target storage system,
The file information table is used to record the attribute information of the file destination, and the attribute information of the file destination includes the text
The store path of part mark and the file destination；
Obtain the metadata of the file destination, wherein the metadata of the file destination includes the user group
Table, the user message table, the storing device information table and the file information table.
Optionally, before the metadata for obtaining the file destination, further comprise:
Generate the file sharing information table for corresponding to the file destination, wherein the file sharing information table is for remembering
Record the shared information of the file destination；
Obtain the metadata of the file destination, wherein the metadata of the file destination includes the user group
Table, the user message table, the storing device information table, the file information table and the file sharing information table.
Optionally, after the metadata by the file destination is stored into metadata storage cluster, further
Include:
Receive the read requests being read out to the file destination；
According to the file identification that the read requests carry, the target is obtained from the metadata storage cluster
The metadata of file；
The store path for the file destination that metadata according to the file destination includes stores from the file and collects
The file destination is read in group.
Second aspect, the embodiment of the invention also provides a kind of document management apparatus, comprising: file reception unit, mark
Allocation unit, file storage unit and metadata storage unit；
The file reception unit, for receiving the file destination to be stored of user's upload；
The mark allocation unit, the file destination distribution for receiving for the file reception unit are corresponding
File identification, wherein the file identification be used for the file destination carry out identity be identified；
The file storage unit, for at least two different types of storage systems included by the file storage cluster
Middle determination target storage system corresponding with the data type of the file destination received by the file reception unit,
And by file destination storage into the target storage system；
The metadata storage unit, for storing the metadata of the file destination into metadata storage cluster,
Wherein, the metadata of the file destination includes the files-designated that the mark allocation unit is file destination distribution
Know the store path stored with the file storage unit to the file destination.
Optionally, this document managing device further comprises: tabulation unit and metadata acquiring unit；
The tabulation unit, for generate correspond to user group's table of the file destination, user message table, storage are set
Standby information table and file information table, wherein user group's table is used to record the attribute information organized belonging to the user, institute
User message table is stated for recording the identity information of the user, the storing device information table is for recording the target storage
The attribute information of system, the file information table are used to record the attribute information of the file destination, the category of the file destination
Property information includes the store path of the file identification and the file destination；
The metadata acquiring unit, for obtaining the metadata of the file destination, wherein the member of the file destination
Data include user group's table, the user message table, the storing device information table being stated tabulation unit and being generated
With the file information table.
Optionally,
The tabulation unit is further used for generating the file sharing information table for corresponding to the file destination, wherein institute
File sharing information table is stated for recording the shared information of the file destination；
The metadata acquiring unit is further used for obtaining the metadata of the file destination, wherein the target text
The metadata of part includes user group's table, the user message table, the storage equipment that the tabulation unit generates
Information table, the file information table and the file sharing information table.
Optionally, this document managing device further comprises: request reception unit, meta-data read unit and file are read
Unit；
The request reception unit, for receiving the read requests being read out to the file destination；
The meta-data read unit, what the read requests for being received according to the request reception unit carried
The file identification obtains the metadata of the file destination from the metadata storage cluster；
The document reading unit, first number of the file destination for being read according to the meta-data read unit
According to the store path for the file destination for including, the file destination is read from the file storage cluster.
The third aspect, the embodiment of the invention also provides a kind of file management systems, comprising: metadata storage cluster, text
Any one document management apparatus that part storage cluster and second aspect provide；
The metadata storage cluster, for storing metadata accessed by the document management apparatus；
The file storage cluster includes at least two different types of storage systems, for filling to the file management
The file received is set to be stored.
Optionally,
The metadata storage cluster includes: odd number management node and at least two back end；
The management node, for that will store from the metadata of the document management apparatus at least two different institutes
It states on back end, and is obtained from least two back end according to the file identification from the document management apparatus
Take corresponding metadata；
The back end, for storing metadata.
File management method provided in an embodiment of the present invention, device and system need to carry out receiving user upload
After the file destination of storage, the file identification for being identified to its identity is distributed for file destination first, later from file
Target corresponding with the data type of file destination is determined in the storage system of multiple and different types included by storage cluster
Storage system, and by file destination storage into the target storage system determined, it later will include first number of file destination
According to storage into metadata storage cluster, and the metadata of file destination includes file identification and the storage road of file destination
Diameter.It can be seen that when user needs to share the file destination that it is uploaded, according to the data type of file destination from file
The target storage system for being suitable for storing file destination is determined in multiple storage systems that storage cluster includes, and then by file destination
It stores in target storage system, the file of corresponding data type is stored using the characteristics of different type storage system, thus
More easily data can be managed.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is the present invention
Some embodiments for those of ordinary skill in the art without creative efforts, can also basis
These attached drawings obtain other attached drawings.
Fig. 1 is a kind of flow chart of file management method provided by one embodiment of the present invention；
Fig. 2 is a kind of flow chart of file reading provided by one embodiment of the present invention；
Fig. 3 is the flow chart of another file management method provided by one embodiment of the present invention；
Fig. 4 is the schematic diagram of equipment where a kind of document management apparatus provided by one embodiment of the present invention；
Fig. 5 is a kind of schematic diagram of document management apparatus provided by one embodiment of the present invention；
Fig. 6 is the schematic diagram of another document management apparatus provided by one embodiment of the present invention；
Fig. 7 is the schematic diagram of another document management apparatus provided by one embodiment of the present invention；
Fig. 8 is a kind of schematic diagram of file management system provided by one embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments, based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without making creative work, shall fall within the protection scope of the present invention.
As shown in Figure 1, this method may comprise steps of the embodiment of the invention provides a kind of file management method:
Step 101: receiving the file destination to be stored that user uploads；
Step 102: distributing corresponding file identification for file destination, wherein file identification is used for file destination
Identity is identified；
Step 103: the determining and target from least two different types of storage systems included by file storage cluster
The corresponding target storage system of the data type of file, and by file destination storage into target storage system；
Step 104: the metadata of file destination being stored into metadata storage cluster, wherein first number of file destination
According to the store path for including file identification and file destination.
The embodiment of the invention provides a kind of file management methods, in the mesh store for receiving user's upload
After marking file, the file identification for being identified to its identity is distributed for file destination first, later from file storage cluster
Target storage system corresponding with the data type of file destination is determined in the storage system of included multiple and different types,
It later will include the metadata storage of file destination to member and by file destination storage into the target storage system determined
In data store set group, and the metadata of file destination includes the file identification and store path of file destination.It can be seen that
When user needs to share the file destination that it is uploaded, include from file storage cluster according to the data type of file destination
Multiple storage systems in determine the target storage system for being suitable for storing file destination, and then file destination is stored and is deposited to target
In storage system, the file of corresponding data type is stored, using the characteristics of different type storage system so as to more convenient
Ground is managed data.
Optionally, on the basis of file management method shown in Fig. 1, step 104 arrives the metadata storage of file destination
In metadata storage cluster, before this firstly the need of the metadata for getting file destination, the metadata of file destination is specific
It can obtain in the following way:
User group's table, user message table, storing device information table and the file information table for corresponding to file destination are generated,
And then get include user group's table generated, user message table, storing device information table and file information table member
Data, wherein
User group's table is used to record the attribute information organized belonging to user, and user group's table specifically can be such as the following table 1 institute
Show, be used for the information such as record organization number, the organization number, title, the level of affiliated tissue is used for tree structure identification
Relationship does not consider administrator；
Table 1
| Field name | Type | Description |
| orgid | string | Major key, organization number |
| porgid | string | Affiliated organization number |
| Orgname | string | Title |
User message table is used to record the identity information of user, and user message table specifically can be as shown in table 2 below, for remembering
The information such as number, the file directory of user number, the user name of tissue belonging to Customs Assigned Number, user are recorded, other can extended
Customer management information, such as last login IP, login time, logout time etc.；
Table 2
| Field name | Type | Description |
| userid | String | Major key, Customs Assigned Number |
| Orgid | string | Organization number belonging to user |
| Folder | String | Unique key, the file directory number of active user, uses uuid |
| username | String | User name |
| used | bool | Whether use |
Storing device information table is used to record the attribute information of target storage system, and storing device information table specifically can be as
Shown in the following table 2, for record storage equipment unique number, device type, information of connection storage etc., multiple types can be used
Storage equipment, individually storage in order to update device metadata information；
Table 3
| Field name | Type | Description |
| fstoreid | int | Major key stores equipment unique number |
| stype | int | Device type, such as ftp, kudu, hdfs, file etc. |
| sinfo | string | The information of storage is connected, for example ftp is address and port etc. |
File information table is used to record the attribute information of file destination, and file information table specifically can be as shown in table 4 below, uses
In record parent information, file ID (file identification), file type, filename, file size, creation time, renewal time
Etc. information, file managed with tree, most higher level is the root of user, lower to have subdirectory and file, specific item
Can there are subdirectory and file again under record, Fid uniformly uses uuid, and for impracticable real name to avoid conflict, frefid is mesh
The store path for marking file, is combined with fstoreid to identify the more specific location information of file；
Table 4
The metadata of file destination includes user group's table, user message table, storing device information table and the file information
Table, the information organized belonging to record user, the identity information of user, the information of storage equipment for storing file destination and
The attribute information of file destination itself.In this way, when user needs to be read out file destination, according to first number of file destination
According to can quickly read file destination, and more easily file destination can be managed, so as to promote logarithm
According to the convenience and efficiency being managed.
Specifically, in the management of metadata, the basic logic organized to file is that file belongs to some text
Part folder, highest file belong to user, and user belongs to some tissue.File storage cluster includes that multiple storage systems (are deposited
Store up equipment), file is stored in some storage equipment, and has unique file identification.
For example, user A belongs to tissue B, user A uploads file destination X, according to the data type of file destination X by target
In file X storage to storage equipment C, then the corresponding user group's table of file destination X is used for the attribute information of record organization B, mesh
The corresponding user message table of mark file X is used to record the identity information of user A, the corresponding storing device information table of file destination X
For the attribute information of record storage equipment C, the corresponding file information table of file destination X is used to record the attribute of file destination X
Information.
Optionally, the metadata of file destination include user group's table, user message table, storing device information table and
On the basis of file information table, the metadata of file destination can also include the file sharing information table corresponding to file destination.
File sharing information table is used to record the shared information of file destination, and file sharing information table specifically can be as shown in table 5 below, uses
In information such as record reference number of a document, the number for sharing to user, the number of shared user, shared times；
Table 5
The metadata of file destination includes file sharing information table, and file sharing information table has recorded being total to for file destination
Information is enjoyed, when needing to be read out file destination in the presence of reading user, can include according to the metadata of file destination
File sharing information table, which determines, reads whether user there is permission to be read out to file destination, to realize to file destination
Access authority is managed, and further improves the convenience being managed to data.
Optionally, on the basis of file management method shown in Fig. 1, the metadata of file destination is stored in step 104
After into metadata storage cluster, other users can be read out file destination, as shown in Fig. 2, the process specifically may be used
With the following steps are included:
Step 201: receiving the read requests being read out to file destination；
Step 202: the file identification carried according to read requests obtains the member of file destination from metadata storage cluster
Data；
Step 203: the store path for the file destination that the metadata according to file destination includes, from file storage cluster
Read file destination.
Due to including the file identification and store path of file destination in the metadata of file destination, when user's needs pair
When file destination is read out, user sends the read requests for carrying file identification, after receiving read requests, according to reading
The file identification for taking request to carry match the metadata of acquisition file destination from metadata storage cluster, and then according to getting
The metadata store path that includes read file destination from file storage cluster.In this way, when user needs to file destination
When being read out, it is only necessary to which the read requests for the file identification that input carries file destination can realize the reading of file destination
It takes, further improves the efficiency being managed to data.
Below by taking shared file between two users as an example, file management method provided in an embodiment of the present invention is made into one
Step is described in detail, as shown in figure 3, this method may comprise steps of:
Step 301: receiving the file destination that the first user uploads.
In embodiments of the present invention, when the first user needs the file destination by its user to share to other users,
First user needs to upload file destination, so as to receive the file destination of the first user upload.
For example, receiving the file destination X that the first user uploads.
Step 302: distributing corresponding file identification for file destination.
In embodiments of the present invention, after receiving file destination, corresponding file identification is distributed for file destination,
In, file identification is a unique identification code for being identified to the identity of file destination.
For example, distributing corresponding file identification X for file destination X.
Step 303: the storage system for storing file destination is determined from file storage cluster.
In embodiments of the present invention, file storage cluster includes the storage system of multiple and different types, different types of
Storage system is suitable for storing the file of different types of data, for example including there is the storage systems such as ftp, HDFS, file.For mesh
After marking file distribution file identification, according to the data type of file destination from each storage system that file storage cluster includes
Determine the target file system for being suitable for storing file destination.
For example, the data volume of file destination X is larger, it is suitable for storage into distributed file system HDFS, therefore by file
The HDFS that storage cluster includes is determined as the target file system for storing file destination X.
Step 304: by file destination storage into target storage system.
In embodiments of the present invention, after the target storage system for determining to be suitable for storing file destination, by file destination
It stores in target storage system.
For example, by file destination X storage into HDFS.
Step 305: obtaining the metadata of file destination.
In embodiments of the present invention, the configuration information inputted according to the attribute information of file destination and the first user, it is raw
At user group's table, user message table, storing device information table, file information table and the file-sharing letter for corresponding to file destination
Table is ceased, wherein record has the file identification and store path of file destination in file information table, and then by user group generated
Knit the first number of table, user message table, storing device information table, file information table and file sharing information table as file destination
According to.
For example, obtaining the metadata X of file destination X.
Step 306: the metadata of file destination is stored into metadata storage cluster.
In embodiments of the present invention, it stores in the metadata for getting file destination into metadata storage cluster.Wherein,
Metadata storage cluster can realize that kudu storage system is that a kind of distributed column of open source is deposited by kudu storage system
Storage, data organize in the form of a table, and real-time update, write-in and the rapid data based on major key that can provide data are read and complete
The mode of table scan.
For example, by metadata X storage into metadata storage cluster.
Step 307: receiving the read requests of second user input being read out to file destination.
In embodiments of the present invention, when second user needs to be read out file destination, second user, which is sent, to be carried
Have a read requests of the file identification of file destination, thus receive can receive second user transmission carry file destination
The read requests of file identification.
For example, receiving the read requests X for carrying file identification X that second user is sent.
Step 308: the metadata of file destination is obtained according to the file identification that read requests carry.
In embodiments of the present invention, it after the read requests for receiving second user transmission, is parsed from read requests
File identification entrained by it, and then the file identification that will acquire and the metadata stored in metadata storage cluster progress
Match, obtains the metadata of file destination.
For example, file identification X is parsed from read requests X, by what is stored in file identification X and metadata storage cluster
Each metadata is matched, and the metadata X to match with file identification X is obtained.
Step 309: file destination is read from file storage cluster according to the metadata obtained all.
In embodiments of the present invention, it after getting metadata, according to the store path for including in metadata, is deposited from file
The second user file destination to be read is read in accumulation.
For example, reading file destination X from HDFS according to the store path recorded in metadata X.
Step 310: the file destination read is sent to second user.
It in embodiments of the present invention, will after the file destination needed for reading second user in file storage cluster
The file destination read issues second user.
For example, the file destination X read from HDFS is sent to second user.
As shown in Figure 4, Figure 5, the embodiment of the invention provides a kind of document management apparatus.Installation practice can be by soft
Part is realized, can also be realized by way of hardware or software and hardware combining.For hardware view, as shown in figure 4, being this hair
A kind of hardware structure diagram of equipment where the document management apparatus that bright embodiment provides, in addition to processor shown in Fig. 4, memory,
Except network interface and nonvolatile memory, the equipment in embodiment where device usually can also include other hardware,
Such as it is responsible for the forwarding chip of processing message.Taking software implementation as an example, as shown in figure 5, as the dress on a logical meaning
It sets, is that computer program instructions corresponding in nonvolatile memory are read into memory by fortune by the CPU of equipment where it
What row was formed.Document management apparatus provided in this embodiment, comprising: file reception unit 501, mark allocation unit 502, file
Storage unit 503 and metadata storage unit 504；
File reception unit 501, for receiving the file destination to be stored of user's upload；
Allocation unit 502 is identified, the file destination for receiving for file reception unit 501 distributes corresponding file
Mark, wherein file identification is used to carry out identity to file destination to be identified；
File storage unit 503, for at least two different types of storage systems included by the file storage cluster
Middle determination target storage system corresponding with the data type of file destination received by file reception unit 501, and will
File destination is stored into target storage system；
Metadata storage unit 504, for storing the metadata of file destination into metadata storage cluster, wherein
The metadata of file destination includes to identify the file identification and file storage unit that allocation unit 502 is file destination distribution
The store path that 503 pairs of file destinations are stored.
Optionally, on the basis of document management apparatus shown in Fig. 5, as shown in fig. 6, document management apparatus can be further
It include: tabulation unit 505 and metadata acquiring unit 506；
Tabulation unit 505, for generating the user group's table for corresponding to file destination, user message table, storage equipment letter
Cease table and file information table, wherein user group's table is used to record the attribute information organized belonging to user, and user message table is used for
The identity information of user is recorded, storing device information table is used to record the attribute information of target storage system, and file information table is used
In the attribute information of record file destination, the attribute information of file destination includes the store path of file identification and file destination；
Metadata acquiring unit 506, for obtaining the metadata of file destination, wherein the metadata of file destination includes
User group's table, user message table, storing device information table and the file information table for thering is tabulation unit to generate.
Optionally, on the basis of document management apparatus shown in Fig. 6,
Tabulation unit 505 is further used for generating the file sharing information table for corresponding to file destination, wherein file is total
Information table is enjoyed for recording the shared information of file destination；
Metadata acquiring unit 506 is further used for obtaining the metadata of file destination, wherein first number of file destination
It is total to according to the user group's table, user message table, storing device information table, file information table and the file that include tabulation unit generation
Enjoy information table.
Optionally, on the basis of the document management apparatus shown in Fig. 5 or Fig. 6, this document managing device can be further
It include: request reception unit 507, meta-data read unit 508 and document reading unit 509；
Request reception unit 507, for receiving the read requests being read out to file destination；
Meta-data read unit 508, the files-designated that the read requests for being received according to request reception unit 507 carry
Know, the metadata of file destination is obtained from metadata storage cluster；
The metadata of document reading unit 509, the file destination for being read according to meta-data read unit 508 includes
File destination store path, read file destination from file storage cluster.
It should be noted that the contents such as information exchange, implementation procedure between each unit in above-mentioned apparatus, due to this
Inventive method embodiment is based on same design, and for details, please refer to the description in the embodiment of the method for the present invention, no longer superfluous herein
It states.
As shown in figure 8, one embodiment of the invention provides a kind of file management system, comprising: metadata storage cluster
801, the document management apparatus 803 that file storage cluster 802 and any of the above-described embodiment provide；
Metadata storage cluster 801, for metadata accessed by storage file managing device 803；
File storage cluster 802 includes at least two different types of storage systems, for document management apparatus 803
The file received is stored.
In file management system provided in an embodiment of the present invention, document management apparatus 803 is that user and metadata store
Middleware between cluster 801 and file storage cluster 802, when user needs storage file, document management apparatus 803 will be used
The gone up transmitting file storage in family is stored to file storage cluster 802, and by the storage of the metadata of the gone up transmitting file of user to metadata
In cluster 801, when user needs to read file, read requests that document management apparatus 803 is inputted according to user are from metadata
Corresponding metadata is obtained in storage cluster 801, and then is read from file storage cluster 802 according to the metadata obtained all
File needed for user.
Optionally, on the basis of file management system shown in Fig. 8, metadata storage cluster 801 may include odd number
Management node and at least two back end, wherein
Management node, for that will store from the metadata of document management apparatus at least two different back end
On, and corresponding metadata obtained from least two back end according to the file identification from document management apparatus；
Back end, for storing metadata.
In metadata storage cluster, management node is responsible for the metadata information of Maintenance Table, and management node is one or more
Odd number node, 3 or 5 nodes can be used, generally to re-elect out new master after a wherein node failure
Management node.User is accessed by the address of service of management node, and the mode of multiple management nodes can guarantee the height of service
Availability and reliability.
In metadata storage cluster, back end is responsible for the storage of specific data, guarantees number by the way of more copies
According to that will not lose, data distribution formula provides the high-performance of reading and writing data from structure.
Optionally, metadata storage cluster can be realized by kudu storage system, and kudu is a kind of distribution of open source
Column storage, data carry out tissue in the form of a table, can provide real-time update, write-in and the rapid data based on major key of data
The mode of reading and full table scan, can provide storage capacity more higher than traditional Relational DataBase and read or write speed.kudu
It is divided into master node (management node) and tserver node (back end) from system architecture；Master node is responsible for dimension
The metadata information of table is protected, is one or more odd number node, generally uses 3 or 5 nodes, user is saved by master
The address of service of point accesses, and the mode of more master nodes ensure that the high availability and reliability of service；Tserver node
It is responsible for the storage of specific data, guarantees that data will not lose by the way of more copies, the distribution of data is provided from structure
The high-performance of reading and writing data.
A set of object storage is developed based on CMSP, is stored in by API and returns to a mark this document after a file
Uuid character string can be used to obtain file using this character string.It, can be by file in the realization of network disk file management system
Real name, into database, the storage with file separates the information preservations such as this uuid.
Using kudu cluster as metadata storage cluster, the process of deployment file management system be may comprise steps of:
(a) need to dispose a kudu cluster, kudu is deployed in Linux server, than if any 5 hosts, Ke Yixuan
Wherein 3 conduct master nodes are selected, 5 are used as tserver node.The access of kudu passes through kudu master node.?
Build table in kudu, kudu provides data storage, externally provides api interface, can dispose an Impala cluster again and come pair
Kudu is met, provides the SQL interface of data query analysis for kudu, user can grasp by JDBC/ODBC interface using SQL
Make the data in kudu and builds table, modification table etc..Impala major deployments are on the tserver node server of kudu.Pass through
The SQL that impala builds table userorginfo (user group's table) in kudu is as follows:
create table userorginfo(orgid string,porgid string,orgname string,
primary key(orgid))partition by hash(orgid)partitions 5stored as kudu；
(b) deployment storage equipment (file storage cluster), can according to need and build hdfs cluster, ftp, network file is deposited
The servers such as storage；
(c) network disk file management system WEB application is built, application and development can choose oneself known development language and open
Frame is sent out, pays attention to storing the calling interface that equipment provides.For example selection uses Java and Spring Development of Framework, impala is provided
JDBC interface, HDFS also provide Java api interface；
(d) function that file management system is realized includes storage device management (the management necessary letter of storage device access
Breath), user group's management (forms the institutional framework of user), and user (is suspended to tissue) by user management, and the additions and deletions of file change
It looks into and shares and the functions such as share with checking.
The embodiment of the invention also provides a kind of readable mediums, including execute instruction, when the processor of storage control is held
When executing instruction described in row, the storage control executes the file management method that above-mentioned each embodiment provides.
The embodiment of the invention also provides a kind of storage controls, comprising: processor, memory and bus；
The memory is executed instruction for storing, and the processor is connect with the memory by the bus, when
When the storage control is run, the processor executes the described of memory storage and executes instruction, so that the storage
Controller executes the file management method that above-mentioned each embodiment provides.
In conclusion the file management method of each embodiment offer of the present invention, device and system, at least have has as follows
Beneficial effect:
1, in embodiments of the present invention, receive user upload the file destination store after, first for
File destination distributes file identification for being identified to its identity, multiple and different included by the file storage cluster later
Target storage system corresponding with the data type of file destination is determined in the storage system of type, and file destination is stored
Into the target storage system determined, later by include file destination metadata store into metadata storage cluster,
And the metadata of file destination includes the file identification and store path of file destination.It can be seen that user needs to thereon
When the file destination of biography is shared, multiple storage systems for including from file storage cluster according to the data type of file destination
Middle determination is suitable for storing the target storage system of file destination, and then file destination storage is utilized into target storage system
The characteristics of different type storage system, stores the file of corresponding data type, so as to more easily carry out pipe to data
Reason.
2, in embodiments of the present invention, the metadata of file destination includes user group's table, and user message table, storage are set
Standby information table and file information table record the identity information of the information, user organized belonging to user, for storing file destination
Store the information of equipment and the attribute information of file destination itself.In this way, when user needs to be read out file destination,
File destination can be quickly read according to the metadata of file destination, and pipe more easily can be carried out to file destination
Reason, so as to promote the convenience and efficiency that are managed to data.
3, in embodiments of the present invention, the metadata of file destination includes file sharing information table, file sharing information
Table has recorded the shared information of file destination, can be according to mesh when needing to be read out file destination in the presence of reading user
The file sharing information table that the metadata of mark file includes, which determines, reads whether user there is permission to be read out to file destination,
The access authority of file destination is managed to realize, further improves the convenience being managed to data.
4, in embodiments of the present invention, due to including the file identification of file destination in the metadata of file destination and depositing
Path is stored up, when user needs to be read out file destination, user sends the read requests for carrying file identification, is receiving
To after read requests, the member for obtaining file destination is matched from metadata storage cluster according to the file identification that read requests carry
Data, and then the store path for including according to the metadata got reads file destination from file storage cluster.In this way, working as
When user needs to be read out file destination, it is only necessary to which input carries the read requests of the file identification of file destination
To realize the reading of file destination, the efficiency being managed to data is further improved.
It should be noted that, in this document, such as first and second etc relational terms are used merely to an entity
Or operation is distinguished with another entity or operation, is existed without necessarily requiring or implying between these entities or operation
Any actual relationship or order.Moreover, the terms "include", "comprise" or its any other variant be intended to it is non-
It is exclusive to include, so that the process, method, article or equipment for including a series of elements not only includes those elements,
It but also including other elements that are not explicitly listed, or further include solid by this process, method, article or equipment
Some elements.In the absence of more restrictions, the element limited by sentence " including one ", is not arranged
Except there is also other identical factors in the process, method, article or apparatus that includes the element.
Those of ordinary skill in the art will appreciate that: realize that all or part of the steps of above method embodiment can pass through
The relevant hardware of program instruction is completed, and program above-mentioned can store in computer-readable storage medium, the program
When being executed, step including the steps of the foregoing method embodiments is executed；And storage medium above-mentioned includes: ROM, RAM, magnetic disk or light
In the various media that can store program code such as disk.
Finally, it should be noted that the foregoing is merely presently preferred embodiments of the present invention, it is merely to illustrate skill of the invention
Art scheme, is not intended to limit the scope of the present invention.Any modification for being made all within the spirits and principles of the present invention,
Equivalent replacement, improvement etc., are included within the scope of protection of the present invention.
Claims (10)
1. a kind of file management method characterized by comprising
Receive the file destination to be stored that user uploads；
Corresponding file identification is distributed for the file destination, wherein the file identification is used for the file destination
Identity is identified；
The determining number with the file destination from least two different types of storage systems included by file storage cluster
According to the corresponding target storage system of type, and by file destination storage into the target storage system；
The metadata of the file destination is stored into metadata storage cluster, wherein the metadata packet of the file destination
Include the store path of the file identification and the file destination.
2. the method according to claim 1, wherein in the metadata storage by the file destination to member
Before in data store set group, further comprise:
User group's table, user message table, storing device information table and the file information table for corresponding to the file destination are generated,
Wherein, user group's table is used to record the attribute information organized belonging to the user, and the user message table is for recording
The identity information of the user, the storing device information table is used to record the attribute information of the target storage system, described
File information table is used to record the attribute information of the file destination, and the attribute information of the file destination includes the files-designated
Know the store path with the file destination；
Obtain the metadata of the file destination, wherein the metadata of the file destination includes user group's table, institute
State user message table, the storing device information table and the file information table.
3. according to the method described in claim 2, it is characterized in that, before the metadata for obtaining the file destination,
Further comprise:
Generate the file sharing information table for corresponding to the file destination, wherein the file sharing information table is for recording institute
State the shared information of file destination；
Obtain the metadata of the file destination, wherein the metadata of the file destination includes user group's table, institute
State user message table, the storing device information table, the file information table and the file sharing information table.
4. method according to any one of claims 1 to 3, which is characterized in that in first number by the file destination
After storing into metadata storage cluster, further comprise:
Receive the read requests being read out to the file destination；
According to the file identification that the read requests carry, the file destination is obtained from the metadata storage cluster
Metadata；
The store path for the file destination that metadata according to the file destination includes, from the file storage cluster
Read the file destination.
5. a kind of document management apparatus characterized by comprising file reception unit, mark allocation unit, file storage unit
And metadata storage unit；
The file reception unit, for receiving the file destination to be stored of user's upload；
The mark allocation unit, the file destination for receiving for the file reception unit distribute corresponding text
Part mark, wherein the file identification is used to carry out identity to the file destination to be identified；
The file storage unit, for true from least two different types of storage systems included by file storage cluster
Fixed target storage system corresponding with the data type of the file destination received by the file reception unit, and will
The file destination storage is into the target storage system；
The metadata storage unit, for storing the metadata of the file destination into metadata storage cluster, wherein
The metadata of the file destination include it is described mark allocation unit be the file destination distribution the file identification and
The store path that the file storage unit stores the file destination.
6. device according to claim 5, which is characterized in that further comprise: tabulation unit and metadata acquiring unit；
The tabulation unit, for generating the user group's table for corresponding to the file destination, user message table, storage equipment letter
Cease table and file information table, wherein user group's table is used to record the attribute information organized belonging to the user, the use
Family information table is used to record the identity information of the user, and the storing device information table is for recording the target storage system
Attribute information, the file information table is used to record the attribute information of the file destination, the attribute letter of the file destination
Breath includes the store path of the file identification and the file destination；
The metadata acquiring unit, for obtaining the metadata of the file destination, wherein the metadata of the file destination
It include user group's table, the user message table, the storing device information table and the institute that the tabulation unit generates
State file information table.
7. device according to claim 6, which is characterized in that
The tabulation unit is further used for generating the file sharing information table for corresponding to the file destination, wherein the text
Part shared information table is used to record the shared information of the file destination；
The metadata acquiring unit is further used for obtaining the metadata of the file destination, wherein the file destination
Metadata includes user group's table, the user message table, the storing device information that the tabulation unit generates
Table, the file information table and the file sharing information table.
8. according to the device any in claim 5 to 7, which is characterized in that further comprise: request reception unit, member
Data-reading unit and document reading unit；
The request reception unit, for receiving the read requests being read out to the file destination；
The meta-data read unit, described in read requests for being received according to the request reception unit carry
File identification obtains the metadata of the file destination from the metadata storage cluster；
The document reading unit, the metadata packet of the file destination for being read according to the meta-data read unit
The store path of the file destination included reads the file destination from the file storage cluster.
9. a kind of file management system characterized by comprising metadata storage cluster, file storage cluster and claim 5
To the document management apparatus any in 8；
The metadata storage cluster, for storing metadata accessed by the document management apparatus；
The file storage cluster includes at least two different types of storage systems, for connecing to the document management apparatus
The file received is stored.
10. system according to claim 9, which is characterized in that
The metadata storage cluster includes: odd number management node and at least two back end；
The management node, for that will store from the metadata of the document management apparatus at least two different numbers
Phase is obtained from least two back end according on node, and according to the file identification from the document management apparatus
Corresponding metadata；
The back end, for storing metadata.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811325312.4A CN109542861B (en) | 2018-11-08 | 2018-11-08 | File management method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811325312.4A CN109542861B (en) | 2018-11-08 | 2018-11-08 | File management method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109542861A true CN109542861A (en) | 2019-03-29 |
| CN109542861B CN109542861B (en) | 2023-06-09 |
Family
ID=65844669
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811325312.4A Active CN109542861B (en) | 2018-11-08 | 2018-11-08 | File management method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109542861B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110515896A (en) * | 2019-08-29 | 2019-11-29 | 网易（杭州）网络有限公司 | Model resource management method, model file production method, device and system |
| CN110928497A (en) * | 2019-11-15 | 2020-03-27 | 浪潮电子信息产业股份有限公司 | Metadata processing method, device and equipment and readable storage medium |
| CN111782886A (en) * | 2020-06-28 | 2020-10-16 | 杭州海康威视数字技术股份有限公司 | Method and device for managing metadata |
| CN113590543A (en) * | 2020-04-30 | 2021-11-02 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for information processing |
| CN114416728A (en) * | 2021-12-27 | 2022-04-29 | 炫彩互动网络科技有限公司 | Server archiving and file reading method |
| CN114840488A (en) * | 2022-07-04 | 2022-08-02 | 柏科数据技术（深圳）股份有限公司 | Distributed storage method, system and storage medium based on super-fusion structure |
| CN115623081A (en) * | 2021-07-16 | 2023-01-17 | 广州视源电子科技股份有限公司 | Data downloading method, data uploading method and distributed storage system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7844582B1 (en) * | 2004-10-28 | 2010-11-30 | Stored IQ | System and method for involving users in object management |
| CN104537076A (en) * | 2014-12-31 | 2015-04-22 | 北京奇艺世纪科技有限公司 | File reading and writing method and device |
-
2018
- 2018-11-08 CN CN201811325312.4A patent/CN109542861B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7844582B1 (en) * | 2004-10-28 | 2010-11-30 | Stored IQ | System and method for involving users in object management |
| CN104537076A (en) * | 2014-12-31 | 2015-04-22 | 北京奇艺世纪科技有限公司 | File reading and writing method and device |
Non-Patent Citations (1)
| Title |
|---|
| 林媛: "云存储中小文件元数据管理研究与优化", 《电脑知识与技术》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110515896A (en) * | 2019-08-29 | 2019-11-29 | 网易（杭州）网络有限公司 | Model resource management method, model file production method, device and system |
| CN110515896B (en) * | 2019-08-29 | 2021-10-26 | 网易（杭州）网络有限公司 | Model resource management method, model file manufacturing method, device and system |
| CN110928497A (en) * | 2019-11-15 | 2020-03-27 | 浪潮电子信息产业股份有限公司 | Metadata processing method, device and equipment and readable storage medium |
| CN110928497B (en) * | 2019-11-15 | 2021-06-15 | 浪潮电子信息产业股份有限公司 | Metadata processing method, device and equipment and readable storage medium |
| CN113590543A (en) * | 2020-04-30 | 2021-11-02 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for information processing |
| CN111782886A (en) * | 2020-06-28 | 2020-10-16 | 杭州海康威视数字技术股份有限公司 | Method and device for managing metadata |
| CN115623081A (en) * | 2021-07-16 | 2023-01-17 | 广州视源电子科技股份有限公司 | Data downloading method, data uploading method and distributed storage system |
| CN114416728A (en) * | 2021-12-27 | 2022-04-29 | 炫彩互动网络科技有限公司 | Server archiving and file reading method |
| CN114840488A (en) * | 2022-07-04 | 2022-08-02 | 柏科数据技术（深圳）股份有限公司 | Distributed storage method, system and storage medium based on super-fusion structure |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109542861B (en) | 2023-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11816126B2 (en) | Large scale unstructured database systems | |
| CN109542861A (en) | File management method, device and system | |
| CN103812939B (en) | Big data storage system | |
| CN104067216B (en) | System and method for implementing expansible data storage service | |
| US8676951B2 (en) | Traffic reduction method for distributed key-value store | |
| CN105324770B (en) | Effectively read copy | |
| CN106255967B (en) | NameSpace management in distributed memory system | |
| CN104657459B (en) | A kind of mass data storage means based on file granularity | |
| US9244958B1 (en) | Detecting and reconciling system resource metadata anomolies in a distributed storage system | |
| US20130110873A1 (en) | Method and system for data storage and management | |
| CN104618482B (en) | Access method, server, conventional memory device, the system of cloud data | |
| CN102708165B (en) | Document handling method in distributed file system and device | |
| CN108763436A (en) | A kind of distributed data-storage system based on ElasticSearch and HBase | |
| CN104462185B (en) | A kind of digital library's cloud storage system based on mixed structure | |
| CN103605698A (en) | Cloud database system used for distributed heterogeneous data resource integration | |
| US11803572B2 (en) | Schema-based spatial partitioning in a time-series database | |
| US10812543B1 (en) | Managed distribution of data stream contents | |
| CN102664914A (en) | IS/DFS-Image distributed file storage query system | |
| US11263270B1 (en) | Heat balancing in a distributed time-series database | |
| CN107291876A (en) | A kind of DDM method | |
| CN108268614A (en) | A kind of distribution management method of forest reserves spatial data | |
| CN109597903A (en) | Image file processing apparatus and method, document storage system and storage medium | |
| US9898614B1 (en) | Implicit prioritization to rate-limit secondary index creation for an online table | |
| US9231957B2 (en) | Monitoring and controlling a storage environment and devices thereof | |
| US11366598B1 (en) | Dynamic lease assignments in a time-series database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |