CN104462563A - File storage method and system - Google Patents
File storage method and system Download PDFInfo
- Publication number
- CN104462563A CN104462563A CN201410833341.7A CN201410833341A CN104462563A CN 104462563 A CN104462563 A CN 104462563A CN 201410833341 A CN201410833341 A CN 201410833341A CN 104462563 A CN104462563 A CN 104462563A
- Authority
- CN
- China
- Prior art keywords
- small documents
- write
- large files
- index
- index information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a file storage method which is applied to a storage device. The file storage method includes the steps that large files for merging and storing small files written by a front-end device are created in local; when the small files written by the front-end device are received, index information of the small files is cached in local; the large files are appointed for the small files, the small files and corresponding description information are written into the appointed large files according to the written-in sequence, and the cached indexing information of the small files is written into small file indexing areas in batch according to the written-in sequence when new data cannot be written into the large files any more or writing is timed out, wherein the indexing information comprises access paths of the appointed large files and starting offset of the small files in the large files. By means of the file storage method, the written-in performance of the small files can be improved.
Description
Technical field
The present invention relates to field of video monitoring, particularly relate to a kind of file memory method and system.
Background technology
Along with the fast development of bayonet socket monitoring business, a great problem be stored as in order to industry of the small documents such as single picture.Current industry is to the storage of small documents, and existing solution is mainly realized by file system or cloud storage scheme.
But, when small documents being stored by file system, after long-time full covering, easily there is file fragmentation, and the degraded performance stored; And when small documents being stored by cloud storage scheme, such as, adopt hadoop scheme, use cluster device or multiple copy to store and small documents is stored, still can there is the low problem of memory property, and very waste storage space.
Summary of the invention
In view of this, the present invention proposes a kind of small documents storage means, is applied on memory device, and described method comprises:
Create the small documents being used for being write by described front-end equipment to carry out merging the large files stored in this locality;
When receiving the small documents of front-end equipment write, the index information of described small documents is carried out buffer memory in this locality;
For described small documents specifies large files, according to write sequence, the descriptor of described small documents and correspondence is write the described large files of specifying, and when this large files cannot write new data or write time-out again, the index information of the described small documents of buffer memory is written to small documents index area according to write sequence batch;
Wherein, described index information comprises access path and the beginning side-play amount of described small documents in this large files of described large files of specifying.
Optionally, described small documents index area comprises catalogue subregion;
Described is that described small documents specifies large files to comprise:
For described small documents specifies large files;
The directory information of described small documents and the directory information of described large files of specifying are write described catalogue subregion, and in catalogue subregion, preserves the mapping relations of the directory information of described small documents and the directory information of described large files; The directory information of wherein said large files comprises the access path of the catalogue of described large files.
Optionally, described according to write sequence the descriptor of described small documents and correspondence write the described large files of specifying before, comprising:
The index information of first small documents write by described front-end equipment and the mark of described large files of specifying write described small documents index area, can carry out abnormal restoring so that follow-up according to the mark of the index information of described first small documents and described large files of specifying to the data in small documents index area.
Optionally, described small documents index area also comprises index subregion; Wherein said index subregion comprises the file grouping block of at least one default size;
Describedly the index information of the described small documents of buffer memory be written to described small documents index area according to write sequence batch comprise:
For described small documents distributes a file grouping block;
The index information of described small documents is write described file grouping block according to write sequence batch, and preserves the mapping relations of described small documents and described large files of specifying;
When described file grouping block cannot write new data again, from the end position of described file grouping block, a new file grouping block is again marked off in described index subregion, described new file grouping block is distributed to described small documents, and continue the fileinfo of described small documents batch to write described new file grouping block, until the index information of described small documents has all write.
Optionally, described method also comprises:
The index information of the small documents of buffer memory is being written in the process of small documents index area according to write sequence batch, if equipment generation power-off or restart, after equipment is restarted, according to the mark of the described large files stored in described small documents index area and the index information of described first small documents, find corresponding large files, and travel through this large files, read in this large files the described descriptor stored, after recovering according to the index information of described descriptor to described small documents, again described small documents index area is write according to write sequence batch.
Optionally, described method also comprises:
After the index information of the small documents of described buffer memory is successfully written to described small documents index area, discharge the described little civilian index information of buffer memory.
The present invention also proposes a kind of small documents storage system, is applied on memory device, and described system comprises:
Creation module, carries out merging the large files stored for creating the small documents being used for being write by described front-end equipment in this locality;
Cache module, for when receiving the small documents of front-end equipment write, carries out buffer memory by the index information of described small documents in this locality;
Writing module, for specifying large files for described small documents, according to write sequence, the descriptor of described small documents and correspondence is write the described large files of specifying, and when this large files cannot write new data or write time-out again, the index information of the described small documents of buffer memory is written to small documents index area according to write sequence batch;
Wherein, described index information comprises access path and the beginning side-play amount of described small documents in this large files of described large files of specifying.
Optionally, described small documents index area comprises catalogue subregion;
Said write module specifically for:
For described small documents specifies large files;
The directory information of described small documents and the directory information of described large files of specifying are write described catalogue subregion, and in catalogue subregion, preserves the mapping relations of the directory information of described small documents and the directory information of described large files; The directory information of wherein said large files comprises the access path of the catalogue of described large files.
Optionally, said write module is further used for:
Before according to write sequence the descriptor of described small documents and correspondence being write the described large files of specifying; the index information of first small documents write by described front-end equipment and the mark of described large files of specifying write described small documents index area, can carry out abnormal restoring so that follow-up according to the mark of the index information of described first small documents and described large files of specifying to the data in small documents index area.
Optionally, described small documents index area also comprises index subregion; Wherein said index subregion comprises the file grouping block of at least one default size;
Said write module is further used for:
For described small documents distributes a file grouping block;
The index information of described small documents is write described file grouping block according to write sequence batch, and preserves the mapping relations of described small documents and described large files of specifying;
When described file grouping block cannot write new data again, from the end position of described file grouping block, a new file grouping block is again marked off in described index subregion, described new file grouping block is distributed to described small documents, and continue the fileinfo of described small documents batch to write described new file grouping block, until the index information of described small documents has all write.
Optionally, said write module is further used for:
The index information of the small documents of buffer memory is being written in the process of small documents index area according to write sequence batch, if equipment generation power-off or restart, after equipment is restarted, according to the mark of the described large files stored in described small documents index area and the index information of described first small documents, find corresponding large files, and travel through this large files, read in this large files the described descriptor stored, after recovering according to the index information of described descriptor to described small documents, again described small documents index area is write according to write sequence batch.
Optionally, described system also comprises:
Release module, after being successfully written to described small documents index area, discharges the index information of the described small documents of buffer memory for the index information of the small documents at described buffer memory.
The present invention program is used for the small documents that front-end equipment writes to carry out merging the large files stored by creating in this locality, when receiving the small documents of front-end equipment write, the index information of the small documents received is carried out buffer memory in this locality, then according to write sequence, the descriptor of described small documents and correspondence is write described large files, and when described large files cannot write new data or write time-out again, the index information of the small documents of buffer memory is written to small documents index area according to write sequence batch.Because described index information comprises access path and the beginning side-play amount of described small documents in this large files of described large files of specifying, user the beginning side-play amount in this large files directly can access small documents according to the access path of described large files and small documents on large files, therefore can realize that small documents is become large files to store, avoid and directly store small documents and produce file fragmentation, and saved storage space; Simultaneously owing to have employed the scheme of batch write index information, the write performance of small documents significantly can also be promoted.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of a kind of file memory method proposed in an exemplifying embodiment;
Fig. 2 is a kind of small documents storage scheme schematic diagram proposed in an exemplifying embodiment;
Fig. 3 is the storage node composition of a kind of small documents index area proposed in an exemplifying embodiment;
Fig. 4 is the storage node composition of a kind of large files proposed in an exemplifying embodiment;
Fig. 5 is the building-block of logic of a kind of document storage system proposed in an exemplifying embodiment.
Embodiment
The present invention is intended to that the small documents that front-end equipment writes is mapped to large files and stores, and has the memory property of large files.
The present invention program is used for the small documents that front-end equipment writes to carry out merging the large files stored by creating in this locality, when receiving the small documents of front-end equipment write, the index information of the small documents received is carried out buffer memory in this locality, then according to write sequence, the descriptor of described small documents and correspondence is write described large files, and when described large files cannot write new data or write time-out again, the index information of the small documents of buffer memory is written to small documents index area according to write sequence batch.Because described index information comprises access path and the beginning side-play amount of described small documents in this large files of described large files of specifying, user the beginning side-play amount in this large files directly can access small documents according to the access path of described large files and small documents on large files, therefore can realize that small documents is become large files to store, avoid and directly store small documents and produce file fragmentation, and saved storage space; Simultaneously owing to have employed the scheme of batch write index information, the write performance of small documents significantly can also be promoted.
In order to make technical scheme of the present invention clearly understand, describe the present invention below in conjunction with the accompanying drawing embodiment that develops simultaneously.
In exemplifying a kind of embodiment, propose a kind of file memory method, be applied on memory device, refer to Fig. 1, described method comprises:
Step S101, to create in this locality and be used for the small documents that write by described front-end equipment and carry out merging the large files stored;
Step S102, when receive front-end equipment write small documents time, the index information of described small documents is carried out buffer memory in this locality;
When realizing, can creating in advance of described large files, establishment of also can setting out again when receiving the small documents of front-end equipment write.
Step S103, be that described small documents specifies large files, according to write sequence, the descriptor of described small documents and correspondence is write the described large files of specifying, and when this large files cannot write new data or write time-out again, the index information of the described small documents of buffer memory is written to small documents index area according to write sequence batch; Wherein, described index information comprises access path and the beginning side-play amount of described small documents in this large files of described large files of specifying.
In the present embodiment, described front-end equipment can be the front end surveillance device in supervisory system, such as, in the supervisory system based on bayonet socket monitoring business, described front-end equipment can be front end bayonet socket watch-dog (such as bayonet camera) in this supervisory system.
Refer to Fig. 2, Fig. 2 is the small documents storage scheme schematic diagram shown in the present embodiment.
As shown in Figure 2, in whole storage scheme, comprise front end for storing the small documents index area of the index data of small documents, and the large files of rear end; Described large files is actual is the set of a collection of small documents, there are mapping relations with the small documents of front end.Wherein said large files can be stored in independently on block memory device, also can leave on file system in such as existing realization or cloud storage system, not limited in the present embodiment.
In the present embodiment, described small documents index area, for describing the index structure of small documents, ensures the performance of index upgrade and the reliability of index itself.Described small documents index area can be divided out a catalogue subregion and an index subregion.Described catalogue subregion is for depositing the directory information of described small documents, and described index subregion is for depositing the index information of described small documents.
The directory information of described small documents, can comprise the directory information that the directory name (i.e. the directory name of user) of small documents, creation-time, father's layer catalogue and small documents catalogue are mapped to the large files in described large files; The directory information of wherein said large files can comprise the information such as the access path of described large files catalogue; The index information of described small documents, can comprise the information such as the access path of large files, the beginning side-play amount in large files after the title of small documents, file size, creation-time, father's layer catalogue mark, the mapping of described small documents.
By above index structure, can realize that the small documents that front-end equipment writes is mapped as a large files and store on a storage device.User is when checking the small documents that rear end memory device stores, first can by the mapping relations between small documents and large files, search the access path of the large files catalogue after the mapping of small documents catalogue, then under this large files catalogue, then the information searching such as the access path of the large files in described large files and the beginning side-play amount in large files is mapped to corresponding small documents according to small documents.Wherein said beginning side-play amount is for describing the position in small documents large files in the mapped.
Refer to Fig. 3, Fig. 3 is the storage node composition of the described small documents index area shown in the present embodiment.
As shown in Figure 3, the storage organization of described small documents index area in the present embodiment, storage organization comparatively general at present can be continued to use, mainly comprise four parts such as super (superblock), Inode (index node), Dir-block (directory block), File-block (data block).Super is for describing the information such as starting position and end position of Inode, Dir-block, File-block.Inode, for describing the occupation condition in Dir-block and File-block, when realizing, can adopt bitmap to take situation to what describe resource in Dir-block and File-block.Dir-block is the catalogue subregion of described small documents, for storing the directory information of small documents; Each Dir in this subregion represents an independently storage directory respectively.File-block is described index subregion, for depositing the index information of described small documents.
Please continue see Fig. 3, in the present embodiment, in order to improve the write performance of small documents, can by File-block being divided into several file grouping blocks, and specify at least corresponding file grouping block of the directory information of each small documents stored in Dir-block, thus when upgrading the index information of small documents, batch updating can be carried out in units of file grouping block, when index information by small documents writes described small documents index area, can by the index data of the small documents in same file grouping block batch write small documents index area, therefore when the quantity of small documents is enough large, the performance loss of an index upgrade is negligible, thus the write performance of small documents can be made to reach suitable with the write performance of large files realize effect.
Wherein, what deserves to be explained is, the size of described file grouping block can combine actual service conditions by user and carry out equipment, and the quantity of described File-block file grouping block, then depend on the quantity of the size of file grouping block and the small documents of front-end equipment write.Such as, for bayonet socket business, suppose front end bayonet socket watch-dog each hour produce picture (small documents) be 100000, the size of described file grouping block is set to storage 2000 picture indices, when bayonet socket watch-dog is by the memory device of the picture of generation per hour batch write rear end, so the quantity of the required file grouping block of this write is then 50.
Refer to Fig. 4, Fig. 4 is the storage node composition of the described large files shown in the present embodiment.
As shown in Figure 4, large files is actual is the set of a collection of small documents, there are mapping relations with small documents, each small documents (SF-N) in large files comprises data division (Data) and descriptor (Desc) two parts, described descriptor can comprise user file title and file size when realizing, when the index data of the small documents stored in small documents index area is damaged, can recover according to the index data of described descriptor to the small documents in small documents index area stored in large files.
Below with bayonet socket business for example describes the storing process of this programme in detail.
The feature of bayonet socket business is that the flow of business is very large, the bayonet socket watch-dog of its business front end can produce the small documents such as a large amount of pictures or picture frame at short notice, when the small documents produced business front end stores, usually be all by setting a time cycle (such as 1 hour), the small documents collected in time cycle is stored in an independently storage directory, then by the memory device of the small documents batch write rear end in this storage directory.
Therefore, for the business characteristic of bayonet socket business, the small documents of the bayonet socket watch-dog of front end batch write can merge in this locality by rear end memory device, becomes a large files and stores.
When realizing, first can create some small documents for being write by the bayonet socket watch-dog of front end in memory device this locality and carrying out merging the large files stored; A wherein storage directory of the corresponding bayonet socket watch-dog this locality of each large files; The size of described large files can be arranged according to the service conditions of reality by user; Such as, when realizing, in order to ensure by the small documents compound mapping of front-end equipment write-once in same large files as much as possible, the size of large files can be set according to total size of storage directory corresponding to large files.
Memory device, when receiving the small documents of bayonet socket watch-dog write of front end, first can set up affairs in this locality; Described affairs can be the internal storage structure of inner definition, and for describing the process of a small documents index batch updating, these affairs namely by setting up can realize the index data of small documents batch to write described small documents index area.After affairs have been set up, the title (mark) of the index information of the small documents received and described large files can be cached in affairs.Memory device is when the small documents write large files starting front end bayonet socket watch-dog to write, first from the large files created, a large files can be specified for described small documents, then by the directory information of described small documents, and the information such as the access path of the catalogue of the described large files of specifying is written in the Dir-block in the small documents index structure gone out as shown in Figure 3, and in Dir-block, preserve the mapping relations of the directory information of described small documents and the directory information of described large files.Now the catalogue of this large files is the mapping directory of described small documents, follow-uply according to these mapping relations, can be written in the large files of specifying by described small documents according to the access path of the catalogue of described large files.
After described large files has been specified, the index information of small documents that can first be write by the described front-end equipment first of buffer memory in affairs and the title of this large files have write described small documents index area, for follow-up abnormal restoring.After write completes, the file index information of small documents of described first write of buffer memory in affairs and the title of described large files can be discharged.
Next, the small documents that can be write by described front-end equipment according to write sequence and the descriptor of correspondence write described large files according to storage organization illustrated in fig. 4, and wherein said descriptor can comprise the information such as title, size of small documents; When described large files cannot write new data again, such as the size of current large files has reached setting size, can trigger during in affairs, the index information of the described small documents of buffer memory is written in the index structure of the small documents index area gone out as shown in Figure 3 File-block according to write sequence batch, to complete the renewal of disk.
But above scheme is for some special application scenarioss, can cause because write the duration oversize and disk that causes upgrades problem not in time, if disk upgrades not in time, if so there is power-off or reboot event in ablation process, then have to re-start recovery to the data in small documents index area.
Such as, for the road monitoring scene of bayonet socket watch-dog, because road car is less time at dead of night, the picture number that in the bayonet socket watch-dog time cycle, (such as per hour interior) collects is limited, when therefore the picture collected in time cycle batch being written to rear end memory device, if large files is write full after trigger the index information of small documents write small documents index area again, the write duration then can be caused long, so within the time period of this write, if equipment generation power-off or restart, after then having to that recovery is re-started to the data in small documents index area, could upgrade disk.
Therefore, for above scene, when the descriptor of the small documents write by described front-end equipment according to write sequence and correspondence writes described large files according to storage organization illustrated in fig. 4, a time-out time can be set, if this write duration is when reaching time-out time, can trigger during in affairs, the index information of the described small documents of buffer memory is written in the index structure of the small documents index area gone out as shown in Figure 3 File-block according to write sequence batch immediately, thus the renewal of disk can be completed timely, to reduce the number of times data in small documents index area being re-started to recovery.
Please continue see Fig. 3, when the index information batch of the small documents by buffer memory in Current transaction is written to described small documents index area, suppose that this small documents write is the small documents in catalogue Dir1, first can for Dir1 distributes a file grouping block FileGrp1 from the File-block of small documents index area, by in the catalogue Dir1 of small documents described in the positional information of this file grouping block FileGrp1 write Dir-block, then the index information of described small documents is write in described file grouping block FileGrp1 according to write sequence batch, and preserve the mapping relations of described small documents and described large files, now described large files be described small documents map after large files.
Certainly, if in the process of this write, FileGrp1 has reached setting size, when cannot write data again, from the end position of described FileGrp1, a new file grouping block FileGrp2 (not shown in Fig. 3) again can be marked off in index subregion, FileGrp2 is distributed to Dir1, and by the positional information of FileGrp2 write FileGrp1, to form the chained list of a file grouping block, then the index information of described small documents batch is continued to write in described FileGrp2, and repeat above process, until the index information of described small documents has write.In the present embodiment, the index information of the small documents of buffer memory is being written in the process of small documents index area according to write sequence batch, if equipment generation power-off or restart, after equipment is restarted, can according to the index information of the title of the described large files stored in described small documents index area and described first small documents, (when realizing, large files may have multiple to find corresponding large files, therefore the large files title by having recorded accurately can find corresponding large files), and travel through this large files, read in this large files the described descriptor stored, then after recovering according to the index information of described descriptor to described small documents, again described small documents index area is write according to write sequence batch.
As previously mentioned, described index information can comprise the title of small documents, file size, creation-time, father's layer catalogue mark, described small documents map after the information such as the access path of large files, the beginning side-play amount in large files.Described descriptor can comprise the information such as title, size of small documents.
Therefore, when recovering described index information, the information such as the access path of the large files in the index information of described first small documents that can store from small documents index area after direct copying creation-time, father's layer catalogue mark and the mapping of described small documents, then recover user file title and file size according to described descriptor.Finally, when recovering the beginning side-play amount of each small documents in large files, can according to the beginning side-play amount in the file index information of first small documents recorded, and obtain after calculating in conjunction with the actual size of described first small documents.Such as, suppose that the beginning side-play amount of first small documents is 0, size is 40, show that in large files, front 40 bytes are first small documents of write, so when recovering the beginning side-play amount of second small documents, the size of its reality can be added by the beginning side-play amount of described first small documents, the beginning side-play amount obtaining second small documents is 40, now show in large files to be second small documents from the 40th byte, the like, the beginning side-play amount of other each small documents can be recovered in order.After in described small documents index area, the index information of each small documents successfully recovers, because the index data of buffer memory in affairs just can discharge after write completes, therefore again according to write sequence, the index data of buffer memory in affairs batch can be write described small documents index area.
Known by the description of above embodiment, the present invention program is used for the small documents that front-end equipment writes to carry out merging the large files stored by creating in this locality, when receiving the small documents of front-end equipment write, the index information of the small documents received is carried out buffer memory in this locality, then according to write sequence, the descriptor of described small documents and correspondence is write described large files, and when described large files cannot write new data or write time-out again, the index information of the small documents of buffer memory is written to small documents index area according to write sequence batch.Because described index information comprises access path and the beginning side-play amount of described small documents in this large files of described large files of specifying, user the beginning side-play amount in this large files directly can access small documents according to the access path of described large files and small documents on large files, therefore can realize that small documents is become large files to store, avoid and directly store small documents and produce file fragmentation, and saved storage space; Simultaneously owing to have employed the scheme of batch write index information, the write performance of small documents significantly can also be promoted.
Simultaneously, because small documents index area have employed special index structure, file grouping block is divided to the fileinfo in index area, and specify at least corresponding file grouping block of the directory information of each small documents stored in index area, thus when upgrading the index information of small documents, batch updating can be carried out in units of file grouping block, therefore significantly can promote the write performance of small documents, thus the write performance of small documents can be made to reach suitable with the write performance of large files realize effect; And, file grouping block is divided to the fileinfo in index area, can also realize when deleted file index, only need simply in the index area of front end, to delete several file grouping block, and delete the large files catalogue after mapping in rear end, whole efficiency is higher.
In addition, due to small documents store in large files time, save the descriptor of small documents, therefore once the index information of small documents occurs abnormal, can recover according to the descriptor of small documents, effectively ensure that the safety of index information.And owing to preserving together with data in the descriptor of large files small file, if there is in the situations such as Bad Track, when recovering small documents, normal sector need not rely on the data of the sector of bad track, can recover separately.
Refer to Fig. 5, in a kind of exemplary embodiment, the present invention also proposes a kind of document storage system 50, is applied on memory device, and described system 50 comprises:
Creation module 501, carries out merging the large files stored for creating the small documents being used for being write by described front-end equipment in this locality;
Cache module 502, for when receiving the small documents of front-end equipment write, carries out buffer memory by the index information of described small documents in this locality;
Writing module 503, for specifying large files for described small documents, according to write sequence, the descriptor of described small documents and correspondence is write the described large files of specifying, and when this large files cannot write new data or write time-out again, the index information of the described small documents of buffer memory is written to small documents index area according to write sequence batch;
Wherein, described index information comprises access path and the beginning side-play amount of described small documents in this large files of described large files of specifying.
In the present embodiment, described small documents index area comprises catalogue subregion;
Said write module 503 specifically for:
For described small documents specifies large files;
The directory information of described small documents and the directory information of described large files of specifying are write described catalogue subregion, and in catalogue subregion, preserves the mapping relations of the directory information of described small documents and the directory information of described large files; The directory information of wherein said large files comprises the access path of the catalogue of described large files.
In the present embodiment, said write module 503 is further used for:
Before according to write sequence the descriptor of described small documents and correspondence being write the described large files of specifying; the index information of first small documents write by described front-end equipment and the mark of described large files of specifying write described small documents index area, can carry out abnormal restoring so that follow-up according to the mark of the index information of described first small documents and described large files of specifying to the data in small documents index area.
In the present embodiment, described small documents index area also comprises index subregion; Wherein said index subregion comprises the file grouping block of at least one default size;
Said write module 503 is further used for:
For described small documents distributes a file grouping block;
The index information of described small documents is write described file grouping block according to write sequence batch, and preserves the mapping relations of described small documents and described large files of specifying;
When described file grouping block cannot write new data again, from the end position of described file grouping block, a new file grouping block is again marked off in described index subregion, described new file grouping block is distributed to described small documents, and continue the fileinfo of described small documents batch to write described new file grouping block, until the index information of described small documents has all write.
In the present embodiment, said write module 503 is further used for:
The index information of the small documents of buffer memory is being written in the process of small documents index area according to write sequence batch, if equipment generation power-off or restart, after equipment is restarted, according to the mark of the described large files stored in described small documents index area and the index information of described first small documents, find corresponding large files, and travel through this large files, read in this large files the described descriptor stored, after recovering according to the index information of described descriptor to described small documents, again described small documents index area is write according to write sequence batch.
In the present embodiment, described system 50 also comprises:
Release module 504, after being successfully written to described small documents index area, discharges the index information of the described small documents of buffer memory for the index information of the small documents at described buffer memory.
By the description of above embodiment, it will be appreciated by those skilled in the art that the module in embodiment in device can merge into a module, also can split into multiple submodule further.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, within the spirit and principles in the present invention all, any amendment made, equivalent replacement, improvement etc., all should be included within the scope of protection of the invention.
Claims (12)
1. a small documents storage means, is applied on memory device, it is characterized in that, described method comprises:
Create the small documents being used for being write by described front-end equipment to carry out merging the large files stored in this locality;
When receiving the small documents of front-end equipment write, the index information of described small documents is carried out buffer memory in this locality;
For described small documents specifies large files, according to write sequence, the descriptor of described small documents and correspondence is write the described large files of specifying, and when this large files cannot write new data or write time-out again, the index information of the described small documents of buffer memory is written to small documents index area according to write sequence batch;
Wherein, described index information comprises access path and the beginning side-play amount of described small documents in this large files of described large files of specifying.
2. the method for claim 1, is characterized in that, described small documents index area comprises catalogue subregion;
Described is that described small documents specifies large files to comprise:
For described small documents specifies large files;
The directory information of described small documents and the directory information of described large files of specifying are write described catalogue subregion, and in catalogue subregion, preserves the mapping relations of the directory information of described small documents and the directory information of described large files; The directory information of wherein said large files comprises the access path of the catalogue of described large files.
3. the method for claim 1, is characterized in that, described according to write sequence the descriptor of described small documents and correspondence write the described large files of specifying before, comprising:
The index information of first small documents write by described front-end equipment and the mark of described large files of specifying write described small documents index area, can carry out abnormal restoring so that follow-up according to the mark of the index information of described first small documents and described large files of specifying to the data in small documents index area.
4. the method for claim 1, is characterized in that, described small documents index area also comprises index subregion; Wherein said index subregion comprises the file grouping block of at least one default size;
Describedly the index information of the described small documents of buffer memory be written to described small documents index area according to write sequence batch comprise:
For described small documents distributes a file grouping block;
The index information of described small documents is write described file grouping block according to write sequence batch, and preserves the mapping relations of described small documents and described large files of specifying;
When described file grouping block cannot write new data again, from the end position of described file grouping block, a new file grouping block is again marked off in described index subregion, described new file grouping block is distributed to described small documents, and continue the fileinfo of described small documents batch to write described new file grouping block, until the index information of described small documents has all write.
5. method as claimed in claim 3, it is characterized in that, described method also comprises:
The index information of the small documents of buffer memory is being written in the process of small documents index area according to write sequence batch, if equipment generation power-off or restart, after equipment is restarted, according to the mark of the described large files stored in described small documents index area and the index information of described first small documents, find corresponding large files, and travel through this large files, read in this large files the described descriptor stored, after recovering according to the index information of described descriptor to described small documents, again described small documents index area is write according to write sequence batch.
6. the method for claim 1, is characterized in that, described method also comprises:
After the index information of the small documents of described buffer memory is successfully written to described small documents index area, discharge the described little civilian index information of buffer memory.
7. a small documents storage system, is applied on memory device, it is characterized in that, described system comprises:
Creation module, carries out merging the large files stored for creating the small documents being used for being write by described front-end equipment in this locality;
Cache module, for when receiving the small documents of front-end equipment write, carries out buffer memory by the index information of described small documents in this locality;
Writing module, for specifying large files for described small documents, according to write sequence, the descriptor of described small documents and correspondence is write the described large files of specifying, and when this large files cannot write new data or write time-out again, the index information of the described small documents of buffer memory is written to small documents index area according to write sequence batch;
Wherein, described index information comprises access path and the beginning side-play amount of described small documents in this large files of described large files of specifying.
8. system as claimed in claim 7, it is characterized in that, described small documents index area comprises catalogue subregion;
Said write module specifically for:
For described small documents specifies large files;
The directory information of described small documents and the directory information of described large files of specifying are write described catalogue subregion, and in catalogue subregion, preserves the mapping relations of the directory information of described small documents and the directory information of described large files; The directory information of wherein said large files comprises the access path of the catalogue of described large files.
9. method as claimed in claim 7, it is characterized in that, said write module is further used for:
Before according to write sequence the descriptor of described small documents and correspondence being write the described large files of specifying; the index information of first small documents write by described front-end equipment and the mark of described large files of specifying write described small documents index area, can carry out abnormal restoring so that follow-up according to the mark of the index information of described first small documents and described large files of specifying to the data in small documents index area.
10. system as claimed in claim 7, it is characterized in that, described small documents index area also comprises index subregion; Wherein said index subregion comprises the file grouping block of at least one default size;
Said write module is further used for:
For described small documents distributes a file grouping block;
The index information of described small documents is write described file grouping block according to write sequence batch, and preserves the mapping relations of described small documents and described large files of specifying;
When described file grouping block cannot write new data again, from the end position of described file grouping block, a new file grouping block is again marked off in described index subregion, described new file grouping block is distributed to described small documents, and continue the fileinfo of described small documents batch to write described new file grouping block, until the index information of described small documents has all write.
11. systems as claimed in claim 9, it is characterized in that, said write module is further used for:
The index information of the small documents of buffer memory is being written in the process of small documents index area according to write sequence batch, if equipment generation power-off or restart, after equipment is restarted, according to the mark of the described large files stored in described small documents index area and the index information of described first small documents, find corresponding large files, and travel through this large files, read in this large files the described descriptor stored, after recovering according to the index information of described descriptor to described small documents, again described small documents index area is write according to write sequence batch.
12. systems as claimed in claim 7, it is characterized in that, described system also comprises:
Release module, after being successfully written to described small documents index area, discharges the index information of the described small documents of buffer memory for the index information of the small documents at described buffer memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410833341.7A CN104462563B (en) | 2014-12-26 | 2014-12-26 | A kind of file memory method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410833341.7A CN104462563B (en) | 2014-12-26 | 2014-12-26 | A kind of file memory method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104462563A true CN104462563A (en) | 2015-03-25 |
| CN104462563B CN104462563B (en) | 2019-04-30 |
Family
ID=52908598
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410833341.7A Active CN104462563B (en) | 2014-12-26 | 2014-12-26 | A kind of file memory method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104462563B (en) |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104715070A (en) * | 2015-04-01 | 2015-06-17 | 武汉中科通达高新技术股份有限公司 | Large image file as well as generation and read method thereof |
| CN105868286A (en) * | 2016-03-23 | 2016-08-17 | 中国科学院计算技术研究所 | Parallel adding method and system for merging small files on basis of distributed file system |
| CN106021585A (en) * | 2016-06-02 | 2016-10-12 | 同济大学 | Traffic incident video access method and system based on time-space characteristics |
| CN106155845A (en) * | 2016-08-02 | 2016-11-23 | 四川效率源信息安全技术股份有限公司 | A kind of restructuring restoration methods based on XFS file system data |
| CN106326292A (en) * | 2015-06-29 | 2017-01-11 | 杭州海康威视数字技术股份有限公司 | Data structure and file aggregation and reading methods and apparatuses |
| CN106897391A (en) * | 2017-01-24 | 2017-06-27 | 北京百悟科技有限公司 | File memory method and device |
| CN107506447A (en) * | 2017-08-25 | 2017-12-22 | 郑州云海信息技术有限公司 | A kind of small documents reading/writing method and system based on local file system |
| CN107509085A (en) * | 2017-08-24 | 2017-12-22 | 山东浪潮商用系统有限公司 | The storage method and device of a kind of files in stream media |
| CN107644056A (en) * | 2017-08-04 | 2018-01-30 | 武汉烽火众智数字技术有限责任公司 | A kind of file memory method, apparatus and system |
| CN107665092A (en) * | 2016-07-28 | 2018-02-06 | 华为技术有限公司 | A kind of storage method and device |
| CN108319541A (en) * | 2017-01-16 | 2018-07-24 | 上海大唐移动通信设备有限公司 | A kind of log file analysis method and device |
| CN109101598A (en) * | 2018-07-31 | 2018-12-28 | 成都华栖云科技有限公司 | A kind of small page picture rendering method |
| CN109947721A (en) * | 2017-12-01 | 2019-06-28 | 北京安天网络安全技术有限公司 | A kind of small documents treating method and apparatus |
| CN109994131A (en) * | 2019-04-10 | 2019-07-09 | 广西电网有限责任公司玉林供电局 | A kind of power frequency recorded wave file compression and storage method and system based on index |
| CN110147203A (en) * | 2019-05-16 | 2019-08-20 | 北京金山云网络技术有限公司 | A kind of file management method, device, electronic equipment and storage medium |
| CN110555000A (en) * | 2019-09-05 | 2019-12-10 | 重庆紫光华山智安科技有限公司 | Card port picture metadata concurrent writing and reading method |
| CN110968549A (en) * | 2019-11-18 | 2020-04-07 | Oppo（重庆）智能科技有限公司 | File storage method and device, electronic equipment and medium |
| CN111258955A (en) * | 2018-11-30 | 2020-06-09 | 北京白山耘科技有限公司 | File reading method and system, storage medium and computer equipment |
| CN111966845A (en) * | 2020-08-31 | 2020-11-20 | 重庆紫光华山智安科技有限公司 | Picture management method and device, storage node and storage medium |
| CN113110801A (en) * | 2021-04-15 | 2021-07-13 | 山东英信计算机技术有限公司 | Method, system, equipment and storage medium for accelerating small file reading speed |
| CN115080526A (en) * | 2022-08-22 | 2022-09-20 | 四川蜀天信息技术有限公司 | Method for storing large file based on IPFS |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103577123A (en) * | 2013-11-12 | 2014-02-12 | 河海大学 | Small file optimization storage method based on HDFS |
| US20140108402A1 (en) * | 2004-01-23 | 2014-04-17 | Hand Held Products, Inc. | System and Method to Store and Retrieve Indentifier Associated Information Content |
| CN103856567A (en) * | 2014-03-26 | 2014-06-11 | 西安电子科技大学 | Small file storage method based on Hadoop distributed file system |
| CN103970869A (en) * | 2014-05-12 | 2014-08-06 | 浙江宇视科技有限公司 | Large file storage method |
-
2014
- 2014-12-26 CN CN201410833341.7A patent/CN104462563B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140108402A1 (en) * | 2004-01-23 | 2014-04-17 | Hand Held Products, Inc. | System and Method to Store and Retrieve Indentifier Associated Information Content |
| CN103577123A (en) * | 2013-11-12 | 2014-02-12 | 河海大学 | Small file optimization storage method based on HDFS |
| CN103856567A (en) * | 2014-03-26 | 2014-06-11 | 西安电子科技大学 | Small file storage method based on Hadoop distributed file system |
| CN103970869A (en) * | 2014-05-12 | 2014-08-06 | 浙江宇视科技有限公司 | Large file storage method |
Cited By (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104715070A (en) * | 2015-04-01 | 2015-06-17 | 武汉中科通达高新技术股份有限公司 | Large image file as well as generation and read method thereof |
| CN106326292A (en) * | 2015-06-29 | 2017-01-11 | 杭州海康威视数字技术股份有限公司 | Data structure and file aggregation and reading methods and apparatuses |
| CN106326292B (en) * | 2015-06-29 | 2020-05-19 | 杭州海康威视数字技术股份有限公司 | Data structure and file aggregation and reading method and device |
| CN105868286B (en) * | 2016-03-23 | 2019-03-12 | 中国科学院计算技术研究所 | The parallel method of adding and system merged based on distributed file system small documents |
| CN105868286A (en) * | 2016-03-23 | 2016-08-17 | 中国科学院计算技术研究所 | Parallel adding method and system for merging small files on basis of distributed file system |
| CN106021585A (en) * | 2016-06-02 | 2016-10-12 | 同济大学 | Traffic incident video access method and system based on time-space characteristics |
| CN107665092A (en) * | 2016-07-28 | 2018-02-06 | 华为技术有限公司 | A kind of storage method and device |
| CN107665092B (en) * | 2016-07-28 | 2019-11-12 | 华为技术有限公司 | A kind of storage method and device |
| CN106155845A (en) * | 2016-08-02 | 2016-11-23 | 四川效率源信息安全技术股份有限公司 | A kind of restructuring restoration methods based on XFS file system data |
| CN108319541A (en) * | 2017-01-16 | 2018-07-24 | 上海大唐移动通信设备有限公司 | A kind of log file analysis method and device |
| CN106897391A (en) * | 2017-01-24 | 2017-06-27 | 北京百悟科技有限公司 | File memory method and device |
| CN107644056A (en) * | 2017-08-04 | 2018-01-30 | 武汉烽火众智数字技术有限责任公司 | A kind of file memory method, apparatus and system |
| CN107509085A (en) * | 2017-08-24 | 2017-12-22 | 山东浪潮商用系统有限公司 | The storage method and device of a kind of files in stream media |
| CN107509085B (en) * | 2017-08-24 | 2019-12-24 | 山东浪潮商用系统有限公司 | Storage method and device of streaming media file |
| CN107506447A (en) * | 2017-08-25 | 2017-12-22 | 郑州云海信息技术有限公司 | A kind of small documents reading/writing method and system based on local file system |
| CN109947721A (en) * | 2017-12-01 | 2019-06-28 | 北京安天网络安全技术有限公司 | A kind of small documents treating method and apparatus |
| CN109947721B (en) * | 2017-12-01 | 2021-08-17 | 北京安天网络安全技术有限公司 | Small file processing method and device |
| CN109101598A (en) * | 2018-07-31 | 2018-12-28 | 成都华栖云科技有限公司 | A kind of small page picture rendering method |
| CN111258955A (en) * | 2018-11-30 | 2020-06-09 | 北京白山耘科技有限公司 | File reading method and system, storage medium and computer equipment |
| CN111258955B (en) * | 2018-11-30 | 2023-09-19 | 北京白山耘科技有限公司 | File reading method and system, storage medium and computer equipment |
| CN109994131A (en) * | 2019-04-10 | 2019-07-09 | 广西电网有限责任公司玉林供电局 | A kind of power frequency recorded wave file compression and storage method and system based on index |
| CN109994131B (en) * | 2019-04-10 | 2021-10-22 | 广西电网有限责任公司玉林供电局 | Index-based power frequency wave recording file compression storage method and system |
| CN110147203A (en) * | 2019-05-16 | 2019-08-20 | 北京金山云网络技术有限公司 | A kind of file management method, device, electronic equipment and storage medium |
| CN110555000A (en) * | 2019-09-05 | 2019-12-10 | 重庆紫光华山智安科技有限公司 | Card port picture metadata concurrent writing and reading method |
| CN110968549A (en) * | 2019-11-18 | 2020-04-07 | Oppo（重庆）智能科技有限公司 | File storage method and device, electronic equipment and medium |
| CN110968549B (en) * | 2019-11-18 | 2024-03-29 | Oppo（重庆）智能科技有限公司 | File storage method, device, electronic equipment and medium |
| CN111966845A (en) * | 2020-08-31 | 2020-11-20 | 重庆紫光华山智安科技有限公司 | Picture management method and device, storage node and storage medium |
| CN111966845B (en) * | 2020-08-31 | 2023-11-17 | 重庆紫光华山智安科技有限公司 | Picture management method, device, storage node and storage medium |
| CN113110801A (en) * | 2021-04-15 | 2021-07-13 | 山东英信计算机技术有限公司 | Method, system, equipment and storage medium for accelerating small file reading speed |
| CN115080526A (en) * | 2022-08-22 | 2022-09-20 | 四川蜀天信息技术有限公司 | Method for storing large file based on IPFS |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104462563B (en) | 2019-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104462563A (en) | File storage method and system | |
| US8250033B1 (en) | Replication of a data set using differential snapshots | |
| US10496599B1 (en) | Cloud data archiving using chunk-object mapping and synthetic full backup | |
| CN108319602B (en) | Database management method and database system | |
| US9336095B2 (en) | Computing system and related data management method thereof | |
| US9304998B2 (en) | Main-memory database checkpointing | |
| US10146631B1 (en) | Incremental forever backups for exchange | |
| JP6264666B2 (en) | Data storage method, data storage device, and storage device | |
| CN101808137B (en) | Data transmission method, device and system | |
| CN113722275B (en) | Object storage space management method, device, server and storage medium | |
| CN106951375B (en) | Method and device for deleting snapshot volume in storage system | |
| CN109391647B (en) | Storage resource recovery method, device and system | |
| CN103473277A (en) | Snapshot method and device for file systems | |
| CN102955720A (en) | Method for improving stability of EXT (extended) file system | |
| WO2018141304A1 (en) | Flash file system and data management method thereof | |
| JP5868986B2 (en) | Recovery by item | |
| CN105824572A (en) | Disk storage space managing method, apparatus and storage device | |
| CN104461773A (en) | Backup deduplication method of virtual machine | |
| US10977143B2 (en) | Mirrored write ahead logs for data storage system | |
| CN102314503A (en) | Indexing method | |
| CN103164528B (en) | A kind of index establishing method of audio, video data | |
| CN110597762A (en) | File processing method, device, equipment and storage medium | |
| KR101674176B1 (en) | Method and apparatus for fsync system call processing using ordered mode journaling with file unit | |
| US10701153B1 (en) | Cloud data archiving using latest snapshot metadata | |
| CN115617264A (en) | Distributed storage method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |