CN107016130A - A kind of method and device of file storage - Google Patents
A kind of method and device of file storage Download PDFInfo
- Publication number
- CN107016130A CN107016130A CN201710351825.1A CN201710351825A CN107016130A CN 107016130 A CN107016130 A CN 107016130A CN 201710351825 A CN201710351825 A CN 201710351825A CN 107016130 A CN107016130 A CN 107016130A
- Authority
- CN
- China
- Prior art keywords
- file
- attribute
- stored
- aggregate
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method and device of file storage, whether by detecting the directory attribute of file to be stored, it is the Aggregate attribute set previously according to application scenarios to judge directory attribute；If, function is write in file opening polymerization, and file to be stored is stored to aggregate file, when the residual memory space of the OBJ objects in aggregate file is equal to or less than preset capacity, by multiple file to be stored batch write-in disks in OBJ objects, aggregate file includes multiple OBJ objects；If it is not, file to be stored is write into disk.The application is by judging whether directory attribute is Aggregate attribute, and to determine that current file is that file polymerization is write, or commonly non-polymeric is write, realize different according to scene, directory attribute is preset, function is write in selectively unlocking or closing file polymerization, to reduce the write-in pressure of bottom cluster.
Description
Technical field
The present invention relates to field of filesystems, more particularly to a kind of method and device of file storage.
Background technology
With the development of file system technology, the application of distributed storage file system is also more and more extensive.
In distributed storage file system, in order to tackle mass small documents storage scenarios, typically small documents are polymerize and write
In disk, to reduce the number of times of write-in disk, file write performance is improved.But, when file be it is larger, with either quantity not
When many, it is not necessary to recycle file polymerization to write function to realize that file is stored.
When file is larger or during negligible amounts, the write-in pressure of bottom cluster can be caused by writing function using small documents polymerization
It is larger.
The content of the invention
It is an object of the invention to provide a kind of method and device of file storage, it is therefore intended that solves file in the prior art
During larger and negligible amounts, the problem of write-in pressure of bottom cluster is larger can be caused by writing function using small documents polymerization.
In order to solve the above technical problems, the present invention provides a kind of method of file storage, this method includes：
The directory attribute of file to be stored is detected, whether judge the directory attribute is to be set previously according to application scenarios
Aggregate attribute；
If so, function is write in file opening polymerization, the file to be stored is stored to aggregate file, when the aggregate file
, will be multiple described to be stored in the OBJ objects when residual memory space of interior OBJ objects is equal to or less than preset capacity
Files in batch writes disk, and the aggregate file includes multiple OBJ objects；
If it is not, the file to be stored is write into disk.
Alternatively, it is described detection file to be stored directory attribute, judge the directory attribute whether be previously according to
Also include before the Aggregate attribute of application scenarios setting：
Receive and instruction is set；
Instructed according to described set, the field value in current file extended attribute is set to default value, described will worked as
The directory attribute of preceding document is set to Aggregate attribute.
Alternatively, it is described detection file to be stored directory attribute, judge the directory attribute whether be previously according to answer
Included with the Aggregate attribute of scene settings：
Whether the field value for judging the file to be stored is the default value；
If so, the directory attribute for determining the file to be stored is the Aggregate attribute；
If it is not, the directory attribute for determining the file to be stored is non-polymeric attribute.
In addition, present invention also offers a kind of device of file storage, the device includes：
Judge module, the directory attribute for detecting file to be stored, judge the directory attribute whether be previously according to
The Aggregate attribute of application scenarios setting；
Polymerize writing module, for if so, file opening polymerization write function, by the file to be stored store to polymerization text
Part, when the residual memory space of the OBJ objects in the aggregate file is equal to or less than preset capacity, by the OBJ objects
Interior multiple file to be stored batch write-in disks, the aggregate file includes multiple OBJ objects；
Non-polymeric writing module, for if it is not, the file to be stored is write into disk.
Alternatively, in addition to：
Receiving module, instruction is set for receiving；
Numerical value setup module, for being instructed according to described set, is set to pre- by the field value in current file extended attribute
If numerical value, the directory attribute of the current file is set to Aggregate attribute.
Alternatively, the judge module includes：
Numerical value judging unit, for judging whether the field value of the file to be stored is the default value；
First determining unit, for if so, the directory attribute for determining the file to be stored is the Aggregate attribute；
Second determining unit, for if it is not, the directory attribute for determining the file to be stored is non-polymeric attribute.
A kind of method and device of file storage provided by the present invention, by detecting the directory attribute of file to be stored,
Whether judge directory attribute is the Aggregate attribute set previously according to application scenarios；If so, function is write in file opening polymerization, it will treat
Storage file is stored to aggregate file, when the residual memory space of the OBJ objects in aggregate file is equal to or less than preset capacity
When, by multiple file to be stored batch write-in disks in OBJ objects, aggregate file includes multiple OBJ objects；If it is not,
File to be stored is write into disk.The application is by judging whether directory attribute is Aggregate attribute, to determine that current file is text
Part polymerization is write, or commonly non-polymeric is write, and realizes according to scene difference, presets directory attribute, selectively unlocking or pass
Function is write in closed file polymerization, to reduce the write-in pressure of bottom cluster.
Brief description of the drawings
, below will be to embodiment or existing for the clearer explanation embodiment of the present invention or the technical scheme of prior art
The accompanying drawing used required in technology description is briefly described, it should be apparent that, drawings in the following description are only this hair
Some bright embodiments, for those of ordinary skill in the art, on the premise of not paying creative work, can be with root
Other accompanying drawings are obtained according to these accompanying drawings.
A kind of schematic flow sheet of the embodiment for the file memory method that Fig. 1 is provided by the embodiment of the present invention；
The structured flowchart for the file storage device that Fig. 2 is provided by the embodiment of the present invention.
Embodiment
In order that those skilled in the art more fully understand the present invention program, with reference to the accompanying drawings and detailed description
The present invention is described in further detail.Obviously, described embodiment is only a part of embodiment of the invention, rather than
Whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art are not making creative work premise
Lower obtained every other embodiment, belongs to the scope of protection of the invention.
Refer to Fig. 1, a kind of stream of the embodiment for the file memory method that Fig. 1 is provided by the embodiment of the present invention
Journey schematic diagram, this method comprises the following steps：
Step 101：The directory attribute of file to be stored is detected, whether judge the directory attribute is previously according to applied field
The Aggregate attribute of scape setting.
It should be noted that above-mentioned file to be stored can refer to small documents or refer to big file.Above-mentioned applied field
Scape can refer to mass small documents storage scenarios and file is larger and negligible amounts storage scenarios in one.
Above-mentioned directory attribute can refer to whether the catalogue of file is polymerization catalogue.Specifically, can be by the expansion of file
Open up the respective field in attribute to set whether the catalogue of this document polymerize catalogue, that is, the numeral of field is set, it is current to set
Whether catalogue is polymerization catalogue.Catalogue is set to after polymerization catalogue, this document then can write flow storage according to file polymerization,
And catalogue can then be stored for the file of general catalogue according to ordinary file Stored Procedure.
Specifically, operation processing function, handle the current file to be stored of function check directory attribute, judge its whether be
It polymerize catalogue.
It is also poly- in the subdirectory of the polymerization directory creating it is understood that catalogue is set as after polymerization catalogue
Close catalogue.
The attribute of file directory can be set by the user in advance, can specifically be set by setting field values.
As a kind of embodiment, in the directory attribute of above-mentioned detection file to be stored, the directory attribute is judged
It can also include before whether being the Aggregate attribute set previously according to application scenarios：Receive and instruction is set；Set according to described
Instruction, is set to default value by the field value in current file extended attribute, the directory attribute of the current file is set to
Aggregate attribute.
It should be noted that above-mentioned setting instruction can be the setting instruction of user, i.e., user can specify particular category
Whether it is polymerization catalogue, that is, it is file polymerization write-in disk to specify current file to be stored, or in non-polymeric write-in disk.
Above-mentioned default value can be embodied in 0 or 1, for example, i.e. can be by the specific fields in file extent attribute
Value be set to 0, represent that function is write in shielding polymerization, when being set to 1, represent that function is write in file opening polymerization.Specifically, can be by small text
The catalogue of part is set to Aggregate attribute, and big file is set to non-polymeric attribute.
After field value is set, that is, after setting the directory attribute of file, during detection can using detected field value as
How much, and then judge that what the directory attribute of file is.
As a kind of embodiment, the directory attribute of above-mentioned detection file to be stored judges that the directory attribute is
No is that the process of the Aggregate attribute set previously according to application scenarios can be specially：Judge the word of the file to be stored
Whether segment value is the default value；If so, the directory attribute for determining the file to be stored is the Aggregate attribute；If
No, the directory attribute for determining the file to be stored is non-polymeric attribute.
Specifically, processing function can detect whether the field values of current file are 1, if 1, then judge current file
Directory attribute be Aggregate attribute, if 0, then judge the directory attribute of current file for non-polymeric attribute.
Step 102：If so, function is write in file opening polymerization, the file to be stored is stored to aggregate file, when described
When the residual memory space of OBJ objects in aggregate file is equal to or less than preset capacity, by multiple institutes in the OBJ objects
File to be stored batch write-in disk is stated, the aggregate file includes multiple OBJ objects.
It should be noted that above-mentioned aggregate file can include storing in multiple OBJ objects, and each OBJ objects
Multiple small documents, now, aggregate file can have n OBJ object, and each OBJ objects can store m source file.And it is above-mentioned
The memory space of OBJ objects is fixed, for example, 4MB.
Above-mentioned preset capacity can be 0 or other numerical value, no longer limit herein.Specifically, when OBJ objects exist
, can be by the unified write-in of all source files to be updated in the OBJ objects to disk, i.e. batch when being fully written or being nearly fully written
Disk is write, the purpose of batch updating source file is realized.
Step 103：If it is not, the file to be stored is write into disk.
Specifically, when file directory attribute is non-polymeric attribute, then file to be stored is carried out according to generic storage flow
Storage.Specifically, file to be stored is stored to OBJ objects, then again write the file in OBJ objects in disk, and
Without being write again after OBJ objects are filled with.
The file memory method that the embodiment of the present invention is provided, by detecting the directory attribute of file to be stored, judges mesh
Whether record attribute is the Aggregate attribute set previously according to application scenarios；If so, function is write in file opening polymerization, by text to be stored
Part is stored to aggregate file, will when the residual memory space of the OBJ objects in aggregate file is equal to or less than preset capacity
Multiple file to be stored batch write-in disks in OBJ objects, aggregate file includes multiple OBJ objects；If it is not, will wait to deposit
Store up file write-in disk.This method is by judging whether directory attribute is Aggregate attribute, to determine that current file is file polymerization
Write, or commonly non-polymeric is write, and realizes according to scene difference, presets directory attribute, selectively unlocking or closing file
Function is write in polymerization, to reduce the write-in pressure of bottom cluster.
File storage device provided in an embodiment of the present invention is introduced below, file storage device described below with
Above-described file memory method can be mutually to should refer to.
The structured flowchart for the file storage device that Fig. 2 is provided by the embodiment of the present invention, the file storage device of reference picture 2 can
With including：
Judge module 21, the directory attribute for detecting file to be stored, judge directory attribute whether be previously according to should
With the Aggregate attribute of scene settings；
Polymerize writing module 22, for if so, file opening polymerization write function, file to be stored is stored to aggregate file,
When the residual memory space of the OBJ objects in aggregate file is equal to or less than preset capacity, multiple in OBJ objects are waited to deposit
Files in batch write-in disk is stored up, aggregate file includes multiple OBJ objects；
Non-polymeric writing module 23, for if it is not, file to be stored is write into disk.
Alternatively, in addition to：
Receiving module, instruction is set for receiving；
Numerical value setup module, for according to instruction is set, the field value in current file extended attribute to be set into present count
Value, Aggregate attribute is set to by the directory attribute of current file.
Alternatively, above-mentioned judge module includes：
Numerical value judging unit, for judging whether the field value of file to be stored is default value；
First determining unit, for if so, the directory attribute for determining file to be stored is Aggregate attribute；
Second determining unit, for if it is not, the directory attribute for determining file to be stored is non-polymeric attribute.
The file storage device that the embodiment of the present invention is provided, by detecting the directory attribute of file to be stored, judges mesh
Whether record attribute is the Aggregate attribute set previously according to application scenarios；If so, function is write in file opening polymerization, by text to be stored
Part is stored to aggregate file, will when the residual memory space of the OBJ objects in aggregate file is equal to or less than preset capacity
Multiple file to be stored batch write-in disks in OBJ objects, aggregate file includes multiple OBJ objects；If it is not, will wait to deposit
Store up file write-in disk.The device is by judging whether directory attribute is Aggregate attribute, to determine that current file is file polymerization
Write, or commonly non-polymeric is write, and realizes according to scene difference, presets directory attribute, selectively unlocking or closing file
Function is write in polymerization, to reduce the write-in pressure of bottom cluster.
The embodiment of each in this specification is described by the way of progressive, what each embodiment was stressed be with it is other
Between the difference of embodiment, each embodiment same or similar part mutually referring to.For being filled disclosed in embodiment
For putting, because it is corresponded to the method disclosed in Example, so description is fairly simple, related part is referring to method part
Explanation.
Professional further appreciates that, with reference to the unit of each example of the embodiments described herein description
And algorithm steps, can be realized with electronic hardware, computer software or the combination of the two, in order to clearly demonstrate hardware and
The interchangeability of software, generally describes the composition and step of each example according to function in the above description.These
Function is performed with hardware or software mode actually, depending on the application-specific and design constraint of technical scheme.Specialty
Technical staff can realize described function to each specific application using distinct methods, but this realization should not
Think beyond the scope of this invention.
Directly it can be held with reference to the step of the method or algorithm that the embodiments described herein is described with hardware, processor
Capable software module, or the two combination are implemented.Software module can be placed in random access memory (RAM), internal memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of storage medium well known in field.
The method and device that file provided by the present invention is stored is described in detail above.Tool used herein
Body example is set forth to the principle and embodiment of the present invention, and the explanation of above example is only intended to help and understands this hair
Bright method and its core concept.It should be pointed out that for those skilled in the art, not departing from the present invention
On the premise of principle, some improvement and modification can also be carried out to the present invention, these are improved and modification also falls into right of the present invention
It is required that protection domain in.
Claims (6)
1. a kind of method of file storage, it is characterised in that including：
The directory attribute of file to be stored is detected, whether judge the directory attribute is the polymerization set previously according to application scenarios
Attribute；
If so, function is write in file opening polymerization, the file to be stored is stored to aggregate file, when in the aggregate file
When the residual memory space of OBJ objects is equal to or less than preset capacity, by multiple files to be stored in the OBJ objects
Batch write-in disk, the aggregate file includes multiple OBJ objects；
If it is not, the file to be stored is write into disk.
2. the method as described in claim 1, it is characterised in that in the directory attribute of the detection file to be stored, judge institute
Stating before whether directory attribute is the Aggregate attribute set previously according to application scenarios also includes：
Receive and instruction is set；
Instructed according to described set, the field value in current file extended attribute is set to default value, ought be above by described in
The directory attribute of part is set to Aggregate attribute.
3. method as claimed in claim 2, it is characterised in that the directory attribute of the detection file to be stored, judges described
Whether directory attribute is that the Aggregate attribute set previously according to application scenarios includes：
Whether the field value for judging the file to be stored is the default value；
If so, the directory attribute for determining the file to be stored is the Aggregate attribute；
If it is not, the directory attribute for determining the file to be stored is non-polymeric attribute.
4. a kind of device of file storage, it is characterised in that including：
Judge module, the directory attribute for detecting file to be stored, whether judge the directory attribute is previously according to application
The Aggregate attribute of scene settings；
Polymerize writing module, for if so, file opening polymerization write function, the file to be stored is stored to aggregate file, when
, will be many in the OBJ objects when residual memory space of OBJ objects in the aggregate file is equal to or less than preset capacity
The individual file to be stored batch write-in disk, the aggregate file includes multiple OBJ objects；
Non-polymeric writing module, for if it is not, the file to be stored is write into disk.
5. device as claimed in claim 4, it is characterised in that also include：
Receiving module, instruction is set for receiving；
Numerical value setup module, for being instructed according to described set, present count is set to by the field value in current file extended attribute
Value, Aggregate attribute is set to by the directory attribute of the current file.
6. device as claimed in claim 5, it is characterised in that the judge module includes：
Numerical value judging unit, for judging whether the field value of the file to be stored is the default value；
First determining unit, for if so, the directory attribute for determining the file to be stored is the Aggregate attribute；
Second determining unit, for if it is not, the directory attribute for determining the file to be stored is non-polymeric attribute.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710351825.1A CN107016130A (en) | 2017-05-18 | 2017-05-18 | A kind of method and device of file storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710351825.1A CN107016130A (en) | 2017-05-18 | 2017-05-18 | A kind of method and device of file storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107016130A true CN107016130A (en) | 2017-08-04 |
Family
ID=59449176
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710351825.1A Pending CN107016130A (en) | 2017-05-18 | 2017-05-18 | A kind of method and device of file storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107016130A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101567001A (en) * | 2009-05-22 | 2009-10-28 | 清华大学 | Method for managing metadata file layout of parallel file system |
| CN102075581A (en) * | 2011-01-25 | 2011-05-25 | 中国科学院计算技术研究所 | Data transmission method and device oriented to distributed file system |
| CN102708062A (en) * | 2012-04-10 | 2012-10-03 | 深圳市佳信捷电子有限公司 | Multimedia storage method and multimedia storage device based on data block |
| CN103577123A (en) * | 2013-11-12 | 2014-02-12 | 河海大学 | Small file optimization storage method based on HDFS |
| US9405997B1 (en) * | 2014-06-17 | 2016-08-02 | Amazon Technologies, Inc. | Optical character recognition |
| CN106603729A (en) * | 2017-02-23 | 2017-04-26 | 郑州云海信息技术有限公司 | Distributed-file-system multi-client synchronization method and system thereof |
-
2017
- 2017-05-18 CN CN201710351825.1A patent/CN107016130A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101567001A (en) * | 2009-05-22 | 2009-10-28 | 清华大学 | Method for managing metadata file layout of parallel file system |
| CN102075581A (en) * | 2011-01-25 | 2011-05-25 | 中国科学院计算技术研究所 | Data transmission method and device oriented to distributed file system |
| CN102708062A (en) * | 2012-04-10 | 2012-10-03 | 深圳市佳信捷电子有限公司 | Multimedia storage method and multimedia storage device based on data block |
| CN103577123A (en) * | 2013-11-12 | 2014-02-12 | 河海大学 | Small file optimization storage method based on HDFS |
| US9405997B1 (en) * | 2014-06-17 | 2016-08-02 | Amazon Technologies, Inc. | Optical character recognition |
| CN106603729A (en) * | 2017-02-23 | 2017-04-26 | 郑州云海信息技术有限公司 | Distributed-file-system multi-client synchronization method and system thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8595202B2 (en) | Method and system for intra-document object locking in collaborative authoring | |
| US8291186B2 (en) | Volume record data set optimization apparatus and method | |
| JP2008547074A (en) | Virtualized file system | |
| CN104268473B (en) | Method and device for detecting application programs | |
| CN104281535B (en) | A kind for the treatment of method and apparatus of mapping table in internal memory | |
| CN107562913A (en) | The date storage method and device of a kind of distributed file system | |
| CN108573019B (en) | Data migration method and device, electronic equipment and readable storage medium | |
| CN109597578A (en) | A kind of date storage method, system, equipment and computer readable storage medium | |
| CN109086462A (en) | The management method of metadata in a kind of distributed file system | |
| US7725507B1 (en) | Dynamic directories | |
| JPS63255748A (en) | Storage device | |
| CN110019527B (en) | Slave library reading method, related device and equipment | |
| CN109522273A (en) | A kind of method and device for realizing data write-in | |
| CN107016130A (en) | A kind of method and device of file storage | |
| CN110288340B (en) | Data set storage method based on Java intelligent contract | |
| CN106528876A (en) | Information processing method for distributed system and distributed information processing system | |
| US20120144136A1 (en) | Restoration of data from a backup storage volume | |
| US9116915B1 (en) | Incremental scan | |
| CN110222123A (en) | The method and electronic equipment that Hive dynamic partition for teaching platform loads | |
| CN109597587A (en) | A kind of method for writing data, medium and Nonvolatile memory | |
| US9552347B1 (en) | Data grid cell styling systems and methods | |
| US9547582B2 (en) | Selectable data on file viewing in a debugger | |
| CN109684235A (en) | A kind of method, device and equipment of computer system application cache | |
| Kiltz et al. | A new forensic model and its application to the collection, extraction and long term storage of screen content off a memory dump | |
| CN106469020B (en) | Cache element and control method and its application system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170804 |