CN107168802A - The merging method and device of a kind of cloud storage small file - Google Patents
The merging method and device of a kind of cloud storage small file Download PDFInfo
- Publication number
- CN107168802A CN107168802A CN201710352406.XA CN201710352406A CN107168802A CN 107168802 A CN107168802 A CN 107168802A CN 201710352406 A CN201710352406 A CN 201710352406A CN 107168802 A CN107168802 A CN 107168802A
- Authority
- CN
- China
- Prior art keywords
- sfq
- stored
- call number
- load
- signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of merging method of cloud storage small file and device, this method includes：Receive the queue full signal that SFQ is sent；Pass through the load of system load prediction algorithm computing system；Judge whether load is less than first threshold；If it is not, then the call number of the SFQ at the current time small documents stored is stored to a corresponding BQ；If so, then merging the corresponding small documents of call number of SFQ storages, or merge SFQ small documents corresponding with the call number that whole BQ is each stored；The present invention passes through when load is less than first threshold, merge the corresponding small documents of call number of SFQ storages, or merge SFQ small documents corresponding with the call number that whole BQ is each stored, can be in the case where system load be high, each self-corresponding small documents of whole call numbers for being stored SFQ or each BQ using sequential file technology are merged into one big file, realize the load balancing of system and improve systematic function.
Description
Technical field
The present invention relates to technical field of data storage, the merging method and dress of more particularly to a kind of cloud storage small file
Put.
Background technology
With the development of modern society's science and technology, cloud storage has been widely applied to as a kind of emerging Network storage technology
In the life of people.HDFS (Hadoop distributed file system) is a kind of distribution with Error Tolerance property
Formula file system model, can be deployed on the common machines or virtual machine for supporting JAVA running environment, using the teaching of the invention it is possible to provide height is handled up
The data access of amount, is especially suitable for disposing cloud storage platform.
HDFS is using master-slave architecture design pattern (master/slave architecture), a name node
(Name Node) and some back end (Data Node) constitute HDFS clusters.The design pole of HDFS this single name node
The earth simplifies the structure of file system, but has also therefore triggered the problem of HDFS small-file storage efficiency is low.
In the prior art, the main flow of the cloud storage small file storage efficiency of such as storage system based on HDFS is solved the problems, such as
Thought is to merge or be combined as big file by small documents, and method main at present is divided into 2 kinds, and one kind is to utilize Hadoop filings
Technologies such as (Hadoop archive, HAR) realizes the method that small documents merge, another, is proposed for specific application
File combined method.Both approaches are all based on the merging or combination of file, and to solve, small-file storage efficiency is not high to ask
Topic, but there is problems with both approaches：The load state of cloud storage system is not considered, and cloud storage is complete as one
System, while small-file storage efficiency is improved, it is also contemplated that the load state of system, because either file is closed
And still file is combined, to being all an extra operation for such as HDFS cloud storage system；And small documents are not closed
And scale is studied, i.e., not yet determine that a how many small documents merge into one big file and can be optimal systematic function.
Therefore, how according to the load state of cloud storage system, to reaching that certain scale small documents are merged, cloud storage system is realized
Load balancing, optimize small-file storage efficiency, be urgent problem now.
The content of the invention
It is an object of the invention to provide a kind of merging method of cloud storage small file and device, to use sequential file skill
Small documents are merged into big file by art, and realize the load balancing of system using system load prediction algorithm, are optimized small documents and are deposited
Store up efficiency.
In order to solve the above technical problems, the present invention provides a kind of merging method of cloud storage small file, including：
Receive the queue full signal that SFQ is sent；
Pass through the load of system load prediction algorithm computing system；
Judge whether the load is less than first threshold；
If it is not, then the call number of the SFQ at the current time small documents stored is stored to a corresponding BQ；Its
In, each BQ stores the SFQ at each self-corresponding moment call number；
If so, then merging the corresponding small documents of call number of the SFQ storages, or merge the SFQ and whole BQ each
The corresponding small documents of call number of storage.
Optionally, the call number for the small documents that the SFQ by current time is stored is stored to a corresponding BQ,
Including：
Judge whether the BQ for having stored call number reaches predetermined number；
If so, the small documents corresponding with the call number that whole BQ is each stored that then merge the SFQ；
If it is not, then the SFQ at the current time call numbers stored are stored to a corresponding BQ.
Optionally, before the queue full signal that the reception SFQ is sent, in addition to：
Based on sequential file technology, the call number for the small documents that user is uploaded is stored to the SFQ；
When the quantity of the call number of the SFQ storages reaches Second Threshold, the queue full signal is sent.
Optionally, the load by system load prediction algorithm computing system, including：
The load is calculated by the system load prediction algorithm based on analytic hierarchy process (AHP).
Optionally, the queue full signal that the reception SFQ is sent, including：
Recognize the signal received；Wherein, the signal include time that the queue full signal and timer send to
Signal；
If the signal is the queue full signal, perform described by the negative of system load prediction algorithm computing system
The step of load；
If the signal is the time to signal, passes through the system load prediction algorithm and calculate the load；
Judge whether the load is less than the first threshold；
If the load is less than the first threshold, merge the corresponding small documents of call number that whole BQ is each stored.
Optionally, before the signal that the identification is received, in addition to：
The timer is spaced the transmission time to signal at preset timed intervals.
In addition, present invention also offers a kind of merging device of cloud storage small file, including：
Receiving module, the queue full signal for receiving SFQ transmissions；
Computing module, for the load by system load prediction algorithm computing system；
Judge module, for judging whether the load is less than first threshold；
Memory module, for when the load is not less than the first threshold, the SFQ at current time to be stored
The call number of small documents is stored to a corresponding BQ；Wherein, each BQ stores the SFQ's at each self-corresponding moment
Call number；
Merging module, for when the load is less than the first threshold, merging the call number correspondence of the SFQ storages
Small documents, or merge SFQ small documents corresponding with the call number that whole BQ is each stored.
Optionally, the memory module, including：
Whether the first judging submodule, the BQ for judging to have stored call number reaches predetermined number；
First merges submodule, for when the BQ for having stored call number reaches the predetermined number, merge the SFQ and
The corresponding small documents of call number that whole BQ are each stored；
Sub-module stored, for when the BQ for having stored call number is not up to the predetermined number, by the institute at current time
The call number for stating SFQ storages is stored to a corresponding BQ.
Optionally, the device also includes：
Classification memory module, for based on sequential file technology, the call number for the small documents that user is uploaded to be stored
To the SFQ；
Sending module, when the quantity for the SFQ call numbers stored reaches Second Threshold, sends the queue
Full signal.
Optionally, the receiving module, including：
Submodule is recognized, for recognizing the signal received；If the signal is the queue full signal, calculated to described
Module sends the first enabling signal；If the signal is the time to signal, sends second to calculating sub module and start letter
Number；
The calculating sub module, for calculating the load by the system load prediction algorithm；
Second judging submodule, for judging whether the load is less than the first threshold；
Second merges submodule, for when the load is less than the first threshold, merging what whole BQ was each stored
The corresponding small documents of call number.
A kind of merging method of cloud storage small file provided by the present invention, including：Receive the queue full letter that SFQ is sent
Number；Pass through the load of system load prediction algorithm computing system；Judge whether the load is less than first threshold；If it is not, then will
The call number of the small documents of the SFQ storages at current time is stored to a corresponding BQ；Wherein, each BQ storages are each right
The SFQ at the moment answered call number；If so, then merging the corresponding small documents of call number of the SFQ storages, or close
And SFQ small documents corresponding with the call number that whole BQ is each stored；
It can be seen that, the present invention is by receiving the queue full signal that SFQ is sent, it is possible to use SFQ (Sequence File
Queue, sequential file queue) storage sequential file technology under small documents call number；Pass through system load prediction algorithm meter
The load of calculation system, can be calculated the load state of cloud storage system；By when load is less than first threshold, merging
The corresponding small documents of call number of SFQ storages, or merge SFQ and whole BQ (Backup queue, standby queue) each storages
The corresponding small documents of call number, can be in the case where system load be high, using sequential file technology by SFQ or each BQ
Each self-corresponding small documents of whole call numbers of storage are merged into one big file, realize the load balancing of system and improve
Systematic function, optimizes small-file storage efficiency, improves Consumer's Experience.In addition, present invention also offers small text in cloud storage
The merging device of part, equally with above-mentioned beneficial effect.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is the accompanying drawing used required in technology description to be briefly described, it should be apparent that, drawings in the following description are only this
The embodiment of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can also basis
The accompanying drawing of offer obtains other accompanying drawings.
A kind of flow chart of the merging method for cloud storage small file that Fig. 1 is provided by the embodiment of the present invention；
The flow chart of the merging method for another cloud storage small file that Fig. 2 is provided by the embodiment of the present invention；
The handling process signal of the merging method for another cloud storage small file that Fig. 3 is provided by the embodiment of the present invention
Figure；
A kind of structure chart of the merging device for cloud storage small file that Fig. 4 is provided by the embodiment of the present invention.
Embodiment
To make the purpose, technical scheme and advantage of the embodiment of the present invention clearer, below in conjunction with the embodiment of the present invention
In accompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is
A part of embodiment of the present invention, rather than whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art
The every other embodiment obtained under the premise of creative work is not made, belongs to the scope of protection of the invention.
It refer to Fig. 1, a kind of flow of the merging method for cloud storage small file that Fig. 1 is provided by the embodiment of the present invention
Figure.This method can include：
Step 101：Receive the queue full signal that SFQ is sent.
Wherein, when queue full signal can reach annexable threshold value for the quantity of the call number of the SFQ small documents stored
The signal of transmission.For the particular content of queue full signal, that is, the content that SFQ is sent, can voluntarily it be set by designer
Put, the present embodiment does not do any limitation to this.
It is understood that what the method that the present embodiment is provided can be merged with control small documents in cloud storage system
Method performed by controller, can also add other moulds in cloud storage system on the basis of the method that the present embodiment is provided
The step of block is performed, such as receiving module are based on sequential file technology, the call numbers of the small documents that user is uploaded store to SFQ and
SFQ modules are when the quantity of the SFQ call numbers stored reaches Second Threshold, the full signal of transmit queue.The present embodiment is not done to this
Any limitation.
It should be noted that the method that the present embodiment is provided can expire signal to small documents for controller receiving queue
Merge, so as to reach the purpose of optimization small-file storage efficiency；Can also be receiving queue completely signal and time to signal
Small documents are merged, that is, also include being spaced at preset timed intervals to controller in cloud storage system sending the time to signal
Timer so that further improve cloud storage system load balancing.The present embodiment does not do any limitation to this.
Step 102：Pass through the load of system load prediction algorithm computing system.
Wherein, system load prediction algorithm can be based on analytic hierarchy process (AHP) (analytic hierarchy process,
AHP system load prediction algorithm), or other algorithms, as long as system load prediction algorithm computing system can be passed through
Load, for the particular content and type of system load prediction algorithm, the present embodiment does not do any limitation.
As long as it is understood that the load of cloud storage system can be calculated, for passing through system load prediction algorithm
The concrete mode of the load of cloud storage system is calculated, the present embodiment does not do any limitation.
Step 103：Judge whether load is less than first threshold；If it is not, then entering step 104；If so, then entering step
105。
Wherein, the numerical value that first threshold can voluntarily be set for designer, when cloud storage system load reaches the numerical value
When, it was demonstrated that the load of cloud storage is higher.For the setting of the concrete numerical value of first threshold, the present embodiment does not do any limitation.
Step 104：The call number of the SFQ at the current time small documents stored is stored to a corresponding BQ；Wherein, often
Individual BQ stores the SFQ at each self-corresponding moment call number.
Wherein, the purpose of this step can be, when the load of system reaches first threshold, the SFQ at current time to be stored
Call number be put into a BQ, it is to avoid when system load is higher, carry out small documents merging the step of.
It is understood that a SFQ and multiple BQ can be set up in cloud storage system for each user, each BQ storages
Be not in the same time SFQ storage multiple small documents call number.It is many that the SFQ or each BQ of the full signal of transmission storage are stored
The corresponding multiple small documents of individual call number can be to merge into one big file.
Can be by SFQ it should be noted that storing the call number that SFQ is stored to a corresponding BQ in this step
The call number of storage stores the BQ for not storing call number to one established, or the call number for storing SFQ is stored
To a newly-built BQ, the present embodiment is unrestricted to this.
It is preferred that, in order to avoid the BQ quantity in system is excessive, this step can include the BQ for judging to have stored call number
Whether predetermined number is reached；If the BQ for having stored call number reaches predetermined number, merge the rope that SFQ and whole BQ are each stored
The corresponding small documents of quotation marks；If having stored the BQ of call number to reach predetermined number, the index that the SFQ at current time is stored
Number store to a corresponding BQ.The present embodiment is unrestricted to this.
Step 105：Merge the corresponding small documents of call number of SFQ storages, or merge the rope that SFQ and whole BQ are each stored
The corresponding small documents of quotation marks.
Wherein, the step of this step can be to merge small documents, can be according to having stored call number in system
BQ correspondences merge small documents, if having stored the BQ of call number in system, the call number pair that can be stored SFQ
The small documents answered are merged into one big file；If having stored the BQ of call number in system, call number will have each been stored
The corresponding small documents of call number that each store of BQ be merged into one big file, and the call number that SFQ is stored is corresponding small
Piece file mergence is into one big file.
It is understood that the concrete mode for merging small documents in this step, can be the call number for storing SFQ
The call number of one big file is merged into, and the content of the corresponding small documents of each call number is merged into one big file
Hold, or other modes, as long as the SFQ or each BQ corresponding small documents of call number stored can be merged into one big
File, merges mode, the present embodiment does not do any limitation for specific.
In the present embodiment, the embodiment of the present invention is by receiving the queue full signal that SFQ is sent, it is possible to use SFQ stores sequence
The call number of small documents under row File Technology；, can be to cloud storage by the load of system load prediction algorithm computing system
The load state of system is calculated；By when load is less than first threshold, merging the corresponding small text of call number of SFQ storages
Part, or merge corresponding with the call number that whole BQ is each the stored small documents of SFQ, can in the case where system load is not high,
Each self-corresponding small documents of whole call numbers for being stored SFQ or each BQ using sequential file technology are merged into one big text
Part, realizes the load balancing of system and improves systematic function, optimize small-file storage efficiency, improve Consumer's Experience.
It refer to Fig. 2 and Fig. 3, the merging side for another cloud storage small file that Fig. 2 is provided by the embodiment of the present invention
The flow chart of method；The handling process of the merging method for another cloud storage small file that Fig. 3 is provided by the embodiment of the present invention
Schematic diagram.This method can include：
Step 201：Based on sequential file technology, the call number for the small documents that user is uploaded is stored to SFQ.
Wherein, sequential file (Sequence File) is a kind of binary file technology that HDFS is provided, this binary system
File directly will<Key, value>File is arrived to serializing, file can realize the compression based on record or data block when serializing.
In cloud storage system, big file is merged into by small documents using Sequence File technologies to binary file, with small documents
Call number be key, content is merged for value form, and the compression based on data block is realized while merging, so,
The disk space of back end is also saved while name node memory headroom is saved.
Specifically, in cloud storage system, the handling process for the method that the present embodiment is provided can be as shown in Figure 3.For
The treatment effeciency to small documents is improved, cloud storage system is that each user can establish 3 kinds of queues:1st kind can be sequence
Document queue (Sequence File queue, SFQ), the 2nd kind can be sequential file operation queue (Sequence File
Operation queue, SFOQ), the 3rd kind can be standby queue (Backup queue, BQ).Wherein, SFQ is used for small documents
Merging, SFOQ is used for operation to small documents after merging, and BQ is used for feelings of the small documents number more than SFQ or SFOQ length operated
Condition.The length of 3 kinds of queues can be consistent, and the number of call number is stored for the concrete numerical value of queue length, that is, each queue
Amount can voluntarily be set by designer according to practical scene and user's request, and the present embodiment does not do any limitation to this.
It should be noted that this step can be with as shown in figure 3, the server of cloud storage system receives the local of user's upload
File (process 1), the type to this document is judged (process 2), if small documents, then is deposited the call number of this document
Storage is into SFQ (process 3).For the detailed process in this step, the concrete mode such as judged file type, can use with
The same or analogous mode of prior art, can also use other modes, as long as can be by the call number of the small documents received
Store into the corresponding SFQ of each user, for specific storing process, the present embodiment does not do any limitation.
Step 202：When the quantity of the call number of SFQ storages reaches Second Threshold, the full signal of transmit queue.
Wherein, Second Threshold can be the quantity for the small documents for being merged into one big file, for the specific number of the quantity
The setting of value, can voluntarily be set, the present embodiment is to this not by any limit by designer according to practical scene and user's request
System.
Specifically, this step can be as shown in dotted line a in Fig. 3, when the quantity of the SFQ call numbers stored reaches Second Threshold
When, that is, SFQ it is full when, to the full signal (QF signals) of controller transmit queue.
Step 203：Timer is spaced the transmission time to signal at preset timed intervals.
Wherein, prefixed time interval can be that designer judges whether conjunction according to what practical scene or user's request were set
And the time interval of small documents, such as 30min.The present embodiment does not do any limitation to this.
Specifically, this step can as shown in dotted line b in Fig. 3, when timer reaches the time point of prefixed time interval,
The time is sent to signal (TU signals) to controller.
Step 204：Recognize the signal received.
It is understood that this step can recognize the signal received for controller, for specific identification process and knowledge
Other mode, can voluntarily be set, the present embodiment is unrestricted to this by designer.
Step 205：When signal is queue full signal, pass through the load of system load prediction algorithm computing system.
Step 206：Judge whether load is less than first threshold；If it is not, then entering step 207；If so, then entering step
210。
Wherein, step 205 and step 206 are similar to step 102 and step 103, will not be repeated here.
It is understood that this step can be with as shown in figure 3, controller be calculated system load (process 4.2), certainly
The fixed merging (process 5) for whether carrying out small documents.
Step 207：Judge whether the BQ for having stored call number reaches predetermined number；If so, then entering step 208；If it is not,
Then enter step 209.
It is understood that the quantity of the BQ in order to avoid having stored call number in system is excessive, can be by this step
Whether the quantity of the BQ to having stored call number in system reaches that predetermined number judges, then by step 208, is depositing
When the quantity for storing up the BQ of call number reaches predetermined number, merge SFQ small documents corresponding with the call number that whole BQ is each stored.
It should be noted that the setting of the numerical value for predetermined number, can be by designer according to practical scene and use
Family demand is voluntarily set, and the present embodiment is unrestricted to this.
Specifically, this step can be with as shown in figure 3, controller reading SFQ and BQ relevant information (process 4.1), be determined
Whether the merging (process 5) of small documents is carried out.
Step 208：Merge SFQ small documents corresponding with the call number that whole BQ is each stored.
It is understood that the purpose of this step is by cloud storage system, the whole that the user uploads is annexable small
Piece file mergence is into corresponding big file.
Step 209：The SFQ at the current time call numbers stored are stored to a corresponding BQ.
Wherein, the purpose of this step be system load higher and system in BQ quantity it is few when, by this step
Avoid the merging to small documents.
Step 210：Merge the corresponding small documents of call number of SFQ storages, or merge the rope that SFQ and whole BQ are each stored
The corresponding small documents of quotation marks.
Wherein, this step is similar to step 105, will not be repeated here.
Step 211：When signal be the time to signal when, pass through system load prediction algorithm computational load.
Step 212：Judge whether load is less than first threshold；If so, then entering step 213；If it is not, then terminating this stream
Journey.
Wherein, step 211 and step 212 are similar to step 102 and step 103, will not be repeated here.
Step 213：Merge the corresponding small documents of call number that whole BQ is each stored.
It is understood that the purpose of this step be time for being sent by timer to signal, the index stored in SFQ
Number not up to the corresponding small documents of BQ that call number has been stored in system are closed by Second Threshold and during not high system load
And, it is further ensured that the load balancing of system.
In the present embodiment, whether the present invention reaches predetermined number by the BQ for judging to have stored call number, can avoid cloud
The quantity that the BQ of call number has been stored in storage system is excessive；By when signal be the time to signal when, it is pre- by system load
Method of determining and calculating computational load, can SFQ store call number be not up to Second Threshold and not high system load when, in system
The corresponding small documents of BQ of storage call number are merged, and further ensure the load balancing of system.
It refer to Fig. 4, a kind of structure of the merging device for cloud storage small file that Fig. 4 is provided by the embodiment of the present invention
Figure.The device can include：
Receiving module 100, the queue full signal for receiving SFQ transmissions；
Computing module 200, for the load by system load prediction algorithm computing system；
Judge module 300, for judging whether load is less than first threshold；
Memory module 400, for when load is not less than first threshold, by the SFQ at the current time small documents stored
Call number is stored to a corresponding BQ；Wherein, each BQ stores the SFQ at each self-corresponding moment call number；
Merging module 500, for when load is less than first threshold, merging the corresponding small documents of call number of SFQ storages,
Or merge SFQ small documents corresponding with the call number that whole BQ is each stored.
Optionally, memory module 400, can include：
Whether the first judging submodule, the BQ for judging to have stored call number reaches predetermined number；
First merges submodule, for when the BQ for having stored call number reaches predetermined number, merging SFQ and whole BQ each
From the corresponding small documents of the call number of storage；
Sub-module stored, for when the BQ for having stored call number is not up to predetermined number, the SFQ at current time to be stored
Call number store to a corresponding BQ.
Optionally, the device can also include：
Classify memory module, for based on sequential file technology, the call numbers of the small documents that user is uploaded store to
SFQ；
Sending module, when the quantity for the SFQ call numbers stored reaches Second Threshold, the full signal of transmit queue.
Optionally, receiving module 100, can include：
Submodule is recognized, for recognizing the signal received；If signal is queue full signal, first is sent to computing module
Enabling signal；If signal is the time to signal, the second enabling signal is sent to calculating sub module；
Calculating sub module, for passing through system load prediction algorithm computational load；
Second judging submodule, for judging whether load is less than first threshold；
Second merges submodule, for when load is less than first threshold, merging the call number pair that whole BQ is each stored
The small documents answered.
In the present embodiment, the present invention receives the queue full signal that SFQ is sent by receiving module 100, it is possible to use SFQ is deposited
Store up the call number of the small documents under sequential file technology；System load prediction algorithm computing system is utilized by computing module 200
Load, the load state of cloud storage system can be calculated；By merging module 500 when load is less than first threshold
When, the corresponding small documents of call number of merging SFQ storages, or the call number that merging SFQ is each stored with whole BQ are corresponding small
File, can be in the case where system load be high, the whole call numbers for being stored SFQ or each BQ using sequential file technology
Each self-corresponding small documents are merged into one big file, realize the load balancing of system and improve systematic function, optimize
Small-file storage efficiency, improves Consumer's Experience.
The embodiment of each in specification is described by the way of progressive, and what each embodiment was stressed is and other realities
Apply the difference of example, between each embodiment identical similar portion mutually referring to.For device disclosed in embodiment
Speech, because it is corresponded to the method disclosed in Example, so description is fairly simple, related part is referring to method part illustration
.
Professional further appreciates that, with reference to the unit of each example of the embodiments described herein description
And algorithm steps, can be realized with electronic hardware, computer software or the combination of the two, in order to clearly demonstrate hardware and
The interchangeability of software, generally describes the composition and step of each example according to function in the above description.These
Function is performed with hardware or software mode actually, depending on the application-specific and design constraint of technical scheme.Specialty
Technical staff can realize described function to each specific application using distinct methods, but this realization should not
Think beyond the scope of this invention.
Directly it can be held with reference to the step of the method or algorithm that the embodiments described herein is described with hardware, processor
Capable software module, or the two combination are implemented.Software module can be placed in random access memory (RAM), internal memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of storage medium well known in field.
The merging method and device to cloud storage small file provided by the present invention are described in detail above.Herein
In apply specific case the principle and embodiment of the present invention be set forth, the explanation of above example is only intended to side
The method and its core concept of the assistant solution present invention.It should be pointed out that for those skilled in the art, not
On the premise of departing from the principle of the invention, some improvement and modification can also be carried out to the present invention, these are improved and modification is also fallen into
In the protection domain of the claims in the present invention.
Claims (10)
1. a kind of merging method of cloud storage small file, it is characterised in that including：
Receive the queue full signal that SFQ is sent；
Pass through the load of system load prediction algorithm computing system；
Judge whether the load is less than first threshold；
If it is not, then the call number of the SFQ at the current time small documents stored is stored to a corresponding BQ；Wherein, often
Individual BQ stores the SFQ at each self-corresponding moment call number；
If so, then merge the corresponding small documents of call number of the SFQ storages, or the merging SFQ and whole BQ is each stored
The corresponding small documents of call number.
2. the merging method of cloud storage small file according to claim 1, it is characterised in that described by current time
The call number of the small documents of the SFQ storages is stored to a corresponding BQ, including：
Judge whether the BQ for having stored call number reaches predetermined number；
If so, the small documents corresponding with the call number that whole BQ is each stored that then merge the SFQ；
If it is not, then the SFQ at the current time call numbers stored are stored to a corresponding BQ.
3. the merging method of cloud storage small file according to claim 2, it is characterised in that the reception SFQ is sent
Queue full signal before, in addition to：
Based on sequential file technology, the call number for the small documents that user is uploaded is stored to the SFQ；
When the quantity of the call number of the SFQ storages reaches Second Threshold, the queue full signal is sent.
4. the merging method of cloud storage small file according to claim 3, it is characterised in that described to pass through system load
The load of prediction algorithm computing system, including：
The load is calculated by the system load prediction algorithm based on analytic hierarchy process (AHP).
5. the merging method of the cloud storage small file according to Claims 1-4, it is characterised in that the reception SFQ hairs
The queue full signal sent, including：
Recognize the signal received；Wherein, the signal includes time that the queue full signal and timer send to signal；
If the signal is the queue full signal, the load by system load prediction algorithm computing system is performed
Step；
If the signal is the time to signal, passes through the system load prediction algorithm and calculate the load；
Judge whether the load is less than the first threshold；
If the load is less than the first threshold, merge the corresponding small documents of call number that whole BQ is each stored.
6. the merging method of cloud storage small file according to claim 5, it is characterised in that what the identification was received
Before signal, in addition to：
The timer is spaced the transmission time to signal at preset timed intervals.
7. a kind of merging device of cloud storage small file, it is characterised in that including：
Receiving module, the queue full signal for receiving SFQ transmissions；
Computing module, for the load by system load prediction algorithm computing system；
Judge module, for judging whether the load is less than first threshold；
Memory module, for when the load is not less than the first threshold, by the small text of the SFQ storages at current time
The call number of part is stored to a corresponding BQ；Wherein, each BQ stores the SFQ at each self-corresponding moment index
Number；
Merging module, for when the load is less than the first threshold, the call number for merging the SFQ storages to be corresponding small
File, or merge SFQ small documents corresponding with the call number that whole BQ is each stored.
8. the merging device of cloud storage small file according to claim 7, it is characterised in that the memory module, bag
Include：
Whether the first judging submodule, the BQ for judging to have stored call number reaches predetermined number；
First merges submodule, for when the BQ for having stored call number reaches the predetermined number, merging the SFQ and whole
The corresponding small documents of call number that BQ is each stored；
Sub-module stored, for when the BQ for having stored call number is not up to the predetermined number, by the SFQ at current time
The call number of storage is stored to a corresponding BQ.
9. the merging device of cloud storage small file according to claim 8, it is characterised in that also include：
Classification memory module, for based on sequential file technology, the call number for the small documents that user is uploaded to be stored to institute
State SFQ；
Sending module, when the quantity for the SFQ call numbers stored reaches Second Threshold, sends the queue full letter
Number.
10. the merging method of the cloud storage small file according to claim 7 to 9, it is characterised in that the reception mould
Block, including：
Submodule is recognized, for recognizing the signal received；If the signal is the queue full signal, to the computing module
Send the first enabling signal；If the signal is the time to signal, the second enabling signal is sent to calculating sub module；
The calculating sub module, for calculating the load by the system load prediction algorithm；
Second judging submodule, for judging whether the load is less than the first threshold；
Second merges submodule, for when the load is less than the first threshold, merging the index that whole BQ is each stored
Number corresponding small documents.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710352406.XA CN107168802A (en) | 2017-05-18 | 2017-05-18 | The merging method and device of a kind of cloud storage small file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710352406.XA CN107168802A (en) | 2017-05-18 | 2017-05-18 | The merging method and device of a kind of cloud storage small file |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107168802A true CN107168802A (en) | 2017-09-15 |
Family
ID=59816215
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710352406.XA Pending CN107168802A (en) | 2017-05-18 | 2017-05-18 | The merging method and device of a kind of cloud storage small file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107168802A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679177A (en) * | 2017-09-29 | 2018-02-09 | 郑州云海信息技术有限公司 | A kind of small documents storage optimization method based on HDFS, device, equipment |
| CN108595567A (en) * | 2018-04-13 | 2018-09-28 | 郑州云海信息技术有限公司 | A kind of merging method of small documents, device, equipment and readable storage medium storing program for executing |
| CN109766318A (en) * | 2018-12-17 | 2019-05-17 | 新华三大数据技术有限公司 | File reading and device |
| CN110968272A (en) * | 2019-12-16 | 2020-04-07 | 华中科技大学 | Time sequence prediction-based method and system for optimizing storage performance of mass small files |
| CN112486697A (en) * | 2020-11-16 | 2021-03-12 | 杭州电魂网络科技股份有限公司 | Data transmission method, system, electronic device and storage medium |
| CN113127548A (en) * | 2019-12-31 | 2021-07-16 | 奇安信科技集团股份有限公司 | File merging method, device, equipment and storage medium |
| CN113722274A (en) * | 2021-08-09 | 2021-11-30 | 河南农业大学 | Efficient R-tree index remote sensing data storage model |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110295855A1 (en) * | 2010-05-31 | 2011-12-01 | Microsoft Corporation | Graph-Processing Techniques for a MapReduce Engine |
| CN103678579A (en) * | 2013-12-12 | 2014-03-26 | 浪潮电子信息产业股份有限公司 | Optimizing method for small-file storage efficiency |
| CN105183371A (en) * | 2015-08-14 | 2015-12-23 | 山东大学 | Migration balancing policy based electricity-consuming information distributed file storage method and apparatus |
| CN105243109A (en) * | 2015-09-25 | 2016-01-13 | 杭州华为数字技术有限公司 | Data backup method and data processing system |
| CN106599184A (en) * | 2016-12-13 | 2017-04-26 | 西北师范大学 | Hadoop system optimization method |
-
2017
- 2017-05-18 CN CN201710352406.XA patent/CN107168802A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110295855A1 (en) * | 2010-05-31 | 2011-12-01 | Microsoft Corporation | Graph-Processing Techniques for a MapReduce Engine |
| CN103678579A (en) * | 2013-12-12 | 2014-03-26 | 浪潮电子信息产业股份有限公司 | Optimizing method for small-file storage efficiency |
| CN105183371A (en) * | 2015-08-14 | 2015-12-23 | 山东大学 | Migration balancing policy based electricity-consuming information distributed file storage method and apparatus |
| CN105243109A (en) * | 2015-09-25 | 2016-01-13 | 杭州华为数字技术有限公司 | Data backup method and data processing system |
| CN106599184A (en) * | 2016-12-13 | 2017-04-26 | 西北师范大学 | Hadoop system optimization method |
Non-Patent Citations (1)
| Title |
|---|
| 余思 等: "一种提高云存储中小文件存储效率的方案", 《西安交通大学学报》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679177A (en) * | 2017-09-29 | 2018-02-09 | 郑州云海信息技术有限公司 | A kind of small documents storage optimization method based on HDFS, device, equipment |
| CN108595567A (en) * | 2018-04-13 | 2018-09-28 | 郑州云海信息技术有限公司 | A kind of merging method of small documents, device, equipment and readable storage medium storing program for executing |
| CN109766318A (en) * | 2018-12-17 | 2019-05-17 | 新华三大数据技术有限公司 | File reading and device |
| CN109766318B (en) * | 2018-12-17 | 2021-03-02 | 新华三大数据技术有限公司 | File reading method and device |
| CN110968272A (en) * | 2019-12-16 | 2020-04-07 | 华中科技大学 | Time sequence prediction-based method and system for optimizing storage performance of mass small files |
| CN113127548A (en) * | 2019-12-31 | 2021-07-16 | 奇安信科技集团股份有限公司 | File merging method, device, equipment and storage medium |
| CN113127548B (en) * | 2019-12-31 | 2023-10-31 | 奇安信科技集团股份有限公司 | File merging method, device, equipment and storage medium |
| CN112486697A (en) * | 2020-11-16 | 2021-03-12 | 杭州电魂网络科技股份有限公司 | Data transmission method, system, electronic device and storage medium |
| CN113722274A (en) * | 2021-08-09 | 2021-11-30 | 河南农业大学 | Efficient R-tree index remote sensing data storage model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107168802A (en) | The merging method and device of a kind of cloud storage small file | |
| WO2019128547A1 (en) | Neural network model training method and apparatus | |
| CN107045422A (en) | Distributed storage method and equipment | |
| CN108089814A (en) | A kind of date storage method and device | |
| CN109144791A (en) | Data conversion storage method, apparatus and data management server | |
| CN109213792A (en) | Method, server-side, client, device and the readable storage medium storing program for executing of data processing | |
| CN108062243A (en) | Generation method, task executing method and the device of executive plan | |
| CN103218260A (en) | Virtual machine migration method and device | |
| CN107948084B (en) | Current limiting method and device | |
| CN110928950A (en) | Block chain transaction information management method and device, computer equipment and storage medium | |
| CN111369009A (en) | Distributed machine learning method capable of tolerating untrusted nodes | |
| CN107589990A (en) | A kind of method and system of the data communication based on thread pool | |
| CN104348852B (en) | A kind of method, apparatus and system for realizing telecommunication capability mass-sending | |
| CN109743202A (en) | Management method, device, equipment and the readable storage medium storing program for executing of data | |
| CN107453948A (en) | The storage method and system of a kind of network measurement data | |
| CN116720132A (en) | Power service identification system, method, device, medium and product | |
| EP2982133B1 (en) | Method and system for managing data | |
| CN104866375B (en) | A kind of method and device for migrating virtual machine | |
| CN102480502A (en) | I/O load equilibrium method and I/O server | |
| CN108090186A (en) | A kind of electric power data De-weight method on big data platform | |
| CN109118361A (en) | Quota control method, apparatus and system | |
| CN106982126A (en) | A kind of resource-sharing charging method and message accounting, memory bank | |
| CN104050100B (en) | A kind of data flow memory management method and system suitable for big data environment | |
| US20200234180A1 (en) | Cognitive machine learning for semantic network | |
| CN110351189B (en) | Routing control method facing real-time edge calculation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170915 |