CN108595567A - A kind of merging method of small documents, device, equipment and readable storage medium storing program for executing - Google Patents
A kind of merging method of small documents, device, equipment and readable storage medium storing program for executing Download PDFInfo
- Publication number
- CN108595567A CN108595567A CN201810331806.7A CN201810331806A CN108595567A CN 108595567 A CN108595567 A CN 108595567A CN 201810331806 A CN201810331806 A CN 201810331806A CN 108595567 A CN108595567 A CN 108595567A
- Authority
- CN
- China
- Prior art keywords
- small documents
- load
- threshold
- merging
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses a kind of merging methods of small documents, can be by judging whether the load value of system judges less than preset load threshold the working condition of system when the quantity of the small documents of acquisition reaches preset first threshold.When the load value of system is less than load threshold, mean that the currently performed business of system can't consume many resources, whole system also has surplus energy to merge the small documents of acquisition at this time, at this time can't impact the merging of the small documents of acquisition to the business being carrying out in system.By the judgement to system load value size it is possible to prevente effectively from when merging small documents to system in the business that is carrying out impact.Merge device, a kind of merging equipment of small documents and a kind of computer readable storage medium the invention also discloses a kind of small documents, equally there is above-mentioned advantageous effect.
Description
Technical field
The present invention relates to file field of storage, more particularly to a kind of merging method of small documents, device, equipment and readable
Storage medium.
Background technology
As the continuous progress of science and technology, application of the distributed memory system in life are more and more extensive in recent years.Dividing
In cloth storage system, it will usually HDFS (distributed file system) be set, the file to store is managed.
Single name node is only existed in HDFS under normal conditions, but stores a large amount of small text within the storage system
When part, usually there is a problem of that storage efficiency is low.The small documents refer to the file size file smaller than data block capacity, are led to
It is often that file size is known as small documents less than the file of 64MB.
At this stage, mainly by the way that a large amount of small documents to be merged to or are combined into big file, small documents are solved with this
The low problem of storage efficiency.
But in the prior art, during small documents are merged into big file, it will usually to distributed storage system
The business being carrying out in system impacts, and the speed of service for being carrying out business in entire distributed memory system can be caused to become
Slowly, there is the problems such as interim card.
Invention content
The object of the present invention is to provide a kind of merging methods of small documents, it is possible to prevente effectively from when merging small documents to being
Influence caused by current business of uniting；Another object of the present invention is to provide a kind of merging device of small documents, a kind of small texts
The merging equipment of part and a kind of computer readable storage medium, it is possible to prevente effectively from when merging small documents to system current business
Caused influence.
In order to solve the above technical problems, the present invention provides a kind of merging method of small documents, the method includes：
Obtain the small documents that user sends；
When the number of the small documents reaches preset first threshold, it is preset to judge whether the load value of system is less than
Load threshold；
If so, the small documents of acquisition are merged.
Optionally, after the small documents that the acquisition user sends, the method further includes：
The corresponding identification information of the small documents is added to document queue；Wherein, when the number of the small documents is more than
When the length of the document queue, the identification information is added to the document queue and at least one spare queue successively；
It is described when the number of the small documents reaches preset first threshold, it is pre- to judge whether the load value of system is less than
If load threshold include：
When the identification information in the document queue is booked, judge whether the load value is less than the load threshold.
Optionally, after whether the load value for judging system is less than preset load threshold, the method is also wrapped
It includes：
If it is not, then judging whether the quantity of the spare queue is more than preset second threshold；If so, by the institute of acquisition
Small documents are stated to merge.
Optionally, after whether the quantity for judging the spare queue is more than preset second threshold, the side
Method further includes：
If it is not, then create a spare queue, and the identification information in the document queue is transferred to described newly-built standby
Use queue；
S308：It controls timer and carries out timing；
S309：When reaching preset time threshold between timer institute timing, judge that the load value of the system is
It is no to be less than the load threshold；If so, executing S310；If it is not, then re-executing S308；
S310：The small documents of acquisition are merged.
Optionally, it is described by the small documents of acquisition merge including：
The small documents are serialized, to merge the small documents.
Optionally, the method for calculating the load value includes：
The load attribute value of multigroup system is obtained according to the sequencing of time；
The corresponding decision attribute values of multigroup load attribute value are calculated separately out by AHP；
According to multiple decision attribute values, the load value is calculated by AHP.
Optionally, the identification information is the call number of the small documents.
The present invention also provides a kind of merging device of small documents, described device includes：
Acquisition module：Small documents for obtaining user's transmission；
First judgment module：For when the number of the small documents reaches preset first threshold, judging the negative of system
Whether load value is less than preset load threshold；
Merging module：For when the judging result of first judgment module is to be, the small documents of acquisition to be closed
And.
The present invention also provides a kind of merging equipment of small documents, including：
Memory：For storing computer program；
Processor：The merging method of small documents as described in any of the above-described being realized when for executing the computer program
Step.
The present invention also provides a kind of computer readable storage medium, meter is stored on the computer readable storage medium
Calculation machine program realizes the step of the merging method of small documents as described in any of the above-described when the computer program is executed by processor
Suddenly.
The merging method of a kind of small documents provided by the present invention, when the quantity of the small documents of acquisition reaches preset first
It, can be by judging whether the load value of system judges less than preset load threshold the working condition of system when threshold value.When
When the load value of system is less than load threshold, it is meant that the currently performed business of system can't consume many resources, whole at this time
A system also has surplus energy to merge the small documents of acquisition, at this time by the small documents of acquisition merge can't in system just
It is impacted in the business of execution.By the judgement to system load value size it is possible to prevente effectively from when merging small documents to being
The business being carrying out in system impacts.
The present invention also provides a kind of merging device of small documents, a kind of merging equipment of small documents and a kind of computers can
Storage medium is read, equally there is above-mentioned advantageous effect, no longer repeated herein.
Description of the drawings
It, below will be to embodiment or existing for the clearer technical solution for illustrating the embodiment of the present invention or the prior art
Attached drawing is briefly described needed in technology description, it should be apparent that, the accompanying drawings in the following description is only this hair
Some bright embodiments for those of ordinary skill in the art without creative efforts, can be with root
Other attached drawings are obtained according to these attached drawings.
A kind of flow chart for small documents merging method that Fig. 1 is provided by the embodiment of the present invention；
The flow chart for another small documents merging method that Fig. 2 is provided by the embodiment of the present invention；
The flow chart for another small documents merging method that Fig. 3 is provided by the embodiment of the present invention；
A kind of flow chart for specific small documents merging method that Fig. 4 is provided by the embodiment of the present invention；
Fig. 5 is merged the structure diagram of device by a kind of small documents that the embodiment of the present invention provides；
Fig. 6 is merged the structure diagram of equipment by a kind of small documents that the embodiment of the present invention provides.
Specific implementation mode
Core of the invention is to provide a kind of merging method of small documents.In the prior art, small when what is stored in system
When the quantity of file reaches certain number, the small documents of acquisition will be merged into big file, the storage to improve system is imitated
Rate.But during merging small documents, the load of current system can't be considered, i.e., no matter executing in current system is
Which kind of business, no matter remaining how many resource in current system, small documents can all be merged, this will merge small documents
The currently performed business of system is impacted in the process.
And a kind of merging method of small documents provided by the present invention, when the quantity of the small documents of acquisition reaches preset
It, can be by judging whether the load value of system judges less than preset load threshold the working condition of system when one threshold value.
When the load value of system is less than load threshold, it is meant that the currently performed business of system can't consume many resources, at this time
Whole system also has surplus energy to merge the small documents of acquisition, at this time can't be in system by the merging of the small documents of acquisition
The business being carrying out impacts.By the judgement to system load value size it is possible to prevente effectively from when merging small documents pair
The business being carrying out in system impacts.
In order to enable those skilled in the art to better understand the solution of the present invention, with reference to the accompanying drawings and detailed description
The present invention is described in further detail.Obviously, described embodiments are only a part of the embodiments of the present invention, rather than
Whole embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art are not making creative work premise
Lower obtained every other embodiment, shall fall within the protection scope of the present invention.
Referring to FIG. 1, a kind of flow chart for small documents merging method that Fig. 1 is provided by the embodiment of the present invention.
Referring to Fig. 1, in embodiments of the present invention, small documents merging method includes：
S101：Obtain the small documents that user sends.
In this step, typically user stores a large amount of small documents into distributed memory system.With the angle of system
For degree, that is, obtain the small documents that user sends.
S102：When the number of small documents reaches preset first threshold, it is default to judge whether the load value of system is less than
Load threshold.If so, executing S103.
In this step, when the number of the small documents of acquisition reaches preset first threshold, it can judge the load of system
Whether value is less than preset load threshold.The load value is the current load value of system or is predicted by calculating next
When etching system load value.
In embodiments of the present invention, it can be calculated by the load value of other thread real time computation systems in system
The step of load value can also be in this step, when obtaining with the method parallel processing provided in the embodiment of the present invention
When the small documents taken reach a certain number of, then computing system load value, in embodiments of the present invention and be not specifically limited.
Specific algorithm in relation to calculating the load value please refers to following inventive embodiments, is no longer repeated herein.
S103：The small documents of acquisition are merged.
The load value size of system can be judged in S102.When the load value is less than preset load threshold
When, it is meant that the currently performed business of system can't consume many resources, and whole system also has surplus energy that can will obtain at this time
Small documents merge.I.e. in this step, when the judging result in S102 is to be, then the small documents of acquisition are closed
And.
In this step, the step of small documents being merged usually is specially to serialize the small documents, described in merging
Small documents.
Sequential file (SequenceFile) is a kind of binary file technology that HDFS is provided, this binary file skill
Art is directly by ＜ key, and for value ＞ to serializing at file, file can realize the compression based on record or data block when serializing.
In embodiments of the present invention, small documents can be merged small documents using SequenceFile technologies, typically with small
The form that the call number of file is key, content is value merges, and the pressure based on data block may be implemented in merging while
Contracting, to solve the problems, such as that small-file storage efficiency is low.
When the load value of system is greater than or equal to preset load threshold, it is meant that there is no extra moneys for system at this time
Source can merge the small documents of acquisition.I.e. when the judging result in S102 is no, usually without merging small documents
This operation, but directly terminate flow, the business being carrying out in system is impacted to avoid small documents are merged.
Certainly, when the load value of system is greater than or equal to preset load threshold, i.e., when the judging result in S102 is
When no, it is possible that a variety of situations, need to execute different flows for different situations, detailed content please refers to following hairs
Bright embodiment.
The merging method for a kind of small documents that the embodiment of the present invention is provided, when the quantity of the small documents of acquisition reaches default
First threshold when, can be by judging whether the load value of system judges less than preset load threshold the work shape of system
State.When the load value of system is less than load threshold, it is meant that the currently performed business of system can't consume many resources, this
When whole system also have surplus energy to merge the small documents of acquisition, at this time by the small documents of acquisition merging can't be to system
In the business that is carrying out impact.By the judgement to system load value size it is possible to prevente effectively from when merging small documents
The business being carrying out in system is impacted.
The small documents sent for convenience to user count, be provided in the present invention document queue of certain length with
The identification information of small documents is stored in spare queue, the document queue and spare queue.Detailed content please refers to following hairs
Bright embodiment.
Referring to FIG. 2, the flow chart for another small documents merging method that Fig. 2 is provided by the embodiment of the present invention.
Referring to Fig. 2, in embodiments of the present invention, small documents merging method includes：
S201：Obtain the small documents that user sends.
This step and S101 in foregoing invention embodiment are essentially identical, and detailed content in foregoing invention embodiment to do in detail
It is thin to introduce, it is no longer repeated herein.
S202：The corresponding identification information of small documents is added to document queue.
In this step, when the number of the small documents is more than the length of the document queue, by the identification information
It is added to the document queue and at least one spare queue successively.
In embodiments of the present invention, the document queue of certain length can be pre-established.It is a large amount of small when being got in S201
After file, the identification information of the small documents got document queue can be added in this step.The identification information is usual
For the call number of small documents, naturally it is also possible to do identification information, the identification information with the other information of the small documents got
As long as can be corresponded with small documents, in embodiments of the present invention and it is not specifically limited.
In this step, when the number of the small documents is more than the length of the document queue, i.e., when in document queue
When can not leave the identification information of whole small documents, can the identification information of small documents be added to document queue and extremely successively
A few spare queue, i.e., be first added to document queue by the identification information of small documents, when document queue is booked, then by small text
The identification information of part is added to spare queue.The spare queue can be the spare queue pre-established, can also be to work as
The spare queue resettled when document queue is booked, in embodiments of the present invention and is not specifically limited.Under normal conditions, described
The length of spare queue is certain, when a spare queue is filled up by the identification information of small documents, then by remaining identification information
It is added to next spare queue, until all indicating that information is added into document queue or spare queue.
In embodiments of the present invention, document queue is usually consistent with the length of spare queue, but for document queue with it is standby
It in embodiments of the present invention and is not specifically limited, is determined on a case-by-case basis with the specific length of queue.
S203：When the identification information in document queue is booked, judge whether load value is less than load threshold.If so,
Execute S204.
This step is substantially the same with S102 in foregoing invention embodiment.In this step, when the mark in document queue is believed
When breath is booked, i.e., when the identification information of other small documents can not be added in document queue again, it is meant that got foot
The small documents of enough amounts can as the case may be merge small documents at this time, and small documents are merged into big file.When
When identification information in document queue is booked, it can judge whether the load value of system is less than preset load threshold in this step
Value.Detailed content in relation to the load value is described in detail in foregoing invention embodiment, is no longer repeated herein.Have
The specific algorithm for closing the calculating load value please refers to following inventive embodiments, is not also repeated again herein.
S204：The small documents of acquisition are merged.
In this step, the small documents corresponding to the identification information in document queue can be merged respectively, and
Small documents corresponding to identification information in spare queue are merged；Can also be will be in document queue and spare queue
Small documents corresponding to identification information merge.The particular number of small documents is in the embodiment of the present invention when in relation to merging every time
In and be not specifically limited.
This step and S103 in foregoing invention embodiment are essentially identical, and detailed content has been done in foregoing invention embodiment in detail
It is thin to introduce, it is no longer repeated herein.
The merging method for a kind of small documents that the embodiment of the present invention is provided, by adding into the document queue pre-established
The small documents that the identification information of corresponding small documents can easily send user are added to count, and then convenient for subsequent step
It executes.
When system has got a large amount of small documents, but the load value of system is greater than or equal to preset load threshold
When, it is possible that a variety of situations, need to execute different flows for different situations, detailed content please refers to following inventions
Embodiment.
Referring to FIG. 3, the flow chart for another small documents merging method that Fig. 3 is provided by the embodiment of the present invention.
Referring to Fig. 3, in embodiments of the present invention, small documents merging method includes：
S301：Obtain the small documents that user sends.
S302：The corresponding identification information of small documents is added to document queue.
In this step, when document queue is booked by the identification information of small documents, the spare team of a preset length can be created
Row, the spare queue is added to by the identification information of remaining small documents；When a spare queue is filled out by the identification information of small documents
Man Shi can create another spare queue, then remaining identification information is added to next spare queue.
S303：When the identification information in document queue is booked, judge whether load value is less than load threshold.If so,
Execute S304；If it is not, then executing S305.
In this step, when the identification information in document queue is booked, it will usually send " queue full " (QF) to system
Signal, the at this time main control part in system, such as controller can judge whether load value is less than load threshold according to QF signals
Value.
S304：The small documents of acquisition are merged.
In embodiments of the present invention, S301 to S304 and S201 to S204 in foregoing invention embodiment are essentially identical, in detail
Content please refers to foregoing invention embodiment, no longer carries out expansion description herein.
S305：Judge whether the quantity of spare queue is more than preset second threshold.If so, executing S306；If it is not, then
Execute S307.
In this step, when the load value of system is greater than or equal to load threshold, further judge the number of spare queue
Whether amount is more than preset second threshold.This step judge spare queue quantity whether be more than second threshold purpose be substantially
Judge the quantity of small documents obtained.When the quantity of spare queue is less than second threshold, illustrate the small text that system obtains
The quantity of part is also less big, the storage efficiency of influence whole system that will not be excessive.When the quantity of spare queue is greater than or equal to
When second threshold, illustrates that the quantity of the small documents stored in system is excessive, has seriously affected the storage efficiency of whole system,
It needs immediately to merge small documents to improve the storage efficiency of system at this time.
S306：The small documents of acquisition are merged.
In this step, when the result judged in S305 is is, illustrate that the quantity of the small documents stored in system is excessive,
The storage efficiency of whole system has been seriously affected, has needed immediately to merge small documents at this time, i.e., in this step will
The small documents of acquisition merge.In remaining content and the embodiment of the present invention of this step in S304, with foregoing invention embodiment
S204 is essentially identical, and detailed content please refers to foregoing invention embodiment, no longer carries out expansion description herein.
S307：A newly-built spare queue, and the identification information in document queue is transferred to newly-built spare queue.
In this step, when the judging result of S305 is no, it is meant that the quantity for the small documents that system obtains is also less
Greatly, the storage efficiency of influence whole system that will not be excessive.At this time in this step, a spare queue can be created again, and
Identification information in document queue is transferred to the newly-built spare queue, to empty document queue so that in document queue
It can be added to the identification information of the small documents newly stored, and postpone the time for merging small documents.
S308：It controls timer and carries out timing.
After emptying document queue, timing can be carried out with the timer in control system in this step, for determining
The time for merging small documents is postponed.
S309：When reaching preset time threshold between the timing of timer institute, it is negative to judge whether the load value of system is less than
Carry threshold value.If so, executing S310；If it is not, then re-executing S308.
In this step, when reaching preset time threshold between the timing of timer institute, it is meant that will merge small text
The step for part, has postponed time enough, and the load value that can rejudge system in this step at this time whether small states load
Threshold value.S303 and S203 in foregoing invention embodiment are essentially identical in judgement and the embodiment of the present invention in relation to load value, in detail
Content please refers to foregoing invention embodiment, is no longer repeated herein.
In this step, when reaching preset time threshold between the timing of timer institute, it will usually send " the time to system
To " (TU) signal, the main control part in system at this time, such as controller can judge that the load value of system is according to TU signals
It is no to be less than load threshold.
In this step, when by will merge small documents the step for postponed preset time after, if at this time system bear
Load value is still greater than or equal to preset load threshold, illustrates that the resource that system is not still extra at this time can be by the small of acquisition
File merges, and needs to recycle above-mentioned S308 to S309 at this time, until system load value be less than preset load threshold, and
Go to following S310.
S310：The small documents of acquisition are merged.
This step and S304 in the embodiment of the present invention and S204 in foregoing invention embodiment are essentially identical, and detailed content please join
According to foregoing invention embodiment, no longer repeated herein.It in this step, can be only by the identification information institute in spare queue
Corresponding small documents merge, can also be by the small documents corresponding to identification information whole in document queue and spare queue
It merges, in embodiments of the present invention and is not specifically limited.
This flow of above-mentioned S308 to S310 can execute in other threads, i.e. S308 to S310 can be with above-mentioned S301
The execution parallel to S307.During the time is to this when waiting for timer i.e. in S309, still S301 can be executed extremely
The identification information of the small documents got is added to document queue by this flow of S307, and when document queue is full, to system
Send QF signals so that control unit in system judges whether the load value of system is less than load threshold.When load value is small
When load threshold, the small documents of acquisition are merged.It should be noted that if being that time when timer is waited for arrive in S309
During, system has merged small documents according to QF signals, needs to close the timer in system at this time, and terminates
State this performed flow in other threads of S308 to S310.
The merging method for a kind of small documents that the embodiment of the present invention is provided, when the load value of system is more than the load of setting
When threshold value, corresponding detailed process is provided for different situations, to optimize the merging flow of entire small documents.
Specific algorithm in relation to computing system load value will be described in detail in following inventive embodiments.
Referring to FIG. 4, a kind of flow chart for specific small documents merging method that Fig. 4 is provided by the embodiment of the present invention.
Difference and foregoing invention embodiment, the embodiment of the present invention is on the basis of foregoing invention embodiment, further
The specific algorithm of system load value is defined.Remaining content to be described in detail in foregoing invention embodiment,
This is no longer repeated.
Referring to Fig. 4, in embodiments of the present invention, the specific method of the load value of computing system includes：
S401：The load attribute value of multigroup system is obtained according to the sequencing of time.
In embodiments of the present invention, it will usually obtain the load attribute of whole system in real time according to the sequencing of time
Value.The load attribute value has generally included cpu busy percentage, memory usage, bandwidth availability ratio, system average throughput etc.
Deng.Certain load attribute value can also be other other parameters that can be used for characterizing system load situation, in this hair
It for the specific type of load attribute value and is not specifically limited, is determined on a case-by-case basis in bright embodiment.
Under normal conditions, in embodiments of the present invention, the negative of multigroup whole system can be obtained according to the sequencing of time
Carry attribute value.It is negative that each in system is inscribed at the time of wherein every group of data correspond to different, when including corresponding in every group of data
Carry the concrete numerical value of attribute value.
S402：The corresponding decision attribute values of multigroup load attribute value are calculated separately out by AHP.
It in embodiments of the present invention, can be by AHP (analytic hierarchy process (AHP)), according to the multigroup load attribute obtained in S401
Value, calculates the decision attribute values corresponding to each group of load attribute value.
AHP is a kind of analytic hierarchy process, is a kind of criteria decision-making method.AHP can be passed through in this step
First each group of load attribute value is permeated decision attribute values.Since different types of load attribute value has different weights
The property wanted first can be permeated each group of load attribute value by AHP according to the importance of load attribute value in this step
Decision attribute values, the specific method is as follows：
The first step：Construct comparator matrix A.Each group of the load attribute value obtained in S401 is defined as follows：
Pi(i=1,2 ..., n)；Wherein i is different types of load attribute value, a shared n kinds load attribute value.
Each element a in comparator matrix AijDefinition it is as shown in table 1：
The definition of 1 comparator matrix element of table
| Load attribute value relationship | Pi＜ Pj | Pi=Pj | Pi＞ Pj |
| aij | 0 | 1 | 2 |
Wherein i and j respectively represents different types of load attribute value Pi。
Second step：Judgement Matricies A*。
Above-mentioned comparator matrix A is passed through as down conversion obtains judgment matrix A*：
In above formula：rmax=max (ri)；rmin=min (ri)；bm=rmax/rmin。
Third walks：Calculate optimum transfer matrix C and weight wj.Element c in optimum transfer matrix CijCalculation formula it is as follows：
Calculate to obtain weights
4th step：Calculate decision attribute values di.Its calculation formula is as follows：
S403：According to multiple decision attribute values, load value is calculated by AHP.
Multiple decision attribute values d at corresponding each moment are calculated by AHP in above-mentioned S402i.With S402 phases
It is similar, in this step, load value can be calculated according to decision attribute values by AHP.
Concretely, it will usually which first the importance according to decision attribute values about the time constructs comparator matrix A, then successively
Corresponding t is calculated to third step according to the first step described in S402, second step1, t2..., tnThe decision attribute at moment
Value d1, d2..., dnCorresponding weights Wi(t in system is finally calculated in i=1,2 ..., n(n+1)The load value at moment
dn+1, load value dn+1Calculation formula it is as follows：
The computational methods of load value in a kind of system that the embodiment of the present invention is provided belong to according to load caused by system
The historical information of property value, is analyzed by secondary AHP, can predict the load value of subsequent time system, and according to this load value
It can effectively judge whether subsequent time is convenient to merge small documents.
A kind of merging device of small documents provided in an embodiment of the present invention is introduced below, small documents described below
Reference can be corresponded with above-described small documents merging method by merging device.
Fig. 5 is merged the structure diagram of device by a kind of small documents that the embodiment of the present invention provides, described small with reference to Fig. 5
Piece file mergence device may include：
Acquisition module 100：Small documents for obtaining user's transmission.
First judgment module 200：For when the number of the small documents reaches preset first threshold, judging system
Whether load value is less than preset load threshold.
Merging module 300：For when the judging result of first judgment module is to be, the small documents of acquisition to be carried out
Merge.
The small documents of the present embodiment merge device for realizing small documents merging method above-mentioned, therefore small documents merge dress
The embodiment part of the visible small documents merging method hereinbefore of specific implementation mode in setting, for example, acquisition module 100, the
One judgment module 200, merging module 300, be respectively used to realize in above-mentioned small documents merging method step S101, S102 and
S103, so, specific implementation mode is referred to the description of corresponding various pieces embodiment, and details are not described herein.
Merge equipment to a kind of small documents provided in an embodiment of the present invention below to be introduced, small documents described below close
And equipment can correspond reference with above-described small documents merging method and small documents merging device.
Referring to FIG. 6, Fig. 6 is merged the structure diagram of equipment by a kind of small documents that the embodiment of the present invention provides.
With reference to Fig. 6, it may include processor 400 and memory 500 which, which merges equipment,.
The memory 500 is for storing computer program；The processor 400 is for when executing the computer program
Realize the small documents merging method described in foregoing invention embodiment.
The small documents of the present embodiment merge processor 400 in equipment and are used to install the small text described in above-mentioned inventive embodiments
Part merge device, simultaneous processor 400 with memory 500 be combined may be implemented it is small described in any of the above-described inventive embodiments
Piece file mergence method.Therefore small documents merge the reality of the visible small documents merging method hereinbefore of specific implementation mode in equipment
A part is applied, specific implementation mode is referred to the description of corresponding various pieces embodiment, and details are not described herein.
The present invention also provides a kind of computer readable storage medium, meter is stored on the computer readable storage medium
Calculation machine program, the computer program realize that the small documents described in any of the above-described inventive embodiments close when being executed by processor
And method.Remaining content is referred to the prior art, no longer carries out expansion description herein.
Each embodiment is described by the way of progressive in this specification, the highlights of each of the examples are with it is other
The difference of embodiment, just to refer each other for same or similar part between each embodiment.For being filled disclosed in embodiment
For setting, since it is corresponded to the methods disclosed in the examples, so description is fairly simple, related place is referring to method part
Explanation.
Professional further appreciates that, unit described in conjunction with the examples disclosed in the embodiments of the present disclosure
And algorithm steps, can be realized with electronic hardware, computer software, or a combination of the two, in order to clearly demonstrate hardware and
The interchangeability of software generally describes each exemplary composition and step according to function in the above description.These
Function is implemented in hardware or software actually, depends on the specific application and design constraint of technical solution.Profession
Technical staff can use different methods to achieve the described function each specific application, but this realization is not answered
Think beyond the scope of this invention.
The step of method described in conjunction with the examples disclosed in this document or algorithm, can directly be held with hardware, processor
The combination of capable software module or the two is implemented.Software module can be placed in random access memory (RAM), memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of storage medium well known in field.
Finally, it is to be noted that, herein, relational terms such as first and second and the like be used merely to by
One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation
Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning
Covering non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only that
A little elements, but also include other elements that are not explicitly listed, or further include for this process, method, article or
The intrinsic element of equipment.In the absence of more restrictions, the element limited by sentence " including one ... ", not
There is also other identical elements in the process, method, article or apparatus that includes the element for exclusion.
A kind of merging method of small documents provided by the present invention, device, equipment and readable storage medium storing program for executing are carried out above
It is discussed in detail.Principle and implementation of the present invention are described for specific case used herein, above example
Explanation be merely used to help understand the present invention method and its core concept.It should be pointed out that for the common of the art
, without departing from the principle of the present invention, can be with several improvements and modifications are made to the present invention for technical staff, these
Improvement and modification are also fallen within the protection scope of the claims of the present invention.
Claims (10)
1. a kind of merging method of small documents, which is characterized in that the method includes：
Obtain the small documents that user sends；
When the number of the small documents reaches preset first threshold, judge whether the load value of system is less than preset load
Threshold value；
If so, the small documents of acquisition are merged.
2. according to the method described in claim 1, it is characterized in that, it is described acquisition user send small documents after, it is described
Method further includes：
The corresponding identification information of the small documents is added to document queue；Wherein, described in being more than when the number of the small documents
When the length of document queue, the identification information is added to the document queue and at least one spare queue successively；
It is described when the number of the small documents reaches preset first threshold, judge system load value whether be less than it is preset
Load threshold includes：
When the identification information in the document queue is booked, judge whether the load value is less than the load threshold.
3. according to the method described in claim 2, it is characterized in that, whether being less than in the load value for judging system preset
After load threshold, the method further includes：
If it is not, then judging whether the quantity of the spare queue is more than preset second threshold；If so, by the described small of acquisition
File merges.
4. according to the method described in claim 3, it is characterized in that, judging whether the quantity of the spare queue is more than described
After preset second threshold, the method further includes：
If it is not, then creating a spare queue, and the identification information in the document queue is transferred to the newly-built spare team
Row；
S308：It controls timer and carries out timing；
S309：When reaching preset time threshold between timer institute timing, judge whether the load value of the system is small
In the load threshold；If so, executing S310；If it is not, then re-executing S308；
S310：The small documents of acquisition are merged.
5. according to the method described in claim 1, it is characterized in that, it is described by the small documents of acquisition merge including：
The small documents are serialized, to merge the small documents.
6. according to the method described in any one of claim 1 to 5 claim, which is characterized in that calculate the side of the load value
Method includes：
The load attribute value of multigroup system is obtained according to the sequencing of time；
The corresponding decision attribute values of multigroup load attribute value are calculated separately out by AHP；
According to multiple decision attribute values, the load value is calculated by AHP.
7. according to the method described in claim 2, it is characterized in that, the identification information is the call number of the small documents.
8. a kind of merging device of small documents, which is characterized in that described device includes：
Acquisition module：Small documents for obtaining user's transmission；
First judgment module：For when the number of the small documents reaches preset first threshold, judging the load value of system
Whether preset load threshold is less than；
Merging module：For when the judging result of first judgment module is to be, the small documents of acquisition to be merged.
9. a kind of merging equipment of small documents, which is characterized in that including：
Memory：For storing computer program；
Processor：The merging side of the small documents as described in any one of claim 1 to 7 is realized when for executing the computer program
The step of method.
10. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium
Program realizes the merging method of the small documents as described in any one of claim 1 to 7 when the computer program is executed by processor
The step of.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810331806.7A CN108595567A (en) | 2018-04-13 | 2018-04-13 | A kind of merging method of small documents, device, equipment and readable storage medium storing program for executing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810331806.7A CN108595567A (en) | 2018-04-13 | 2018-04-13 | A kind of merging method of small documents, device, equipment and readable storage medium storing program for executing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108595567A true CN108595567A (en) | 2018-09-28 |
Family
ID=63622456
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810331806.7A Pending CN108595567A (en) | 2018-04-13 | 2018-04-13 | A kind of merging method of small documents, device, equipment and readable storage medium storing program for executing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108595567A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109766318A (en) * | 2018-12-17 | 2019-05-17 | 新华三大数据技术有限公司 | File reading and device |
| CN109994131A (en) * | 2019-04-10 | 2019-07-09 | 广西电网有限责任公司玉林供电局 | A kind of power frequency recorded wave file compression and storage method and system based on index |
| CN111538702A (en) * | 2020-04-20 | 2020-08-14 | 北京京安佳新技术有限公司 | Hadoop-based massive small file processing method and device |
| CN112948330A (en) * | 2021-02-26 | 2021-06-11 | 拉卡拉支付股份有限公司 | Data merging method, device, electronic equipment, storage medium and program product |
| CN113568742A (en) * | 2021-07-22 | 2021-10-29 | 四川英得赛克科技有限公司 | File compression method, device and system, electronic equipment and storage medium |
| CN113722072A (en) * | 2021-09-14 | 2021-11-30 | 华瑞指数云(河南)科技有限公司 | Storage system file merging method and device based on intelligent distribution |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678579A (en) * | 2013-12-12 | 2014-03-26 | 浪潮电子信息产业股份有限公司 | Optimizing method for small-file storage efficiency |
| CN104536959A (en) * | 2014-10-16 | 2015-04-22 | 南京邮电大学 | Optimized method for accessing lots of small files for Hadoop |
| US20160321089A1 (en) * | 2015-04-30 | 2016-11-03 | Microsoft Technology Licensing, Llc | Binary file generation |
| CN106909651A (en) * | 2017-02-23 | 2017-06-30 | 郑州云海信息技术有限公司 | A kind of method for being write based on HDFS small documents and being read |
| CN107168802A (en) * | 2017-05-18 | 2017-09-15 | 郑州云海信息技术有限公司 | The merging method and device of a kind of cloud storage small file |
| CN107679177A (en) * | 2017-09-29 | 2018-02-09 | 郑州云海信息技术有限公司 | A kind of small documents storage optimization method based on HDFS, device, equipment |
-
2018
- 2018-04-13 CN CN201810331806.7A patent/CN108595567A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678579A (en) * | 2013-12-12 | 2014-03-26 | 浪潮电子信息产业股份有限公司 | Optimizing method for small-file storage efficiency |
| CN104536959A (en) * | 2014-10-16 | 2015-04-22 | 南京邮电大学 | Optimized method for accessing lots of small files for Hadoop |
| US20160321089A1 (en) * | 2015-04-30 | 2016-11-03 | Microsoft Technology Licensing, Llc | Binary file generation |
| CN106909651A (en) * | 2017-02-23 | 2017-06-30 | 郑州云海信息技术有限公司 | A kind of method for being write based on HDFS small documents and being read |
| CN107168802A (en) * | 2017-05-18 | 2017-09-15 | 郑州云海信息技术有限公司 | The merging method and device of a kind of cloud storage small file |
| CN107679177A (en) * | 2017-09-29 | 2018-02-09 | 郑州云海信息技术有限公司 | A kind of small documents storage optimization method based on HDFS, device, equipment |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109766318A (en) * | 2018-12-17 | 2019-05-17 | 新华三大数据技术有限公司 | File reading and device |
| CN109766318B (en) * | 2018-12-17 | 2021-03-02 | 新华三大数据技术有限公司 | File reading method and device |
| CN109994131A (en) * | 2019-04-10 | 2019-07-09 | 广西电网有限责任公司玉林供电局 | A kind of power frequency recorded wave file compression and storage method and system based on index |
| CN109994131B (en) * | 2019-04-10 | 2021-10-22 | 广西电网有限责任公司玉林供电局 | Index-based power frequency wave recording file compression storage method and system |
| CN111538702A (en) * | 2020-04-20 | 2020-08-14 | 北京京安佳新技术有限公司 | Hadoop-based massive small file processing method and device |
| CN112948330A (en) * | 2021-02-26 | 2021-06-11 | 拉卡拉支付股份有限公司 | Data merging method, device, electronic equipment, storage medium and program product |
| CN113568742A (en) * | 2021-07-22 | 2021-10-29 | 四川英得赛克科技有限公司 | File compression method, device and system, electronic equipment and storage medium |
| CN113568742B (en) * | 2021-07-22 | 2024-04-19 | 四川英得赛克科技有限公司 | File compression method, device and system, electronic equipment and storage medium |
| CN113722072A (en) * | 2021-09-14 | 2021-11-30 | 华瑞指数云(河南)科技有限公司 | Storage system file merging method and device based on intelligent distribution |
| CN113722072B (en) * | 2021-09-14 | 2024-02-13 | 华瑞指数云科技(深圳)有限公司 | Storage system file merging method and device based on intelligent shunting |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108595567A (en) | A kind of merging method of small documents, device, equipment and readable storage medium storing program for executing | |
| CN107659433B (en) | Cloud resource scheduling method and equipment | |
| CN104102543B (en) | The method and apparatus of adjustment of load in a kind of cloud computing environment | |
| CN107239336A (en) | A kind of method and device for realizing task scheduling | |
| US9582338B2 (en) | Calculating a dispatcher's relative share based on relative queue length and capacity value of a plurality of workload types and computing systems combinations | |
| US9374314B2 (en) | QoS aware balancing in data centers | |
| CN104268018B (en) | Job scheduling method and job scheduler in a kind of Hadoop clusters | |
| CN107656813A (en) | The method, apparatus and terminal of a kind of load dispatch | |
| CN104731642A (en) | Backfill scheduling method and system for embarrassingly parallel jobs | |
| CN112559182B (en) | Resource allocation method, device, equipment and storage medium | |
| CN102012891B (en) | Computer cluster management method, device and system | |
| CN105373432B (en) | A kind of cloud computing resource scheduling method based on virtual resource status predication | |
| CN104572106A (en) | Concurrent program developing method for processing of large-scale data based on small memory | |
| CN105117285B (en) | A kind of nonvolatile memory method for optimizing scheduling based on mobile virtual system | |
| CN111506423A (en) | Resource scheduling method and device, electronic equipment and storage medium | |
| CN106897136A (en) | A kind of method for scheduling task and device | |
| CN107273182A (en) | A kind of resources of virtual machine dynamic expanding method and system | |
| CN116680060B (en) | Task allocation method, device, equipment and medium for heterogeneous computing system | |
| CN109062666A (en) | A kind of cluster virtual machine management method and relevant apparatus | |
| CN107479949A (en) | Low energy consumption cloud computing method for scheduling task | |
| CN103106116B (en) | Dynamic resource management method based on sliding window | |
| CN108984298A (en) | A kind of resource regulating method and system of cloud computing platform | |
| CN108897622A (en) | A kind of dispatching method and relevant apparatus of task run | |
| CN108255595A (en) | A kind of dispatching method of data task, device, equipment and readable storage medium storing program for executing | |
| CN112416596A (en) | Node scheduling method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180928 |
|
| RJ01 | Rejection of invention patent application after publication |