CN107463332A - A kind of file splitting method and device - Google Patents
A kind of file splitting method and device Download PDFInfo
- Publication number
- CN107463332A CN107463332A CN201610390987.1A CN201610390987A CN107463332A CN 107463332 A CN107463332 A CN 107463332A CN 201610390987 A CN201610390987 A CN 201610390987A CN 107463332 A CN107463332 A CN 107463332A
- Authority
- CN
- China
- Prior art keywords
- file
- segmentation
- threshold
- internal memory
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1847—File system types specifically adapted to static storage, e.g. adapted to flash memory or SSD
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application discloses a kind of file splitting method and device.Server system resources are obtained, current segmentation threshold is calculated according to the system resource；Pending file is split according to the current segmentation threshold, file process is carried out to the segmentation result；The result for obtaining and preserving the file process realizes the dynamic partition of calculation document.
Description
Technical field
The application belongs to data calculating field, specifically, is related to a kind of file splitting method and device.
Background technology
In big data processing procedure, when handling big file calculating, calculating directly is carried out to big file and normally results in behaviour
Make Installed System Memory spilling.Therefore, can be to text in order to prevent big file is loaded into when internal memory is calculated from causing internal memory to overflow
Part is split, and way commonly used in the prior art is that big file to be calculated is split according to the value of some fixation, example
Such as split according to glue file size or glue file line number, finally being calculated in real-time calculate could export after terminating
Result of calculation.
However, this file splitting method of prior art does not have the free memory and CPU for considering application server
Tight situation, in real-time calculating process, it is possible to cause internal memory to overflow because file cutting is excessive, and due to being to load
File is to be calculated in order, and any one file can be caused to calculate failure and can not obtain the result of calculation of stage.
Therefore, a kind of new file splitting method urgently proposes.
The content of the invention
In view of this, technical problems to be solved in this application there is provided a kind of file splitting method and device.
In order to solve the above-mentioned technical problem, the application has opened a kind of file splitting method and device.
A kind of file splitting method of the application, including the steps：
Server system resources are obtained, current segmentation threshold is calculated according to the system resource；
Pending file is split according to the current segmentation threshold, file process is carried out to the segmentation result；
Obtain and preserve the result of the file process.
Wherein, the system resource specifically includes：Server CPU idleness and the free value of internal memory.
Wherein, current segmentation threshold is calculated according to the system resource, specifically included：To the sky of CPU described in current time
The free value of not busy rate and the internal memory is weighted summation, and using the result of the weighted sum as the current segmentation threshold
Value.
Wherein, pending file is split according to the current segmentation threshold, specifically included：Using order line work
Tool, the pending file is split according to the current segmentation threshold.
A kind of file splitting method of the application, including the steps：
Pending file is split according to preset segmentation threshold and obtains partitioning portion and remainder；To the segmentation
Part carries out the file process, segmentation correction value is obtained according to the system resource in the file process moment server, with
The segmentation correction value updates the preset segmentation threshold to be split to the remainder next time；
Obtain and preserve the result that the partitioning portion carries out the file process.
The system resource specifically includes：Server CPU idleness and the free value of internal memory.
Wherein, segmentation correction value is obtained, is specifically included：The free value of idleness and the internal memory to the CPU enters
Row weighted sum, and using the result of the weighted sum as the segmentation correction value.
Wherein, pending file is split according to preset segmentation threshold and obtains partitioning portion and remainder, specifically
Including：Using command-line tool, the pending file is split according to the file division threshold value.
Wherein, segmentation correction value is obtained according to the system resource, in addition to：When the free time for judging the server CPU
Rate be more than default first threshold and, when the free value for judging the internal memory is more than default Second Threshold, then with described pre-
The segmentation threshold put is as the segmentation correction value.
A kind of file division device of the application, including following module：
Pretreatment module, for obtaining server system resources, current segmentation threshold is calculated according to the system resource；
First segmentation module, for splitting pending file according to the current segmentation threshold；
First processing module, for carrying out file process to the segmentation result；
First memory module, for obtaining and preserving the result of the file process.
Wherein, the system resource specifically includes：Server CPU idleness and the free value of internal memory.
Further, the pretreatment module is specifically used for：Idleness and the internal memory to CPU described in current time
Free value be weighted summation, and using the result of the weighted sum as the current segmentation threshold.
Further, the first segmentation module is specifically used for：Using command-line tool, according to the current segmentation threshold
The pending file is split.
A kind of file division device of the application, including following module：
Second segmentation module, for obtaining partitioning portion according to the pending file of preset segmentation threshold segmentation and remaining
Remaining part point；Second processing module, for carrying out file process to the partitioning portion, taken according at the file process moment
The system resource of business device obtains segmentation correction value, updates the preset segmentation threshold with the segmentation correction value to described
Remainder is split next time；
Second memory module, for obtaining and preserving the result of the file process.
Wherein, the system resource specifically includes：Server CPU idleness and the free value of internal memory.
Wherein, the Second processing module is specifically used for：The free value of idleness and the internal memory to the CPU enters
Row weighted sum, and using the result of the weighted sum as the segmentation correction value.
Wherein, the second segmentation module is specifically used for：Using command-line tool, treated according to the segmentation threshold to described
The file of processing is split.
Wherein, the Second processing module is specifically additionally operable to：When judgement server CPU idleness is more than default
First threshold and, when the free value for judging the internal memory is more than default Second Threshold, then with the preset segmentation threshold
As the segmentation correction value.
Compared with prior art, the application can be obtained including following technique effect：
1) dynamically big file to be calculated is divided by calculation server free memory and CPU loading condition
Cut, the situation for avoiding causing server memory to be overflowed because excessive file is loaded occurs；
2) real-time collecting result of calculation and result of calculation is preserved in real time, avoids leading to not output because calculating failure appointing
What result of calculation.
Certainly, implementing any product of the application must be not necessarily required to reach all the above technique effect simultaneously.
Brief description of the drawings
Accompanying drawing described herein is used for providing further understanding of the present application, forms the part of the application, this Shen
Schematic description and description please is used to explain the application, does not form the improper restriction to the application.In the accompanying drawings：
Fig. 1 is the techniqueflow chart of the embodiment of the present application one；
Fig. 2 is the techniqueflow chart of the embodiment of the present application two；
Fig. 3 is the apparatus structure schematic diagram of the embodiment of the present application three；
Fig. 4 is the apparatus structure schematic diagram of the embodiment of the present application four.
Embodiment
Presently filed embodiment is described in detail below in conjunction with drawings and Examples, and thereby how the application is applied
Technological means can fully understand and implement according to this to solve technical problem and reach the implementation process of technical effect.
In a typical configuration, computing device includes one or more processors (CPU), input/output interface, net
Network interface and internal memory.
Internal memory may include computer-readable medium in volatile memory, random access memory (RAM) and/or
The forms such as Nonvolatile memory, such as read-only storage (ROM) or flash memory (flash RAM).Internal memory is computer-readable medium
Example.
Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by any method
Or technology come realize information store.Information can be computer-readable instruction, data structure, the module of program or other data.
The example of the storage medium of computer includes, but are not limited to phase transition internal memory (PRAM), static RAM (SRAM), moved
State random access memory (DRAM), other kinds of random access memory (RAM), read-only storage (ROM), electric erasable
Programmable read only memory (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc read-only storage (CD-ROM),
Digital versatile disc (DVD) or other optical storages, magnetic cassette tape, the storage of tape magnetic rigid disk or other magnetic storage apparatus
Or any other non-transmission medium, the information that can be accessed by a computing device available for storage.Define, calculate according to herein
Machine computer-readable recording medium does not include non-temporary computer readable media (transitory media), such as the data-signal and carrier wave of modulation.
Some vocabulary has such as been used to censure specific components among specification and claim.Those skilled in the art should
It is understood that hardware manufacturer may call same component with different nouns.This specification and claims are not with name
The difference of title is used as the mode for distinguishing component, but is used as the criterion of differentiation with the difference of component functionally.Such as logical
The "comprising" of piece specification and claim mentioned in is an open language, therefore should be construed to " include but do not limit
In "." substantially " refer in receivable error range, those skilled in the art can be described within a certain error range solution
Technical problem, basically reach the technique effect.In addition, " coupling " one word is herein comprising any direct and indirect electric property coupling
Means.Therefore, if the first device of described in the text one is coupled to a second device, representing the first device can directly electrical coupling
The second device is connected to, or the second device is electrically coupled to indirectly by other devices or coupling means.Specification
Subsequent descriptions for implement the application better embodiment, so it is described description be for the purpose of the rule for illustrating the application,
It is not limited to scope of the present application.The protection domain of the application is worked as to be defined depending on appended claims institute defender.
It should also be noted that, term " comprising ", "comprising" or its any other variant are intended to nonexcludability
Comprising, so that commodity or system including a series of elements not only include those key elements, but also including without clear and definite
The other element listed, or also include for this commodity or the intrinsic key element of system.In the feelings not limited more
Under condition, the key element that is limited by sentence "including a ...", it is not excluded that in the commodity including the key element or system also
Other identical element be present.
Fig. 1 is the techniqueflow chart of the embodiment of the present application one, with reference to Fig. 1, a kind of file splitting method of the application, and specific bag
Include following steps：
Step 110：Server system resources are obtained, current segmentation threshold is calculated according to the system resource；
Step 120：Pending file is split according to the current segmentation threshold, and style of writing is entered to the result of the segmentation
Part processing；
Step 130：Obtain and preserve the result of the file process.
In step 110, the server refers to carry out all devices of data calculating, and its hardware composition comprises at least
Central processor CPU, the memory headroom that data processing is provided and the hard drive space for preserving result.The acquisition server
System resource, refer specifically to obtain server CPU idleness and internal memory free value size.
Specifically, CPU idleness is generally represented with the size of idleCPU value, idleCPU i.e. system idle,
Its complete name is System Idle Process, i.e. system free value, and the bigger explanation CPU of IdleCPU values takes lower.Example
Such as, when " System Idle Process " processes take resource be 2% when, illustrate machine at present only 2% resource be free time
, i.e., machine may be infected virus or occupied by other programs 98% resource.In other words, " System Idle
It is more that Process " processes take the more big then system available resources of resource.In the present embodiment, shell scripts can be used real
When acquisition server CPU idleness, certainly, it is any can realize the embodiment of the present application server resource utilization rate collection meter
Calculation machine language, software or hardware are within the protection domain of the application.
At the same time, the embodiment of the present application is while server current time CPU idleness is obtained, it is also necessary to obtains current
Moment takes the remaining internal storage data of server.Server memory is primarily used to store ephemeral data, does caching, will be interim
Processing data be put into inside internal memory, avoid directly reading hard disk, the reading speed of file during so as to improve data processing.Server
Shared internal memory is fewer in itself for system, and internal memory needs the size consumed, the data volume handled depending on server it is big
It is small.Therefore, before file division, the foundation using internal memory surplus value as file division in data handling procedure can dynamic
Adjustment is loaded the size of file, so as to avoid load document is excessive from causing internal memory to overflow.
Specifically, after getting the system resource at the current time of server, idleness to the CPU and it is described in
The free value deposited is weighted summation, and using the result of the weighted sum as the current time segmentation threshold.It is specific to add
The formula for weighing summation is as follows：
Free value/64*N of segmentation threshold=CPU idleness+internal memory, wherein, N is integer, represents the block number of internal memory,
The unit of the internal memory free value is MB.
Wherein, calculating of the idleness of the CPU on segmentation threshold influences smaller, so, the formula can be with following
Mode is present：
Free value/64*N of segmentation threshold=internal memory, wherein, N is integer, represents the block number of internal memory, and the internal memory is idle
The unit of value is MB.For example, on an application server, a certain free time at moment idle value is 80%, and free memory value is
8000MB, then this moment file division segmentation threshold：0.8+8000MB/64*2=250.8MB, then by pending file
When being divided into 250.8MB, treatment effeciency highest, if certainly pending file size is less than 250.8MB, then do not influenceing to locate
Manage fireballing while add number of processes, if pending file size is more than 250.8MB, be then likely to result in CPU
Load is superfluous and internal memory overflows.
In the step 120, after getting the current segmentation threshold, file is split, specific dividing method
The command-line tool of system can be used to be split.
Command-line tool is in an operating system, to prompt a kind of work prompt into line command input.In different behaviour
Make under system environments, CMD is different.Under windows environment, order line program is cmd.exe, is one 32
The order line program of position, Microsoft's Windows systems are operated based on the command interpreter on Windows similar to the DOS of Microsoft
System.Number order is inputted, cmd.exe can be performed, for example input shutdown-s will shut down after 30 seconds.The application is real
Apply the segmentation that cut command-line tools can be used to realize file in example, the partition tools carried using Microsoft's Windows systems, letter
It is single easy-to-use and free, realize the file division of low cost.
The size to be split known to can directly be split by file in step 120, be carried out after segmentation at file
Reason.The file process can include any type of processing such as compression, calculating, storage, transmission and the broadcasting of file.Example
Such as, when the file process is that file calculates, before the calculating of file, first according to the use of current time system resource
Situation obtains current segmentation threshold, by pending file division is first for calculating according to current file segmentation threshold
Point and remaining Part II, the Part I for calculating, which can be loaded into internal memory, to be calculated.Continue acquisition first
The behaviour in service of part system resource at the time of calculating, and calculated next time according to the behaviour in service of this moment system resource
The segmentation threshold that file calculates, after obtaining new segmentation threshold, splits according to new segmentation threshold to Part II, so as to
Most rational file size during file calculating is carried out each time.
It should be noted that in the embodiment of the present application, the result for splitting to obtain is the minimum list of file process each time
Member, limited by system resource, current segmentation threshold of each moment is not necessarily identical, thus be accordingly used in file process most
Junior unit is not usually fixed.
In step 130, increase the result of a thread real-time collecting server and be stored in result firmly
Among disk, overall result is exported again after the completion of all divided parts are all handled, so as to avoid a certain ring
Any result can not be exported caused by the processing failure or processing time-out of section.
In the present embodiment, by the system resource of advance acquisition server and according to the remaining free memory of server and
Cpu load situation so as to handle, is avoided because loading caused by excessive file in server to be split to file
The situation for depositing spilling occurs, and at the same time, real-time collecting result of calculation is simultaneously preserved result, relative to existing skill
Art, the embodiment of the present application avoid the situation that any result may can not be exported caused by processing failure.
Fig. 2 is the techniqueflow chart of the embodiment of the present application two, with reference to Fig. 2, a kind of file splitting method of the application, and specific bag
Include following steps：
Step S210：Pending file is split according to preset segmentation threshold and obtains partitioning portion and remainder；
Step S220：The file process is carried out to the partitioning portion, according in the file process moment server
System resource obtain segmentation correction value, the preset segmentation threshold is updated to the residue with the segmentation correction value
Split next time part；
Step S230：Obtain and preserve the result of the file process.
Specifically, in step S210, wherein, it can be set in advance, Huo Zhegen that the preset segmentation threshold, which is,
Obtained according to the system resource of current server.Wherein, obtained according to the system resource of current server, there are two kinds of possible sides
Formula, one is, before pending file is integrally not processed, calculated according to the system resource of server current state
Learn；The second is during file is processed, while the file for being divided into fritter is carrying out file process,
According to system when handling file remaining system resource calculate to file process next time when rational file size cut
Score value.
Wherein, the partitioning portion is to be currently needed for processed small documents, and the remainder is that big file excision is small
Remaining part after file, this part will re-start segmentation after the file division threshold value is updated, then be calculated.
Specifically, in step S220, the file process can include the compression of file, calculating, storage, transmission and
Any type of processing such as broadcasting.While the small documents obtained to first step cutting carry out processing processing, when obtaining current
The amount at quarter.The segmentation each time of pending file all needs to handle again should divided threshold it is handled next time when
It is worth size.For divided current small documents when participating in data processing, synchronize progress is next file division threshold value
Calculating, i.e., described segmentation correction value, and the preset segmentation threshold is updated with the segmentation correction value, so as to current small text
Directly the remaining remainder of pending file can be carried out according to the segmentation threshold for being updated over obtaining after the completion of the processing of part
Split next time and load, it is time saving and efficient.
The system resource for obtaining server, refer specifically to obtain the CPU idleness of server and the free value of internal memory
Size.
Specifically, CPU idleness is generally represented with the size of idleCPU value, idleCPU i.e. system idle,
Its complete name is System Idle Process, i.e. system free value, and the bigger explanation CPU of IdleCPU values takes lower.Example
Such as, when " System Idle Process " processes take resource be 2% when, illustrate machine at present only 2% resource be free time
, i.e., machine may be infected virus or occupied by other programs 98% resource.In other words, " System Idle
It is more that Process " processes take the more big then system available resources of resource.In the present embodiment, shell scripts can be used real
When acquisition server CPU idleness, certainly, it is any can realize the embodiment of the present application server resource utilization rate collection place
Reason machine language, software or hardware are within the protection domain of the application.
At the same time, the embodiment of the present application is while server CPU idleness is obtained, it is also necessary to obtains server residue
Internal storage data.Server memory is primarily used to store ephemeral data, does caching, interim processing data is put into interior
The inside is deposited, avoids directly reading hard disk, the reading speed of file during so as to improve data processing.Server system is shared interior in itself
Deposit the size fewer, internal memory needs consume, the size depending on the data volume that server is handled.Therefore, in file point
Before cutting, the foundation using internal memory surplus value as file division in data handling procedure, it can dynamically adjust and be loaded file
Size, so as to avoid load document is excessive from causing internal memory to overflow.
Specifically, after getting the system resource of server, the free value of idleness and the internal memory to the CPU
Summation is weighted, and using the result of the weighted sum as the file division threshold value.The formula of specific weighted sum is such as
Under：
Free value/64*N of segmentation threshold=CPU idleness+internal memory, wherein, N is integer, represents the block number of internal memory,
The unit of the internal memory free value is MB.
Wherein, calculating of the idleness of the CPU on file division threshold value influences smaller, so, the formula can be with
Following manner is present：
Free value/64*N of segmentation threshold=internal memory, wherein, N is integer, represents the block number of internal memory, and the internal memory is idle
The unit of value is MB.
For example, on an application server, idle idle values are 80%, and free memory value is 8000MB, then splits
File threshold value：0.8+8000MB/64*2=250.8, then it is to be segmented into 250.8MB by the partitioning portion of pending file division
When, treatment effeciency highest, if pending file size be less than 250.8MB certainly, then do not influenceing processing speed soon same
When add number of processes, if pending file size is more than 250.8MB, then be likely to result in cpu load it is superfluous and
Internal memory overflows.
While the partitioning portion that size is 250.8MB is in processing, continue to obtain the current surplus of server system resources
Surplus, with increasing for processing task, it is assumed that now idle values are 85%, and free memory value is 7500MB, then splits correction value
For：0.75+7500/64*2=235.125MB.The small documents for participating in file process next time should be 235.125MB in cutting
When, its treatment effeciency highest.
While the partitioning portion that size is 235.125MB is in processing, continue to obtain the current of server system resources
Surplus, the file division threshold value remaining remainder of pending file is divided next time when is calculated, so as to reasonable basis
The surplus of server current system resource constantly dynamically updates the size of file division threshold value.Specifically, in step S230,
Increase the result of a thread real-time collecting server and result is stored among hard disk, when all divided
Part exports overall result again after the completion of all handling, so as to avoid the processing failure of a certain link or processing time-out
It is caused to export any result.
It should be noted that in the embodiment of the present application, obtained while carrying out the file process to the result of the segmentation
Take the system resource of server, when the idleness for judging the server CPU be more than default first threshold and, when judging institute
The free value for stating internal memory is more than default Second Threshold, then is used as the segmentation correction value using the preset segmentation threshold.That is,
If the surplus value for the internal memory of file process is more and change floating is smaller, then can be after single treatment, use is identical
The file division threshold value of size is calculated pending big file division into small documents of a size, so as to save
For calculating the time of segmentation correction value, CPU caused by also avoiding excessive calculating takes.It should be noted that the application is real
Apply in example, the result for splitting to obtain is the minimum unit of file process each time, is limited by system resource, and each moment works as
Preceding segmentation threshold is not necessarily identical, and the minimum unit that thus be accordingly used in file process is not usually fixed.In this implementation,
During separation calculation is carried out to big file, the system resource of server is constantly obtained, and it is idle by calculation server
Internal memory and CPU loading condition are dynamically split to big file to be calculated, avoid because loading excessive file and
The situation for causing server memory to be overflowed occurs；At the same time, if the surplus value of Installed System Memory it is more and change float it is smaller when,
Big file is split with fixed file division threshold value, time-saving and efficiency.
Fig. 3 is the apparatus structure schematic diagram of the embodiment of the present application three, with reference to Fig. 3, a kind of file division device of the application, and tool
Body includes：
Pretreatment module 310, for obtaining server system resources, current segmentation threshold is calculated according to the system resource
Value；
First segmentation module 320, for splitting pending file according to the current segmentation threshold；
First processing module 330 is used to carry out file process to the segmentation result；
First memory module 340, for obtaining and preserving the result of the file process.
Wherein, the system resource specifically includes：Server CPU idleness and the free value of internal memory.
The pretreatment module 310 is specifically used for：The free time of idleness and the internal memory to CPU described in current time
Value is weighted summation, and using the result of the weighted sum as the current segmentation threshold.
The segmentation module 320 is specifically used for：Using command-line tool, wait to locate to described according to the current segmentation threshold
The file of reason is split.
Embodiment illustrated in fig. 3 can perform embodiment illustrated in fig. 1 methods described, and its implementing principle and technical effect is not done superfluous
State.
Fig. 4 is the apparatus structure schematic diagram of the embodiment of the present application four, with reference to Fig. 4, a kind of file division device of the application, and tool
Body includes：
Second segmentation module 410, for according to preset segmentation threshold split pending file obtain partitioning portion with
And remainder；Second processing module 420, for carrying out file process to the partitioning portion, according in the file process
The system resource of moment server obtains segmentation correction value, with the segmentation correction value update the preset segmentation threshold to
The remainder is split next time；
Second memory module 430, the result of the file process is carried out for obtaining and preserving.
Wherein, the system resource specifically includes：Server CPU idleness and the free value of internal memory.
Wherein, the Second processing module 420 is specifically used for：The free time of idleness and the internal memory to the CPU
Value is weighted summation, and using the result of the weighted sum as the segmentation correction value.
Wherein, the second segmentation module 410 is specifically used for：Using command-line tool, according to the segmentation threshold to institute
Pending file is stated to be split.
Wherein, the Second processing module 420 is specifically additionally operable to：When the idleness for judging the server CPU is more than in advance
If first threshold and, when the free value for judging the internal memory is more than default Second Threshold, then with the preset segmentation
Threshold value is as the segmentation correction value.
Embodiment illustrated in fig. 4 can perform embodiment illustrated in fig. 2 methods described, and its implementing principle and technical effect is not done superfluous
State.
Some preferred embodiments of the present invention have shown and described in described above, but as previously described, it should be understood that the present invention
Be not limited to form disclosed herein, be not to be taken as the exclusion to other embodiment, and available for various other combinations,
Modification and environment, and above-mentioned teaching or the technology or knowledge of association area can be passed through in the scope of the invention is set forth herein
It is modified., then all should be in this hair and the change and change that those skilled in the art are carried out do not depart from the spirit and scope of the present invention
In the protection domain of bright appended claims.
Claims (18)
- A kind of 1. file splitting method, it is characterised in that including：Server system resources are obtained, current segmentation threshold is calculated according to the system resource；Pending file is split according to the current segmentation threshold, file process is carried out to the segmentation result；Obtain and preserve the result of the file process.
- 2. the method as described in claim 1, it is characterised in that the system resource specifically includes：Server CPU idleness And the free value of internal memory.
- 3. method as claimed in claim 2, it is characterised in that current segmentation threshold is calculated according to the system resource, specifically Including：The free value of idleness and the internal memory to CPU described in current time is weighted summation, and the weighting is asked The result of sum is as the current segmentation threshold.
- 4. the method as described in claim 1, it is characterised in that according to the file of the current segmentation threshold processing to be split, Specifically include：Using command-line tool, the pending file is split according to the current segmentation threshold.
- A kind of 5. file splitting method, it is characterised in that including：Split pending file according to preset segmentation threshold, obtain partitioning portion and remainder；File process is carried out to the partitioning portion, obtaining segmentation according to the system resource of the server in the file process repaiies On the occasion of updating the preset segmentation threshold with the segmentation correction value to be split to the remainder next time；Obtain and preserve the result of the file process.
- 6. method as claimed in claim 5, it is characterised in that the system resource specifically includes：Server CPU idleness And the free value of internal memory.
- 7. method as claimed in claim 6, it is characterised in that obtain segmentation correction value, specifically include：The free value of idleness and the internal memory to the CPU is weighted summation, and by the result of the weighted sum As the segmentation correction value.
- 8. method as claimed in claim 5, it is characterised in that split pending file according to preset segmentation threshold, obtain To partitioning portion and remainder, specifically include：Using command-line tool, the pending file is split according to the segmentation threshold.
- 9. method as claimed in claim 6, it is characterised in that segmentation correction value is obtained according to the system resource, in addition to：When the idleness for judging the server CPU be more than default first threshold and, when the free value for judging the internal memory More than default Second Threshold, then the segmentation correction value is used as using the preset segmentation threshold.
- A kind of 10. file division device, it is characterised in that including：Pretreatment module, for obtaining server system resources, current segmentation threshold is calculated according to the system resource；First segmentation module, for splitting pending file according to the current segmentation threshold；First processing module, for carrying out file process to the segmentation result；First memory module, for obtaining and preserving the result of the file process.
- 11. device as claimed in claim 10, it is characterised in that the system resource specifically includes：Server CPU free time The free value of rate and internal memory.
- 12. device as claimed in claim 11, it is characterised in that the pretreatment module is specifically used for：The free value of idleness and the internal memory to CPU described in current time is weighted summation, and the weighting is asked The result of sum is as the current segmentation threshold.
- 13. device as claimed in claim 10, it is characterised in that the first segmentation module is specifically used for：Using command-line tool, the pending file is split according to the current segmentation threshold.
- 14. a kind of file division device, it is characterised in that including following module：Second segmentation module, partitioning portion and remainder are obtained for splitting pending file according to preset segmentation threshold Point；Second processing module, for carrying out file process to the partitioning portion, according in the file process moment server System resource obtain segmentation correction value, the preset segmentation threshold is updated to the residue with the segmentation correction value Split next time part；Second memory module, for obtaining and preserving the result of the file process.
- 15. device as claimed in claim 14, it is characterised in that the system resource specifically includes：Server CPU free time The free value of rate and internal memory.
- 16. device as claimed in claim 15, it is characterised in that the Second processing module is specifically used for：The free value of idleness and the internal memory to the CPU is weighted summation, and by the result of the weighted sum As the segmentation correction value.
- 17. device as claimed in claim 14, it is characterised in that the second segmentation module is specifically used for：Using command-line tool, the pending file is split according to the segmentation threshold.
- 18. device as claimed in claim 15, it is characterised in that the Second processing module is specifically additionally operable to：When the idleness for judging the server CPU be more than default first threshold and, when the free value for judging the internal memory More than default Second Threshold, then the segmentation correction value is used as using the preset segmentation threshold.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610390987.1A CN107463332B (en) | 2016-06-03 | 2016-06-03 | File segmentation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610390987.1A CN107463332B (en) | 2016-06-03 | 2016-06-03 | File segmentation method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107463332A true CN107463332A (en) | 2017-12-12 |
| CN107463332B CN107463332B (en) | 2020-12-29 |
Family
ID=60545791
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610390987.1A Active CN107463332B (en) | 2016-06-03 | 2016-06-03 | File segmentation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107463332B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109101531A (en) * | 2018-06-22 | 2018-12-28 | 联想(北京)有限公司 | Document handling method, apparatus and system |
| CN109190094A (en) * | 2018-09-05 | 2019-01-11 | 盈嘉互联（北京）科技有限公司 | Building Information Model file cutting method based on IFC standard |
| CN109413509A (en) * | 2018-12-06 | 2019-03-01 | 武汉微梦文化科技有限公司 | A kind of HD video processing method |
| CN110515964A (en) * | 2019-08-30 | 2019-11-29 | 百度在线网络技术（北京）有限公司 | A kind of file updating method, device, electronic equipment and medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101557407A (en) * | 2008-04-11 | 2009-10-14 | 盛大计算机（上海）有限公司 | Transmission and storage method of program-ordering data of high definition media P2P |
| CN105528371A (en) * | 2014-09-30 | 2016-04-27 | 北京金山云网络技术有限公司 | Method, device, and system for executing writing task |
| US20160154604A1 (en) * | 2014-12-02 | 2016-06-02 | International Business Machines Corporation | Writing files to a storage medium |
-
2016
- 2016-06-03 CN CN201610390987.1A patent/CN107463332B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101557407A (en) * | 2008-04-11 | 2009-10-14 | 盛大计算机（上海）有限公司 | Transmission and storage method of program-ordering data of high definition media P2P |
| CN105528371A (en) * | 2014-09-30 | 2016-04-27 | 北京金山云网络技术有限公司 | Method, device, and system for executing writing task |
| US20160154604A1 (en) * | 2014-12-02 | 2016-06-02 | International Business Machines Corporation | Writing files to a storage medium |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109101531A (en) * | 2018-06-22 | 2018-12-28 | 联想(北京)有限公司 | Document handling method, apparatus and system |
| CN109101531B (en) * | 2018-06-22 | 2022-05-31 | 联想(北京)有限公司 | File processing method, device and system |
| CN109190094A (en) * | 2018-09-05 | 2019-01-11 | 盈嘉互联（北京）科技有限公司 | Building Information Model file cutting method based on IFC standard |
| CN109190094B (en) * | 2018-09-05 | 2023-03-10 | 盈嘉互联（北京）科技有限公司 | Building information model file segmentation method based on IFC standard |
| CN109413509A (en) * | 2018-12-06 | 2019-03-01 | 武汉微梦文化科技有限公司 | A kind of HD video processing method |
| CN110515964A (en) * | 2019-08-30 | 2019-11-29 | 百度在线网络技术（北京）有限公司 | A kind of file updating method, device, electronic equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107463332B (en) | 2020-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107463332A (en) | A kind of file splitting method and device | |
| US11340803B2 (en) | Method for configuring resources, electronic device and computer program product | |
| US11544300B2 (en) | Reducing storage required for an indexing structure through index merging | |
| WO2016205977A1 (en) | Techniques to run one or more containers on virtual machine | |
| US8667489B2 (en) | Systems and methods for sharing the results of analyses among virtual machines | |
| WO2022105440A1 (en) | Hybrid quantum-classical cloud platform and task execution method | |
| US11831410B2 (en) | Intelligent serverless function scaling | |
| US20150046620A1 (en) | Pessimistic interrupt affinity for devices | |
| US11960941B2 (en) | Managing virtual machine memory balloon using time series predictive data | |
| US10831532B2 (en) | Updating a nested virtualization manager using live migration of virtual machines | |
| CN111679860B (en) | Distributed information processing method and device | |
| US20120324194A1 (en) | Adjusting the amount of memory allocated to a call stack | |
| US9983908B1 (en) | Adjusting allocation of selected resources for capped and uncapped virtual machines | |
| US11106380B2 (en) | Migration of storage for workloads between desktop and cloud environments | |
| WO2021094859A1 (en) | Region constrained regularized adversarial examples for model interpretability | |
| US9880740B2 (en) | Rapid virtual machine power-on with iterative swap space reservation | |
| CN112286638A (en) | Method for realizing hot plug between CPU and memory of Openstack virtual machine | |
| US20210182191A1 (en) | Free memory page hinting by virtual machines | |
| US10108773B1 (en) | Partitioning circuit designs for implementation within multi-die integrated circuits | |
| US10810129B2 (en) | Application memory organizer | |
| CN110825660A (en) | HBase-based optimization method, system and equipment for Julient suspension | |
| US9917598B1 (en) | Implementing preemptive customized codeset converter selection on SAAS | |
| US9734270B2 (en) | Control path power adjustment for chip design | |
| CN111381974A (en) | Method for optimizing Drools response time in large-scale high-concurrency scene | |
| CN110928673A (en) | Task allocation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |