CN105224291B

CN105224291B - A kind of data processing method and device

Info

Publication number: CN105224291B
Application number: CN201510632540.6A
Authority: CN
Inventors: 许晓炜
Original assignee: Beijing QIYI Century Science and Technology Co Ltd
Current assignee: Beijing QIYI Century Science and Technology Co Ltd
Priority date: 2015-09-29
Filing date: 2015-09-29
Publication date: 2017-12-08
Anticipated expiration: 2035-09-29
Also published as: CN105224291A

Abstract

The embodiment of the invention discloses a kind of data processing method and device, method includes：Sampled for initial data, and according to preset rules, sampled data is divided into M the first data blocks；Count the first handling duration of M the first data blocks；According to preset rules, initial data is divided into M the second data blocks；According to the handling duration of the first data block, the handling duration in the second data block is determined；Handling duration is more than to the data point reuse in the second data block of given threshold to handling duration to be less than in the second data block of given threshold；After the completion of adjustment, M the second data blocks of parallel processing.The handling duration for each data block that the embodiment of the present invention divides according to sampled data, determine the handling duration of each data block of initial data division, and then the data volume of the data block of initial data division is adjusted, realize that the handling duration of the data block of initial data division is more uniform, further reduce the disposed of in its entirety duration of system.

Description

A kind of data processing method and device

Technical field

The present invention relates to distributed computing technology field, more particularly to a kind of data processing method and device.

Background technology

Some current distributed computing systems, when carrying out data processing, it is generally the case that need to divide data Block, parallel processing is carried out to data block.

For the above-mentioned distributed computing system based on deblocking parallel processing, the handling duration of whole system data takes Certainly in the most long data block of handling duration.

If the handling duration of divided data block is uneven, the handling duration of some data blocks is longer, can cause whole The handling duration of system data extends, so as to influence the overall performance of system.Such as：Be intended to find out comprising 250,360,600,960, 251st, the prime number in 359,601 and 953 data, then 250,360,600,960,251,359,601 and 953 data will be included 2 two data blocks of data block 1 and data block are divided into, wherein, the data in data block 1 are { 250,360,600,960 }, data Data in block 2 are { 251,359,601,953 }.Although the size of data block 1 and data block 2 is just the same, it is apparent that data The handling duration of block 2 is longer, handles the handling duration that the duration of all data is depended in processing data block 2.

The content of the invention

The purpose of the embodiment of the present invention is to provide a kind of data processing method and device, to improve the performance of system.

To reach above-mentioned purpose, the embodiment of the invention discloses a kind of data processing method, applied to Distributed Calculation system System, methods described include：

Sampled for initial data, obtain sampled data；

According to preset data block division rule, the sampled data is divided into M the first data blocks；

Count the first handling duration of each first data block in the M the first data blocks；

According to the preset data block division rule, initial data is divided into M the second data blocks, wherein, the M Second data block corresponds with the M the first data blocks；

The first handling duration of each first data block, determines institute in the M the first data blocks obtained according to statistics State the second processing duration of each second data block in M the second data blocks；

According to the second processing duration of each second data block of determination, first threshold is determined, so that second processing duration The second data block and second processing duration more than the first threshold are less than the second data block of the first threshold respectively extremely Rare one；

By second processing duration more than the data point reuse in the second data block of the first threshold to second processing duration Less than in the second data block of the first threshold；

After the completion of adjustment, M described in parallel processing the second data blocks.

Preferably, second processing duration be more than the first threshold the second data block have it is multiple in the case of,

Data point reuse second processing duration being more than in the second data block of the first threshold is to second processing Duration is less than in the second data block of the first threshold, including：

According to the order of second processing duration from long to short, second processing duration is more than the of the first threshold successively Data point reuse in two data blocks is less than to second processing duration in the second data block of the first threshold.

Preferably, data point reuse second processing duration being more than in the second data block of the first threshold is to the Two handling durations are less than in the second data block of the first threshold, including：

It is more than the data that predetermined number is extracted in each second data block of the first threshold from second processing duration；

A random number is generated in total value scope, wherein, the total value scope is less than institute for second processing duration The number range that number range corresponding to each second data block of first threshold is formed is stated, the second processing duration is less than institute It is to be less than the first threshold according to second processing duration to state number range corresponding to each second data block of first threshold What the second processing duration of each second data block determined, and the second processing duration is less than each the of the first threshold Number range corresponding to two data blocks is continuous and not overlapping；

Number range corresponding to each second data block for being less than the first threshold according to the second processing duration of determination And the random number, the second data block corresponding to the data area that the random number is in, it is defined as data to be received Second data block；

The data of the predetermined number extracted are adjusted into the second data block of the data to be received of determination.

Preferably, second processing duration be less than the first threshold the second data block have it is multiple in the case of,

Data point reuse second processing duration being more than successively in the second data block of the first threshold is to second Handling duration is less than in the second data block of the first threshold, including：

Never adjust successively data and second processing duration be more than it is in the second data block of the first threshold, the Extract the data of predetermined number in the second most long data block of two handling durations, and adjusted to do not adjust data and Second processing duration is less than in the second data block in the second data block of the first threshold, second processing duration is most short；

If the quantity that second processing duration is more than the second data block of the first threshold is small more than second processing duration In the quantity of the second data block of the first threshold, continue to be more than from remaining do not adjust data and second processing duration The number of predetermined number is extracted in the second data block in second data block of the first threshold, second processing duration is most long According to, and adjusted to second processing duration and be less than in the second data block of the first threshold.

The data of the predetermined number extracted are adjusted to second processing duration at random and are less than the first threshold In second data block.

The embodiment of the invention also discloses a kind of data processing method, and applied to distributed computing system, method includes：

Initial data is divided into N number of 3rd data block；

Sampled for each 3rd data block, obtain the sampled data of each 3rd data block；

For the sampled data of each 3rd data block, a newly-generated data block；

3rd handling duration of each newly-generated data block of statistics；

3rd handling duration of each newly-generated data block obtained according to statistics, determines N number of 3rd data block In each 3rd data block fourth process duration；

According to the fourth process duration of each 3rd data block of determination, Second Threshold is determined, so that fourth process duration The 3rd data block and fourth process duration more than the Second Threshold are less than the 3rd data block of the Second Threshold respectively extremely Rare one；

By fourth process duration more than the data point reuse in the 3rd data block of the Second Threshold to fourth process duration Less than in the 3rd data block of the Second Threshold；

After the completion of adjustment, N number of 3rd data block described in parallel processing.

The embodiment of the invention also discloses a kind of data processing equipment, and applied to distributed computing system, described device can With including：First sampling module, the first data block division module, the first handling duration statistical module, the second data block division mould Block, second processing duration determining module, first threshold determining module, the first adjusting module and first processing module, wherein,

First sampling module, for being sampled for initial data, obtains sampled data；

The first data block division module, for according to preset data block division rule, the sampled data to be divided For M the first data blocks；

The first handling duration statistical module, for counting each first data block in the M the first data blocks First handling duration；

The second data block division module, for according to the preset data block division rule, initial data to be divided For M the second data blocks, wherein, the M the second data blocks correspond with the M the first data blocks；

The second processing duration determining module, for according in the obtained M the first data blocks of statistics each the First handling duration of one data block, determine the second processing duration of each second data block in the M the second data blocks；

The first threshold determining module, for the second processing duration of each second data block according to determination, it is determined that First threshold, so that second processing duration is more than the second data block of the first threshold and second processing duration is less than described the Second data block of one threshold value distinguishes at least one；

First adjusting module, for the number being more than second processing duration in the second data block of the first threshold It is less than according to adjusting to second processing duration in the second data block of the first threshold；

The first processing module, after the completion of adjustment, M described in parallel processing the second data blocks.

Preferably, second processing duration be more than the first threshold the second data block have it is multiple in the case of, it is described First adjusting module, is specifically used for：

Preferably, first adjusting module, including：First extracting sub-module, the first generation submodule, first determine son Module and the first adjustment submodule, wherein,

First extracting sub-module, for being more than each second data block of the first threshold from second processing duration In extract the data of predetermined number；

The first generation submodule, for generating a random number in total value scope, wherein, the total value model The number range for being less than number range corresponding to each second data block of the first threshold for second processing duration and forming is enclosed, It is according to second processing that the second processing duration, which is less than number range corresponding to each second data block of the first threshold, What the second processing duration that duration is less than each second data block of the first threshold determined, and the second processing duration is small Number range corresponding to each second data block in the first threshold is continuous and not overlapping；

First determination sub-module, for being less than each the of the first threshold according to the second processing duration of determination Number range corresponding to two data blocks and the random number of the first generation submodule generation, submodule is generated by described first Second data block corresponding to the data area that the random number of generation is in, it is defined as the second data block of data to be received；

The first adjustment submodule, for the data for the predetermined number for extracting first extracting sub-module, is adjusted In second data block of the data to be received that first determination sub-module determines.

First adjusting module, including：Second extracting sub-module and the second adjustment submodule, wherein,

Second extracting sub-module, for successively never adjust data and second processing duration be more than described first The data of predetermined number are extracted in the second data block in second data block of threshold value, second processing duration is most long；

The second adjustment submodule, for the data for the predetermined number for extracting second extracting sub-module, according to Secondary adjust to do not adjust data and second processing duration is less than in the second data block of the first threshold, second processing In the most short data block of duration；

The quantity for being more than the second data block of the first threshold in second processing duration is less than more than second processing duration In the case of the quantity of second data block of the first threshold,

Second extracting sub-module, it is additionally operable to continue to be more than from remaining do not adjust data and second processing duration The number of predetermined number is extracted in the second data block in second data block of the first threshold, second processing duration is most long According to；

The second adjustment submodule, it is additionally operable to second extracting sub-module continuing the number of predetermined number that extracts It is less than according to adjusting to second processing duration in the second data block of the first threshold.

Preferably, first adjusting module, including：3rd extracting sub-module and the 3rd adjustment submodule, wherein,

3rd extracting sub-module, for being more than each second data block of the first threshold from second processing duration In extract the data of predetermined number；

The 3rd adjustment submodule, for the data of the predetermined number extracted to be adjusted to second processing at random Duration is less than the second data block of the first threshold.

The embodiment of the invention also discloses a kind of data processing equipment, and applied to distributed computing system, device includes：The Three data block division modules, the second sampling module, new data block generation module, the 3rd handling duration statistical module, fourth process Duration determining module, Second Threshold determining module, the second adjusting module and Second processing module, wherein,

The 3rd data block division module, for initial data to be divided into N number of 3rd data block；

Second sampling module, for being sampled for each 3rd data block, obtain each described 3rd The sampled data of data block；

The new data block generation module, for the sampled data for each 3rd data block, a newly-generated number According to block；

The 3rd handling duration statistical module, for counting the 3rd handling duration of each newly-generated data block；

The fourth process duration determining module, for obtained according to statistics the 3rd of each newly-generated data block at Duration is managed, determines the fourth process duration of each 3rd data block in N number of 3rd data block；

The Second Threshold determining module, for the fourth process duration of each 3rd data block according to determination, it is determined that Second Threshold, so that fourth process duration is more than the 3rd data block of the Second Threshold and fourth process duration is less than described the 3rd data block of two threshold values distinguishes at least one；

Second adjusting module, for the number being more than fourth process duration in the 3rd data block of the Second Threshold It is less than according to adjusting to fourth process duration in the 3rd data block of the Second Threshold；

The Second processing module, after the completion of adjustment, N number of 3rd data block described in parallel processing.

As seen from the above technical solutions, the embodiment of the present invention samples to initial data, by sampled data according to pre- If data block division rule is divided into multiple data blocks, the handling duration of each data block is counted.Then preset according to identical Data block division rule, initial data is divided into the one-to-one data block of multiple data blocks with sampled data division, and The handling duration of each data block divided according to sampled data, when determining the processing of each data block of initial data division It is long, and then the data volume of the data block of initial data division is adjusted, to realize the place of the data block of initial data division Reason duration is more uniform, the disposed of in its entirety duration of system is further reduced, so as to improve the performance of system.Certainly, this hair is implemented Bright any product or method must be not necessarily required to reach all the above advantage simultaneously.

Brief description of the drawings

In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing There is the required accompanying drawing used in technology description to be briefly described, it should be apparent that, drawings in the following description are only this Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with Other accompanying drawings are obtained according to these accompanying drawings.

Fig. 1 is a kind of schematic flow sheet of data processing method provided in an embodiment of the present invention；

Fig. 2 is another schematic flow sheet of data processing method provided in an embodiment of the present invention；

Fig. 3 is a kind of structural representation of data processing equipment provided in an embodiment of the present invention；

Fig. 4 is another structural representation of data processing equipment provided in an embodiment of the present invention.

Embodiment

Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, rather than whole embodiments.It is based on Embodiment in the present invention, those of ordinary skill in the art are obtained every other under the premise of creative work is not made Embodiment, belong to the scope of protection of the invention.

In order to solve prior art problem, the embodiments of the invention provide a kind of data processing method and device.Lower kept man of a noblewoman First data processing method provided in an embodiment of the present invention is described in detail.

Fig. 1 is a kind of schematic flow sheet of data processing method provided in an embodiment of the present invention, can be included：

S101：Sampled for initial data, obtain sampled data.

S102：According to preset data block division rule, sampled data is divided into M the first data blocks.

S103：Count the first handling duration of each first data block in M the first data blocks.

S104：According to preset data block division rule, initial data is divided into M the second data blocks.

Wherein, above-mentioned M the second data blocks correspond with above-mentioned M the first data blocks.

S105：The first handling duration of each first data block, determines M in M the first data blocks obtained according to statistics The second processing duration of each second data block in individual second data block.

S106：According to the second processing duration of each second data block of determination, first threshold is determined.

S107：By second processing duration more than the data point reuse in the second data block of first threshold to second processing duration Less than in the second data block of first threshold.

S108：After the completion of adjustment, M the second data blocks of parallel processing.

Assuming that initial data there are 100,000, sampled for initial data, obtain 1000 sampled datas.Preset data Block division rule can be：By data divided by 7, remainder identical data are divided into same data block.

, by each sampled data in 1000 sampled datas divided by 7, will according to above-mentioned preset data block division rule Remainder identical sampled data is divided into same first data block, i.e. has divided 7 the first data blocks：Remainder is 0 sampling First data block of data composition is M0, and the first data block of the sampled data composition that remainder is 1 is M1, and remainder is 2 sampling First data block of data composition is M2, and the first data block of the sampled data composition that remainder is 3 is M3, and remainder is 4 sampling First data block of data composition is M4, and the first data block of the sampled data composition that remainder is 5 is M5, and remainder is 6 sampling First data block of data composition is M6.

Count the first handling duration of each data block in this 7 first data blocks, it is assumed that the first data block counted on M0 the first handling duration is 0.4 second, and in order to which following statement is convenient, the first data block M0 the first handling duration can for 0.4 second To be abbreviated as M0：0.4 second, the first handling duration according to the first data block M0 was the shorthand way of 0.4 second, and other first are counted It is abbreviated as successively according to the first handling duration of block：

M1：0.6 second, M2：0.3 second, M3：0.5 second, M4：0.2 second, M5：0.7 second, M6：0.1 second.

According to identical preset data block division rule, by each initial data in 100,000 initial data divided by 7, Remainder identical initial data is divided into same second data block, i.e. initial data is divided into 7 the second data blocks： Second data block of the initial data composition that remainder is 0 is N0, and the second data block of the initial data composition that remainder is 1 is N1, Second data block of the initial data composition that remainder is 2 is N2, and the second data block of the initial data composition that remainder is 3 is N3, Second data block of the initial data composition that remainder is 4 is N4, and the second data block of the initial data composition that remainder is 5 is N5, Second data block of the initial data composition that remainder is 6 is N6.This 7 second data blocks and above-mentioned 7 the first data blocks are one by one It is corresponding.Wherein, M0 is corresponding with N0, and M1 is corresponding with N1, and M2 is corresponding with N2, and M3 is corresponding with N3, and M4 is corresponding with N4, and M5 is corresponding with N5, M6 is corresponding with N6.

According to the first handling duration of 7 the first data blocks, the second processing duration of 7 the second data blocks is determined.Second The determination method of handling duration can have two kinds.

The first determination method, the second processing duration of this 7 the second data blocks can be defined as and 7 the first data First handling duration of block is equal, according to above-mentioned shorthand way, successively by the second processing duration letter of each second data block It is written as：

N0：0.4 second, N1：0.6 second, N2：0.3 second, N3：0.5 second, N4：0.2 second, N5：0.7 second, N6：0.1 second.

Second of determination method, can be according to sampled data and the ratio of the data volume of initial data, and 7 the first data First handling duration of block, to determine the second processing duration of this 7 the second data blocks.Sampled data has 1000, original number According to there is 100,000, the data volume ratio of sampled data and initial data is 1：100, when the first of 7 the first data blocks is handled It is long to be multiplied by the second processing duration that 100 obtained handling durations are corresponding 7 the second data blocks.According to the above-mentioned side of writing a Chinese character in simplified form Formula, the second processing duration of each second data block is abbreviated as successively：

N0：40 seconds, N1：60 seconds, N2：30 seconds, N3：50 seconds, N4：20 seconds, N5：70 seconds, N6：10 seconds.

According to the second processing duration of each second data block of determination, first threshold is determined., should when determining first threshold Make first threshold as far as possible close to the average value of the second processing duration of this 7 the second data blocks, so that second processing duration is more than The second data block that the second data block and second processing duration of first threshold are less than first threshold distinguishes at least one, and Quantity is roughly equal.

Second of determination method of second processing duration is taken, and first threshold is defined as 40 seconds, therefore, second processing Duration be more than first threshold the second data block be：N1, N3 and N5, second processing duration are less than the second data block of first threshold For：N2, N4 and N6.

By the data point reuse in the second data block N1, N3, N5 into the second data block N2, N4, N6.

After the completion of adjustment, this 7 second data blocks of parallel processing.

The embodiment of the present invention is sampled to initial data, and sampled data is divided into according to preset data block division rule Multiple data blocks, count the handling duration of each data block.Then according to identical preset data block division rule, by original number According to the one-to-one data block of multiple data blocks being divided into sampled data division, and according to each number of sampled data division According to the handling duration of block, the handling duration of each data block of initial data division is determined, and then to the number of initial data division It is adjusted according to the data volume of block, to realize that the handling duration of the data block of initial data division is more uniform, further reduces The disposed of in its entirety duration of system, so as to improve the performance of system.

In actual applications, it is multiple to there is a situation where that second processing duration has more than the second data block of first threshold, this When by second processing duration be more than first threshold the second data block in data point reuse to second processing duration be less than the first threshold In second data block of value, order that can be according to second processing duration from long to short, second processing duration is more than the successively Data point reuse in second data block of one threshold value is less than to second processing duration in the second data block of first threshold.

Exemplary, also illustrated with above-mentioned example, first threshold is defined as 40 seconds；Second processing duration is more than first Second data block of threshold value is：N1, N3 and N5, the second data block that second processing duration is less than first threshold are：N2, N4 and N6。

Second processing duration be more than first threshold the second data block have it is multiple, can according to second processing duration by grow to Short order, N5 is first adjusted, then adjust N1, finally adjust N3, such a adjustment order is more logical, naturally it is also possible to according to other Sequentially, it is not limited herein.

Specifically, when second processing duration to be more than to data point reuse to the second processing in the second data block of first threshold In long the second data block less than first threshold, method of adjustment can have three kinds.

The first method of adjustment, it can include：

It is more than the data that predetermined number is extracted in each second data block of first threshold from second processing duration；

A random number is generated in total value scope, wherein, total value scope is that second processing duration is less than the first threshold The number range that number range corresponding to each second data block of value is formed, second processing duration are less than each of first threshold Number range corresponding to second data block is the second of each second data block for being less than first threshold according to second processing duration What handling duration determined, and second processing duration be less than number range corresponding to each second data block of first threshold it is continuous and It is not overlapping；

According to the second processing duration of determination be less than first threshold each second data block corresponding to number range and Random number, the second data block corresponding to the data area that random number is in, it is defined as the second data block of data to be received；

Assuming that the data of predetermined number are 1000 data.

1000 data are extracted from N5.

It is less than each second the data block N2, N4, N6 of first threshold second processing duration according to second processing duration, N2：30 seconds, N4：20 seconds, N6：10 seconds, determine that second processing duration is less than each second the data block N2, N4, N6 of first threshold Corresponding number range.These three number ranges are continuous and not overlapping, and meet the second shorter data of second processing duration Number range is bigger corresponding to block.These three number ranges form a total size, and a random number is generated in total size, should Which number range is random number is in, and just extracted from N5 1000 data are sent to corresponding to that number range Second data block.

More reasonably the data volume of the second data block is adjusted in order to realize, the second shorter number of second processing duration The probability that the data that predetermined number is extracted from N5 are received according to block should be bigger.Improving second data block reception, this is extracted The probability of data, it can be realized by expanding the number range of the second data block.

Such as N2：30 seconds, N4：20 seconds, N6：10 seconds, N6 second processing duration was most short, receives what is extracted from N5 The probability of 1000 data should be bigger, and secondly, N2 should be minimum by N4.Accordingly, it is determined that number range corresponding to N6 is 1-50, really It is 51-80 to determine number range corresponding to N4, and it is 81-100 to determine number range corresponding to N2.Total value scope is 1-100, in 1- A random number is generated in 100.Assuming that generation random number for 30,30 be in 1-50 between, by what is extracted from N5 1000 data are sent in the second data block corresponding to 1-50 number range, i.e. in N6.

Above-mentioned determination second processing duration is less than the mode of number range corresponding to each second data block of first threshold, There can be two kinds.The mode of the first determination number range, only it can be determined once before data are adjusted, afterwards each number During according to adjustment, a random number only need to be generated in total value scope, judges which number range is the random number be in.The The mode of two kinds of determination number ranges, second processing duration can be all redefined less than first threshold during each data point reuse Number range corresponding to each second data block, regenerates random number.

In the present embodiment, by the way of the first determination number range.

Then, then N1 is adjusted, 1000 data is extracted from N1.

Assuming that generation random number 62,62 is between 51-80, extracted from N1 1000 data are sent to 51- In second data block corresponding to 80 number range, i.e. in N4.

Finally, N3 is adjusted, 1000 data are extracted from N3.

Assuming that the random number of generation for 45,45 be in 1-50 between, extracted from N3 1000 data are sent To in the second data block corresponding to 1-50 number range, i.e. in N6.

According to the first method of adjustment, the second shorter data block reception of second processing duration extracts the number of predetermined number According to probability it is bigger, rather than the data extracted are all sent to the second most short data block of second processing duration.When treating When second data block of adjustment is a lot, the data of the second data block can be more uniformly adjusted using probability, avoid a certain second Data block adjustment pre-treatment duration is very short, but due to receiving the data extracted from other second data blocks too much, causes to adjust Whole post processing duration becomes oversize, occurs so as to influence the situation of the disposed of in its entirety duration of system.

Second of method of adjustment, it can include：

Successively never adjust data and second processing duration be more than first threshold the second data block in, at second Extract the data of predetermined number in most long the second data block of reason duration, and adjusted to do not adjust data and second Handling duration is less than in the second data block in the second data block of first threshold, second processing duration is most short.

Assuming that the data of predetermined number are 1000 data.

Also illustrated with above-mentioned example, second processing duration is more than the second data block N1, N3, N5 of first threshold；The Two handling durations are less than the second data block N2, N4, N6 of first threshold.

Before adjustment, N1, N3, the second most long data block of second processing duration in N5 is N5；In N2, N4, N6 The second most short data block of two handling durations is N6.1000 data are extracted from N5, and by 1000 data point reuses extremely N6。

Now, do not adjust data and second processing duration be more than in the second data block of first threshold, second processing The second most long data block of duration is N1, do not adjust data and second processing duration be less than the second data block of first threshold In, the second data block that second processing duration is most short be N4.Extract 1000 data from N1, and by 1000 data Adjust to N4.

Now, do not adjust data and second processing duration be more than in the second data block of first threshold, second processing The second most long data block of duration is N3；Do not adjust data and second processing duration be less than the second data block of first threshold In, the second data block that second processing duration is most short be N2.Extract 1000 data from N3, and by 1000 data Adjust to N2.

As it was previously stated, when determining first threshold, should try one's best the second data block for making second processing duration be more than first threshold The the second data number of blocks for being less than first threshold with second processing duration is roughly equal.But if second processing duration is more than When the second data block and second processing duration of first threshold are unequal less than the second data number of blocks of first threshold：

If the quantity that second processing duration is more than the second data block of first threshold is less than second processing duration and is less than the The quantity of second data block of one threshold value, still according to above-mentioned second of method of adjustment, it is adjusted.

It is assumed that the second data block that second processing duration is less than first threshold removes N2, and outside N4, N6, also NX and NY, its In, NX：15 seconds, NY：25 seconds.That is, second processing duration is more than the second data block N1, N3, N5 of first threshold；During second processing Long the second data block N2, N4, N6, NX, NY less than first threshold.Then still according to second of method of adjustment：

Before adjustment, do not adjust data and second processing duration be more than it is in the second data block of first threshold, second The second most long data block of handling duration is N5；Do not adjust data and second processing duration be less than the second number of first threshold It is N6 according to the second most short data block of in block, second processing duration.1000 data are extracted from N5, and by this 1000 Data point reuse is to N6.

Now, do not adjust data and second processing duration be more than in the second data block of first threshold, second processing The second most long data block of duration is N1；Do not adjust data and second processing duration be less than the second data block of first threshold In, the second data block that second processing duration is most short be NX.Extract 1000 data from N1, and by 1000 data Adjust to NX.

Do not adjust now data and second processing duration be more than in the second data block of first threshold, second processing The second most long data block of duration is N3；Do not adjust data and second processing duration be less than the second data block of first threshold In, the second data block that second processing duration is most short be N4.Extract 1000 data from N3, and by 1000 data Adjust to N4.

If the quantity that second processing duration is more than the second data block of first threshold is more than second processing duration and is less than the The quantity of second data block of one threshold value, continue to be more than first threshold from remaining do not adjust data and second processing duration The second data block in, extract the data of predetermined number in the second data block that second processing duration is most long, and adjusted It is whole to be less than to second processing duration in the second data block of first threshold.

If the quantity that second processing duration is more than the second data block of first threshold is more than second processing duration and is less than the The quantity of second data block of one threshold value, the equal second processing duration of quantity is more than the according to above-mentioned second of method of adjustment After the second data block and second processing duration of one threshold value are less than the second data block adjustment of first threshold, continue from residue The second processing duration for not adjusting data be more than first threshold the second data block in second processing duration it is most long second The data of predetermined number are extracted in data block, and the data point reuse of the predetermined number to second processing duration is less than the first threshold In second data block of value.

It is assumed that the second data block that second processing duration is more than first threshold removes N1, and outside N3, N5, also NA and NB, its In, NA：55 seconds, NB：65 seconds.That is, second processing duration is more than the second data block N1, N3, N5, NA, NB of first threshold；Second Handling duration is less than the second data block N2, N4, N6 of first threshold.

Before adjustment, N1, N3, N5, NA, the second most long data block of second processing duration in NB is N5；N2, N4, N6 In most short the second data block of second processing duration be N6.Extract 1000 data from N5, and by 1000 data Adjust to N6.

Now, do not adjust data and second processing duration be more than in the second data block of first threshold, second processing The second most long data block of duration is NB；Do not adjust data and second processing duration be less than the second data block of first threshold In, the second data block that second processing duration is most short be N4.Extract 1000 data from NB, and by 1000 data Adjust to N4.

Now, do not adjust data and second processing duration be more than in the second data block of first threshold, second processing The second most long data block of duration is N1；Do not adjust data and second processing duration be less than the second data block of first threshold In, the second data block that second processing duration is most short be N2.Extract 1000 data from N1, and by 1000 data Adjust to N2.

Now, the data that second processing duration is less than in the second data block N2, N4, N6 of first threshold are made toning It is whole, and second processing duration is more than the second data block N1, N3, N5 of first threshold, there are data in NA and N3 in NA, NB not Carried out adjustment.Continue never to adjust and extracted in the second most long data block of the second processing duration in the NA and N3 of data 1000 data, and by 1000 data point reuses to N2, in N4, N6.

For the sake of rationally, the second data block that second processing duration is less than first threshold can be according to second processing duration Order from short to long receives 1000 data extracted successively.That is, by extract 1000 data, it is sent to successively N6, N4, N2.

NA is the second most long data block of the second processing duration in the NA and N3 for do not adjust data.It will can be carried from NA 1000 data point reuses taken out are to N6.

Now, N3 is unique the second data block for not adjusting data, can be by extracted from N3 1000 data Adjust to N4.

According to second of method of adjustment, when second processing duration is less than the second data block of first threshold according to second processing Long order from short to long receives the data of the predetermined number extracted in turn successively, and it is shorter to avoid a certain second processing duration The data that extract too much of the second data block reception, so as to which handling duration becomes oversize, influence the feelings of system disposed of in its entirety duration Condition occurs.

The third method of adjustment, it can include：

The data of the predetermined number extracted are adjusted to second processing duration to the second data for being less than first threshold at random In block.

Also illustrated with above-mentioned example, the second data block that second processing duration is more than first threshold is：N1, N3 and N5；Second processing duration be less than first threshold the second data block be：N2, N4 and N6.

Assuming that the data of predetermined number can be 1000 data.

1000 data are respectively extracted from N1, N3, N5, are adjusted at random to N2, N4, in N6.

Using embodiment illustrated in fig. 1 of the present invention, initial data is sampled, sampled data is drawn according to preset data block Divider is then divided into multiple data blocks, counts the handling duration of each data block.Then divided according to identical preset data block Rule, initial data is divided into the one-to-one data block of multiple data blocks with sampled data division, and according to hits According to the handling duration of each data block of division, the handling duration of each data block of initial data division is determined, and then to original The data volume of the data block of beginning data division is adjusted, to realize that the handling duration of the data block of initial data division is more equal It is even, the disposed of in its entirety duration of system is further reduced, so as to improve the performance of system.

Fig. 2 is another schematic flow sheet of data processing method provided in an embodiment of the present invention, can be included：

S201：Initial data is divided into N number of 3rd data block.

S202：Sampled for each 3rd data block, obtain the sampled data of each 3rd data block.

S203：For the sampled data of each 3rd data block, a newly-generated data block.

S204：3rd handling duration of each newly-generated data block of statistics.

S205：3rd handling duration of each newly-generated data block obtained according to statistics, determines N number of 3rd data block In each 3rd data block fourth process duration.

S206：According to the fourth process duration of each 3rd data block of determination, Second Threshold is determined.

S207：By fourth process duration more than the data point reuse in the 3rd data block of Second Threshold to fourth process duration Less than in the 3rd data block of Second Threshold.

S208：After the completion of adjustment, N number of 3rd data block of parallel processing.

Assuming that initial data there are 100,000, initial data is divided into 5 the 3rd data blocks：P1、P2……P5.

In actual applications, above-mentioned 100,000 former number data can be divided into 5 the 3rd data blocks, each 3rd data Block includes 20,000 initial data.

Sampled for this 5 the 3rd data blocks, obtain the sampled data of each 3rd data block in P1-P5.

For the sampled data of each 3rd data block in P1-P5, a respective newly-generated data block：Q1、Q2……Q5.

Count Q1-Q5 the 3rd handling duration, it is assumed that Q1 the first handling duration is 0.4 second, according to the above-mentioned side of writing a Chinese character in simplified form Formula, it is assumed that Q1-Q5 the 3rd handling duration is followed successively by：Q1：0.4 second, Q2：0.6 second, Q3：0.9 second, Q4：0.5 second, Q5：0.2 Second.

The Q1-Q5 obtained according to statistics the 3rd handling duration, determine P1-P5 fourth process duration.During fourth process Long determination method can have two kinds.

The first determination method, P1-P5 fourth process duration can be defined as to the 3rd handling duration phase with Q1-Q5 Deng according to above-mentioned shorthand way, the fourth process duration of each 3rd data block is abbreviated as successively：P1：0.4 second, P2： 0.6 second, P3：0.9 second, P4：0.5 second, P5：0.2 second.

Second of determination method, can be according to the 3rd of sampled data and the ratio of the data volume of initial data, and Q1-Q5 the Handling duration, to determine P1-P5 fourth process duration：Assuming that sampled data has 1000, initial data has 100,000, sampling The data volume ratio of data and initial data is 1：100, Q1-Q5 the 3rd handling duration is multiplied by 100 obtained handling durations For P1-P5 fourth process duration.According to above-mentioned shorthand way, successively by the fourth process duration letter of each 3rd data block It is written as：

P1：40 seconds, P2：60 seconds, P3：90 seconds, P4：50 seconds, P5：20 seconds.

According to the fourth process duration of each 3rd data block of determination, Second Threshold is determined., should when determining Second Threshold Make Second Threshold as far as possible close to the average value of the fourth process duration of this 5 the 3rd data blocks, so that fourth process duration is more than The 3rd data block that the 3rd data block and fourth process duration of Second Threshold are less than Second Threshold distinguishes at least one, and Quantity is roughly equal.

Second of determination method of fourth process duration is taken, and Second Threshold is defined as 50 seconds, therefore, fourth process Duration be more than Second Threshold the 3rd data block be：P2 and P3；Fourth process duration be less than Second Threshold the 3rd data block be： P1 and P5.

By the 3rd data block P2, data point reuse in P3 to the 3rd data block P1, in P5.

After the completion of adjustment, this 5 the 3rd data blocks of parallel processing.

In actual applications, it is multiple to there is a situation where that fourth process duration has more than the 3rd data block of Second Threshold, this When by fourth process duration be more than Second Threshold the 3rd data block in data point reuse to fourth process duration be less than the second threshold In 3rd data block of value, order that can be according to fourth process duration from long to short, fourth process duration is more than the successively Data point reuse in 3rd data block of two threshold values is less than to fourth process duration in the 3rd data block of Second Threshold.

Exemplary, also illustrated with above-mentioned example, Second Threshold is defined as 50 seconds；Fourth process duration is more than second 3rd data block of threshold value is：P2 and P3；Fourth process duration be less than Second Threshold the 3rd data block be：P1 and P5.

Fourth process duration be more than Second Threshold the 3rd data block have it is multiple, can according to fourth process duration by grow to Short order, P3 is first adjusted, then adjust P2, such a adjustment order is more logical, naturally it is also possible in other sequences, herein not It is limited.

Specifically, when fourth process duration to be more than to data point reuse to the fourth process in the 3rd data block of Second Threshold In long the 3rd data block less than Second Threshold, method of adjustment can have three kinds.

The first method of adjustment, it can include：

It is more than the data that predetermined number is extracted in each 3rd data block of Second Threshold from fourth process duration；

A random number is generated in total value scope, wherein, total value scope is that fourth process duration is less than the second threshold The number range that number range corresponding to each 3rd data block of value is formed, fourth process duration are less than each of Second Threshold Number range corresponding to 3rd data block is the 4 of each 3rd data block for being less than Second Threshold according to fourth process duration What handling duration determined, and fourth process duration be less than number range corresponding to each 3rd data block of Second Threshold it is continuous and It is not overlapping；

According to the fourth process duration of determination be less than Second Threshold each 3rd data block corresponding to number range and Random number, the 3rd data block corresponding to the data area that random number is in, it is defined as the 3rd data block of data to be received；

The data of the predetermined number extracted are adjusted into the 3rd data block of the data to be received of determination.

Assuming that the data of predetermined number are 1000 data.

1000 data are extracted from P3.

It is less than each 3rd data block P1 of Second Threshold, P5 fourth process duration, P1 according to fourth process duration：40 Second, P5：20 seconds, determine that fourth process duration is less than each 3rd data block P1 of Second Threshold, number range corresponding to P5.This Two values scope is continuous and not overlapping, and meets that number range is more corresponding to the 3rd shorter data block of fourth process duration Greatly.The two number ranges form a total size, and a random number is generated in total size, and which numerical value is the random number be in In scope, extracted from P3 1000 data are just sent to the 3rd data block corresponding to that number range.

More reasonably the data volume of the 3rd data block is adjusted in order to realize, the 3rd shorter number of fourth process duration The probability that 1000 data extracted from P3 are received according to block should be bigger.Improve the 3rd data block reception number that extracts According to probability, can be realized by expanding the number range of the 3rd data block.

Such as P1：40 seconds, P5：20 seconds, P5 fourth process duration was most short, and P5 receives 1000 extracted from P3 The probability of data should be more than the probability that P1 receives 1000 data extracted from P3.Accordingly, it is determined that numerical value corresponding to P5 Scope is 1-75, and it is 75-100 to determine number range corresponding to P1.Total value scope is 1-100, in 1-100 generate one with Machine number.Assuming that generation random number for 30,30 be in 1-75 between, extracted from P3 1000 data are sent to 1- In 3rd data block corresponding to 75 number range, i.e. in P5.

Above-mentioned determination fourth process duration is less than the mode of number range corresponding to each 3rd data block of Second Threshold, There can be two kinds.The mode of the first determination number range, only it can be determined once before data are adjusted, afterwards each number During according to adjustment, a random number only need to be generated in total value scope, judges which number range is the random number be in.The The mode of two kinds of determination number ranges, fourth process duration can be all redefined less than Second Threshold during each data point reuse Number range corresponding to each 3rd data block, regenerates random number.

In the present embodiment, by the way of the first determination number range.

Then, then P2 is adjusted, 1000 data is extracted from P2.

Assuming that generation random number 62,62 is between 1-75, extracted from P2 1000 data are sent to 1-75 Number range corresponding in the 3rd data block, i.e. in P5.

According to the first method of adjustment, the 3rd shorter data block reception of fourth process duration extracts the number of predetermined number According to probability it is bigger, rather than the data extracted are all sent to the 3rd most short data block of fourth process duration.When treating When 3rd data block of adjustment is a lot, the data of the 3rd data block can be more uniformly adjusted using probability, avoid a certain 3rd Data block adjustment pre-treatment duration is very short, but due to receiving the data extracted from other 3rd data blocks too much, causes to adjust Whole post processing duration becomes oversize, occurs so as to influence the situation of the disposed of in its entirety duration of system.

Second of method of adjustment, can apply to fourth process duration more than Second Threshold the 3rd data block have it is multiple Situation：

Wherein, successively by fourth process duration more than the data point reuse in the 3rd data block of Second Threshold to fourth process Duration is less than in the 3rd data block of Second Threshold, can include：

Never adjust successively data and fourth process duration be more than it is in the 3rd data block of Second Threshold, the everywhere Extract the data of predetermined number in most long the 3rd data block of reason duration, and adjusted to do not adjust data and the 4th Handling duration is less than in the 3rd data block in the 3rd data block of Second Threshold, fourth process duration is most short.

Assuming that the data of predetermined number are 1000 data.

Also illustrated with above-mentioned example, fourth process duration is more than the 3rd data block P2, P3 of Second Threshold；Everywhere Manage the 3rd data block P1, P5 that duration is less than Second Threshold.

Before adjustment, the 3rd most long data block of fourth process duration in P2, P3 is P3；Fourth process in P1, P5 The 3rd most short data block of duration is P5.Extract 1000 data from P3, and by 1000 data point reuses to P5.

Now, most long the 3rd data block of fourth process duration for not adjusting data is P2；The of data is not adjusted everywhere It is P1 to manage the 3rd most short data block of duration.Extract 1000 data from P2, and by 1000 data point reuses to P1.

As it was previously stated, when determining Second Threshold, should try one's best the 3rd data block for making fourth process duration be more than Second Threshold The 3rd data number of blocks for being less than Second Threshold with fourth process duration is roughly equal.But if fourth process duration is more than When the 3rd data block and fourth process duration of Second Threshold are unequal less than the 3rd data number of blocks of Second Threshold：

If the quantity that fourth process duration is more than the 3rd data block of Second Threshold is less than fourth process duration and is less than the The quantity of 3rd data block of two threshold values, still according to above-mentioned second of method of adjustment, it is adjusted.

It is assumed that the 3rd data block that fourth process duration is less than Second Threshold removes P1, and outside P5, also PX and PY, wherein, PX：15 seconds, PY：25 seconds.That is, fourth process duration is more than the 3rd data block P2, P3 of Second Threshold；Fourth process duration is less than 3rd data block P1 of Second Threshold, P5, PX, PY.Then still according to second of method of adjustment：

Before adjustment, the 3rd most long data block of fourth process duration in P2, P3 is P3；In P1, P5, PX, PY The second most short data block of two handling durations is PX.1000 data are extracted from P3, and by 1000 data point reuses extremely PX。

Now, most long the second data block of second processing duration for not adjusting data is P2；At do not adjust data second It is P5 to manage the second most long data block of duration.Extract 1000 data from P2, and by 1000 data point reuses to P5.

If the quantity that fourth process duration is more than the 3rd data block of Second Threshold is more than fourth process duration and is less than the The quantity of 3rd data block of two threshold values, continue to be more than Second Threshold from remaining do not adjust data and fourth process duration The 3rd data block in, extract the data of predetermined number in the 3rd data block that fourth process duration is most long, and adjusted It is whole to be less than to fourth process duration in the 3rd data block of Second Threshold.

If the quantity that fourth process duration is more than the 3rd data block of Second Threshold is more than fourth process duration and is less than the The quantity of 3rd data block of two threshold values, the equal fourth process duration of quantity is more than the according to above-mentioned second of method of adjustment After the 3rd data block and fourth process duration of two threshold values are more than the 3rd data block adjustment of Second Threshold, continue from residue Do not adjust data, fourth process duration be more than that in the 3rd data block of Second Threshold, fourth process duration is most long the 1000 data are extracted in three data blocks, and this 1000 are adjusted to fourth process duration the 3rd number for being less than Second Threshold According in block.

It is assumed that the 3rd data block that fourth process duration is more than Second Threshold removes P2, and outside P3, also PA and PB, wherein, PA：55 seconds, PB：65 seconds.That is, fourth process duration is more than the 4th data block P2, P3, PA, PB of Second Threshold；During fourth process Long the 3rd data block P1, P5 less than Second Threshold.

Before adjustment, do not adjust data and fourth process duration be more than it is in the 3rd data block of Second Threshold, the 4th The 3rd most long data block of handling duration is P3；Do not adjust data and fourth process duration be less than the 3rd number of Second Threshold It is P5 according to the 3rd most short data block of in block, fourth process duration.1000 data are extracted from P3, and by this 1000 Data point reuse is to P5.

Now, do not adjust data and fourth process duration be more than in the 3rd data block of Second Threshold, fourth process The 3rd most long data block of duration is PB；Do not adjust data and fourth process duration be less than the 3rd data block of Second Threshold In, the 3rd data block that fourth process duration is most short be P1.Extract 1000 data from PB, and by 1000 data Adjust to P1.

Now, fourth process duration is less than the 3rd data block P1 of Second Threshold, and the data in P5 were made adjustment, And fourth process duration is more than the 3rd data block P2, P3, PA of Second Threshold, there are the data in P2 and PA not carry out in PB Adjustment.Continue never to adjust in the 3rd most long data block of the fourth process duration in the P2 and PA of data and extract 1000 numbers According to, and by 1000 data point reuses to P1, in P5.

For the sake of rationally, the 3rd data block that fourth process duration is less than Second Threshold can be according to fourth process duration Order from short to long receives 1000 data extracted successively.That is, by extract 1000 data, it is sent to successively P5, P1.

P2 is the 3rd most long data block of the fourth process duration in the P2 and PA for do not adjust data.It will can be carried from P2 1000 data point reuses taken out are to P5.

Now, PA is unique the 3rd data block for not adjusting data, can be by extracted from PA 1000 data Adjust to P1.

According to second of method of adjustment, when fourth process duration is less than the 3rd data block of Second Threshold according to fourth process Long order from short to long receives the data of the predetermined number extracted in turn successively, and it is shorter to avoid a certain fourth process duration The data that extract too much of the 3rd data block reception, so as to which handling duration becomes oversize, influence the feelings of system disposed of in its entirety duration Condition occurs.

The third method of adjustment, it can include：

It is more than the data that predetermined number is extracted in each 3rd data block of the Second Threshold from fourth process duration；

The data of the predetermined number extracted are adjusted to fourth process duration at random and are less than the Second Threshold In 3rd data block.

Also illustrated with above-mentioned example, it is assumed that fourth process duration be more than Second Threshold the 3rd data block be：P2 and P3；Fourth process duration be less than Second Threshold the 3rd data block be：P1 and P5.

Assuming that the data of predetermined number can be 1000 data.

1000 data are respectively extracted from P2, P3, are adjusted at random to P1, in P5.

Using embodiment illustrated in fig. 2 of the present invention, initial data is divided into multiple data blocks, carried out for each data block Sampling, for the newly-generated data block of sampled data of each data block, count the handling duration of each new data block, and according to The handling duration of each new data block, the handling duration of each data block of initial data division is determined, and then to initial data The data volume of the data block of division is adjusted, and to realize that the handling duration of the data block of initial data division is more uniform, is entered One step reduces the disposed of in its entirety duration of system, so as to improve the performance of system.

Corresponding with above-mentioned embodiment of the method, the embodiment of the present invention additionally provides a kind of data processing equipment.

Fig. 3 is a kind of structural representation of data processing equipment provided in an embodiment of the present invention, can be included：First sampling Module 301, the first data block division module 302, the first handling duration statistical module 303, the second data block division module 304, Second processing duration determining module 305, first threshold determining module 306, the first adjusting module 307 and first processing module 308, Wherein,

First sampling module 301, for being sampled for initial data, obtains sampled data.

First data block division module 302, for according to preset data block division rule, sampled data to be divided into M First data block.

First handling duration statistical module 303, for counting each first data block in M the first data blocks first at Manage duration.

Second data block division module 304, for according to preset data block division rule, initial data to be divided into M Second data block, wherein, M the second data blocks correspond with M the first data blocks.

Second processing duration determining module 305, for according to each first data in obtained M the first data blocks of statistics First handling duration of block, determine the second processing duration of each second data block in M the second data blocks.

First threshold determining module 306, for the second processing duration of each second data block according to determination, determine One threshold value, so that second processing duration is more than the second data block of first threshold and second processing duration is less than the of first threshold Two data blocks distinguish at least one.

First adjusting module 307, the data for being more than second processing duration in the second data block of first threshold are adjusted It is whole to be less than to second processing duration in the second data block of first threshold.

First processing module 308, after the completion of adjustment, M the second data blocks of parallel processing.

Wherein, second processing duration be more than first threshold the second data block have it is multiple in the case of, first adjustment mould Block 307, specifically can be used for：

According to the order of second processing duration from long to short, second processing duration is more than into first threshold successively second counts It is less than according to the data point reuse in block to second processing duration in the second data block of first threshold.

In actual applications, the first adjusting module 307, can include：First extracting sub-module, first generation submodule, First determination sub-module and the first adjustment submodule (not shown), wherein,

First extracting sub-module, extracted for being more than from second processing duration in each second data block of first threshold The data of predetermined number；

First generation submodule, for generating a random number in total value scope, wherein, total value scope is second Handling duration is less than the number range that number range corresponding to each second data block of first threshold is formed, second processing duration It is that the every of first threshold is less than according to second processing duration less than number range corresponding to each second data block of first threshold What the second processing duration of individual second data block determined, and second processing duration is less than each second data block pair of first threshold The number range answered is continuous and not overlapping；

First determination sub-module, for being less than each second data block of first threshold according to the second processing duration of determination Corresponding number range and the random number of the first generation submodule generation, the random number of the first generation submodule generation is in Data area corresponding to the second data block, be defined as the second data block of data to be received；

First adjustment submodule, for the data for the predetermined number for extracting the first extracting sub-module, is adjusted to first In second data block of the data to be received that determination sub-module determines.

In actual applications, second processing duration be less than first threshold the second data block have it is multiple in the case of, the One adjusting module 307, can include：Second extracting sub-module and the second adjustment submodule (not shown), wherein,

Second extracting sub-module, it is more than the of first threshold for never adjust data successively and second processing duration The data of predetermined number are extracted in the second data block in two data blocks, second processing duration is most long；

Second adjustment submodule, for the data for the predetermined number for extracting the second extracting sub-module, be sequentially adjusted in Do not adjust data and second processing duration be less than number in the second data block of first threshold, second processing duration is most short According in block；

The quantity for being more than the second data block of first threshold in second processing duration is more than second processing duration less than first In the case of the quantity of second data block of threshold value,

Second extracting sub-module, it is additionally operable to continue to be more than first from remaining do not adjust data and second processing duration The data of predetermined number are extracted in the second data block in second data block of threshold value, second processing duration is most long；

Second adjustment submodule, is additionally operable to the second extracting sub-module continuing the data point reuse of predetermined number extracted extremely Second processing duration is less than in the second data block of first threshold.

In actual applications, the first adjusting module 307, can include：3rd extracting sub-module and the 3rd adjustment submodule (not shown), wherein,

3rd extracting sub-module, extracted for being more than from second processing duration in each second data block of first threshold The data of predetermined number；

3rd adjustment submodule, is less than for the data of the predetermined number extracted to be adjusted to second processing duration at random Second data block of first threshold.

Using embodiment illustrated in fig. 3 of the present invention, initial data is sampled, sampled data is drawn according to preset data block Divider is then divided into multiple data blocks, counts the handling duration of each data block.Then divided according to identical preset data block Rule, initial data is divided into the one-to-one data block of multiple data blocks with sampled data division, and according to hits According to the handling duration of each data block of division, the handling duration of each data block of initial data division is determined, and then to original The data volume of the data block of beginning data division is adjusted, to realize that the handling duration of the data block of initial data division is more equal It is even, the disposed of in its entirety duration of system is further reduced, so as to improve the performance of system.

Fig. 4 is another structural representation of data processing equipment provided in an embodiment of the present invention, can be included：3rd number According to block division module 401, the second sampling module 402, new data block generation module 403, the 3rd handling duration statistical module 404, Fourth process duration determining module 405, Second Threshold determining module 406, the second adjusting module 407 and Second processing module 408, Wherein,

3rd data block division module 401, for initial data to be divided into N number of 3rd data block.

Second sampling module 402, for being sampled for each 3rd data block, obtain adopting for each 3rd data block Sample data.

New data block generation module 403, for the sampled data for each 3rd data block, a newly-generated data block.

3rd handling duration statistical module 404, for counting the 3rd handling duration of each newly-generated data block.

Fourth process duration determining module 405, for obtained according to statistics the 3rd of each newly-generated data block at Duration is managed, determines the fourth process duration of each 3rd data block in N number of 3rd data block.

Second Threshold determining module 406, for the fourth process duration of each 3rd data block according to determination, determine Two threshold values, so that fourth process duration is more than the 3rd data block of the Second Threshold and fourth process duration is less than described second 3rd data block of threshold value distinguishes at least one.

Second adjusting module 407, for the number being more than fourth process duration in the 3rd data block of the Second Threshold It is less than according to adjusting to fourth process duration in the 3rd data block of the Second Threshold.

Second processing module 408, after the completion of adjustment, N number of 3rd data block described in parallel processing.

Wherein, fourth process duration be more than Second Threshold the 3rd data block have it is multiple in the case of, second adjustment mould Block 407, specifically can be used for：

According to the order of fourth process duration from long to short, fourth process duration is more than into Second Threshold successively the 3rd counts It is less than according to the data point reuse in block to fourth process duration in the 3rd data block of Second Threshold.

In actual applications, the second adjusting module 407, can include：4th extracting sub-module, second generation submodule, Second determination sub-module and the 4th adjustment submodule (not shown), wherein,

4th extracting sub-module, extracted for being more than from fourth process duration in each 3rd data block of Second Threshold The data of predetermined number；

Second generation submodule, for generating a random number in total value scope, wherein, total value scope is the 4th Handling duration is less than the number range that number range corresponding to each 3rd data block of Second Threshold is formed, fourth process duration It is that the every of Second Threshold is less than according to fourth process duration less than number range corresponding to each 3rd data block of Second Threshold What the fourth process duration of individual 3rd data block determined, and fourth process duration is less than each 3rd data block pair of Second Threshold The number range answered is continuous and not overlapping；

Second determination sub-module, for being less than each 3rd data block of Second Threshold according to the fourth process duration of determination Corresponding number range and the random number of the second generation submodule generation, the random number of the second generation submodule generation is in Data area corresponding to the second data block, be defined as the second data block of data to be received；

4th adjustment submodule, for the data for the predetermined number for extracting the 4th extracting sub-module, is adjusted to second In second data block of the data to be received that determination sub-module determines.

In actual applications, fourth process duration be less than Second Threshold the 3rd data block have it is multiple in the case of, the Two adjusting modules 407, can include：5th extracting sub-module and the 5th adjustment submodule (not shown), wherein,

5th extracting sub-module, it is more than the of Second Threshold for never adjust data successively and fourth process duration The data of predetermined number are extracted in the 3rd data block in three data blocks, fourth process duration is most long；

5th adjustment submodule, for the data for the predetermined number for extracting the 5th extracting sub-module, be sequentially adjusted in Do not adjust data and fourth process duration be less than number in the 3rd data block of Second Threshold, fourth process duration is most short According in block；

The quantity for being more than the 3rd data block of Second Threshold in fourth process duration is more than fourth process duration less than second In the case of the quantity of 3rd data block of threshold value,

5th extracting sub-module, it is additionally operable to continue to be more than second from remaining do not adjust data and fourth process duration The data of predetermined number are extracted in the 3rd data block in 3rd data block of threshold value, fourth process duration is most long；

5th adjustment submodule, is additionally operable to the 5th extracting sub-module continuing the data point reuse of predetermined number extracted extremely Fourth process duration is less than in the 3rd data block of Second Threshold.

In actual applications, the second adjusting module 407, can include：6th extracting sub-module and the 6th adjustment submodule (not shown), wherein,

6th extracting sub-module, extracted for being more than from fourth process duration in each 3rd data block of Second Threshold The data of predetermined number；

6th adjustment submodule, is less than for the data of the predetermined number extracted to be adjusted to fourth process duration at random 3rd data block of Second Threshold.

Using embodiment illustrated in fig. 4 of the present invention, initial data is divided into multiple data blocks, carried out for each data block Sampling, for the newly-generated data block of sampled data of each data block, count the handling duration of each new data block, and according to The handling duration of each new data block, the handling duration of each data block of initial data division is determined, and then to initial data The data volume of the data block of division is adjusted, and to realize that the handling duration of the data block of initial data division is more uniform, is entered One step reduces the disposed of in its entirety duration of system, so as to improve the performance of system.

It should be noted that herein, such as first and second or the like relational terms are used merely to a reality Body or operation make a distinction with another entity or operation, and not necessarily require or imply and deposited between these entities or operation In any this actual relation or order.Moreover, term " comprising ", "comprising" or its any other variant are intended to Nonexcludability includes, so that process, method, article or equipment including a series of elements not only will including those Element, but also the other element including being not expressly set out, or it is this process, method, article or equipment also to include Intrinsic key element.In the absence of more restrictions, the key element limited by sentence "including a ...", it is not excluded that Other identical element also be present in process, method, article or equipment including the key element.

Each embodiment in this specification is described by the way of related, identical similar portion between each embodiment Divide mutually referring to what each embodiment stressed is the difference with other embodiment.It is real especially for device For applying example, because it is substantially similar to embodiment of the method, so description is fairly simple, related part is referring to embodiment of the method Part explanation.

Can one of ordinary skill in the art will appreciate that realizing that all or part of step in above method embodiment is To instruct the hardware of correlation to complete by program, described program can be stored in computer read/write memory medium, The storage medium designated herein obtained, such as：ROM/RAM, magnetic disc, CD etc..

The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all Any modification, equivalent substitution and improvements made within the spirit and principles in the present invention etc., are all contained in protection scope of the present invention It is interior.

Claims

1. a kind of data processing method, it is characterised in that applied to distributed computing system, methods described includes：

Sampled for initial data, obtain sampled data；

According to preset data block division rule, the sampled data is divided into M the first data blocks, the M is positive integer；

The first handling duration of each first data block, determines the M in the M the first data blocks obtained according to statistics The second processing duration of each second data block in second data block；

Wherein, in the M the first data blocks the first handling duration of each first data block respectively with the M the second data The second processing duration of each second data block is equal in block；Or the M obtained according to statistics first data block In each first data block the first handling duration, at determine each second data block in the M the second data blocks second Duration is managed, including：

The M first number obtained according to the ratio and statistics of the sampled data and the data volume of the initial data According to the first handling duration of each first data block in block, the of each second data block in the M the second data blocks is determined Two handling durations；

According to the second processing duration of each second data block of determination, first threshold is determined, so that second processing duration is more than The second data block and second processing duration of the first threshold are less than the second data block difference of the first threshold at least One；

Data point reuse to second processing duration second processing duration being more than in the second data block of the first threshold is less than In second data block of the first threshold；

2. according to the method for claim 1, it is characterised in that be more than the second of the first threshold in second processing duration Data block have it is multiple in the case of,

Data point reuse second processing duration being more than in the second data block of the first threshold is to second processing duration Less than in the second data block of the first threshold, including：

According to the order of second processing duration from long to short, second processing duration is more than into the first threshold successively second counts It is less than according to the data point reuse in block to second processing duration in the second data block of the first threshold.

3. according to the method for claim 1, it is characterised in that described that second processing duration is more than the first threshold Data point reuse in second data block is less than to second processing duration in the second data block of the first threshold, including：

A random number is generated in total value scope, wherein, the total value scope is that second processing duration is less than described the The number range that number range corresponding to each second data block of one threshold value is formed, the second processing duration are less than described the Number range corresponding to each second data block of one threshold value is to be less than each of the first threshold according to second processing duration What the second processing duration of the second data block determined, and the second processing duration is less than each second number of the first threshold It is continuous according to number range corresponding to block and not overlapping；

According to the second processing duration of determination be less than the first threshold each second data block corresponding to number range and The random number, the second data block corresponding to the data area that the random number is in, is defined as the second of data to be received Data block；

4. according to the method for claim 2, it is characterised in that be less than the second of the first threshold in second processing duration Data block have it is multiple in the case of,

Data point reuse second processing duration being more than successively in the second data block of the first threshold is to second processing Duration is less than in the second data block of the first threshold, including：

Successively never adjust data and second processing duration be more than the first threshold the second data block in, at second Extract the data of predetermined number in most long the second data block of reason duration, and adjusted to do not adjust data and second Handling duration is less than in the second data block in the second data block of the first threshold, second processing duration is most short；

If the quantity that second processing duration is more than the second data block of the first threshold is less than institute more than second processing duration The quantity of the second data block of first threshold is stated, continues to be more than from remaining do not adjust data and second processing duration described The data of predetermined number are extracted in the second data block in second data block of first threshold, second processing duration is most long, And adjusted to second processing duration and be less than in the second data block of the first threshold.

5. according to the method for claim 1, it is characterised in that described that second processing duration is more than the first threshold Data point reuse in second data block is less than to second processing duration in the second data block of the first threshold, including：

The data of the predetermined number extracted are adjusted to second processing duration at random and are less than the second of the first threshold In data block.

6. a kind of data processing method, it is characterised in that applied to distributed computing system, methods described includes：

Initial data is divided into N number of 3rd data block, the N is positive integer；

For the sampled data of each 3rd data block, a newly-generated data block；

3rd handling duration of each newly-generated data block of statistics；

3rd handling duration of each newly-generated data block obtained according to statistics, determine every in N number of 3rd data block The fourth process duration of individual 3rd data block；

Wherein, the 3rd handling duration of each newly-generated data block is respectively with N number of 3rd data block each the The fourth process duration of three data blocks is equal；Or at the 3rd of each newly-generated data block obtained according to statistics Duration is managed, determines the fourth process duration of each 3rd data block in N number of 3rd data block, including：

The each newly-generated number obtained according to the ratio and statistics of the sampled data and the data volume of the initial data According to the 3rd handling duration of block, the fourth process duration of each 3rd data block in N number of 3rd data block is determined；

According to the fourth process duration of each 3rd data block of determination, Second Threshold is determined, so that fourth process duration is more than The 3rd data block and fourth process duration of the Second Threshold are less than the 3rd data block difference of the Second Threshold at least One；

Data point reuse to fourth process duration fourth process duration being more than in the 3rd data block of the Second Threshold is less than In 3rd data block of the Second Threshold；

7. a kind of data processing equipment, it is characterised in that applied to distributed computing system, described device includes：First sampling Module, the first data block division module, the first handling duration statistical module, the second data block division module, second processing duration Determining module, first threshold determining module, the first adjusting module and first processing module, wherein,

The first data block division module, for according to preset data block division rule, the sampled data to be divided into M First data block, the M are positive integer；

The first handling duration statistical module, for counting first of each first data block in the M the first data blocks Handling duration；

The second data block division module, for according to the preset data block division rule, initial data to be divided into M Second data block, wherein, the M the second data blocks correspond with the M the first data blocks；

The second processing duration determining module, for according to each first number in the obtained M the first data blocks of statistics According to the first handling duration of block, the second processing duration of each second data block in the M the second data blocks is determined；

Wherein, in the M the first data blocks the first handling duration of each first data block respectively with the M the second data The second processing duration of each second data block is equal in block；Or the second processing duration determining module, it is specifically used for：

The first threshold determining module, for the second processing duration of each second data block according to determination, determine first Threshold value, so that second processing duration is more than the second data block of the first threshold and second processing duration is less than first threshold Second data block of value distinguishes at least one；

First adjusting module, the data for being more than second processing duration in the second data block of the first threshold are adjusted It is whole to be less than to second processing duration in the second data block of the first threshold；

8. device according to claim 7, it is characterised in that be more than the second of the first threshold in second processing duration Data block have it is multiple in the case of, first adjusting module, be specifically used for：

9. device according to claim 7, it is characterised in that first adjusting module, including：First extraction submodule Block, the first generation submodule, the first determination sub-module and the first adjustment submodule, wherein,

First extracting sub-module, carried for being more than from second processing duration in each second data block of the first threshold Take out the data of predetermined number；

The first generation submodule, for generating a random number in total value scope, wherein, the total value scope is Second processing duration is less than the number range that number range corresponding to each second data block of the first threshold is formed, described It is according to second processing duration that second processing duration, which is less than number range corresponding to each second data block of the first threshold, Second processing duration less than each second data block of the first threshold determines that and the second processing duration is less than institute It is continuous and not overlapping to state number range corresponding to each second data block of first threshold；

First determination sub-module, for being less than each second number of the first threshold according to the second processing duration of determination According to number range corresponding to block and the random number of the first generation submodule generation, by the described first generation submodule generation The data area that is in of random number corresponding to the second data block, be defined as the second data block of data to be received；

The first adjustment submodule, for the data for the predetermined number for extracting first extracting sub-module, adjust to In second data block of the data to be received that first determination sub-module determines.

10. device according to claim 8, it is characterised in that be less than the of the first threshold in second processing duration Two data blocks have it is multiple in the case of,

Second extracting sub-module, for successively never adjust data and second processing duration be more than the first threshold The second data block in, data that predetermined number is extracted in the second data block that second processing duration is most long；

The second adjustment submodule, for the data for the predetermined number for extracting second extracting sub-module, is adjusted successively It is whole to be less than in the second data block of the first threshold, second processing duration to do not adjust data and second processing duration In most short data block；

The quantity for being more than the second data block of the first threshold in second processing duration is more than second processing duration less than described In the case of the quantity of second data block of first threshold,

Second extracting sub-module, it is additionally operable to continue to be more than from remaining do not adjust data and second processing duration described The data of predetermined number are extracted in the second data block in second data block of first threshold, second processing duration is most long；

The second adjustment submodule, the data for the predetermined number for being additionally operable to continue to extract by second extracting sub-module are adjusted It is whole to be less than to second processing duration in the second data block of the first threshold.

11. device according to claim 7, it is characterised in that first adjusting module, including：3rd extraction submodule Block and the 3rd adjustment submodule, wherein,

3rd extracting sub-module, carried for being more than from second processing duration in each second data block of the first threshold Take out the data of predetermined number；

The 3rd adjustment submodule, for the data of the predetermined number extracted to be adjusted to second processing duration at random Less than the second data block of the first threshold.

12. a kind of data processing equipment, it is characterised in that applied to distributed computing system, described device includes：3rd data Block division module, the second sampling module, new data block generation module, the 3rd handling duration statistical module, fourth process duration are true Cover half block, Second Threshold determining module, the second adjusting module and Second processing module, wherein,

The 3rd data block division module, for initial data to be divided into N number of 3rd data block, the N is positive integer；

Second sampling module, for being sampled for each 3rd data block, obtain each 3rd data The sampled data of block；

The new data block generation module, for the sampled data for each 3rd data block, a newly-generated data block；

The fourth process duration determining module, for according to statistics obtain each newly-generated data block the 3rd processing when It is long, determine the fourth process duration of each 3rd data block in N number of 3rd data block；

Wherein, the 3rd handling duration of each newly-generated data block is respectively with N number of 3rd data block each the The fourth process duration of three data blocks is equal；Or the fourth process duration determining module, it is specifically used for：

The Second Threshold determining module, for the fourth process duration of each 3rd data block according to determination, determine second Threshold value, so that fourth process duration is more than the 3rd data block of the Second Threshold and fourth process duration is less than second threshold 3rd data block of value distinguishes at least one；

Second adjusting module, the data for being more than fourth process duration in the 3rd data block of the Second Threshold are adjusted It is whole to be less than to fourth process duration in the 3rd data block of the Second Threshold；