CN104504020B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN104504020B CN104504020B CN201410766560.8A CN201410766560A CN104504020B CN 104504020 B CN104504020 B CN 104504020B CN 201410766560 A CN201410766560 A CN 201410766560A CN 104504020 B CN104504020 B CN 104504020B
- Authority
- CN
- China
- Prior art keywords
- data processing
- processing amount
- amount
- single data
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 238000012545 processing Methods 0.000 claims abstract description 606
- 238000000034 method Methods 0.000 claims abstract description 11
- 230000009471 action Effects 0.000 claims description 2
- 238000005303 weighing Methods 0.000 claims 1
- 230000008676 import Effects 0.000 description 10
- 238000012360 testing method Methods 0.000 description 8
- 238000011161 development Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of data processing method and device.Wherein, this method includes：Obtain pending data；Single datum target treating capacity is obtained according to the data processing time of pending data single data processing；Pending data is handled according to single datum target treating capacity.The present invention is solved using the low technical problem of the data-handling efficiency caused by existing data processing method.
Description
Technical field
The present invention relates to computer realm, in particular to a kind of data processing method and device.
Background technology
Nowadays, with the development of technology, increasing enterprise or factory all begin to use electronic management, in this process
In will necessarily just produce substantial amounts of data, how these data are rapidly imported into database to carry out data processing just in time
Become one it is in the urgent need to address the problem of.
At present, in order to solve the above problems, one kind is provided in the prior art mass data is subjected to batch processing, then will
Batch data imported into the mode of database, to improve the importing efficiency of data.However, import the process of data in above-mentioned batch
In, set single data import volume is a fixed value being manually set according to personal experience, and then is fixed according to this
Value performs the importing of batch data.That is, the batch data lead-in mode that prior art provides is more subjective, if this amount
What is set is too small, and data volume per treatment will increase number of processes, that is, sum in the case where total amount is constant with regard to small
According to the interaction times in storehouse, increase the cost of data interaction；And if this amount set it is excessive, occupancy that will be excessive too long is delayed
Resource is deposited, forms the competition deadlock of resource.In other words, if the inaccuracy that above-mentioned single data import volume is set, by direct shadow
The efficiency that data import is rung, and then influences the treatment effeciency of data.
The problem of in correlation technique, effective solution is not yet proposed at present.
The content of the invention
It is a primary object of the present invention to provide a kind of data processing method and device, to solve due to using existing number
According to the low technical problem of the data-handling efficiency caused by processing mode.
According to an aspect of the invention, there is provided a kind of data processing method, this method include：Obtain pending number
According to；Single datum target treating capacity is obtained according to the data processing time of above-mentioned pending data single data processing；According to upper
State single datum target treating capacity and handle above-mentioned pending data.
Alternatively, it is above-mentioned that single data mesh is obtained according to the data processing time of above-mentioned pending data single data processing
Mark treating capacity includes：Following steps are repeated to above-mentioned pending data, until obtaining above-mentioned single datum target treating capacity：
In above-mentioned pending data single data processing amount is selected according to the first predetermined condition；Obtained according to above-mentioned single data processing amount
Take the single data processing time for the data for handling above-mentioned single data processing amount；Whether judge above-mentioned single data processing time
Less than or equal to the first predetermined threshold；If judging, above-mentioned single data processing time is more than above-mentioned first predetermined threshold, again
Select above-mentioned single data processing amount；If judging, above-mentioned single data processing time is less than or equal to above-mentioned first predetermined threshold,
Then will be as above-mentioned single datum target treating capacity using above-mentioned single data processing amount corresponding to above-mentioned single data processing time.
Alternatively, it is above-mentioned to include in above-mentioned pending data according to the first predetermined condition selection single data processing amount：
It is determined that select the scope of above-mentioned single data processing amount；From above range the first data processing is selected according to the second predetermined condition
Amount, the second data processing amount and the 3rd data processing amount, wherein, above-mentioned first data processing amount is less than above-mentioned second data processing
Amount, above-mentioned second data processing amount are less than above-mentioned 3rd data processing amount；Obtain above-mentioned first data processing amount, above-mentioned second number
According to treating capacity and the above-mentioned single data processing time of above-mentioned 3rd data processing amount, and at the above-mentioned single data to getting
The reason time is ranked up；Using the most short corresponding data processing amount of above-mentioned single data processing time as above-mentioned single data processing
Amount, and the above-mentioned model for selecting above-mentioned single data processing amount is redefined according to the above-mentioned single data processing time got
Enclose.
Alternatively, above range includes the 4th data processing amount to the quantitative value between the 5th data processing amount, wherein, on
State the 4th data processing amount and be less than or equal to above-mentioned first data processing amount, above-mentioned 5th data processing amount is more than or equal to the above-mentioned 3rd
Data processing amount, it is above-mentioned to select the first data processing amount, the second data processing amount according to the second predetermined condition from above range
And the 3rd data processing amount include：The first average value of above-mentioned 4th data processing amount and above-mentioned 5th data processing amount is calculated,
Above-mentioned first average value is as above-mentioned second data processing amount；Calculate above-mentioned 4th data processing amount and above-mentioned second data processing
Second average value of amount, above-mentioned second average value is as above-mentioned first data processing amount；Calculate above-mentioned 5th data processing amount with
3rd average value of above-mentioned second data processing amount, above-mentioned 3rd average value is as above-mentioned 3rd data processing amount.
Alternatively, the above-mentioned single data processing time that above-mentioned basis is got is redefined at the above-mentioned single data of selection
The above range of reason amount includes at least one of：If above-mentioned single data processing time corresponding to above-mentioned second data processing amount
It is most short, then using the quantitative value between above-mentioned first data processing amount and above-mentioned 3rd data processing amount as the scope after renewal；
If above-mentioned single data processing time is most short corresponding to above-mentioned first data processing amount, by above-mentioned 4th data processing amount and on
The quantitative value between the second data processing amount is stated as the scope after above-mentioned renewal；If on corresponding to above-mentioned 3rd data processing amount
State that single data processing time is most short, then by the quantitative value between above-mentioned second data processing amount and above-mentioned 5th data processing amount
As the scope after above-mentioned renewal.
Alternatively, it is above-mentioned to include in above-mentioned pending data according to the first predetermined condition selection single data processing amount：
Above-mentioned single data processing amount is randomly choosed from above range according to pre-determined number, wherein, above-mentioned pre-determined number is more than or equal to
Second predetermined threshold.
Alternatively, before above-mentioned acquisition pending data, in addition to：Establish the fortune to match with above-mentioned pending data
Row environment.
According to another aspect of the present invention, there is provided a kind of data processing equipment, the device include：First acquisition unit,
For obtaining pending data；Second acquisition unit, for the data processing according to above-mentioned pending data single data processing
Time obtains single datum target treating capacity；Processing unit, for handling above-mentioned treat according to above-mentioned single datum target treating capacity
Processing data.
Alternatively, above-mentioned second acquisition unit includes：Processing module, for by obtaining above-mentioned single data with lower module
Target treating capacity：First choice submodule, for selecting single data according to the first predetermined condition in above-mentioned pending data
Treating capacity；Acquisition submodule, the data of above-mentioned single data processing amount are handled for being obtained according to above-mentioned single data processing amount
Single data processing time；Judging submodule, for judging it is pre- whether above-mentioned single data processing time is less than or equal to first
Determine threshold value；Second selection submodule, for when judging that above-mentioned single data processing time is more than above-mentioned first predetermined threshold,
Reselect above-mentioned single data processing amount；Determination sub-module, for judging that above-mentioned single data processing time is less than
, will be as above-mentioned using above-mentioned single data processing amount corresponding to above-mentioned single data processing time when above-mentioned first predetermined threshold
Single datum target treating capacity；Judge module, for judging whether to obtain above-mentioned single datum target treating capacity.
Alternatively, above-mentioned first choice submodule is realized pre- according to first in above-mentioned pending data by following steps
Fixed condition selection single data processing amount includes：It is determined that select the scope of above-mentioned single data processing amount；Pressed from above range
The first data processing amount, the second data processing amount and the 3rd data processing amount are selected according to the second predetermined condition, wherein, above-mentioned first
Data processing amount is less than above-mentioned second data processing amount, and above-mentioned second data processing amount is less than above-mentioned 3rd data processing amount；Obtain
Take the above-mentioned single data processing of above-mentioned first data processing amount, above-mentioned second data processing amount and above-mentioned 3rd data processing amount
Time, and the above-mentioned single data processing time to getting is ranked up；By the most short correspondence of above-mentioned single data processing time
Data processing amount redefined as above-mentioned single data processing amount, and according to the above-mentioned single data processing time got
Select the above range of above-mentioned single data processing amount.
Alternatively, above range includes the 4th data processing amount to the quantitative value between the 5th data processing amount, wherein, on
State the 4th data processing amount and be less than or equal to above-mentioned first data processing amount, above-mentioned 5th data processing amount is more than or equal to the above-mentioned 3rd
Data processing amount, above-mentioned first choice submodule by following steps realize it is above-mentioned from above range according to the second predetermined condition
The first data processing amount, the second data processing amount and the 3rd data processing amount is selected to include：Calculate above-mentioned 4th data processing amount
With the first average value of above-mentioned 5th data processing amount, above-mentioned first average value is as above-mentioned second data processing amount；In calculating
The second average value of the 4th data processing amount and above-mentioned second data processing amount is stated, above-mentioned second average value is as the above-mentioned first number
According to treating capacity；The 3rd average value of above-mentioned 5th data processing amount and above-mentioned second data processing amount is calculated, the above-mentioned 3rd is average
Value is used as above-mentioned 3rd data processing amount.
Alternatively, above-mentioned first choice submodule realizes above-mentioned single data that above-mentioned basis gets by following steps
Processing time, which redefines, selects the above range of above-mentioned single data processing amount to include at least one of：If above-mentioned second number
It is most short according to above-mentioned single data processing time corresponding to treating capacity, then by above-mentioned first data processing amount and above-mentioned 3rd data
Quantitative value between reason amount is as the scope after renewal；If corresponding to above-mentioned first data processing amount during above-mentioned single data processing
Between it is most short, then using the quantitative value between above-mentioned 4th data processing amount and above-mentioned second data processing amount as above-mentioned renewal after
Scope；If above-mentioned single data processing time corresponding to above-mentioned 3rd data processing amount is most short, by above-mentioned second data processing
Quantitative value between amount and above-mentioned 5th data processing amount is as the scope after above-mentioned renewal.
Alternatively, above-mentioned first choice module is realized predetermined according to first in above-mentioned pending data by following steps
Condition selection single data processing amount includes：Above-mentioned single data processing is randomly choosed from above range according to pre-determined number
Amount, wherein, above-mentioned pre-determined number is more than or equal to the second predetermined threshold.
Alternatively, said apparatus also includes：Establish unit, for before above-mentioned acquisition pending data, establish with it is upper
State the running environment that pending data matches.
The embodiment provided by the application, at the data according to the pending data single data processing got
The time is managed, single datum target treating capacity is obtained, is largely treated further according to the single datum target treating capacity batch processing
Processing data, so as to avoid artificially judging that the treatment effeciency caused by single data processing amount is low by experience in the prior art
Problem, so realize it is fast automatic find the more suitable single data processing amount of pending data, with improve data processing effect
Rate, and meet the data processing needs of user.Further, due to without human intervention, the optimal single data of acquisition of automation
Treating capacity, and then also saved during mass data is handled for user at substantial amounts of data processing cost and data
Manage the time.
Further, by pre-establish with the pending running environment to match, it is specific due to departing from so as to avoid
Implementation environment causes the problem of single datum target treating capacity inaccuracy got so that quick and precisely obtains and meets operation ring
The single datum target treating capacity that border needs, and then improve the treatment effeciency of data.
Brief description of the drawings
The accompanying drawing for forming the part of the application is used for providing a further understanding of the present invention, schematic reality of the invention
Apply example and its illustrate to be used to explain the present invention, do not form inappropriate limitation of the present invention.In the accompanying drawings：
Fig. 1 is a kind of flow chart of optional data processing method according to embodiments of the present invention；
Fig. 2 is a kind of schematic diagram of optional data processing equipment according to embodiments of the present invention.
Embodiment
It should be noted that in the case where not conflicting, the feature in embodiment and embodiment in the application can phase
Mutually combination.Describe the present invention in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
Embodiment 1
According to embodiments of the present invention, there is provided a kind of data processing method, as shown in figure 1, this method includes：
S102, obtain pending data；
S104, single datum target treating capacity is obtained according to the data processing time of pending data single data processing；
S106, pending data is handled according to single datum target treating capacity.
Alternatively, in the present embodiment, above-mentioned data processing method can be, but not limited to be applied to batch data importing number
During according to storehouse, for example, it is assumed that certain enterprise needs a large number of users data importing database, then imported to ensure data
Efficiency, it is necessary to calculate the data volume that above-mentioned mass data single to be imported imports.Specifically, data to be imported are obtained,
And data to be imported are carried out in batches, to obtain the processing time imported per batch data single, and the number imported according to single data
The target treating capacity of single data importing is obtained according to processing time, and then controls and is handled when data import according to above-mentioned target
Amount, database is imported by above-mentioned enterprise user data to be imported in batches.The example above is a kind of example, the present embodiment pair
This is not limited in any way.
Alternatively, in the present embodiment, before pending data is obtained, in addition to：Establish and pending data phase
The running environment matched somebody with somebody.That is, before pending data is handled, will also the running environment where pending data be simulated,
To avoid the single datum target treating capacity inaccuracy being calculated due to departing from original performing environment to cause, and then influence number
According to treatment effeciency.
Alternatively, in the present embodiment, the above-mentioned data processing time according to the pending data single data processing
Obtaining single datum target treating capacity includes：Different single data processing amount input data processing systems is repeatedly selected, to obtain
The data processing time of single data processing is taken, and then data processing time is met that the single data processing amount of predetermined condition is made
For single datum target treating capacity.
Alternatively, in the present embodiment, the mode of the selection single data processing amount can include but is not limited to following
At least one：
1) single data processing amount is randomly choosed in predetermined scope；
2) multiple single data processing amounts are selected according to pre-provisioning request in predetermined scope, in more multiple single data
After data processing time corresponding to treating capacity, the most short single data processing amount of data processing time is selected.
Alternatively, in the present embodiment, the predetermined scope of above-mentioned determination single data processing amount can include but unlimited
Determined according to the total amount of pending data.For example, when first time handling pending data, can be according to the total of pending data
S is measured, judges the span of preferably single data processing amount, for example, the single selected by processing pending data for the first time
Data processing amount can be the numerical value in scope [A, B].
The embodiment provided by the application, at the data according to the pending data single data processing got
The time is managed, single datum target treating capacity is obtained, is largely treated further according to the single datum target treating capacity batch processing
Processing data, so as to avoid artificially judging that the treatment effeciency caused by single data processing amount is low by experience in the prior art
Problem, so realize it is fast automatic find the more suitable single data processing amount of pending data, with improve data processing effect
Rate, and meet the data processing needs of user.Further, due to without human intervention, the optimal single data of acquisition of automation
Treating capacity, and then also saved during mass data is handled for user at substantial amounts of data processing cost and data
Manage the time.
As a kind of optional scheme, single number is obtained according to the data processing time of pending data single data processing
Include according to target treating capacity：
S1, following steps are repeated to pending data, until obtaining single datum target treating capacity：
S12, single data processing amount is selected according to the first predetermined condition in pending data；
S14, the single data processing time of the data of processing single data processing amount is obtained according to single data processing amount；
S16, judges whether single data processing time is less than or equal to the first predetermined threshold；
S18, if judging, single data processing time is more than the first predetermined threshold, reselects single data processing
Amount；
S20, if judging, single data processing time is less than or equal to the first predetermined threshold, by single data processing time
Corresponding single data processing amount will be used as single datum target treating capacity.
Alternatively, in the present embodiment, above-mentioned first predetermined threshold can be, but not limited to different according to data handling system
Real-time demand determine.
Specifically illustrated with reference to the example below, in the running environment of simulation, repeatedly input is predetermined according to first respectively
The single data processing amount of condition selection, when obtaining single data processing after above-mentioned single data processing amount test run
Between, the single data processing time and the size of the first predetermined threshold that obtain more every time, if judging to be more than the first predetermined threshold
Value, then it represents that single data processing amount corresponding to the single data processing time is not most suitable single datum target processing
Amount；If conversely, judge to be less than or equal to the first predetermined threshold, then it represents that corresponding to the single data processing time at single data
Reason amount is more suitable single datum target treating capacity.
The embodiment provided by the application, by selecting single data processing amount input data according to the first predetermined condition
Processing system is to obtain the data processing time of single data corresponding to single data processing amount, further, judges above-mentioned single
Whether the data processing time of data is less than or equal to the first predetermined threshold, can when judging to be less than or equal to the first predetermined threshold
Obtain single data processing amount corresponding to the data processing time of above-mentioned single data for batch processing above-mentioned pending data when
More suitable single datum target treating capacity.Pending data further is handled using above-mentioned single datum target treating capacity,
To avoid in the prior art due to artificially judging that single datum target treating capacity is caused at used single data by experience
Reason amount is unreasonable, and then the problem of cause data-handling efficiency to reduce.
As a kind of optional scheme, single data processing amount bag is selected according to the first predetermined condition in pending data
Include：
S1, it is determined that the scope of selection single data processing amount；
S2, the first data processing amount, the second data processing amount and the 3rd number are selected according to the second predetermined condition from scope
According to treating capacity, wherein, the first data processing amount is less than the second data processing amount, and the second data processing amount is less than the 3rd data processing
Amount；
S3, when obtaining the single data processing of the first data processing amount, the second data processing amount and the 3rd data processing amount
Between, and the single data processing time to getting is ranked up；
S4, using data processing amount corresponding to single data processing time is most short as single data processing amount, and according to obtaining
The single data processing time got redefines the scope of selection single data processing amount.
Alternatively, in the present embodiment, the scope of above-mentioned determination single data processing amount can also include according to single number
According to data processing time determine.
Specifically illustrated with reference to the example below, it is assumed that the scope for determining single data processing amount is [A, B], from the scope
The first data processing amount x, the second data processing amount y, the 3rd data processing amount z of interior selection, wherein, x<y<Z, obtain respectively above-mentioned
The data processing time of single data corresponding to single data processing amount is respectively t1, t2, t3.Above-mentioned data processing time is arranged
Sequence, it is assumed that t1<t2<T3, then single data processing amount corresponding to data processing time t1 by it is selected input running environment carry out
Data test, to judge whether the single data processing amount is that most suitable single datum target is handled when batch data is handled
Amount.
The embodiment provided by the application, during by the data processing of single data that obtains multiple data processing amounts
Between, the most short single data processing amount of data processing time is therefrom selected, for judging whether the single data processing amount is several
Most suitable single datum target treating capacity during according to batch processing.Further, selected using above-mentioned from multiple data processing amounts
A more suitable single data processing amount carries out data test, ensure that the accurate of selected single data processing amount
Property.
As a kind of optional scheme, scope includes the 4th data processing amount to the quantity between the 5th data processing amount
Value, wherein, the 4th data processing amount is less than or equal to the first data processing amount, and the 5th data processing amount is more than or equal at the 3rd data
Reason amount, the first data processing amount, the second data processing amount and the 3rd data processing are selected according to the second predetermined condition from scope
Amount includes：
S1, the first average value of the 4th data processing amount and the 5th data processing amount is calculated, the first average value is as second
Data processing amount；
S2, the second average value of the 4th data processing amount and the second data processing amount is calculated, the second average value is as first
Data processing amount；
S3, the 3rd average value of the 5th data processing amount and the second data processing amount is calculated, the 3rd average value is as the 3rd
Data processing amount.
Specifically illustrated with reference to the example below, it is assumed that the initial value [A, B] of the scope of single data processing amount is determined, its
In, the 4th data processing amount is A, and the 5th data processing amount is B, then the second data processing amount is M=(A+B)/2, corresponding to it
Data processing time f (M)；Further, the first data processing amount is a=(A+M)/2, and its corresponding data processing time is f
(a), the 3rd data processing amount is b=(M+B)/2, and its corresponding data processing time is f (b).
The embodiment provided by the application, the list for selecting to be tested is obtained by way of above-mentioned average
The first data processing amount, the second data processing amount, the 3rd data processing amount of secondary data processing amount, so as to ensure selected number
More meet pending data quantative attribute according to treating capacity, further such that selected single data processing amount is closer to single number
According to target treating capacity, so as to save the testing time of selection single data processing amount, the efficiency of data processing is improved.
As a kind of optional scheme, redefined according to the single data processing time got at selection single data
The scope of reason amount includes at least one of：
S1, if single data processing time is most short corresponding to the second data processing amount, by the first data processing amount and
Quantitative value between three data processing amounts is as the scope after renewal；
S2, if single data processing time is most short corresponding to the first data processing amount, by the 4th data processing amount and
Quantitative value between two data processing amounts is as the scope after renewal；
S3, if single data processing time corresponding to the 3rd data processing amount is most short, by the second data processing amount and
Quantitative value between five data processing amounts is as the scope after renewal.
Specifically illustrated with reference to the example below, more above-mentioned first data processing amount, the second data processing amount, the 3rd number
According to the data processing time for the treatment of capacity：f(M)、f(a)、f(b).Further, if the data processing time f of the second data processing amount
(M) it is minimum, then new scope is arranged to A=a, B=b, i.e. single data processing is selected from the quantitative value between [a, b]
Amount；If the data processing time f (a) of the first data processing amount is minimum, new scope is arranged to B=M, A is constant, i.e., from
Single data processing amount is selected in quantitative value between [A, M]；If the data processing time f (b) of the 3rd data processing amount is minimum,
New scope is then arranged to A=M, B is constant, i.e., single data processing amount is selected from the quantitative value between [M, B].
The embodiment provided by the application, by the way that the scope for selecting single data processing amount is gradually reduced, so that
Selected single data processing amount is closer to single datum target treating capacity, so as to save the survey of selection single data processing amount
The time is tried, improves the efficiency of data processing.
As a kind of optional scheme, single data processing amount bag is selected according to the first predetermined condition in pending data
Include：
S1, single data processing amount is randomly choosed from scope according to pre-determined number, wherein, pre-determined number is more than or equal to the
Two predetermined thresholds.
Alternatively, in the present embodiment, above-mentioned pre-determined number can be, but not limited to be determined according to run time, wherein, on
Run time is stated to can be, but not limited to as a cycle of operation.
The embodiment provided by the application, by way of randomly choosing single data processing amount from scope, enter one
Step, the processing routine for obtaining single data processing amount is simplified, saves development cost and resource.
As a kind of optional scheme, before pending data is obtained, in addition to：
S1, establish the running environment to match with pending data.
Specifically illustrated with reference to the example below, it is assumed that need to import creation data for a certain factory, then establish and the work
The similar build environment of factory, in order to obtain accurate single datum target treating capacity.
By the application provide embodiment, by pre-establish with the pending running environment to match, so as to avoid
Due to departing from the problem of specific implementation environment causes the single datum target treating capacity inaccuracy got so that quick accurate
The single datum target treating capacity for meeting running environment needs is really obtained, and then improves the treatment effeciency of data.
It should be noted that can be in such as one group of computer executable instructions the flow of accompanying drawing illustrates the step of
Performed in computer system, although also, show logical order in flow charts, in some cases, can be with not
The order being same as herein performs shown or described step.
Embodiment 2
According to embodiments of the present invention, a kind of data processing equipment for being used to implement above-mentioned data processing method is additionally provided,
As shown in Fig. 2 the device includes：
1) first acquisition unit 202, for obtaining pending data；
2) second acquisition unit 204, it is single for being obtained according to the data processing time of pending data single data processing
Secondary datum target treating capacity；
3) processing unit 206, for handling pending data according to single datum target treating capacity.
Alternatively, in the present embodiment, above-mentioned data processing equipment can be, but not limited to be applied to batch data importing number
During according to storehouse, for example, it is assumed that certain enterprise needs a large number of users data importing database, then imported to ensure data
Efficiency, it is necessary to calculate the data volume that above-mentioned mass data single to be imported imports.Specifically, data to be imported are obtained,
And data to be imported are carried out in batches, to obtain the processing time imported per batch data single, and the number imported according to single data
The target treating capacity of single data importing is obtained according to processing time, and then controls and is handled when data import according to above-mentioned target
Amount, database is imported by above-mentioned enterprise user data to be imported in batches.The example above is a kind of example, the present embodiment pair
This is not limited in any way.
Alternatively, in the present embodiment, before pending data is obtained, in addition to：Establish and pending data phase
The running environment matched somebody with somebody.That is, before pending data is handled, will also the running environment where pending data be simulated,
To avoid the single datum target treating capacity inaccuracy being calculated due to departing from original performing environment to cause, and then influence number
According to treatment effeciency.
Alternatively, in the present embodiment, the above-mentioned data processing time according to the pending data single data processing
Obtaining single datum target treating capacity includes：Different single data processing amount input data processing systems is repeatedly selected, to obtain
The data processing time of single data processing is taken, and then data processing time is met that the single data processing amount of predetermined condition is made
For single datum target treating capacity.
Alternatively, in the present embodiment, the mode of the selection single data processing amount can include but is not limited to following
At least one：
1) single data processing amount is randomly choosed in predetermined scope；
2) multiple single data processing amounts are selected according to pre-provisioning request in predetermined scope, in more multiple single data
After data processing time corresponding to treating capacity, the most short single data processing amount of data processing time is selected.
Alternatively, in the present embodiment, the predetermined scope of above-mentioned determination single data processing amount can include but unlimited
Determined according to the total amount of pending data.For example, when first time handling pending data, can be according to the total of pending data
S is measured, judges the span of preferably single data processing amount, for example, the single selected by processing pending data for the first time
Data processing amount can be the numerical value in scope [A, B].
The embodiment provided by the application, at the data according to the pending data single data processing got
The time is managed, single datum target treating capacity is obtained, is largely treated further according to the single datum target treating capacity batch processing
Processing data, so as to avoid artificially judging that the treatment effeciency caused by single data processing amount is low by experience in the prior art
Problem, so realize it is fast automatic find the more suitable single data processing amount of pending data, with improve data processing effect
Rate, and meet the data processing needs of user.Further, due to without human intervention, the optimal single data of acquisition of automation
Treating capacity, and then also saved during mass data is handled for user at substantial amounts of data processing cost and data
Manage the time.
As a kind of optional scheme, second acquisition unit 204 includes：
1) processing module, for by obtaining single datum target treating capacity with lower module：
(1) first choice submodule, for selecting single data processing according to the first predetermined condition in pending data
Amount；
(2) acquisition submodule, the list of the data for obtaining processing single data processing amount according to single data processing amount
Secondary data processing time；
(3) judging submodule, for judging whether single data processing time is less than or equal to the first predetermined threshold；
(4) second selection submodules, for when judging that single data processing time is more than the first predetermined threshold, again
Select single data processing amount；
(5) determination sub-module, it is single for when judging that single data processing time is less than or equal to the first predetermined threshold, inciting somebody to action
Single data processing amount corresponding to secondary data processing time will be used as single datum target treating capacity；
2) judge module, for judging whether to obtain single datum target treating capacity.
Alternatively, in the present embodiment, above-mentioned first predetermined threshold can be, but not limited to different according to data handling system
Real-time demand determine.
Specifically illustrated with reference to the example below, in the running environment of simulation, repeatedly input is predetermined according to first respectively
The single data processing amount of condition selection, when obtaining single data processing after above-mentioned single data processing amount test run
Between, the single data processing time and the size of the first predetermined threshold that obtain more every time, if judging to be more than the first predetermined threshold
Value, then it represents that single data processing amount corresponding to the single data processing time is not most suitable single datum target processing
Amount；If conversely, judge to be less than or equal to the first predetermined threshold, then it represents that corresponding to the single data processing time at single data
Reason amount is more suitable single datum target treating capacity.
The embodiment provided by the application, by selecting single data processing amount input data according to the first predetermined condition
Processing system is to obtain the data processing time of single data corresponding to single data processing amount, further, judges above-mentioned single
Whether the data processing time of data is less than or equal to the first predetermined threshold, can when judging to be less than or equal to the first predetermined threshold
Obtain single data processing amount corresponding to the data processing time of above-mentioned single data for batch processing above-mentioned pending data when
More suitable single datum target treating capacity.Pending data further is handled using above-mentioned single datum target treating capacity,
To avoid in the prior art due to artificially judging that single datum target treating capacity is caused at used single data by experience
Reason amount is unreasonable, and then the problem of cause data-handling efficiency to reduce.
As a kind of optional scheme, first choice submodule is realized in pending data according to the by following steps
One predetermined condition selection single data processing amount includes：
S1, it is determined that the scope of selection single data processing amount；
S2, the first data processing amount, the second data processing amount and the 3rd number are selected according to the second predetermined condition from scope
According to treating capacity, wherein, the first data processing amount is less than the second data processing amount, and the second data processing amount is less than the 3rd data processing
Amount；
S3, when obtaining the single data processing of the first data processing amount, the second data processing amount and the 3rd data processing amount
Between, and the single data processing time to getting is ranked up；
S4, using data processing amount corresponding to single data processing time is most short as single data processing amount, and according to obtaining
The single data processing time got redefines the scope of selection single data processing amount.
Alternatively, in the present embodiment, the scope of above-mentioned determination single data processing amount can also include according to single number
According to data processing time determine.
Specifically illustrated with reference to the example below, it is assumed that the scope for determining single data processing amount is [A, B], from the scope
The first data processing amount x, the second data processing amount y, the 3rd data processing amount z of interior selection, wherein, x<y<Z, obtain respectively above-mentioned
The data processing time of single data corresponding to single data processing amount is respectively t1, t2, t3.Above-mentioned data processing time is arranged
Sequence, it is assumed that t1<t2<T3, then single data processing amount corresponding to data processing time t1 by it is selected input running environment carry out
Data test, to judge whether the single data processing amount is that most suitable single datum target is handled when batch data is handled
Amount.
The embodiment provided by the application, during by the data processing of single data that obtains multiple data processing amounts
Between, the most short single data processing amount of data processing time is therefrom selected, for judging whether the single data processing amount is several
Most suitable single datum target treating capacity during according to batch processing.Further, selected using above-mentioned from multiple data processing amounts
A more suitable single data processing amount carries out data test, ensure that the accurate of selected single data processing amount
Property.
As a kind of optional scheme, scope includes the 4th data processing amount to the quantity between the 5th data processing amount
Value, wherein, the 4th data processing amount is less than or equal to the first data processing amount, and the 5th data processing amount is more than or equal at the 3rd data
Reason amount, first choice submodule is realized by following steps selects the first data processing from scope according to the second predetermined condition
Amount, the second data processing amount and the 3rd data processing amount include：
S1, the first average value of the 4th data processing amount and the 5th data processing amount is calculated, the first average value is as second
Data processing amount；
S2, the second average value of the 4th data processing amount and the second data processing amount is calculated, the second average value is as first
Data processing amount；
S3, the 3rd average value of the 5th data processing amount and the second data processing amount is calculated, the 3rd average value is as the 3rd
Data processing amount.
Specifically illustrated with reference to the example below, it is assumed that the initial value [A, B] of the scope of single data processing amount is determined, its
In, the 4th data processing amount is A, and the 5th data processing amount is B, then the second data processing amount is M=(A+B)/2, corresponding to it
Data processing time f (M)；Further, the first data processing amount is a=(A+M)/2, and its corresponding data processing time is f
(a), the 3rd data processing amount is b=(M+B)/2, and its corresponding data processing time is f (b).
The embodiment provided by the application, the list for selecting to be tested is obtained by way of above-mentioned average
The first data processing amount, the second data processing amount, the 3rd data processing amount of secondary data processing amount, so as to ensure selected number
More meet pending data quantative attribute according to treating capacity, further such that selected single data processing amount is closer to single number
According to target treating capacity, so as to save the testing time of selection single data processing amount, the efficiency of data processing is improved.
As a kind of optional scheme, first choice submodule is realized according to the single data got by following steps
The scope that processing time redefines selection single data processing amount includes at least one of：
1), if single data processing time is most short corresponding to the second data processing amount, by the first data processing amount and
Quantitative value between three data processing amounts is as the scope after renewal；
2), if single data processing time is most short corresponding to the first data processing amount, by the 4th data processing amount and
Quantitative value between two data processing amounts is as the scope after renewal；
3), if single data processing time corresponding to the 3rd data processing amount is most short, by the second data processing amount and
Quantitative value between five data processing amounts is as the scope after renewal.
Specifically illustrated with reference to the example below, more above-mentioned first data processing amount, the second data processing amount, the 3rd number
According to the data processing time for the treatment of capacity：f(M)、f(a)、f(b).Further, if the data processing time f of the second data processing amount
(M) it is minimum, then new scope is arranged to A=a, B=b, i.e. single data processing is selected from the quantitative value between [a, b]
Amount；If the data processing time f (a) of the first data processing amount is minimum, new scope is arranged to B=M, A is constant, i.e., from
Single data processing amount is selected in quantitative value between [A, M]；If the data processing time f (b) of the 3rd data processing amount is minimum,
New scope is then arranged to A=M, B is constant, i.e., single data processing amount is selected from the quantitative value between [M, B].
The embodiment provided by the application, by the way that the scope for selecting single data processing amount is gradually reduced, so that
Selected single data processing amount is closer to single datum target treating capacity, so as to save the survey of selection single data processing amount
The time is tried, improves the efficiency of data processing
As a kind of optional scheme, first choice module is realized in pending data according to first by following steps
Predetermined condition selection single data processing amount includes：
S1, single data processing amount is randomly choosed from scope according to pre-determined number, wherein, pre-determined number is more than or equal to the
Two predetermined thresholds.
Alternatively, in the present embodiment, above-mentioned pre-determined number can be, but not limited to be determined according to run time, wherein, on
Run time is stated to can be, but not limited to as a cycle of operation.
The embodiment provided by the application, by way of randomly choosing single data processing amount from scope, enter one
Step, the processing routine for obtaining single data processing amount is simplified, saves development cost and resource.
As a kind of optional scheme, said apparatus also includes：
S1, unit is established, for before pending data is obtained, establishing the operation ring to match with pending data
Border.
Specifically illustrated with reference to the example below, it is assumed that need to import creation data for a certain factory, then establish and the work
The similar build environment of factory, in order to obtain accurate single datum target treating capacity.
By the application provide embodiment, by pre-establish with the pending running environment to match, so as to avoid
Due to departing from the problem of specific implementation environment causes the single datum target treating capacity inaccuracy got so that quick accurate
The single datum target treating capacity for meeting running environment needs is really obtained, and then improves the treatment effeciency of data.
Obviously, those skilled in the art should be understood that above-mentioned each module of the invention or each step can be with general
Computing device realize that they can be concentrated on single computing device, or be distributed in multiple computing devices and formed
Network on, alternatively, they can be realized with the program code that computing device can perform, it is thus possible to they are stored
Performed in the storage device by computing device, either they are fabricated to respectively each integrated circuit modules or by they
In multiple modules or step be fabricated to single integrated circuit module to realize.So, the present invention is not restricted to any specific
Hardware and software combines.
The preferred embodiments of the present invention are the foregoing is only, are not intended to limit the invention, for the skill of this area
For art personnel, the present invention can have various modifications and variations.Within the spirit and principles of the invention, that is made any repaiies
Change, equivalent substitution, improvement etc., should be included in the scope of the protection.
Claims (10)
- A kind of 1. data processing method, it is characterised in that including：Obtain pending data；Single datum target treating capacity is obtained according to the data processing time of the pending data single data processing；The pending data is handled according to the single datum target treating capacity,It is described that single datum target treating capacity bag is obtained according to the data processing time of the pending data single data processing Include：Following steps are repeated to the pending data, until obtaining the single datum target treating capacity：In the pending data single data processing amount is selected according to the first predetermined condition；The single data processing time for the data for handling the single data processing amount is obtained according to the single data processing amount；Judge whether the single data processing time is less than or equal to the first predetermined threshold；If judging, the single data processing time is more than first predetermined threshold, reselects at the single data Reason amount；If judging, the single data processing time is less than or equal to first predetermined threshold, by the single data processing The single data processing amount corresponding to time will be used as the single datum target treating capacity.
- 2. according to the method for claim 1, it is characterised in that it is described in the pending data according to the first predetermined bar Part selection single data processing amount includes：It is determined that select the scope of the single data processing amount；Selected from the scope according to the second predetermined condition at the first data processing amount, the second data processing amount and the 3rd data Reason amount, wherein, first data processing amount is less than second data processing amount, and second data processing amount is less than described 3rd data processing amount；Obtain the single number of first data processing amount, second data processing amount and the 3rd data processing amount According to processing time, and the single data processing time to getting is ranked up；Using data processing amount corresponding to the single data processing time is most short as the single data processing amount, and according to obtaining The single data processing time got redefines the scope for selecting the single data processing amount.
- 3. according to the method for claim 2, it is characterised in that the scope includes the 4th data processing amount to the 5th data Quantitative value between treating capacity, wherein, the 4th data processing amount is less than or equal to first data processing amount, and the described 5th Data processing amount is more than or equal to the 3rd data processing amount, described to select first according to the second predetermined condition from the scope Data processing amount, the second data processing amount and the 3rd data processing amount include：Calculate the first average value of the 4th data processing amount and the 5th data processing amount, the first average value conduct Second data processing amount；Calculate the second average value of the 4th data processing amount and second data processing amount, the second average value conduct First data processing amount；Calculate the 3rd average value of the 5th data processing amount and second data processing amount, the 3rd average value conduct 3rd data processing amount.
- 4. according to the method for claim 3, it is characterised in that the single data processing time that the basis is got Redefine and select the scope of the single data processing amount to include at least one of：If the single data processing time is most short corresponding to second data processing amount, by first data processing amount And the quantitative value between the 3rd data processing amount is as the scope after renewal；If the single data processing time is most short corresponding to first data processing amount, by the 4th data processing amount And the quantitative value between second data processing amount is as the scope after the renewal；If the single data processing time corresponding to the 3rd data processing amount is most short, by second data processing amount And the quantitative value between the 5th data processing amount is as the scope after the renewal.
- 5. according to the method for claim 2, it is characterised in that it is described in the pending data according to the first predetermined bar Part selection single data processing amount includes：The single data processing amount is randomly choosed from the scope according to pre-determined number, wherein, the pre-determined number is more than Equal to the second predetermined threshold.
- A kind of 6. data processing equipment, it is characterised in that including：First acquisition unit, for obtaining pending data；Second acquisition unit, for obtaining single data according to the data processing time of the pending data single data processing Target treating capacity；Processing unit, for handling the pending data according to the single datum target treating capacity,The second acquisition unit includes：Processing module, for by obtaining the single datum target treating capacity with lower module：First choice submodule, for selecting single data processing amount according to the first predetermined condition in the pending data；Acquisition submodule, for obtaining the list for the data for handling the single data processing amount according to the single data processing amount Secondary data processing time；Judging submodule, for judging whether the single data processing time is less than or equal to the first predetermined threshold；Second selection submodule, for when judging that the single data processing time is more than first predetermined threshold, weighing Newly select the single data processing amount；Determination sub-module, for when judging that the single data processing time is less than or equal to first predetermined threshold, inciting somebody to action The single data processing amount corresponding to the single data processing time will be used as the single datum target treating capacity；Judge module, for judging whether to obtain the single datum target treating capacity.
- 7. device according to claim 6, it is characterised in that the first choice submodule is realized by following steps Include in the pending data according to the first predetermined condition selection single data processing amount：It is determined that select the scope of the single data processing amount；Selected from the scope according to the second predetermined condition at the first data processing amount, the second data processing amount and the 3rd data Reason amount, wherein, first data processing amount is less than second data processing amount, and second data processing amount is less than described 3rd data processing amount；Obtain the single number of first data processing amount, second data processing amount and the 3rd data processing amount According to processing time, and the single data processing time to getting is ranked up；Using data processing amount corresponding to the single data processing time is most short as the single data processing amount, and according to obtaining The single data processing time got redefines the scope for selecting the single data processing amount.
- 8. device according to claim 7, it is characterised in that the scope includes the 4th data processing amount to the 5th data Quantitative value between treating capacity, wherein, the 4th data processing amount is less than or equal to first data processing amount, and the described 5th Data processing amount is more than or equal to the 3rd data processing amount, the first choice submodule by following steps realize it is described from In the scope the first data processing amount, the second data processing amount and the 3rd data processing amount bag are selected according to the second predetermined condition Include：Calculate the first average value of the 4th data processing amount and the 5th data processing amount, the first average value conduct Second data processing amount；Calculate the second average value of the 4th data processing amount and second data processing amount, the second average value conduct First data processing amount；Calculate the 3rd average value of the 5th data processing amount and second data processing amount, the 3rd average value conduct 3rd data processing amount.
- 9. device according to claim 8, it is characterised in that the first choice submodule realizes institute by following steps State and the scope bag for selecting the single data processing amount is redefined according to the single data processing time got Include at least one of：If the single data processing time is most short corresponding to second data processing amount, by first data processing amount And the quantitative value between the 3rd data processing amount is as the scope after renewal；If the single data processing time is most short corresponding to first data processing amount, by the 4th data processing amount And the quantitative value between second data processing amount is as the scope after the renewal；If the single data processing time corresponding to the 3rd data processing amount is most short, by second data processing amount And the quantitative value between the 5th data processing amount is as the scope after the renewal.
- 10. device according to claim 7, it is characterised in that the first choice module is realized by following steps Include in the pending data according to the first predetermined condition selection single data processing amount：The single data processing amount is randomly choosed from the scope according to pre-determined number, wherein, the pre-determined number is more than Equal to the second predetermined threshold.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410766560.8A CN104504020B (en) | 2014-12-11 | 2014-12-11 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410766560.8A CN104504020B (en) | 2014-12-11 | 2014-12-11 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104504020A CN104504020A (en) | 2015-04-08 |
| CN104504020B true CN104504020B (en) | 2018-02-23 |
Family
ID=52945418
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410766560.8A Active CN104504020B (en) | 2014-12-11 | 2014-12-11 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104504020B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0420419A1 (en) * | 1989-09-20 | 1991-04-03 | Hitachi, Ltd. | Method and apparatus for on-line processing of transaction data |
| CN103618716A (en) * | 2013-11-28 | 2014-03-05 | 福建星网锐捷网络有限公司 | Conversation interaction method, equipment and system of terminal WAN management protocol |
| CN104102646A (en) * | 2013-04-07 | 2014-10-15 | 腾讯科技（深圳）有限公司 | Method, device and system for processing data |
-
2014
- 2014-12-11 CN CN201410766560.8A patent/CN104504020B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0420419A1 (en) * | 1989-09-20 | 1991-04-03 | Hitachi, Ltd. | Method and apparatus for on-line processing of transaction data |
| CN104102646A (en) * | 2013-04-07 | 2014-10-15 | 腾讯科技（深圳）有限公司 | Method, device and system for processing data |
| CN103618716A (en) * | 2013-11-28 | 2014-03-05 | 福建星网锐捷网络有限公司 | Conversation interaction method, equipment and system of terminal WAN management protocol |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104504020A (en) | 2015-04-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12093708B2 (en) | Virtual machine scheduling method and apparatus | |
| CN105138371B (en) | Method for upgrading software and device | |
| CN105224458A (en) | A kind of database method of testing and system | |
| CN104239567B (en) | Dimension treating method and apparatus in data warehouse | |
| CN106372977B (en) | A kind of processing method and equipment of virtual account | |
| CN110262878A (en) | Timed task processing method, device, equipment and computer readable storage medium | |
| CN110389822A (en) | The node scheduling method, apparatus and server of execution task | |
| CN106407203A (en) | Method and device for identifying target terminal | |
| CN108520329B (en) | Accurate automatic allocation method and device for second-hand house clients based on broker portrait | |
| CN107885913B (en) | Feasibility judgment method and device for radiation field shielding scheme, computer equipment and storage medium | |
| CN112333246A (en) | ABtest experiment method and device, intelligent terminal and storage medium | |
| CN110909888A (en) | Method, device and equipment for constructing generic decision tree and readable storage medium | |
| CN110019625B (en) | Text standard address spatialization method and device and computer readable storage medium | |
| CN106156170A (en) | The analysis of public opinion method and device | |
| CN110083506A (en) | The method and device of cluster resource amount optimization | |
| CN114416583A (en) | Workload determination method, device, equipment and storage medium for automatic test | |
| CN104504020B (en) | Data processing method and device | |
| CN107729341A (en) | Electronic installation, information inquiry control method and computer-readable recording medium | |
| CN107104829B (en) | Physical equipment matching distribution method and device based on network topology data | |
| CN111427660A (en) | Scheduling method and device for uploading machine | |
| CN110138892A (en) | Determine the method and device of equipment regional information | |
| CN114238106A (en) | Test time prediction method and device, electronic device and storage medium | |
| CN112529470A (en) | Task execution method, device, equipment and storage medium of website robot | |
| CN112070349A (en) | Order allocation method, device, equipment and storage medium | |
| CN105281977B (en) | A kind of intelligent behaviour method of testing and system based on binary tree algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Improved data with video transmitter Effective date of registration: 20190531 Granted publication date: 20180223 Pledgee: Shenzhen Black Horse World Investment Consulting Co.,Ltd. Pledgor: BEIJING GRIDSUM TECHNOLOGY Co.,Ltd. Registration number: 2019990000503 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| CP02 | Change in the address of a patent holder |
Address after: 100083 No. 401, 4th Floor, Haitai Building, 229 North Fourth Ring Road, Haidian District, Beijing Patentee after: BEIJING GRIDSUM TECHNOLOGY Co.,Ltd. Address before: 100086 Beijing city Haidian District Shuangyushu Area No. 76 Zhichun Road cuigongfandian 8 layer A Patentee before: BEIJING GRIDSUM TECHNOLOGY Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder | ||
| PP01 | Preservation of patent right |
Effective date of registration: 20240604 Granted publication date: 20180223 |
|
| PP01 | Preservation of patent right |