CN104504020A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN104504020A CN104504020A CN201410766560.8A CN201410766560A CN104504020A CN 104504020 A CN104504020 A CN 104504020A CN 201410766560 A CN201410766560 A CN 201410766560A CN 104504020 A CN104504020 A CN 104504020A
- Authority
- CN
- China
- Prior art keywords
- data processing
- processing amount
- single data
- amount
- mentioned
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a data processing method and device, wherein the method comprises the following steps that data to be process is obtained; the single-time data target processing quantity is obtained according to the data processing time for the single-time data processing of the data to be processed; the data to be processed is processed according to the single-time data target processing quantity. The method and the device solve the technical problem of low data processing efficiency due to the existing data processing mode.
Description
Technical field
The present invention relates to computer realm, in particular to a kind of data processing method and device.
Background technology
Nowadays, along with the development of technology, increasing enterprise or factory all bring into use electronic management, in this process, just will inevitably produce a large amount of data, how these data are imported database rapidly to carry out data processing in time just become a problem in the urgent need to address.
At present, in order to solve the problem, provide one in prior art and mass data is carried out batch treatment, then batch data being imported to the mode of database, to improve the importing efficiency of data.But import in the process of data at above-mentioned batch, set single data importing amount is but the fixed value artificially set according to personal experience, and then performs the importing of batch data according to this fixed value.That is, the batch data lead-in mode that prior art provides is comparatively subjective, if this amount setting is too small, the data volume of each process is just little, will number of processes be increased when total amount is constant, namely with the interaction times of database, increase the cost of data interaction; And if that this amount is arranged is excessive, cache resources will be taken after too much for a long time, form the competition deadlock of resource.In other words, if the setting of above-mentioned single data importing amount is inaccurate, will directly affects the efficiency of data importing, and then affect the treatment effeciency of data.
For the problem in correlation technique, at present effective solution is not yet proposed.
Summary of the invention
Fundamental purpose of the present invention is to provide a kind of data processing method and device, to solve the low technical matters of data-handling efficiency owing to adopting existing data processing method to cause.
According to an aspect of the present invention, provide a kind of data processing method, the method comprises: obtain pending data; Single datum target treatment capacity is obtained according to the data processing time of above-mentioned pending data sheet secondary data process; According to the above-mentioned pending data of above-mentioned single datum target treatment capacity process.
Alternatively, the above-mentioned data processing time according to the process of above-mentioned pending data sheet secondary data obtains single datum target treatment capacity and comprises: perform following steps, until obtain above-mentioned single datum target treatment capacity to above-mentioned pending Data duplication: in above-mentioned pending data, select single data processing amount according to the first predetermined condition; The single data processing time of the data of the above-mentioned single data processing amount of process is obtained according to above-mentioned single data processing amount; Judge whether above-mentioned single data processing time is less than or equal to the first predetermined threshold; If judge, above-mentioned single data processing time is greater than above-mentioned first predetermined threshold, then reselect above-mentioned single data processing amount; If judge, above-mentioned single data processing time is less than or equal to above-mentioned first predetermined threshold, then will as above-mentioned single datum target treatment capacity using above-mentioned single data processing amount corresponding for above-mentioned single data processing time.
Alternatively, above-mentionedly in above-mentioned pending data, single data processing amount is selected to comprise according to the first predetermined condition: the scope determining to select above-mentioned single data processing amount; The first data processing amount, the second data processing amount and the 3rd data processing amount is selected according to the second predetermined condition from above-mentioned scope, wherein, above-mentioned first data processing amount is less than above-mentioned second data processing amount, and above-mentioned second data processing amount is less than above-mentioned 3rd data processing amount; Obtain the above-mentioned single data processing time of above-mentioned first data processing amount, above-mentioned second data processing amount and above-mentioned 3rd data processing amount, and the above-mentioned single data processing time got is sorted; Using the data processing amount of the shortest correspondence of above-mentioned single data processing time as above-mentioned single data processing amount, and redefine the above-mentioned scope selecting above-mentioned single data processing amount according to the above-mentioned single data processing time got.
Alternatively, above-mentioned scope comprises the quantitative value between the 4th data processing amount to the 5th data processing amount, wherein, above-mentioned 4th data processing amount is less than or equal to above-mentioned first data processing amount, above-mentioned 5th data processing amount is more than or equal to above-mentioned 3rd data processing amount, above-mentionedly from above-mentioned scope, select the first data processing amount, the second data processing amount and the 3rd data processing amount to comprise according to the second predetermined condition: the first mean value calculating above-mentioned 4th data processing amount and above-mentioned 5th data processing amount, above-mentioned first mean value is as above-mentioned second data processing amount; Calculate the second mean value of above-mentioned 4th data processing amount and above-mentioned second data processing amount, above-mentioned second mean value is as above-mentioned first data processing amount; Calculate the 3rd mean value of above-mentioned 5th data processing amount and above-mentioned second data processing amount, above-mentioned 3rd mean value is as above-mentioned 3rd data processing amount.
Alternatively, the above-mentioned single data processing time that above-mentioned basis gets redefine select the above-mentioned scope of above-mentioned single data processing amount comprise following one of at least: if above-mentioned single data processing time corresponding to above-mentioned second data processing amount is the shortest, then using the quantitative value between above-mentioned first data processing amount and above-mentioned 3rd data processing amount as the scope after upgrading; If the above-mentioned single data processing time that above-mentioned first data processing amount is corresponding is the shortest, then using the quantitative value between above-mentioned 4th data processing amount and above-mentioned second data processing amount as the scope after above-mentioned renewal; If the above-mentioned single data processing time that above-mentioned 3rd data processing amount is corresponding is the shortest, then using the quantitative value between above-mentioned second data processing amount and above-mentioned 5th data processing amount as the scope after above-mentioned renewal.
Alternatively, above-mentionedly in above-mentioned pending data, single data processing amount is selected to comprise according to the first predetermined condition: according to the above-mentioned single data processing amount of pre-determined number Stochastic choice from above-mentioned scope, wherein, above-mentioned pre-determined number is more than or equal to the second predetermined threshold.
Alternatively, before the pending data of above-mentioned acquisition, also comprise: set up the running environment with above-mentioned pending data match.
According to a further aspect in the invention, provide a kind of data processing equipment, this device comprises: the first acquiring unit, for obtaining pending data; Second acquisition unit, for obtaining single datum target treatment capacity according to the data processing time of above-mentioned pending data sheet secondary data process; Processing unit, for according to the above-mentioned pending data of above-mentioned single datum target treatment capacity process.
Alternatively, above-mentioned second acquisition unit comprises: processing module, for by obtaining above-mentioned single datum target treatment capacity with lower module: the first chooser module, for selecting single data processing amount according to the first predetermined condition in above-mentioned pending data; Obtain submodule, for obtaining the single data processing time of the data of the above-mentioned single data processing amount of process according to above-mentioned single data processing amount; Judge submodule, for judging whether above-mentioned single data processing time is less than or equal to the first predetermined threshold; Second chooser module, for when judging that above-mentioned single data processing time is greater than above-mentioned first predetermined threshold, reselects above-mentioned single data processing amount; Determine submodule, for when judging that above-mentioned single data processing time is less than or equal to above-mentioned first predetermined threshold, will as above-mentioned single datum target treatment capacity using above-mentioned single data processing amount corresponding for above-mentioned single data processing time; Judge module, obtains above-mentioned single datum target treatment capacity for judging whether.
Alternatively, above-mentioned first chooser module realizes selecting single data processing amount to comprise according to the first predetermined condition in above-mentioned pending data by following steps: the scope determining to select above-mentioned single data processing amount; The first data processing amount, the second data processing amount and the 3rd data processing amount is selected according to the second predetermined condition from above-mentioned scope, wherein, above-mentioned first data processing amount is less than above-mentioned second data processing amount, and above-mentioned second data processing amount is less than above-mentioned 3rd data processing amount; Obtain the above-mentioned single data processing time of above-mentioned first data processing amount, above-mentioned second data processing amount and above-mentioned 3rd data processing amount, and the above-mentioned single data processing time got is sorted; Using the data processing amount of the shortest correspondence of above-mentioned single data processing time as above-mentioned single data processing amount, and redefine the above-mentioned scope selecting above-mentioned single data processing amount according to the above-mentioned single data processing time got.
Alternatively, above-mentioned scope comprises the quantitative value between the 4th data processing amount to the 5th data processing amount, wherein, above-mentioned 4th data processing amount is less than or equal to above-mentioned first data processing amount, above-mentioned 5th data processing amount is more than or equal to above-mentioned 3rd data processing amount, above-mentioned first chooser module realizes above-mentionedly from above-mentioned scope, selecting the first data processing amount according to the second predetermined condition by following steps, second data processing amount and the 3rd data processing amount comprise: the first mean value calculating above-mentioned 4th data processing amount and above-mentioned 5th data processing amount, above-mentioned first mean value is as above-mentioned second data processing amount, calculate the second mean value of above-mentioned 4th data processing amount and above-mentioned second data processing amount, above-mentioned second mean value is as above-mentioned first data processing amount, calculate the 3rd mean value of above-mentioned 5th data processing amount and above-mentioned second data processing amount, above-mentioned 3rd mean value is as above-mentioned 3rd data processing amount.
Alternatively, above-mentioned first chooser module by following steps realize above-mentioned single data processing time that above-mentioned basis gets redefine select the above-mentioned scope of above-mentioned single data processing amount comprise following one of at least: if above-mentioned single data processing time corresponding to above-mentioned second data processing amount is the shortest, then using the quantitative value between above-mentioned first data processing amount and above-mentioned 3rd data processing amount as the scope after upgrading; If the above-mentioned single data processing time that above-mentioned first data processing amount is corresponding is the shortest, then using the quantitative value between above-mentioned 4th data processing amount and above-mentioned second data processing amount as the scope after above-mentioned renewal; If the above-mentioned single data processing time that above-mentioned 3rd data processing amount is corresponding is the shortest, then using the quantitative value between above-mentioned second data processing amount and above-mentioned 5th data processing amount as the scope after above-mentioned renewal.
Alternatively, above-mentioned first selects module to realize selecting single data processing amount to comprise according to the first predetermined condition in above-mentioned pending data by following steps: according to the above-mentioned single data processing amount of pre-determined number Stochastic choice from above-mentioned scope, wherein, above-mentioned pre-determined number is more than or equal to the second predetermined threshold.
Alternatively, said apparatus also comprises: set up unit, for before the pending data of above-mentioned acquisition, sets up the running environment with above-mentioned pending data match.
By the embodiment that the application provides, by the data processing time according to the pending data sheet secondary data process got, obtain single datum target treatment capacity, further according to the pending data that this single datum target treatment capacity batch treatment is a large amount of, thus avoid artificially judging by experience the problem that treatment effeciency that single data processing amount causes is low in prior art, and then realize the fast automatic single data processing amount finding pending data comparatively suitable, to improve data-handling efficiency, and meet the data processing needs of user.Further, due to without the need to human intervention, the optimum single data processing amount of acquisition of robotization, and then reach in the process of process mass data also for user saves a large amount of data processing cost and data processing time.
Further, by setting up and the pending running environment matched in advance, thus avoid the inaccurate problem of single datum target treatment capacity causing getting owing to departing from concrete implementation environment, make quick and precisely to obtain the single datum target treatment capacity meeting running environment needs, and then improve the treatment effeciency of data.
Accompanying drawing explanation
The accompanying drawing forming a application's part is used to provide a further understanding of the present invention, and schematic description and description of the present invention, for explaining the present invention, does not form inappropriate limitation of the present invention.In the accompanying drawings:
Fig. 1 is the process flow diagram of a kind of optional data processing method according to the embodiment of the present invention;
Fig. 2 is the schematic diagram of a kind of optional data processing equipment according to the embodiment of the present invention.
Embodiment
It should be noted that, when not conflicting, the embodiment in the application and the feature in embodiment can combine mutually.Below with reference to the accompanying drawings and describe the present invention in detail in conjunction with the embodiments.
Embodiment 1
According to the embodiment of the present invention, provide a kind of data processing method, as shown in Figure 1, the method comprises:
S102, obtains pending data;
S104, obtains single datum target treatment capacity according to the data processing time of pending data sheet secondary data process;
S106, according to the pending data of single datum target treatment capacity process.
Alternatively, in the present embodiment, above-mentioned data processing method can be, but not limited to be applied to batch data and imports in the process of database, such as, suppose that certain enterprise needs a large number of users data importing database, then in order to ensure the efficiency of data importing, just need the data volume calculating the importing of above-mentioned mass data single to be imported.Specifically, obtain data to be imported, and data to be imported are carried out in batches, obtain the processing time that every batch data single imports, and the target treatment capacity of single data importing is obtained according to the data processing time of single data importing, and then control, when data importing according to above-mentioned target treatment capacity, user data to be imported for above-mentioned enterprise to be imported database in batches.Above-mentioned citing is a kind of example, and the present embodiment is not limited in any way this.
Alternatively, in the present embodiment, before the pending data of acquisition, also comprise: set up the running environment with pending data match.That is, before the pending data of process, also will the running environment at the pending data place of simulation, inaccurate to avoid owing to departing from the single datum target treatment capacity that original execution environment causes calculating, and then affect data-handling efficiency.
Alternatively, in the present embodiment, the above-mentioned data processing time according to the process of described pending data sheet secondary data obtains single datum target treatment capacity and comprises: repeatedly select different single data processing amount input data processing systems, to obtain the data processing time of single data processing, and then single data processing amount data processing time being met predetermined condition is as single datum target treatment capacity.
Alternatively, in the present embodiment, the mode of described selection single data processing amount can include but not limited to following one of at least:
1) Stochastic choice single data processing amount in predetermined scope;
2) in predetermined scope, select multiple single data processing amount according to pre-provisioning request, after the data processing time that more multiple single data processing amount is corresponding, select the single data processing amount that data processing time is the shortest.
Alternatively, in the present embodiment, above-mentionedly determine that the predetermined scope of single data processing amount can include but not limited to determine according to the total amount of pending data.Such as, when first time processes pending data, can according to the total amount S of pending data, judge the span of preferably single data processing amount, such as, the single data processing amount processed for the first time selected by pending data can for the numerical value in scope [A, B].
By the embodiment that the application provides, by the data processing time according to the pending data sheet secondary data process got, obtain single datum target treatment capacity, further according to the pending data that this single datum target treatment capacity batch treatment is a large amount of, thus avoid artificially judging by experience the problem that treatment effeciency that single data processing amount causes is low in prior art, and then realize the fast automatic single data processing amount finding pending data comparatively suitable, to improve data-handling efficiency, and meet the data processing needs of user.Further, due to without the need to human intervention, the optimum single data processing amount of acquisition of robotization, and then reach in the process of process mass data also for user saves a large amount of data processing cost and data processing time.
As the optional scheme of one, obtain single datum target treatment capacity according to the data processing time of pending data sheet secondary data process and comprise:
S1, performs following steps, until obtain single datum target treatment capacity to pending Data duplication:
S12, selects single data processing amount according to the first predetermined condition in pending data;
S14, obtains the single data processing time of the data of process single data processing amount according to single data processing amount;
S16, judges whether single data processing time is less than or equal to the first predetermined threshold;
S18, if judge, single data processing time is greater than the first predetermined threshold, then reselect single data processing amount;
S20, if judge, single data processing time is less than or equal to the first predetermined threshold, then will as single datum target treatment capacity using single data processing amount corresponding for single data processing time.
Alternatively, in the present embodiment, above-mentioned first predetermined threshold can be, but not limited to the real-time demand decision different according to data handling system.
Specifically be described in conjunction with following example, in the running environment of simulation, repeatedly input the single data processing amount selected according to the first predetermined condition respectively, single data processing time is obtained after the test run of above-mentioned single data processing amount, the single data processing time more at every turn obtained and the size of the first predetermined threshold, be greater than the first predetermined threshold if judge, then represent that single data processing amount corresponding to this single data processing time is not most suitable single datum target treatment capacity; Otherwise, be less than or equal to the first predetermined threshold if judge, then represent that single data processing amount corresponding to this single data processing time is comparatively suitable single datum target treatment capacity.
By the embodiment that the application provides, by selecting single data processing amount input data processing system to obtain the data processing time of single data corresponding to single data processing amount according to the first predetermined condition, further, judge whether the data processing time of above-mentioned single data is less than or equal to the first predetermined threshold, when judging to be less than or equal to the first predetermined threshold, single datum target treatment capacity comparatively suitable when can to obtain single data processing amount corresponding to the data processing time of above-mentioned single data be batch treatment above-mentioned pending data.Utilize the pending data of above-mentioned single datum target treatment capacity process further, to avoid owing to artificially being judged that by experience single datum target treatment capacity causes adopted single data processing amount unreasonable in prior art, and then the problem causing data-handling efficiency to reduce.
As the optional scheme of one, in pending data, single data processing amount is selected to comprise according to the first predetermined condition:
S1, determines the scope selecting single data processing amount;
S2, from scope, select the first data processing amount, the second data processing amount and the 3rd data processing amount according to the second predetermined condition, wherein, the first data processing amount is less than the second data processing amount, and the second data processing amount is less than the 3rd data processing amount;
S3, obtains the single data processing time of the first data processing amount, the second data processing amount and the 3rd data processing amount, and sorts to the single data processing time got;
S4, using the data processing amount of the shortest correspondence of single data processing time as single data processing amount, and redefines the scope selecting single data processing amount according to the single data processing time got.
Alternatively, in the present embodiment, above-mentionedly determine that the scope of single data processing amount can also comprise and determine according to the data processing time of single data.
Specifically be described in conjunction with following example, suppose to determine that the scope of single data processing amount is for [A, B], within the scope of this, select the first data processing amount x, the second data processing amount y, 3rd data processing amount z, wherein, x<y<z, the data processing time obtaining single data corresponding to above-mentioned single data processing amount is respectively respectively t1, t2, t3.Above-mentioned data processing time is sorted, suppose t1<t2<t3, the single data processing amount that then data processing time t1 is corresponding will be carried out data test, most suitable single datum target treatment capacity during to judge whether this single data processing amount is batch data process by selection input running environment.
By the embodiment that the application provides, by obtaining the data processing time of the single data of multiple data processing amount, therefrom select the single data processing amount that data processing time is the shortest, most suitable single datum target treatment capacity during for judging whether this single data processing amount is batch data process.Further, utilize and above-mentionedly from multiple data processing amount, select a comparatively suitable single data processing amount to carry out data test, ensure that the accuracy of selected single data processing amount.
As the optional scheme of one, scope comprises the quantitative value between the 4th data processing amount to the 5th data processing amount, wherein, 4th data processing amount is less than or equal to the first data processing amount, 5th data processing amount is more than or equal to the 3rd data processing amount, from scope, select the first data processing amount, the second data processing amount and the 3rd data processing amount to comprise according to the second predetermined condition:
S1, calculate the first mean value of the 4th data processing amount and the 5th data processing amount, the first mean value is as the second data processing amount;
S2, calculate the second mean value of the 4th data processing amount and the second data processing amount, the second mean value is as the first data processing amount;
S3, calculate the 3rd mean value of the 5th data processing amount and the second data processing amount, the 3rd mean value is as the 3rd data processing amount.
Specifically be described in conjunction with following example, suppose the initial value [A of the scope determining single data processing amount, B], wherein, 4th data processing amount is A, 5th data processing amount is B, then the second data processing amount is M=(A+B)/2, the data processing time f (M) of its correspondence; Further, the first data processing amount is a=(A+M)/2, and the data processing time of its correspondence is f (a), and the 3rd data processing amount is b=(M+B)/2, and the data processing time of its correspondence is f (b).
By the embodiment that the application provides, the first data processing amount, the second data processing amount, the 3rd data processing amount for selecting the single data processing amount carrying out testing is obtained by above-mentioned mode of averaging, thus the data processing amount selected by ensureing more meets pending data bulk feature, make selected single data processing amount closer to single datum target treatment capacity further, thus save the test duration selecting single data processing amount, improve the efficiency of data processing.
As the optional scheme of one, according to the single data processing time got redefine select the scope of single data processing amount comprise following one of at least:
S1, if single data processing time corresponding to the second data processing amount is the shortest, then using the quantitative value between the first data processing amount and the 3rd data processing amount as the scope after upgrading;
S2, if single data processing time corresponding to the first data processing amount is the shortest, then using the quantitative value between the 4th data processing amount and the second data processing amount as the scope after upgrading;
S3, if single data processing time corresponding to the 3rd data processing amount is the shortest, then using the quantitative value between the second data processing amount and the 5th data processing amount as the scope after upgrading.
Specifically be described in conjunction with following example, the data processing time of more above-mentioned first data processing amount, the second data processing amount, the 3rd data processing amount: f (M), f (a), f (b).Further, if the data processing time f of the second data processing amount (M) is minimum, then new scope is set to A=a, B=b, that is, from the quantitative value between [a, b], selects single data processing amount; If the data processing time f (a) of the first data processing amount is minimum, then new scope is set to B=M, A is constant, namely from the quantitative value between [A, M], selects single data processing amount; If the data processing time f (b) of the 3rd data processing amount is minimum, then new scope is set to A=M, B is constant, namely from the quantitative value between [M, B], selects single data processing amount.
By the embodiment that the application provides, by reducing the scope for selecting single data processing amount gradually, to make selected single data processing amount closer to single datum target treatment capacity, thus save the test duration selecting single data processing amount, improve the efficiency of data processing.
As the optional scheme of one, in pending data, single data processing amount is selected to comprise according to the first predetermined condition:
S1, according to pre-determined number Stochastic choice single data processing amount from scope, wherein, pre-determined number is more than or equal to the second predetermined threshold.
Alternatively, in the present embodiment, above-mentioned pre-determined number can be, but not limited to according to determining working time, and wherein, can be, but not limited to above-mentioned working time is a cycle of operation.
By the embodiment that the application provides, by the mode of Stochastic choice single data processing amount from scope, further, simplify the handling procedure obtaining single data processing amount, save cost of development and resource.
As the optional scheme of one, before the pending data of acquisition, also comprise:
S1, sets up the running environment with pending data match.
Specifically be described in conjunction with following example, being assumed to be a certain factory needs to import production data, then set up the build environment similar to this factory, so that obtain single datum target treatment capacity accurately.
By the embodiment that the application provides, by setting up and the pending running environment matched in advance, thus avoid the inaccurate problem of single datum target treatment capacity causing getting owing to departing from concrete implementation environment, make quick and precisely to obtain the single datum target treatment capacity meeting running environment needs, and then improve the treatment effeciency of data.
It should be noted that, can perform in the computer system of such as one group of computer executable instructions in the step shown in the process flow diagram of accompanying drawing, and, although show logical order in flow charts, but in some cases, can be different from the step shown or described by order execution herein.
Embodiment 2
According to the embodiment of the present invention, additionally provide a kind of data processing equipment for implementing above-mentioned data processing method, as shown in Figure 2, this device comprises:
1) the first acquiring unit 202, for obtaining pending data;
2) second acquisition unit 204, for obtaining single datum target treatment capacity according to the data processing time of pending data sheet secondary data process;
3) processing unit 206, for according to the pending data of single datum target treatment capacity process.
Alternatively, in the present embodiment, above-mentioned data processing equipment can be, but not limited to be applied to batch data and imports in the process of database, such as, suppose that certain enterprise needs a large number of users data importing database, then in order to ensure the efficiency of data importing, just need the data volume calculating the importing of above-mentioned mass data single to be imported.Specifically, obtain data to be imported, and data to be imported are carried out in batches, obtain the processing time that every batch data single imports, and the target treatment capacity of single data importing is obtained according to the data processing time of single data importing, and then control, when data importing according to above-mentioned target treatment capacity, user data to be imported for above-mentioned enterprise to be imported database in batches.Above-mentioned citing is a kind of example, and the present embodiment is not limited in any way this.
Alternatively, in the present embodiment, before the pending data of acquisition, also comprise: set up the running environment with pending data match.That is, before the pending data of process, also will the running environment at the pending data place of simulation, inaccurate to avoid owing to departing from the single datum target treatment capacity that original execution environment causes calculating, and then affect data-handling efficiency.
Alternatively, in the present embodiment, the above-mentioned data processing time according to the process of described pending data sheet secondary data obtains single datum target treatment capacity and comprises: repeatedly select different single data processing amount input data processing systems, to obtain the data processing time of single data processing, and then single data processing amount data processing time being met predetermined condition is as single datum target treatment capacity.
Alternatively, in the present embodiment, the mode of described selection single data processing amount can include but not limited to following one of at least:
1) Stochastic choice single data processing amount in predetermined scope;
2) in predetermined scope, select multiple single data processing amount according to pre-provisioning request, after the data processing time that more multiple single data processing amount is corresponding, select the single data processing amount that data processing time is the shortest.
Alternatively, in the present embodiment, above-mentionedly determine that the predetermined scope of single data processing amount can include but not limited to determine according to the total amount of pending data.Such as, when first time processes pending data, can according to the total amount S of pending data, judge the span of preferably single data processing amount, such as, the single data processing amount processed for the first time selected by pending data can for the numerical value in scope [A, B].
By the embodiment that the application provides, by the data processing time according to the pending data sheet secondary data process got, obtain single datum target treatment capacity, further according to the pending data that this single datum target treatment capacity batch treatment is a large amount of, thus avoid artificially judging by experience the problem that treatment effeciency that single data processing amount causes is low in prior art, and then realize the fast automatic single data processing amount finding pending data comparatively suitable, to improve data-handling efficiency, and meet the data processing needs of user.Further, due to without the need to human intervention, the optimum single data processing amount of acquisition of robotization, and then reach in the process of process mass data also for user saves a large amount of data processing cost and data processing time.
As the optional scheme of one, second acquisition unit 204 comprises:
1) processing module, for by obtaining single datum target treatment capacity with lower module:
(1) first chooser module, for selecting single data processing amount according to the first predetermined condition in pending data;
(2) submodule is obtained, for obtaining the single data processing time of the data of process single data processing amount according to single data processing amount;
(3) submodule is judged, for judging whether single data processing time is less than or equal to the first predetermined threshold;
(4) second chooser modules, for when judging that single data processing time is greater than the first predetermined threshold, reselect single data processing amount;
(5) submodule is determined, for when judging that single data processing time is less than or equal to the first predetermined threshold, will as single datum target treatment capacity using single data processing amount corresponding for single data processing time;
2) judge module, obtains single datum target treatment capacity for judging whether.
Alternatively, in the present embodiment, above-mentioned first predetermined threshold can be, but not limited to the real-time demand decision different according to data handling system.
Specifically be described in conjunction with following example, in the running environment of simulation, repeatedly input the single data processing amount selected according to the first predetermined condition respectively, single data processing time is obtained after the test run of above-mentioned single data processing amount, the single data processing time more at every turn obtained and the size of the first predetermined threshold, be greater than the first predetermined threshold if judge, then represent that single data processing amount corresponding to this single data processing time is not most suitable single datum target treatment capacity; Otherwise, be less than or equal to the first predetermined threshold if judge, then represent that single data processing amount corresponding to this single data processing time is comparatively suitable single datum target treatment capacity.
By the embodiment that the application provides, by selecting single data processing amount input data processing system to obtain the data processing time of single data corresponding to single data processing amount according to the first predetermined condition, further, judge whether the data processing time of above-mentioned single data is less than or equal to the first predetermined threshold, when judging to be less than or equal to the first predetermined threshold, single datum target treatment capacity comparatively suitable when can to obtain single data processing amount corresponding to the data processing time of above-mentioned single data be batch treatment above-mentioned pending data.Utilize the pending data of above-mentioned single datum target treatment capacity process further, to avoid owing to artificially being judged that by experience single datum target treatment capacity causes adopted single data processing amount unreasonable in prior art, and then the problem causing data-handling efficiency to reduce.
As the optional scheme of one, the first chooser module realizes selecting single data processing amount to comprise according to the first predetermined condition in pending data by following steps:
S1, determines the scope selecting single data processing amount;
S2, from scope, select the first data processing amount, the second data processing amount and the 3rd data processing amount according to the second predetermined condition, wherein, the first data processing amount is less than the second data processing amount, and the second data processing amount is less than the 3rd data processing amount;
S3, obtains the single data processing time of the first data processing amount, the second data processing amount and the 3rd data processing amount, and sorts to the single data processing time got;
S4, using the data processing amount of the shortest correspondence of single data processing time as single data processing amount, and redefines the scope selecting single data processing amount according to the single data processing time got.
Alternatively, in the present embodiment, above-mentionedly determine that the scope of single data processing amount can also comprise and determine according to the data processing time of single data.
Specifically be described in conjunction with following example, suppose to determine that the scope of single data processing amount is for [A, B], within the scope of this, select the first data processing amount x, the second data processing amount y, 3rd data processing amount z, wherein, x<y<z, the data processing time obtaining single data corresponding to above-mentioned single data processing amount is respectively respectively t1, t2, t3.Above-mentioned data processing time is sorted, suppose t1<t2<t3, the single data processing amount that then data processing time t1 is corresponding will be carried out data test, most suitable single datum target treatment capacity during to judge whether this single data processing amount is batch data process by selection input running environment.
By the embodiment that the application provides, by obtaining the data processing time of the single data of multiple data processing amount, therefrom select the single data processing amount that data processing time is the shortest, most suitable single datum target treatment capacity during for judging whether this single data processing amount is batch data process.Further, utilize and above-mentionedly from multiple data processing amount, select a comparatively suitable single data processing amount to carry out data test, ensure that the accuracy of selected single data processing amount.
As the optional scheme of one, scope comprises the quantitative value between the 4th data processing amount to the 5th data processing amount, wherein, 4th data processing amount is less than or equal to the first data processing amount, 5th data processing amount is more than or equal to the 3rd data processing amount, and the first chooser module realizes selecting the first data processing amount, the second data processing amount and the 3rd data processing amount to comprise according to the second predetermined condition from scope by following steps:
S1, calculate the first mean value of the 4th data processing amount and the 5th data processing amount, the first mean value is as the second data processing amount;
S2, calculate the second mean value of the 4th data processing amount and the second data processing amount, the second mean value is as the first data processing amount;
S3, calculate the 3rd mean value of the 5th data processing amount and the second data processing amount, the 3rd mean value is as the 3rd data processing amount.
Specifically be described in conjunction with following example, suppose the initial value [A of the scope determining single data processing amount, B], wherein, 4th data processing amount is A, 5th data processing amount is B, then the second data processing amount is M=(A+B)/2, the data processing time f (M) of its correspondence; Further, the first data processing amount is a=(A+M)/2, and the data processing time of its correspondence is f (a), and the 3rd data processing amount is b=(M+B)/2, and the data processing time of its correspondence is f (b).
By the embodiment that the application provides, the first data processing amount, the second data processing amount, the 3rd data processing amount for selecting the single data processing amount carrying out testing is obtained by above-mentioned mode of averaging, thus the data processing amount selected by ensureing more meets pending data bulk feature, make selected single data processing amount closer to single datum target treatment capacity further, thus save the test duration selecting single data processing amount, improve the efficiency of data processing.
As the optional scheme of one, the first chooser module by following steps realize according to the single data processing time that gets redefine select the scope of single data processing amount comprise following one of at least:
1), if single data processing time corresponding to the second data processing amount is the shortest, then using the quantitative value between the first data processing amount and the 3rd data processing amount as the scope after upgrading;
2), if single data processing time corresponding to the first data processing amount is the shortest, then using the quantitative value between the 4th data processing amount and the second data processing amount as the scope after upgrading;
3), if single data processing time corresponding to the 3rd data processing amount is the shortest, then using the quantitative value between the second data processing amount and the 5th data processing amount as the scope after upgrading.
Specifically be described in conjunction with following example, the data processing time of more above-mentioned first data processing amount, the second data processing amount, the 3rd data processing amount: f (M), f (a), f (b).Further, if the data processing time f of the second data processing amount (M) is minimum, then new scope is set to A=a, B=b, that is, from the quantitative value between [a, b], selects single data processing amount; If the data processing time f (a) of the first data processing amount is minimum, then new scope is set to B=M, A is constant, namely from the quantitative value between [A, M], selects single data processing amount; If the data processing time f (b) of the 3rd data processing amount is minimum, then new scope is set to A=M, B is constant, namely from the quantitative value between [M, B], selects single data processing amount.
By the embodiment that the application provides, by reducing the scope for selecting single data processing amount gradually, to make selected single data processing amount closer to single datum target treatment capacity, thus save the test duration selecting single data processing amount, improve the efficiency of data processing
As the optional scheme of one, first selects module to realize selecting single data processing amount to comprise according to the first predetermined condition in pending data by following steps:
S1, according to pre-determined number Stochastic choice single data processing amount from scope, wherein, pre-determined number is more than or equal to the second predetermined threshold.
Alternatively, in the present embodiment, above-mentioned pre-determined number can be, but not limited to according to determining working time, and wherein, can be, but not limited to above-mentioned working time is a cycle of operation.
By the embodiment that the application provides, by the mode of Stochastic choice single data processing amount from scope, further, simplify the handling procedure obtaining single data processing amount, save cost of development and resource.
As the optional scheme of one, said apparatus also comprises:
S1, sets up unit, for before the pending data of acquisition, sets up the running environment with pending data match.
Specifically be described in conjunction with following example, being assumed to be a certain factory needs to import production data, then set up the build environment similar to this factory, so that obtain single datum target treatment capacity accurately.
By the embodiment that the application provides, by setting up and the pending running environment matched in advance, thus avoid the inaccurate problem of single datum target treatment capacity causing getting owing to departing from concrete implementation environment, make quick and precisely to obtain the single datum target treatment capacity meeting running environment needs, and then improve the treatment effeciency of data.
Obviously, those skilled in the art should be understood that, above-mentioned of the present invention each module or each step can realize with general calculation element, they can concentrate on single calculation element, or be distributed on network that multiple calculation element forms, alternatively, they can realize with the executable program code of calculation element, thus, they can be stored and be performed by calculation element in the storage device, or they are made into each integrated circuit modules respectively, or the multiple module in them or step are made into single integrated circuit module to realize.Like this, the present invention is not restricted to any specific hardware and software combination.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (14)
1. a data processing method, is characterized in that, comprising:
Obtain pending data;
Single datum target treatment capacity is obtained according to the data processing time of described pending data sheet secondary data process;
According to pending data described in the process of described single datum target treatment capacity.
2. method according to claim 1, is characterized in that, the described data processing time according to the process of described pending data sheet secondary data obtains single datum target treatment capacity and comprises:
Following steps are performed, until obtain described single datum target treatment capacity to described pending Data duplication:
Single data processing amount is selected according to the first predetermined condition in described pending data;
The single data processing time of the data of the described single data processing amount of process is obtained according to described single data processing amount;
Judge whether described single data processing time is less than or equal to the first predetermined threshold;
If judge, described single data processing time is greater than described first predetermined threshold, then reselect described single data processing amount;
If judge, described single data processing time is less than or equal to described first predetermined threshold, then will as described single datum target treatment capacity using described single data processing amount corresponding for described single data processing time.
3. method according to claim 2, is characterized in that, describedly in described pending data, selects single data processing amount to comprise according to the first predetermined condition:
Determine the scope selecting described single data processing amount;
The first data processing amount, the second data processing amount and the 3rd data processing amount is selected according to the second predetermined condition from described scope, wherein, described first data processing amount is less than described second data processing amount, and described second data processing amount is less than described 3rd data processing amount;
Obtain the described single data processing time of described first data processing amount, described second data processing amount and described 3rd data processing amount, and the described single data processing time got is sorted;
Using the data processing amount of the shortest correspondence of described single data processing time as described single data processing amount, and redefine the described scope selecting described single data processing amount according to the described single data processing time got.
4. method according to claim 3, it is characterized in that, described scope comprises the quantitative value between the 4th data processing amount to the 5th data processing amount, wherein, described 4th data processing amount is less than or equal to described first data processing amount, described 5th data processing amount is more than or equal to described 3rd data processing amount, describedly from described scope, selects the first data processing amount, the second data processing amount and the 3rd data processing amount to comprise according to the second predetermined condition:
Calculate the first mean value of described 4th data processing amount and described 5th data processing amount, described first mean value is as described second data processing amount;
Calculate the second mean value of described 4th data processing amount and described second data processing amount, described second mean value is as described first data processing amount;
Calculate the 3rd mean value of described 5th data processing amount and described second data processing amount, described 3rd mean value is as described 3rd data processing amount.
5. method according to claim 4, is characterized in that, the described single data processing time that described basis gets redefine select the described scope of described single data processing amount comprise following one of at least:
If the described single data processing time that described second data processing amount is corresponding is the shortest, then using the quantitative value between described first data processing amount and described 3rd data processing amount as upgrade after scope;
If the described single data processing time that described first data processing amount is corresponding is the shortest, then using the quantitative value between described 4th data processing amount and described second data processing amount as the scope after described renewal;
If the described single data processing time that described 3rd data processing amount is corresponding is the shortest, then using the quantitative value between described second data processing amount and described 5th data processing amount as the scope after described renewal.
6. method according to claim 3, is characterized in that, describedly in described pending data, selects single data processing amount to comprise according to the first predetermined condition:
According to pre-determined number single data processing amount described in Stochastic choice from described scope, wherein, described pre-determined number is more than or equal to the second predetermined threshold.
7. method according to claim 1, is characterized in that, before the pending data of described acquisition, also comprises:
Set up the running environment with described pending data match.
8. a data processing equipment, is characterized in that, comprising:
First acquiring unit, for obtaining pending data;
Second acquisition unit, for obtaining single datum target treatment capacity according to the data processing time of described pending data sheet secondary data process;
Processing unit, for according to pending data described in the process of described single datum target treatment capacity.
9. device according to claim 8, is characterized in that, described second acquisition unit comprises:
Processing module, for by obtaining described single datum target treatment capacity with lower module:
First chooser module, for selecting single data processing amount according to the first predetermined condition in described pending data;
Obtain submodule, for obtaining the single data processing time of the data of the described single data processing amount of process according to described single data processing amount;
Judge submodule, for judging whether described single data processing time is less than or equal to the first predetermined threshold;
Second chooser module, for when judging that described single data processing time is greater than described first predetermined threshold, reselects described single data processing amount;
Determine submodule, for when judging that described single data processing time is less than or equal to described first predetermined threshold, will as described single datum target treatment capacity using described single data processing amount corresponding for described single data processing time;
Judge module, obtains described single datum target treatment capacity for judging whether.
10. device according to claim 9, is characterized in that, described first chooser module realizes selecting single data processing amount to comprise according to the first predetermined condition in described pending data by following steps:
Determine the scope selecting described single data processing amount;
The first data processing amount, the second data processing amount and the 3rd data processing amount is selected according to the second predetermined condition from described scope, wherein, described first data processing amount is less than described second data processing amount, and described second data processing amount is less than described 3rd data processing amount;
Obtain the described single data processing time of described first data processing amount, described second data processing amount and described 3rd data processing amount, and the described single data processing time got is sorted;
Using the data processing amount of the shortest correspondence of described single data processing time as described single data processing amount, and redefine the described scope selecting described single data processing amount according to the described single data processing time got.
11. devices according to claim 10, it is characterized in that, described scope comprises the quantitative value between the 4th data processing amount to the 5th data processing amount, wherein, described 4th data processing amount is less than or equal to described first data processing amount, described 5th data processing amount is more than or equal to described 3rd data processing amount, selects the first data processing amount, the second data processing amount and the 3rd data processing amount to comprise described in described first chooser module is realized by following steps from described scope according to the second predetermined condition:
Calculate the first mean value of described 4th data processing amount and described 5th data processing amount, described first mean value is as described second data processing amount;
Calculate the second mean value of described 4th data processing amount and described second data processing amount, described second mean value is as described first data processing amount;
Calculate the 3rd mean value of described 5th data processing amount and described second data processing amount, described 3rd mean value is as described 3rd data processing amount.
12. devices according to claim 11, it is characterized in that, described first chooser module by following steps realize described single data processing time that described basis gets redefine select the described scope of described single data processing amount comprise following one of at least:
If the described single data processing time that described second data processing amount is corresponding is the shortest, then using the quantitative value between described first data processing amount and described 3rd data processing amount as upgrade after scope;
If the described single data processing time that described first data processing amount is corresponding is the shortest, then using the quantitative value between described 4th data processing amount and described second data processing amount as the scope after described renewal;
If the described single data processing time that described 3rd data processing amount is corresponding is the shortest, then using the quantitative value between described second data processing amount and described 5th data processing amount as the scope after described renewal.
13. devices according to claim 10, is characterized in that, described first selects module to realize selecting single data processing amount to comprise according to the first predetermined condition in described pending data by following steps:
According to pre-determined number single data processing amount described in Stochastic choice from described scope, wherein, described pre-determined number is more than or equal to the second predetermined threshold.
14. devices according to claim 8, is characterized in that, also comprise:
Set up unit, for before the pending data of described acquisition, set up the running environment with described pending data match.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410766560.8A CN104504020B (en) | 2014-12-11 | 2014-12-11 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410766560.8A CN104504020B (en) | 2014-12-11 | 2014-12-11 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104504020A true CN104504020A (en) | 2015-04-08 |
| CN104504020B CN104504020B (en) | 2018-02-23 |
Family
ID=52945418
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410766560.8A Active CN104504020B (en) | 2014-12-11 | 2014-12-11 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104504020B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0420419A1 (en) * | 1989-09-20 | 1991-04-03 | Hitachi, Ltd. | Method and apparatus for on-line processing of transaction data |
| CN103618716A (en) * | 2013-11-28 | 2014-03-05 | 福建星网锐捷网络有限公司 | Conversation interaction method, equipment and system of terminal WAN management protocol |
| CN104102646A (en) * | 2013-04-07 | 2014-10-15 | 腾讯科技（深圳）有限公司 | Method, device and system for processing data |
-
2014
- 2014-12-11 CN CN201410766560.8A patent/CN104504020B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0420419A1 (en) * | 1989-09-20 | 1991-04-03 | Hitachi, Ltd. | Method and apparatus for on-line processing of transaction data |
| CN104102646A (en) * | 2013-04-07 | 2014-10-15 | 腾讯科技（深圳）有限公司 | Method, device and system for processing data |
| CN103618716A (en) * | 2013-11-28 | 2014-03-05 | 福建星网锐捷网络有限公司 | Conversation interaction method, equipment and system of terminal WAN management protocol |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104504020B (en) | 2018-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11325780B2 (en) | Method and device for sorting cargo | |
| CN107301178B (en) | Data query processing method, device and system | |
| CN102446171B (en) | The method and apparatus of keyword quality score is promoted based on the evaluation and test of weighted mean click-through rate | |
| CN109117141B (en) | Method, device, electronic equipment and computer readable storage medium for simplifying programming | |
| CN105224458A (en) | A kind of database method of testing and system | |
| EP2985730A1 (en) | Method and device for partially-upgrading | |
| CN110262878A (en) | Timed task processing method, device, equipment and computer readable storage medium | |
| CN110851987B (en) | Method, apparatus and storage medium for predicting calculated duration based on acceleration ratio | |
| CN114095567A (en) | Data access request processing method and device, computer equipment and medium | |
| CN108023905B (en) | Internet of things application system and method | |
| CN106648839A (en) | Method and device for processing data | |
| CN104484413A (en) | Method and device for obtaining searching results | |
| CN115756812A (en) | Resource adjusting method and device and storage medium | |
| CN107679107B (en) | Graph database-based power grid equipment reachability query method and system | |
| CN106528551A (en) | Memory application method and apparatus | |
| CN106126670B (en) | Operation data sorting processing method and device | |
| CN112333246A (en) | ABtest experiment method and device, intelligent terminal and storage medium | |
| CN104504020A (en) | Data processing method and device | |
| CN113709099B (en) | Mixed cloud firewall rule issuing method, device, equipment and storage medium | |
| CN105487925A (en) | Data scanning method and device | |
| CN112131179B (en) | Task state detection method, device, computer equipment and storage medium | |
| CN107066247B (en) | Patch query method and device | |
| CN113361877B (en) | Statistical analysis method, device, equipment and medium for power grid frequency data | |
| CN103605740B (en) | Data import treating method and apparatus | |
| CN111105059B (en) | Attribute conflict discovery method, device and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Improved data with video transmitter Effective date of registration: 20190531 Granted publication date: 20180223 Pledgee: Shenzhen Black Horse World Investment Consulting Co., Ltd. Pledgor: Beijing Guoshuang Technology Co.,Ltd. Registration number: 2019990000503 |
|
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: 100083 No. 401, 4th Floor, Haitai Building, 229 North Fourth Ring Road, Haidian District, Beijing Patentee after: Beijing Guoshuang Technology Co.,Ltd. Address before: 100086 Beijing city Haidian District Shuangyushu Area No. 76 Zhichun Road cuigongfandian 8 layer A Patentee before: Beijing Guoshuang Technology Co.,Ltd. |