CN110232092A - A kind of asynchronous solution of batch data based on data processing and relevant device - Google Patents
A kind of asynchronous solution of batch data based on data processing and relevant device Download PDFInfo
- Publication number
- CN110232092A CN110232092A CN201910342283.0A CN201910342283A CN110232092A CN 110232092 A CN110232092 A CN 110232092A CN 201910342283 A CN201910342283 A CN 201910342283A CN 110232092 A CN110232092 A CN 110232092A
- Authority
- CN
- China
- Prior art keywords
- task
- downloading
- download
- tasks
- pool
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 44
- 238000000034 method Methods 0.000 claims abstract description 42
- 238000004364 calculation method Methods 0.000 claims description 25
- 238000012216 screening Methods 0.000 claims description 20
- 230000006870 function Effects 0.000 description 7
- 238000013461 design Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/103—Workflow collaboration or project management
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Strategic Management (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Entrepreneurship & Innovation (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Tourism & Hospitality (AREA)
- Quality & Reliability (AREA)
- General Business, Economics & Management (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Computing Systems (AREA)
- Economics (AREA)
- General Engineering & Computer Science (AREA)
- Information Transfer Between Computers (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention relates to the data processing techniques of big data field, and in particular to a kind of asynchronous solution of batch data based on data processing and relevant device, which comprises requests downloading to generate downloading task；It calculates the sequence weight of multiple downloading tasks and arranges task order；Downloading task is executed according to task order, generates assignment file, notify corresponding business personnel and the assignment file being successfully generated is sent to corresponding business personnel.The above method improves the efficiency of task processing, while being directed to multiple tasks asynchronous process, allows to multiple tasks while handling, reduce the time of task processing by obtaining the weight of different task so that task processes are simplified, procedure.
Description
Technical Field
The invention relates to the field of big data, in particular to a batch data asynchronous solution method based on data processing and related equipment.
Background
Along with the continuous promotion of internet technology, the quantity of internet users is also expanding constantly, the data on the internet demonstrates explosive growth by the growth of large-scale user quantity, in order to deal with huge internet data, some problems can be inevitably produced in the processing and conversion process of tasks, especially, the number of newly-added call records of a sales system is huge day by day, the process of acquiring data is tedious, the data extraction logics and formats under different service scenes are different, meanwhile, the time consumed by the process of extracting data is long, the derivation process is complex, and the requirements of different users cannot be met in time.
The existing internet data processing mainly adopts a synchronous processing mode, the condition of small data volume can be normally carried out, but when the data batch is large, the system is easy to crash and cannot normally run, and the situations of repeated calculation, data loss and the like are easy to happen.
Disclosure of Invention
Therefore, it is necessary to provide a batch data asynchronous solution method based on data processing and related devices, aiming at the problem that when batch data is synchronously processed, repeated calculation, data loss and other situations are easily caused.
A batch data asynchronous solution method based on data processing comprises the following steps:
acquiring a downloading request of a service worker, generating a downloading task according to the downloading request, and adding the downloading task into a task downloading pool;
calculating the sequence weights of a plurality of downloading tasks in the task downloading pool, and sequentially arranging the task sequences of the downloading tasks from large to small according to the sequence weights;
executing the downloading tasks according to the task sequence, obtaining task data corresponding to the downloading tasks, adding the downloading tasks which are successfully executed into a file task pool to be generated, and adding the downloading tasks which are failed to be executed into a task completion notification pool;
calculating generation weights of a plurality of downloading tasks in the to-be-generated file task pool, sequentially generating corresponding task data into task files according to the generation weights from large to small, and adding the downloading tasks in the task files into the task completion notification pool;
and calculating notification weights of a plurality of downloading tasks in the task completion notification pool, traversing the task completion notification pool at regular time according to the notification weights, notifying corresponding business personnel of the completed downloading tasks, sending the successfully generated task files to the corresponding business personnel, and removing the notified downloading tasks from the task completion notification pool.
In one possible design, the obtaining the download request of the service person includes:
the method comprises the steps of obtaining an access request of a service worker, presenting a data downloading page to the service worker, and obtaining the downloading request submitted by the service worker through the data downloading page, wherein the downloading request comprises a data screening condition, a statistical dimension and a priority.
In one possible design, the generating a download task according to the download request, and adding the download task to a task download pool includes:
acquiring a position registration form of a human resource system, acquiring positions of the business personnel according to the position registration form, and determining a role of a submitter of the download request;
reading the downloading request, and defining the task type of the downloading request according to the screening condition and the statistical dimension in the downloading request;
and generating a downloading task by the role of the submitter, the data screening condition, the statistical dimension and the task type, adding the downloading task into a task downloading pool, and marking the priority corresponding to the data screening condition and the statistical dimension in the downloading task.
In a possible design, the calculating a sequence weight of the plurality of download tasks in the task download pool, and sequentially ranking the task sequences of the download tasks according to the sequence weight from large to small includes:
acquiring the time of the downloading task entering the downloading task pool, and counting the queuing time of the downloading task;
calculating the sequence weight of the download tasks according to the priority, the queuing time, the role of the submitter and the task type of the download tasks;
and arranging the task sequence of the download tasks from large to small according to the sequence weight of the download tasks in the download task pool.
In one possible design, the calculating the sequence weight of the download task according to the priority, the queuing time, the role of the submitter and the task type of the download task includes:
presetting the priority of the download task as N levels, wherein N is an integer, acquiring the priority of the download task, and recording the level corresponding to the priority as a numerical value N;
calculating the queuing time of the downloading task in minutes, recording the queuing time as a numerical value m, wherein m is an integer, and discarding the downloading task in less than 1 minute;
setting the role of the submitter as Z levels, wherein Z is an integer, acquiring the role of the submitter of the download task, and recording the level corresponding to the role of the submitter as a numerical value Z;
calculating the sequence weight of the ith downloading task, wherein the calculation formula of the sequence weight is as follows:
Ai＝f1ni+f2mi+f3zi+f4ki
f1+f2+f3+f4＝1
wherein ,AiFor the ith said download task order weight, f1As priority coefficient, f2For queuing time factor, f3To submit the human character coefficients, f4Is a task type coefficient, niIs the priority value, m, of the ith download taskiIs the queuing time value, z, of the ith download taskiA presenter role value, k, for the ith said download taskiThe task type value of the ith downloading task is obtained.
In a possible design, the calculating generation weights of a plurality of download tasks in the task pool of the file to be generated, sequentially generating corresponding task data into task files according to the generation weights from large to small, and adding the download tasks in the task files into the task completion notification pool includes:
according to a calculation method of the sequence weight, calculating the generation weight of a download task in a task pool of a file to be generated;
and carrying out format standardization processing on the corresponding task data according to the generated weight from large to small, writing the task data with unified format into a preset task data summary table to generate a task file, and adding a downloading task in the task file into a task completion notification pool.
In one possible design, the calculating notification weights of a plurality of download tasks in the task completion notification pool, traversing the task completion notification pool at regular time according to the notification weights, notifying corresponding business personnel of the completed download tasks, sending the successfully generated task files to the corresponding business personnel, and removing the notified download tasks from the task completion notification pool includes:
according to the calculation method of the sequence weight, calculating the notification weight of the download task in the task completion notification pool;
and traversing the task notification pool at regular time, sequentially notifying corresponding business personnel of the completed downloading task according to the notification weight from large to small, sending the successfully generated task file to the corresponding business personnel, and removing the notified downloading task from the task completion notification pool.
Based on the same technical concept, the invention also provides a batch data asynchronous solving device based on data processing, which comprises:
the download task generation module is set to acquire a download request of a service worker, generate a download task according to the download request and add the download task into a task download pool;
the sequence weight calculation module is used for calculating the sequence weights of the plurality of downloading tasks in the task downloading pool and arranging the task sequences of the downloading tasks in sequence from large to small according to the sequence weights;
the download task execution module is set to execute the download tasks according to the task sequence, acquire the corresponding download tasks, add the successfully executed download tasks into the to-be-generated file task pool, and add the download tasks which are failed to be executed into the task completion notification pool;
the task file generation module is used for calculating the generation weights of a plurality of downloading tasks in the to-be-generated file task pool, sequentially generating corresponding task data into task files from large to small according to the generation weights, and adding the downloading tasks in the task files into the task completion notification pool;
and the task completion notification module is used for calculating notification weights of a plurality of downloading tasks in the task completion notification pool, regularly traversing the task completion notification pool according to the notification weights, notifying corresponding business personnel of the completed downloading tasks, sending the successfully generated task files to the corresponding business personnel, and removing the notified downloading tasks from the task completion notification pool.
Based on the same concept, the present application proposes a computer device, comprising a memory and a processor, wherein the memory stores computer readable instructions, and the computer readable instructions, when executed by one or more of the processors, cause one or more of the processors to execute the steps of the batch data asynchronous solution based on data processing.
Based on the same concept, the present application proposes a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the above-mentioned batch data asynchronous solution based on data processing.
According to the batch data asynchronous solution method based on data processing and the related equipment, a downloading task is generated according to a downloading request of a service worker by obtaining the downloading request, and the downloading task is added into a task downloading pool; calculating the sequence weights of a plurality of downloading tasks in the task downloading pool, and sequentially arranging the task sequences of the downloading tasks from large to small according to the sequence weights; executing the downloading tasks according to the task sequence, obtaining task data corresponding to the downloading tasks, adding the downloading tasks which are successfully executed into a file task pool to be generated, and adding the downloading tasks which are failed to be executed into a task completion notification pool; calculating generation weights of a plurality of downloading tasks in the to-be-generated file task pool, sequentially generating corresponding task data into task files according to the generation weights from large to small, and adding the downloading tasks in the task files into the task completion notification pool; and calculating notification weights of a plurality of downloading tasks in the task completion notification pool, traversing the task completion notification pool at regular time according to the notification weights, notifying corresponding business personnel of the completed downloading tasks, sending the successfully generated task files to the corresponding business personnel, and removing the notified downloading tasks from the task completion notification pool. The invention sets the export and download of data into several independent modules, which is convenient for later maintenance; by acquiring the weights of different tasks, the task processing process is simplified and streamlined, and the task processing efficiency is improved; the asynchronous processing of the tasks enables a plurality of tasks to be processed simultaneously, and the time for processing the tasks is reduced.
Drawings
FIG. 1 is a flow chart of a batch data asynchronous solution based on data processing according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an apparatus for asynchronous resolution of batch data based on data processing according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Fig. 1 is a flowchart of a batch data asynchronous solution method based on data processing according to an embodiment of the present invention, and as shown in fig. 1, the batch data asynchronous solution method based on data processing includes the following steps:
step S1, download task generation: the method comprises the steps of obtaining a downloading request of a service worker, generating a downloading task according to the downloading request, and adding the downloading task into a task downloading pool.
The step generates a downloading task according to the downloading request of the service personnel by interacting with the service personnel, and provides a basis for executing the downloading task.
In one embodiment, step S1 includes the following specific steps:
step S101, acquiring a downloading request: the method comprises the steps of obtaining an access request of a service worker, presenting a data downloading page to the service worker, and obtaining the downloading request submitted by the service worker through the data downloading page, wherein the downloading request comprises a data screening condition, a statistical dimension and a priority.
In this step, the data screening condition refers to a manner and a process of acquiring data, specifically, as a data calculation function, the statistical dimension refers to an attribute of the data, specifically, the statistical dimension of monthly sales is sales data, the statistical dimension of monthly purchase plans is purchase data, the statistical dimension of annual tax amount is financial data, the priority refers to a priority download level preset by different tasks, preferably, 5 levels in total are 1-5, the download task value of the urgent class is set to 4, the download task value of the normal class is set to 3, the download task value of the adjustable class is set to 2, and the download task value of the irrelevant class is set to 1.
In the step, the requirements of business personnel are acquired through the data downloading page, basic data preparation is prepared for generating the downloading task, and the reality and the effectiveness of the process of generating the downloading task according to information are guaranteed.
For example: a sales department supervisor A of a certain company wants to know the task completion condition of a sales group A, B of a month department, the sales group A is a main product sales group and comprises a sales group leader B, a market specialist C and a hijack thereof, wherein the sales group leader B is a sales department sub supervisor, the sales group B is an edge product sales group and comprises a sales group leader E and a market specialist, the sales group leader A informs all department members of the week case to report the month task progress of the main product in detail and report the month task progress of the edge product roughly, after returning to respective offices, the sales group B, the company C, the company D, the company E and the company D submit corresponding download requests through data download pages, the download requests comprise screening conditions which are summation functions, and the statistical dimension is month sales data, wherein the permissions of the download requests in the Enterprise Resource Planning (ERP) are the data statistical permissions of the sales tasks completed by the company A, because the downloading requests of the second, the third, the fourth, the fifth and the sixth are all the requirements of the next week, but the next instance mainly reports the task progress of the main product, the tasks of the second, the third and the sixth are all in an emergency state, the edge product only reports roughly, the task of the fifth and the sixth is in a common state, and if the priority of the downloading task is distinguished according to the preferred 5 levels, the priority of the second, the third and the sixth is 4, and the priority of the fifth and the sixth is 3.
Step S102, the role of the submitter is obtained: acquiring a position registration form of a human resource system, acquiring positions of the business personnel according to the position registration form, and determining a role of a submitter of the download request;
in this step, the role of the submitter is set according to the job registration form, and specifically, 4 levels of 1-4 are preferred, the role of the submitter in the common staff category is 1, the role of the submitter in the team leader or team leader category is 2, the role of the submitter in the data support category or project manager category is 3, and the role of the submitter in the administrative or operation management category is 4.
The method distinguishes the roles of the submitters of different business personnel according to the job registration form, prevents the task weight from being indistinguishable when a plurality of tasks are carried out simultaneously, and ensures the orderly carrying out of the tasks.
For example: assuming that the submitter roles are distinguished according to the preferred 4 levels, in step S101, the respective roles of b, c, d, e, and z are different, the submitter roles are also different, b is a sales group leader, and is also a department assistant, and belongs to a project manager class, the submitter role is 3, ng is a sales group leader, and belongs to a team group leader class, the submitter role is 2, and c is a market specialist but has a higer, so c also belongs to a team group leader class, the submitter role is 2, d is a market specialist, and belongs to a general staff class, the submitter role is 1.
Step S103, task type definition: reading the downloading request, and defining the task type of the downloading request according to the screening condition and the statistical dimension in the downloading request.
The task types in this step are set according to the download request, and particularly, 2 task types of 1-2 are preferred, the detail report value is 1, and the statistic report value is 2.
The task types of a plurality of tasks are defined through the task downloading request, the weights of the tasks of different task types are effectively distinguished, and the situation that the downloading weights cannot be distinguished among different departments to influence the execution of the downloading tasks is prevented.
For example: assuming that the requirements are set according to the preferred task types, the task types of b, c, d, e, and z in step S102 are all statistical types, and are all marked as 2.
Step S104, generating a final task: and generating a downloading task by the role of the submitter, the data screening condition, the statistical dimension and the task type, adding the downloading task into a task downloading pool, and marking the priority corresponding to the data screening condition and the statistical dimension in the downloading task.
In the step, the downloading task is generated by the role of the submitter, the data screening condition, the statistical dimension and the task type which are obtained in the steps from S101 to S103 and is used as a sequential weight calculation basis of the subsequent downloading task, so that a basis is provided for reasonably controlling the downloading priority of the downloading task.
For example: assuming that all of b, c, d, e and z in step S103 have successfully submitted a download request, 5 download tasks are generated, where the download task of b includes that the role of the submitter is 3, the data screening condition is a summation function, the statistical dimension is sales data, the task type is 2, and the priority label is 4; the downloading task of the third step comprises that the role of a submitter is 2, the data screening condition is a summation function, the statistical dimension is sales data, the task type is 2, and the priority mark is 4; the downloading task of the user comprises that the role of a submitter is 1, the data screening condition is a summation function, the statistical dimension is sales data, the task type is 2, and the priority mark is 4; the downloading task of the E-business comprises that the role of a submitter is 2, the data screening condition is a summation function, the statistical dimension is sales data, the task type is 2, and the priority mark is 3; the downloaded task comprises that the role of the submitter is 1, the data screening condition is a summation function, the statistical dimension is sales data, the task type is 2, and the priority is marked as 3.
In this embodiment, a process of generating the download task is expanded, a source of the download task is described in detail, and a detailed source basis is provided for task execution.
Step S2, sequence weight calculation: and calculating the sequence weight of the plurality of downloading tasks in the task downloading pool, and sequentially arranging the task sequence of the downloading tasks according to the sequence weight from large to small.
In the step, the weights of different downloading tasks are distinguished through calculation, the execution sequence of the downloading tasks is reasonably planned, the effective proceeding of the tasks is ensured, and the resource preemption condition is avoided.
In one embodiment, step S2 includes:
step S201, obtaining the time of the downloading task entering the downloading task pool, and counting the queuing time of the downloading task.
In the step, in order to avoid that the resource of a download task pool is occupied due to long-time waiting of part of download tasks in the download task pool to influence the data timeliness, therefore, the queuing time needs to be considered in the process of calculating the task weight of the download tasks, the queuing time is obtained by obtaining the time when the download tasks enter the download task pool and then calculating the time difference with the current time, the queuing time of the download tasks is calculated in minutes and is recorded as a value m, the m is an integer, the discard of less than 1 minute is carried out, if the value of the queuing time is 5 minutes, the value of the queuing time is 4 minutes and 05 seconds is set as 4.
For example: assuming that the download tasks of b, c, d, e, and z in step S1 have been in the download task pool for 5 min 20S, 3 min 05S, 4 min 06S, 8 min 35S, and 9 min 09S, respectively, the queue durations of the download tasks of b, c, d, e, and z are 5, 3, 4, 8, and 9, respectively.
Step S202, calculating the sequence weight of the download task according to the priority, the queuing time, the role of the submitter and the task type of the download task.
The step effectively obtains the execution sequence of the downloading task by calculating the weight, so that the task processing process is simplified and streamlined, and the task processing efficiency is improved.
The specific process of weight calculation in step S202 is as follows:
presetting the priority of the download task as N levels, wherein N is an integer, acquiring the priority of the download task, and recording the level corresponding to the priority as a numerical value N; calculating the queuing time of the downloading task in minutes, recording the queuing time as a numerical value m, wherein m is an integer, and discarding the downloading task in less than 1 minute; setting the role of the submitter as Z levels, wherein Z is an integer, acquiring the role of the submitter of the download task, and recording the level corresponding to the role of the submitter as a numerical value Z; setting the download task as K task types, wherein K is an integer, acquiring the task type of the download task, and recording the grade corresponding to the task type as a numerical value K; calculating the sequence weight of the ith downloading task, wherein the calculation formula of the sequence weight is as follows:
Ai＝f1ni+f2mi+f3zi+f4ki
f1+f2+f3+f4＝1
wherein ,AiIs the ith oneSequence weight of download task, f1As priority coefficient, f2For queuing time factor, f3To submit the human character coefficients, f4Is a task type coefficient, niIs the priority value, m, of the ith download taskiIs the queuing time value, z, of the ith download taskiA presenter role value, k, for the ith said download taskiThe task type value of the ith downloading task is obtained.
For example: case 1, assume that the download tasks of b, c, d, e, and z in step S201 are task 1, task 2, task 3, task 4, and task 5, respectively, and assume that the priority coefficient f is1Is 0.3, queuing duration factor f2At 0.05, the presenter character coefficient f3Is 0.4, the task type coefficient f40.25, the sequence weight of each download task is:
A1＝0.3×4+0.05×5+0.4×3+0.25×2＝3.15
A2＝0.3×4+0.05×3+0.4×2+0.25×2＝2.65
A3＝0.3×4+0.05×4+0.4×1+0.25×2＝2.30
A4＝0.3×3+0.05×8+0.4×2+0.25×2＝2.60
A5＝0.3×3+0.05×9+0.4×1+0.25×2＝2.25
case 2, assuming that a project manager of a certain company wants to know the fund budget use condition of a certain project in the current month in 11/20/month, a financial statement of the company needs to be checked, where the financial statement corresponds to a download task a, a currently preset priority coefficient is 0.3, a queuing time coefficient is 0.05, a role coefficient of a submitter is 0.4, a task type coefficient is 0.25, the download task a in a download task pool is a general-type download task, a corresponding priority value is 3, a task type is a statistical-type report, a corresponding task type value is 2, the download task has been submitted for 2 minutes and 05 seconds, a corresponding queuing time value is 2, the submitter role is a project manager, and a corresponding submitter role value is 3, and then the sequence weight of the download tasks is:
A＝0.3×3+0.05×2+0.4×3+0.25×2＝2.7
step S203, arranging the task sequence of the download tasks according to the sequence weight of the download tasks in the download task pool from big to small.
In the step, the download tasks are sequenced to obtain the context of executing the plurality of download tasks, so that the tasks are ensured to be executed efficiently and orderly, and the resource preemption condition is avoided.
For example: case 1 of step S2021>A2>A4>A3>A5Then, the execution sequence of the download task is task 1, task 2, task 4, task 3, and task 5.
In this embodiment, the step of weight calculation is described in detail through the calculation process of decomposition order weight, so that a theoretical support is provided for the weight calculation of the download task, and a basis for weight calculation is provided.
Step S3, the download task executes: and executing the downloading tasks according to the task sequence, obtaining task data corresponding to the downloading tasks, adding the downloading tasks which are successfully executed into a task pool of the file to be generated, and adding the downloading tasks which are unsuccessfully executed into a task completion notification pool.
Because the downloading task amount in the downloading task pool is huge, and the downloading tasks need to be executed in sequence in the execution process, a file needs to be generated in the file to be generated after the downloading task is executed successfully, and the downloading task which is executed in failure also needs to be sequentially waited for notifying corresponding service personnel in the task completion notification pool.
In the step, the executed downloading task is removed from the downloading task pool, the downloading task which is successfully executed is added into the generated file task pool, the generated file is waited, the failed task is added into the task completion notification pool, and the resource occupation is avoided.
Step S4, generating a task file: and calculating the generation weights of the plurality of downloading tasks in the task pool of the file to be generated, sequentially generating corresponding task data into task files according to the generation weights from large to small, and adding the downloading tasks in the task files into the task completion notification pool.
In the step, the task file is generated, so that the final purpose of the downloading task is finished, and the successfully executed downloading task can smoothly meet the data requirement of business personnel.
In one embodiment, step S4 includes:
according to a calculation method of the sequence weight, calculating the generation weight of a download task in a task pool of a file to be generated; and carrying out format standardization processing on the corresponding task data according to the generated weight from large to small, writing the task data with unified format into a preset task data summary table to generate a task file, and adding a downloading task in the task file into a task completion notification pool.
In this embodiment, the importance of the download task is re-evaluated by calculating and generating a weight, and a task file is generated according to the size of the importance, where the task file includes a text file, a table file, a picture file, and the like, and is specifically written into a preset table or a text file after the download task is executed by the system, and the specific format is, for example, a format such as word, Excel, and the like.
In the embodiment, the generating weight is obtained by using a similar calculation method as that in step S2, but the queuing time is different from that in step S2, and the queuing time in this step is the time difference between the time when the downloading task enters the generated file task pool and the current time.
Step S5, task completion notification: and calculating notification weights of a plurality of downloading tasks in the task completion notification pool, traversing the task completion notification pool at regular time according to the notification weights, notifying corresponding business personnel of the completed downloading tasks, sending the successfully generated task files to the corresponding business personnel, and removing the notified downloading tasks from the task completion notification pool.
In the step, the corresponding service personnel are informed of the completed downloading task, so that the service personnel can know the progress of the task in time, and the condition that the checking time is missed and the working progress is influenced is avoided.
In one embodiment, step S5 includes
According to the calculation method of the sequence weight, calculating the notification weight of the download task in the task completion notification pool; and traversing the task notification pool at regular time, sequentially notifying corresponding business personnel of the completed downloading task according to the notification weight from large to small, sending the successfully generated task file to the corresponding business personnel, and removing the notified downloading task from the task completion notification pool.
In this embodiment, the corresponding service person is notified of the completed download task at regular time, and the specific timing time is preferably 5 minutes, the task completion notification pool is traversed once, the notification manner includes at least one of a mail notification, a short message notification, and a system message notification, the service person can freely select the notification manner when requesting to download, the notification manner corresponding to the service person is obtained in the obtained download request, and the task notification pool simultaneously sends the generated file to the service person through the notification manner selected by the service person.
The present embodiment obtains the notification weight by using a similar calculation method as in step S2, but the queuing time is different from the queuing time in step S2, and the queuing time in this step is the time difference between the time when the download task enters the task completion notification pool and the current time.
According to the embodiment of the invention, the weights of different tasks are obtained, so that the task processing process is simplified and streamlined, the task processing efficiency is improved, and meanwhile, the multiple tasks are processed asynchronously, so that the multiple tasks are processed asynchronously, and the waiting and processing time of the multiple tasks is reasonably and comprehensively planned.
In one embodiment, a batch data asynchronous solution device based on data processing is provided, as shown in fig. 2, which includes:
the download task generation module is set to acquire a download request of a service worker, generate a download task according to the download request and add the download task into a task download pool;
the sequence weight calculation module is used for calculating the sequence weights of the plurality of downloading tasks in the task downloading pool and arranging the task sequences of the downloading tasks in sequence from large to small according to the sequence weights;
the download task execution module is set to execute the download tasks according to the task sequence, obtain task data corresponding to the download tasks, add the download tasks which are successfully executed into a to-be-generated file task pool, and add the download tasks which are failed to be executed into a task completion notification pool;
the task file generation module is used for calculating the generation weights of a plurality of downloading tasks in the to-be-generated file task pool, sequentially generating corresponding task data into task files from large to small according to the generation weights, and adding the downloading tasks in the task files into the task completion notification pool;
and the task completion notification module is used for calculating notification weights of a plurality of downloading tasks in the task completion notification pool, regularly traversing the task completion notification pool according to the notification weights, notifying corresponding business personnel of the completed downloading tasks, sending the successfully generated task files to the corresponding business personnel, and removing the notified downloading tasks from the task completion notification pool.
In one embodiment, a computer device is provided, which includes a memory and a processor, the memory stores computer readable instructions, and the computer readable instructions, when executed by the processor, cause the processor to implement the steps of a batch data asynchronous solution based on data processing in the above embodiments.
In one embodiment, a storage medium storing computer-readable instructions is provided, and the computer-readable instructions, when executed by one or more processors, cause the one or more processors to perform the steps of a batch data asynchronous solution based on data processing in the above embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A batch data asynchronous solution method based on data processing is characterized by comprising the following steps:
acquiring a downloading request of a service worker, generating a downloading task according to the downloading request, and adding the downloading task into a task downloading pool;
calculating the sequence weights of a plurality of downloading tasks in the task downloading pool, and sequentially arranging the task sequences of the downloading tasks from large to small according to the sequence weights;
executing the downloading tasks according to the task sequence, obtaining task data corresponding to the downloading tasks, adding the downloading tasks which are successfully executed into a file task pool to be generated, and adding the downloading tasks which are failed to be executed into a task completion notification pool;
calculating generation weights of a plurality of downloading tasks in the to-be-generated file task pool, sequentially generating corresponding task data into task files according to the generation weights from large to small, and adding the downloading tasks in the task files into the task completion notification pool;
and calculating notification weights of a plurality of downloading tasks in the task completion notification pool, traversing the task completion notification pool at regular time according to the notification weights, notifying corresponding business personnel of the completed downloading tasks, sending the successfully generated task files to the corresponding business personnel, and removing the notified downloading tasks from the task completion notification pool.
2. The asynchronous solution for batch data based on data processing as claimed in claim 1, wherein said obtaining the downloading request of the service person comprises:
the method comprises the steps of obtaining an access request of a service worker, presenting a data downloading page to the service worker, and obtaining the downloading request submitted by the service worker through the data downloading page, wherein the downloading request comprises a data screening condition, a statistical dimension and a priority.
3. The asynchronous solution for batch data based on data processing according to claim 2, wherein said generating a download task according to the download request, and adding the download task to a task download pool comprises:
acquiring a position registration form of a human resource system, acquiring positions of the business personnel according to the position registration form, and determining a role of a submitter of the download request;
reading the downloading request, and defining the task type of the downloading request according to the screening condition and the statistical dimension in the downloading request;
and generating a downloading task by the role of the submitter, the data screening condition, the statistical dimension and the task type, adding the downloading task into a task downloading pool, and marking the priority corresponding to the data screening condition and the statistical dimension in the downloading task.
4. The asynchronous solution method for batch data based on data processing as claimed in claim 3, wherein said calculating the sequence weight of a plurality of said download tasks in said task download pool, and arranging the task sequence of said download tasks in order from big to small according to said sequence weight comprises:
acquiring the time of the downloading task entering the downloading task pool, and counting the queuing time of the downloading task;
calculating the sequence weight of the download tasks according to the priority, the queuing time, the role of the submitter and the task type of the download tasks;
and arranging the task sequence of the download tasks from large to small according to the sequence weight of the download tasks in the download task pool.
5. The asynchronous bulk data solution method as claimed in claim 4, wherein the step of calculating the sequential weight of the download task according to the priority, queuing duration, role of submitter and task type of the download task comprises:
presetting the priority of the download task as N levels, wherein N is an integer, acquiring the priority of the download task, and recording the level corresponding to the priority as a numerical value N;
calculating the queuing time of the downloading task in minutes, recording the queuing time as a numerical value m, wherein m is an integer, and discarding the downloading task in less than 1 minute;
setting the role of the submitter as Z levels, wherein Z is an integer, acquiring the role of the submitter of the download task, and recording the level corresponding to the role of the submitter as a numerical value Z;
setting the download task as K task types, wherein K is an integer, acquiring the task type of the download task, and recording the grade corresponding to the task type as a numerical value K;
calculating the sequence weight of the ith downloading task, wherein the calculation formula of the sequence weight is as follows:
Ai＝f1ni+f2mi+f3zi+f4ki
f1+f2+f3+f4＝1
wherein ,AiFor the ith said download task order weight, f1As priority coefficient, f2For queuing time factor, f3To submit the human character coefficients, f4Is a task type coefficient, niIs the priority value, m, of the ith download taskiIs the queuing time value, z, of the ith download taskiA presenter role value, k, for the ith said download taskiThe task type value of the ith downloading task is obtained.
6. The batch data asynchronous solution method based on data processing according to claim 1, wherein the calculating of the generation weights of the plurality of download tasks in the to-be-generated file task pool, sequentially generating the corresponding task data into task files according to the generation weights from large to small, and adding the download tasks in the task files into the task completion notification pool comprises:
according to a calculation method of the sequence weight, calculating the generation weight of a download task in a task pool of a file to be generated;
and carrying out format standardization processing on the corresponding task data according to the generated weight from large to small, writing the task data with unified format into a preset task data summary table to generate a task file, and adding a downloading task in the task file into a task completion notification pool.
7. The asynchronous batch data solution method based on data processing as claimed in claim 1, wherein the calculating of the notification weight of the plurality of download tasks in the task completion notification pool, the traversing of the task completion notification pool at regular time according to the notification weight, the notifying of the corresponding service personnel of the completed download task and the sending of the successfully generated task file to the corresponding service personnel, and the removing of the notified download task from the task completion notification pool comprises:
according to the calculation method of the sequence weight, calculating the notification weight of the download task in the task completion notification pool;
and traversing the task notification pool at regular time, sequentially notifying corresponding business personnel of the completed downloading task according to the notification weight from large to small, sending the successfully generated task file to the corresponding business personnel, and removing the notified downloading task from the task completion notification pool.
8. A batch data asynchronous solution device based on data processing is characterized by comprising:
the download task generation module is set to acquire a download request of a service worker, generate a download task according to the download request and add the download task into a task download pool;
the sequence weight calculation module is used for calculating the sequence weights of the plurality of downloading tasks in the task downloading pool and arranging the task sequences of the downloading tasks in sequence from large to small according to the sequence weights;
the download task execution module is set to execute the download tasks according to the task sequence, obtain task data corresponding to the download tasks, add the download tasks which are successfully executed into a to-be-generated file task pool, and add the download tasks which are failed to be executed into a task completion notification pool;
the task file generation module is used for calculating the generation weights of a plurality of downloading tasks in the to-be-generated file task pool, sequentially generating corresponding task data into task files from large to small according to the generation weights, and adding the downloading tasks in the task files into the task completion notification pool;
and the task completion notification module is used for calculating notification weights of a plurality of downloading tasks in the task completion notification pool, regularly traversing the task completion notification pool according to the notification weights, notifying corresponding business personnel of the completed downloading tasks, sending the successfully generated task files to the corresponding business personnel, and removing the notified downloading tasks from the task completion notification pool.
9. A computer device comprising a memory and a processor, the memory having stored therein computer-readable instructions, which when executed by one or more of the processors, cause the one or more processors to perform the steps of a data processing-based asynchronous bulk data resolution method as claimed in any one of claims 1 to 7.
10. A storage medium readable and writable by a processor, the storage medium storing computer-readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of a data processing-based asynchronous bulk data resolution method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910342283.0A CN110232092B (en) | 2019-04-26 | 2019-04-26 | Batch data asynchronous solving method based on data processing and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910342283.0A CN110232092B (en) | 2019-04-26 | 2019-04-26 | Batch data asynchronous solving method based on data processing and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110232092A true CN110232092A (en) | 2019-09-13 |
| CN110232092B CN110232092B (en) | 2023-08-01 |
Family
ID=67860400
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910342283.0A Active CN110232092B (en) | 2019-04-26 | 2019-04-26 | Batch data asynchronous solving method based on data processing and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110232092B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113535369A (en) * | 2021-09-15 | 2021-10-22 | 中建电子商务有限责任公司 | Method and system for scheduling tasks |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105389209A (en) * | 2015-12-25 | 2016-03-09 | 中国建设银行股份有限公司 | Asynchronous batch task processing method and system |
| CN105992043A (en) * | 2015-10-15 | 2016-10-05 | 乐视致新电子科技（天津）有限公司 | Data processing method and device |
| US9824094B1 (en) * | 2014-04-24 | 2017-11-21 | Google Inc. | Systems and methods for prioritizing file downloads |
| CN107977444A (en) * | 2017-12-11 | 2018-05-01 | 成都博睿德科技有限公司 | Mass data method for parallel processing based on big data |
-
2019
- 2019-04-26 CN CN201910342283.0A patent/CN110232092B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9824094B1 (en) * | 2014-04-24 | 2017-11-21 | Google Inc. | Systems and methods for prioritizing file downloads |
| CN105992043A (en) * | 2015-10-15 | 2016-10-05 | 乐视致新电子科技（天津）有限公司 | Data processing method and device |
| CN105389209A (en) * | 2015-12-25 | 2016-03-09 | 中国建设银行股份有限公司 | Asynchronous batch task processing method and system |
| CN107977444A (en) * | 2017-12-11 | 2018-05-01 | 成都博睿德科技有限公司 | Mass data method for parallel processing based on big data |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113535369A (en) * | 2021-09-15 | 2021-10-22 | 中建电子商务有限责任公司 | Method and system for scheduling tasks |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110232092B (en) | 2023-08-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11301136B2 (en) | Capacity forecasting based on capacity policies and transactions | |
| US8285703B1 (en) | Document crawling systems and methods | |
| US20110119680A1 (en) | Policy-driven schema and system for managing data system pipelines in multi-tenant model | |
| US12045214B2 (en) | Database validation and repair tool | |
| US20160071033A1 (en) | Methods and apparatus for tracking construction material delivery | |
| CN109978392B (en) | Agile software development management method and device, electronic equipment and storage medium | |
| US10504045B2 (en) | Audit schedule determination | |
| CN109901918A (en) | A kind of method and apparatus handling overtime task | |
| US20200167738A1 (en) | Support for maintenance of a fleet of vehicles with intuitive display of repair analytics | |
| JP7177597B2 (en) | Production planning device, production planning system, production planning method, and program | |
| EP2199903A1 (en) | Metadata model repository | |
| EP3905058A1 (en) | Systems and methods for monitoring user-defined metrics | |
| CN110232092A (en) | A kind of asynchronous solution of batch data based on data processing and relevant device | |
| US20200348926A1 (en) | System and method for automating environment management of one or more software applications | |
| US20110265190A1 (en) | System and method for processing signature-verification operation | |
| CN111191999B (en) | Product research and development management method, device, computer equipment and storage medium | |
| US20130339075A1 (en) | Using past and scheduled events to reduce the cost of mistakes in an ediscovery process management system | |
| Hu | Information lifecycle modeling framework for construction project lifecycle management | |
| US11276017B2 (en) | Method and system for estimating efforts for software managed services production support engagements | |
| CN113419835A (en) | Job scheduling method, device, equipment and medium | |
| US20140089022A1 (en) | Statement of work analysis and resource participation assessment | |
| CN110032703A (en) | A kind of enterprise web site trustship method and system | |
| US20240028823A1 (en) | System and method for maintaining links and revisions | |
| US20200409904A1 (en) | Data migration grouping, planning and tracking | |
| JP5684311B2 (en) | Schedule management apparatus, method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |