CN113986519A - Data scheduling processing method, device, equipment and storage medium - Google Patents
Data scheduling processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113986519A CN113986519A CN202111625744.9A CN202111625744A CN113986519A CN 113986519 A CN113986519 A CN 113986519A CN 202111625744 A CN202111625744 A CN 202111625744A CN 113986519 A CN113986519 A CN 113986519A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- execution
- splitting
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5017—Task decomposition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of data processing, and discloses a data scheduling processing method, a data scheduling processing device, data scheduling processing equipment and a storage medium. The method comprises the following steps: acquiring target processing data; judging whether the target processing data needs to be split or not; if the splitting processing is needed, splitting the target processing data according to a preset splitting algorithm to obtain N queue execution data, wherein N is a positive integer; converting the N queue execution data into N JSON file data; according to a preset distribution execution algorithm, carrying out distribution execution processing on the N JSON file data to generate N queue message processing data; monitoring the message processing data of the N queues to generate a data processing result, wherein the data processing result comprises: a process continuation result and a process completion result.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a data scheduling method, apparatus, device, and storage medium.
Background
Task scheduling, as part of a software system or platform, is often used to execute programs that need to be executed periodically or temporarily in software, and the results of program execution, which usually do not need to be fed back to the user immediately, can be understood as "programs running in the background" at a simple point.
The existing task scheduler has the following problems:
A) stand-alone implementation. Many software or platforms, which only support stand-alone or single-node execution; however, in a large system, in order to support higher concurrency, a distributed architecture is usually adopted, and in this case, a task scheduler that only supports single-machine execution may have fatal problems such as repeated execution.
B) Only timed tasks are supported. The normal task schedulers such as quartz, etc. only support timing execution; for the temporary operation of the user, the executive program is required to be developed again.
C) Heterogeneous systems are not supported. Most task schedulers, which usually do not support tasks in different development languages, such as the scheduler for java development can only execute the task program for java development; some of the components support schedulers for executing tasks developed in different languages, and support local execution only, and cannot execute tasks on different servers or different operating systems.
Therefore, in the face of the data processing scheduling problem of the multi-language system, a technical problem that the current data processing scheduling system lacks universality and reusability is needed to be solved.
Disclosure of Invention
The invention mainly aims to solve the technical problem that a data processing scheduling system is lack of universality and reusability.
A first aspect of the present invention provides a data scheduling processing method, where the data scheduling processing method includes:
acquiring target processing data;
judging whether the target processing data needs to be split or not;
if the splitting processing is needed, splitting the target processing data according to a preset splitting algorithm to obtain N queue execution data, wherein N is a positive integer;
converting the N queue execution data into N JSON file data;
according to a preset distribution execution algorithm, carrying out distribution execution processing on the N JSON file data to generate N queue message processing data;
monitoring the message processing data of the N queues to generate a data processing result, wherein the data processing result comprises: a process continuation result and a process completion result.
Optionally, in a first implementation manner of the first aspect of the present invention, the acquiring target processing data includes:
based on a timing acquisition mode, capturing task data from a preset task table;
receiving a parameter setting instruction, and reading parameter data in the parameter setting instruction;
and matching and combining the task data and the parameter data to obtain target processing data.
Optionally, in a second implementation manner of the first aspect of the present invention, the determining whether the target processing data needs to be split includes:
judging whether parameter data in the target processing data belong to preset splitting setting data or not;
if the data belongs to the splitting setting data, determining that the target processing data needs to be split;
and if the data does not belong to the splitting setting data, determining that the target processing data does not need to be split.
Optionally, in a third implementation manner of the first aspect of the present invention, the splitting the target processing data according to a preset splitting algorithm to obtain N queue execution data includes:
splitting the parameter data of the target processing data into a plurality of minimum execution data based on the minimum splitting unit to obtain a minimum execution data set;
combining the data in the minimum execution data set to obtain an execution combination set;
and determining the elements in the execution combination set as queue execution data to obtain N queue execution data.
Optionally, in a fourth implementation manner of the first aspect of the present invention, the performing, according to a preset allocation execution algorithm, allocation execution processing on the N JSON file data, and generating N queue message processing data includes:
analyzing the N JSON file data based on a preset Java execution database to obtain N execution data sets;
and inputting the N execution data sets into M processing threads based on the execution data set as an object to obtain N queue message processing data, wherein M is a positive integer.
Optionally, in a fifth implementation manner of the first aspect of the present invention, after analyzing the N JSON file data based on the preset Java execution database to obtain N execution data sets, before inputting the N execution data sets into M processing threads and obtaining N queue message processing data, taking the execution data set as an object, the method further includes:
periodically receiving state uploading data of Y processing threads according to a preset maintenance period, wherein Y is a positive integer not less than M;
and analyzing the state uploading data to obtain M executable processing threads in the Y processing threads.
Optionally, in a sixth implementation manner of the first aspect of the present invention, the monitoring the N queue message processing data, and generating a data processing result includes:
sequentially reading the N queue message processing data to obtain N status characters;
judging whether the N state characters belong to processed state characters or not;
if the data processing results belong to the same category, determining the data processing results as processing completion results;
and if the data processing results do not belong to the same category, determining the data processing results as processing continuation results.
A second aspect of the present invention provides a data scheduling processing apparatus, including:
the acquisition module is used for acquiring target processing data;
the judging module is used for judging whether the target processing data needs to be split or not;
the splitting module is used for splitting the target processing data according to a preset splitting algorithm if the splitting processing is needed, so as to obtain N queue execution data, wherein N is a positive integer;
the conversion module is used for converting the N queue execution data into N JSON file data;
the distribution execution module is used for carrying out distribution execution processing on the N JSON file data according to a preset distribution execution algorithm to generate N queue message processing data;
a monitoring module, configured to monitor the N queue message processing data and generate a data processing result, where the data processing result includes: a process continuation result and a process completion result.
A third aspect of the present invention provides a data scheduling processing apparatus, including: a memory having instructions stored therein and at least one processor, the memory and the at least one processor interconnected by a line; the at least one processor calls the instructions in the memory to enable the data scheduling processing equipment to execute the data scheduling processing method.
A fourth aspect of the present invention provides a computer-readable storage medium having stored therein instructions, which, when run on a computer, cause the computer to execute the above-described data scheduling processing method.
In the embodiment of the invention, a JSON universal data format is used, a producer-consumer mode is based on, and the task is supported to be split, so that a distributed architecture can be supported; when the user quantity is increased and the transverse expansion is needed, only a producer or a consumer needs to be added singly or simultaneously, and the expansion is simple and convenient. The message is distributed based on the message queue, no matter timing or manual triggering of a user, only the message task message needs to be added, the task starting mode is flexible, and the technical problem that the data processing and scheduling system is lack of universality and reusability is solved.
Drawings
Fig. 1 is a schematic diagram of an embodiment of a data scheduling processing method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of an embodiment of a data scheduling processing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic diagram of another embodiment of a data scheduling processing apparatus according to an embodiment of the present invention;
fig. 4 is a schematic diagram of an embodiment of a data scheduling processing apparatus in an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a data scheduling processing method, a device, equipment and a storage medium.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of understanding, a specific flow of the embodiment of the present invention is described below, and referring to fig. 1, an embodiment of a data scheduling processing method according to the embodiment of the present invention includes:
101. acquiring target processing data;
in this embodiment, the scheduling Center (Schedule Center/Schedule-service) mainly includes scheduling management and executor management. Actuator Management (Actuator Management) is mainly responsible for communicating with actuators. For different executors (such as part C # of the Check Engine, it can be understood as an executor), the backend only needs to implement the corresponding java executor subsequently, and is managed by an executor manager (administrator Management) in a unified way. Scheduling Management (Scheduler Management) is mainly responsible for the initiation of tasks, the selection of executors and the processing of results. The system is only responsible for scheduling tasks and does not care about the specific execution of the tasks (the specific execution is handed to the executor for management).
Further, at 101, the following steps may be performed:
1011. based on a timing acquisition mode, capturing task data from a preset task table;
1012. receiving a parameter setting instruction, and reading parameter data in the parameter setting instruction;
1013. and matching and combining the task data and the parameter data to obtain target processing data.
In 1011-.
The parameter setting instruction may be data such as a storage address and whether splitting is performed, for example, in the splitting parameter item, 0 indicates splitting is performed, and 1 indicates not splitting is performed. And combining the task data of the execution mode and the parameter data to obtain target data needing to be processed.
102. Judging whether the target processing data needs to be split or not;
in this embodiment, whether the split parameter in the target processing data belongs to 1 or 0 is checked, and based on the definition parameter of 0 or 1, an indication whether the target processing data needs to be split is obtained.
Further, at 102, the following steps may be performed:
1021. judging whether parameter data in target processing data belong to preset splitting setting data or not;
1022. if the data belongs to the splitting setting data, determining that the target processing data needs to be split;
1023. and if the data does not belong to the splitting setting data, determining that the target processing data does not need to be split.
In 1021-. On the contrary, if the splitting parameter in the parameter data belongs to any one of the characters "R", "P", and "Q", it is determined that the target processing data needs to be split.
103. If the splitting processing is needed, splitting the target processing data according to a preset splitting algorithm to obtain N queue execution data, wherein N is a positive integer;
in this embodiment, the parameters of the service delivery are split into the minimum execution unit. For example, the parameters with the least execution units are a and b, and the parameters delivered by the user are [ a1, a2, a3 ] and [ b1, b2, b3 ], so that the user can split the message into 9 combinations of a1b1, a1b2, a1b3, a2b1, a2b2, a2b3, a3b1, a3b2, a3b3 and the like before sending the message. Such splitting may allow tasks to execute at the same time at an increased rate.
Further, 103 may perform the following steps:
1031. splitting parameter data of the target processing data into a plurality of minimum execution data based on the minimum splitting unit to obtain a minimum execution data set;
1032. combining the data in the minimum execution data set to obtain an execution combination set;
1033. and determining the elements in the execution combination set as queue execution data to obtain N queue execution data.
In the 1031-1033 step, the parameters of the target data are [ a1, a2, a3 ] and [ b1, b2, b3 ], and the minimum execution units are a and b, which are divided into a1, a2, a3, b1, b2 and b 3. The split data are combined in a permutation and combination mode to obtain an execution combination set { a1b1, b1a1, a1b2, b2a1, a1b3, b3a1, a2b1, b1a2, a2b2, b2a2, a2b3, b3a2, b1a3, a3b1, b2a3, a3b2, b3a3 and a3b3}, 18 elements are determined as queue execution data, and 18 queue execution data are obtained.
104. Converting the N queue execution data into N JSON file data;
in the embodiment, the data exchange problem of the heterogeneous system is solved by using a JSON (Java Server pages), which is a universal data format. And after the executors execute the tasks, the results are put back to the message queue to be combined. The most important advantages of this design are low coupling and high scalability:
1) low coupling: the scheduling of tasks and the execution of tasks are completely separated. The design ensures that the realization of the scheduler and the executor are not influenced mutually, wherein, the realization logic is upgraded or changed at any end, and the other end is not influenced as long as the JSON format of the exchange data is not modified. This is also the basis for the latter high scalability.
2) High expansibility: according to the actual operation condition, the executor nodes can be dynamically added or the splitting strategy can be modified. For example, when the user concurrency is high and the execution speed of the executor is slower than the added number of tasks, the executor node may be added to increase the execution speed, or the splitting policy may be adjusted, for example, two sets of parameters are distributed each time, such as one time of sending [ a1b1, a2b1] to a scheduler for execution.
105. According to a preset distribution execution algorithm, carrying out distribution execution processing on the N JSON file data to generate N queue message processing data;
in this embodiment, N JSON file data are allocated to multiple threads for processing, and if the number of threads is greater than N, one thread is not repeatedly allocated to each JSON file, so that N queue messages are obtained. However, if the number of threads is less than N, after the JSON files are sorted and allocated, at least more than one JSON file is allocated to one thread, and data is generated for the processing state of each JSON file to obtain N queue message processing data.
Further, 105 may perform the following steps:
1051. analyzing N JSON file data based on a preset Java execution database to obtain N execution data sets;
1052. and inputting the N execution data sets into M processing threads based on the execution data set as an object to obtain N queue message processing data, wherein M is a positive integer.
In the 1051 and 1052 steps, N file data are distributed to M processing threads by Java executors, and N queue message processing data are generated from the processing content of JSON file data.
Preferably, between 1051 and 1052, the following steps may also be performed:
1053. periodically receiving state uploading data of Y processing threads according to a preset maintenance period, wherein Y is a positive integer not less than M;
1054. and analyzing the state uploading data to obtain M executable processing threads in the Y processing threads.
In the steps 1053 and 1054, all of the Y threads are not necessarily valid, and an executor (such as Check Engine) needs to call a heartbeat interface and report its basic information (such as service name, state, etc.) every 1 minute after starting and starting. The execution management will issue tasks according to the maintained state of the execution. When the task is issued, the service name of the executor (the name is provided by the last registration) of the task in the afternoon is provided, so that the executor determines whether the task needs to be executed; if the executor receives a task which does not belong to the executor, the task needs to be returned (such as putting back MQ), and then the Y threads are analyzed to obtain M threads which can execute the task.
106. Monitoring N queue message processing data, and generating a data processing result, wherein the data processing result comprises: a process continuation result and a process completion result.
In this embodiment, the result of task distributed execution is processed in a message queue manner, and the processing conditions indicated by the message processing data in the N queues are periodically monitored to obtain the data processing result.
Further, 106 may perform the following steps:
1061. sequentially reading N queue message processing data to obtain N state characters;
1062. judging whether the N state characters belong to the processed state characters;
1063. if the data processing results belong to the same category, determining the data processing results as processing completion results;
1064. and if the unevenness belongs to the same, determining the data processing result as a processing continuation result.
In the 1061-1064 step, the N queue message processing data are analyzed, and if the processing is set to 0 and the processing is not set to 1, N sequences "00010 … 00" are generated, and then it is determined whether all the N sequences belong to 0. The sequence may also be converted to a value, and a determination is made as to whether the value is zero, and if not, processing continues, and if zero, processing is complete.
In the embodiment of the invention, a JSON universal data format is used, a producer-consumer mode is based on, and the task is supported to be split, so that a distributed architecture can be supported; when the user quantity is increased and the transverse expansion is needed, only a producer or a consumer needs to be added singly or simultaneously, and the expansion is simple and convenient. The message is distributed based on the message queue, no matter timing or manual triggering of a user, only the message task message needs to be added, the task starting mode is flexible, and the technical problem that the data processing and scheduling system is lack of universality and reusability is solved.
With reference to fig. 2, the data scheduling processing method in the embodiment of the present invention is described above, and a data scheduling processing apparatus in the embodiment of the present invention is described below, where an embodiment of the data scheduling processing apparatus in the embodiment of the present invention includes:
an obtaining module 201, configured to obtain target processing data;
a judging module 202, configured to judge whether the target processing data needs to be split;
the splitting module 203 is configured to split the target processing data according to a preset splitting algorithm if splitting processing is required, so as to obtain N queue execution data, where N is a positive integer;
a conversion module 204, configured to convert the N queue execution data into N JSON file data;
the allocation execution module 205 is configured to perform allocation execution processing on the N JSON file data according to a preset allocation execution algorithm, and generate N queue message processing data;
a monitoring module 206, configured to monitor the N queue message processing data, and generate a data processing result, where the data processing result includes: a process continuation result and a process completion result.
In the embodiment of the invention, a JSON universal data format is used, a producer-consumer mode is based on, and the task is supported to be split, so that a distributed architecture can be supported; when the user quantity is increased and the transverse expansion is needed, only a producer or a consumer needs to be added singly or simultaneously, and the expansion is simple and convenient. The message is distributed based on the message queue, no matter timing or manual triggering of a user, only the message task message needs to be added, the task starting mode is flexible, and the technical problem that the data processing and scheduling system is lack of universality and reusability is solved.
Referring to fig. 3, another embodiment of a data scheduling processing apparatus according to the embodiment of the present invention includes:
an obtaining module 201, configured to obtain target processing data;
a judging module 202, configured to judge whether the target processing data needs to be split;
the splitting module 203 is configured to split the target processing data according to a preset splitting algorithm if splitting processing is required, so as to obtain N queue execution data, where N is a positive integer;
a conversion module 204, configured to convert the N queue execution data into N JSON file data;
the allocation execution module 205 is configured to perform allocation execution processing on the N JSON file data according to a preset allocation execution algorithm, and generate N queue message processing data;
a monitoring module 206, configured to monitor the N queue message processing data, and generate a data processing result, where the data processing result includes: a process continuation result and a process completion result.
The obtaining module 201 is specifically configured to:
based on a timing acquisition mode, capturing task data from a preset task table;
receiving a parameter setting instruction, and reading parameter data in the parameter setting instruction;
and matching and combining the task data and the parameter data to obtain target processing data.
The determining module 202 is specifically configured to:
judging whether parameter data in the target processing data belong to preset splitting setting data or not;
if the data belongs to the splitting setting data, determining that the target processing data needs to be split;
and if the data does not belong to the splitting setting data, determining that the target processing data does not need to be split.
Wherein the splitting module 203 is specifically configured to:
splitting the parameter data of the target processing data into a plurality of minimum execution data based on the minimum splitting unit to obtain a minimum execution data set;
combining the data in the minimum execution data set to obtain an execution combination set;
and determining the elements in the execution combination set as queue execution data to obtain N queue execution data.
The allocation execution module 205 is specifically configured to:
analyzing the N JSON file data based on a preset Java execution database to obtain N execution data sets;
and inputting the N execution data sets into M processing threads based on the execution data set as an object to obtain N queue message processing data, wherein M is a positive integer.
The data scheduling processing apparatus further includes a state analysis module 207, where the state analysis module 207 is specifically configured to:
periodically receiving state uploading data of Y processing threads according to a preset maintenance period, wherein Y is a positive integer not less than M;
and analyzing the state uploading data to obtain M executable processing threads in the Y processing threads.
The monitoring module 206 is specifically configured to:
sequentially reading the N queue message processing data to obtain N status characters;
judging whether the N state characters belong to processed state characters or not;
if the data processing results belong to the same category, determining the data processing results as processing completion results;
and if the data processing results do not belong to the same category, determining the data processing results as processing continuation results.
In the embodiment of the invention, a JSON universal data format is used, a producer-consumer mode is based on, and the task is supported to be split, so that a distributed architecture can be supported; when the user quantity is increased and the transverse expansion is needed, only a producer or a consumer needs to be added singly or simultaneously, and the expansion is simple and convenient. The message is distributed based on the message queue, no matter timing or manual triggering of a user, only the message task message needs to be added, the task starting mode is flexible, and the technical problem that the data processing and scheduling system is lack of universality and reusability is solved.
Fig. 2 and fig. 3 describe the data scheduling processing apparatus in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the data scheduling processing apparatus in the embodiment of the present invention is described in detail from the perspective of hardware processing.
Fig. 4 is a schematic structural diagram of a data scheduling processing apparatus 400 according to an embodiment of the present invention, where the data scheduling processing apparatus 400 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 410 (e.g., one or more processors) and a memory 420, and one or more storage media 430 (e.g., one or more mass storage devices) storing an application 433 or data 432. Memory 420 and storage medium 430 may be, among other things, transient or persistent storage. The program stored on the storage medium 430 may include one or more modules (not shown), each of which may include a series of instructions operating on the data scheduling processing device 400. Further, the processor 410 may be configured to communicate with the storage medium 430 to execute a series of instruction operations in the storage medium 430 on the data schedule processing apparatus 400.
The data scheduling based processing apparatus 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input-output interfaces 460, and/or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, and the like. Those skilled in the art will appreciate that the data schedule-based processing apparatus architecture illustrated in fig. 4 does not constitute a limitation of the data schedule-based processing apparatus and may include more or fewer components than those illustrated, or some components may be combined, or a different arrangement of components.
The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein instructions, which, when executed on a computer, cause the computer to perform the steps of the data scheduling processing method.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses, and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A data scheduling processing method is characterized by comprising the following steps:
acquiring target processing data;
judging whether the target processing data needs to be split or not;
if the splitting processing is needed, splitting the target processing data according to a preset splitting algorithm to obtain N queue execution data, wherein N is a positive integer;
converting the N queue execution data into N JSON file data;
according to a preset distribution execution algorithm, carrying out distribution execution processing on the N JSON file data to generate N queue message processing data;
monitoring the message processing data of the N queues to generate a data processing result, wherein the data processing result comprises: a process continuation result and a process completion result.
2. The data scheduling processing method according to claim 1, wherein the acquiring target processing data includes:
based on a timing acquisition mode, capturing task data from a preset task table;
receiving a parameter setting instruction, and reading parameter data in the parameter setting instruction;
and matching and combining the task data and the parameter data to obtain target processing data.
3. The data scheduling processing method according to claim 2, wherein the determining whether the target processing data needs to be split includes:
judging whether parameter data in the target processing data belong to preset splitting setting data or not;
if the data belongs to the splitting setting data, determining that the target processing data needs to be split;
and if the data does not belong to the splitting setting data, determining that the target processing data does not need to be split.
4. The data scheduling processing method according to claim 1, wherein the splitting the target processing data according to a preset splitting algorithm to obtain N queue execution data includes:
splitting the parameter data of the target processing data into a plurality of minimum execution data based on the minimum splitting unit to obtain a minimum execution data set;
combining the data in the minimum execution data set to obtain an execution combination set;
and determining the elements in the execution combination set as queue execution data to obtain N queue execution data.
5. The data scheduling processing method according to claim 1, wherein the performing, according to a preset allocation execution algorithm, allocation execution processing on the N JSON file data, and generating N queue message processing data includes:
analyzing the N JSON file data based on a preset Java execution database to obtain N execution data sets;
and inputting the N execution data sets into M processing threads based on the execution data set as an object to obtain N queue message processing data, wherein M is a positive integer.
6. The data scheduling processing method according to claim 5, wherein after the preset Java-based execution database analyzes the N JSON file data to obtain N execution data sets, before the execution data sets are used as objects, the N execution data sets are input into M processing threads to obtain N queue message processing data, the method further comprises:
periodically receiving state uploading data of Y processing threads according to a preset maintenance period, wherein Y is a positive integer not less than M;
and analyzing the state uploading data to obtain M executable processing threads in the Y processing threads.
7. The data scheduling processing method according to claim 1, wherein the monitoring the N queue messages to process data, and the generating the data processing result includes:
sequentially reading the N queue message processing data to obtain N status characters;
judging whether the N state characters belong to processed state characters or not;
if the data processing results belong to the same category, determining the data processing results as processing completion results;
and if the data processing results do not belong to the same category, determining the data processing results as processing continuation results.
8. A data scheduling processing apparatus, characterized in that the data scheduling processing apparatus comprises:
the acquisition module is used for acquiring target processing data;
the judging module is used for judging whether the target processing data needs to be split or not;
the splitting module is used for splitting the target processing data according to a preset splitting algorithm if the splitting processing is needed, so as to obtain N queue execution data, wherein N is a positive integer;
the conversion module is used for converting the N queue execution data into N JSON file data;
the distribution execution module is used for carrying out distribution execution processing on the N JSON file data according to a preset distribution execution algorithm to generate N queue message processing data;
a monitoring module, configured to monitor the N queue message processing data and generate a data processing result, where the data processing result includes: a process continuation result and a process completion result.
9. A data schedule processing apparatus, characterized in that the data schedule processing apparatus comprises: a memory having instructions stored therein and at least one processor, the memory and the at least one processor interconnected by a line;
the at least one processor invokes the instructions in the memory to cause the data schedule processing apparatus to perform the data schedule processing method of any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a data scheduling processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111625744.9A CN113986519B (en) | 2021-12-29 | 2021-12-29 | Data scheduling processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111625744.9A CN113986519B (en) | 2021-12-29 | 2021-12-29 | Data scheduling processing method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113986519A true CN113986519A (en) | 2022-01-28 |
| CN113986519B CN113986519B (en) | 2022-06-03 |
Family
ID=79734793
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111625744.9A Active CN113986519B (en) | 2021-12-29 | 2021-12-29 | Data scheduling processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113986519B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105138592A (en) * | 2015-07-31 | 2015-12-09 | 武汉虹信技术服务有限责任公司 | Distributed framework-based log data storing and retrieving method |
| CN108959292A (en) * | 2017-05-19 | 2018-12-07 | 北京京东尚科信息技术有限公司 | A kind of data uploading method, system and computer readable storage medium |
| WO2019039958A1 (en) * | 2017-08-25 | 2019-02-28 | Константин Андреевич МАЛЫШЕВ | Method of describing a composite data type |
| CN110019214A (en) * | 2017-12-04 | 2019-07-16 | 北京京东尚科信息技术有限公司 | The method and apparatus that data split result is verified |
| CN110689377A (en) * | 2019-09-30 | 2020-01-14 | 北京达佳互联信息技术有限公司 | Data detection method and device and electronic equipment |
| CN111143468A (en) * | 2019-12-11 | 2020-05-12 | 浙江华云信息科技有限公司 | Multi-database data management method based on MPP distributed technology |
| CN111309722A (en) * | 2018-12-11 | 2020-06-19 | 北京京东尚科信息技术有限公司 | Data processing method and device |
| CN111475309A (en) * | 2019-01-24 | 2020-07-31 | 北京京东尚科信息技术有限公司 | Data processing method, device, block chain service system and storage medium |
| CN112115680A (en) * | 2020-08-18 | 2020-12-22 | 天津洪恩完美未来教育科技有限公司 | Font splitting method and device, computer equipment and computer readable storage medium |
| CN112328607A (en) * | 2020-12-04 | 2021-02-05 | 四三九九网络股份有限公司 | Asynchronous compression processing method of large-volume JSON data |
| CN113064987A (en) * | 2021-04-30 | 2021-07-02 | 中国工商银行股份有限公司 | Data processing method, apparatus, electronic device, medium, and program product |
| CN113535436A (en) * | 2021-07-27 | 2021-10-22 | 苏州摩联通信技术有限公司 | Data processing method, device and storage medium |
-
2021
- 2021-12-29 CN CN202111625744.9A patent/CN113986519B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105138592A (en) * | 2015-07-31 | 2015-12-09 | 武汉虹信技术服务有限责任公司 | Distributed framework-based log data storing and retrieving method |
| CN108959292A (en) * | 2017-05-19 | 2018-12-07 | 北京京东尚科信息技术有限公司 | A kind of data uploading method, system and computer readable storage medium |
| WO2019039958A1 (en) * | 2017-08-25 | 2019-02-28 | Константин Андреевич МАЛЫШЕВ | Method of describing a composite data type |
| CN110019214A (en) * | 2017-12-04 | 2019-07-16 | 北京京东尚科信息技术有限公司 | The method and apparatus that data split result is verified |
| CN111309722A (en) * | 2018-12-11 | 2020-06-19 | 北京京东尚科信息技术有限公司 | Data processing method and device |
| CN111475309A (en) * | 2019-01-24 | 2020-07-31 | 北京京东尚科信息技术有限公司 | Data processing method, device, block chain service system and storage medium |
| CN110689377A (en) * | 2019-09-30 | 2020-01-14 | 北京达佳互联信息技术有限公司 | Data detection method and device and electronic equipment |
| CN111143468A (en) * | 2019-12-11 | 2020-05-12 | 浙江华云信息科技有限公司 | Multi-database data management method based on MPP distributed technology |
| CN112115680A (en) * | 2020-08-18 | 2020-12-22 | 天津洪恩完美未来教育科技有限公司 | Font splitting method and device, computer equipment and computer readable storage medium |
| CN112328607A (en) * | 2020-12-04 | 2021-02-05 | 四三九九网络股份有限公司 | Asynchronous compression processing method of large-volume JSON data |
| CN113064987A (en) * | 2021-04-30 | 2021-07-02 | 中国工商银行股份有限公司 | Data processing method, apparatus, electronic device, medium, and program product |
| CN113535436A (en) * | 2021-07-27 | 2021-10-22 | 苏州摩联通信技术有限公司 | Data processing method, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113986519B (en) | 2022-06-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6411982B2 (en) | Thread based governor for time scheduled process execution | |
| US7689996B2 (en) | Method to distribute programs using remote Java objects | |
| US6571215B1 (en) | System and method for generating a schedule based on resource assignments | |
| US8392572B2 (en) | Method for scheduling cloud-computing resource and system applying the same | |
| US20060268967A1 (en) | Supplying instruction to operational stations | |
| CN111625331B (en) | Task scheduling method, device, platform, server and storage medium | |
| CN112162865A (en) | Server scheduling method and device and server | |
| CN111984390A (en) | Task scheduling method, device, equipment and storage medium | |
| US8266624B2 (en) | Task dispatch utility coordinating the execution of tasks on different computers | |
| CN104915259A (en) | Task scheduling method applied to distributed acquisition system | |
| CN111125444A (en) | Big data task scheduling management method, device, equipment and storage medium | |
| CN107992362A (en) | The method, apparatus and system of automated performance testing | |
| CN101751288A (en) | Method, device and system applying process scheduler | |
| CN112882828B (en) | Method for managing and scheduling a processor in a processor-based SLURM operation scheduling system | |
| CN112307066B (en) | Distributed data aggregation method, system, device and storage medium | |
| CN112365157A (en) | Intelligent dispatching method, device, equipment and storage medium | |
| CN111913784B (en) | Task scheduling method and device, network element and storage medium | |
| CN115470915A (en) | Server system of quantum computer and implementation method thereof | |
| CN113986519B (en) | Data scheduling processing method, device, equipment and storage medium | |
| CN112527642A (en) | Performance test index data display method, device, equipment and storage medium | |
| CN111082964B (en) | Distribution method and device of configuration information | |
| CN112307046A (en) | Data acquisition method and device, computer readable storage medium and electronic equipment | |
| CN109829005A (en) | A kind of big data processing method and processing device | |
| Ostermann et al. | Workflow monitoring and analysis tool for ASKALON | |
| CN114297042A (en) | Interface pressure performance testing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |