CN108829521B - Task processing method and device, computer equipment and storage medium - Google Patents
Task processing method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN108829521B CN108829521B CN201810609360.XA CN201810609360A CN108829521B CN 108829521 B CN108829521 B CN 108829521B CN 201810609360 A CN201810609360 A CN 201810609360A CN 108829521 B CN108829521 B CN 108829521B
- Authority
- CN
- China
- Prior art keywords
- preset
- task
- executed
- queue
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Abstract
The application discloses a task processing method, a task processing device, computer equipment and a storage medium, wherein the method comprises the following steps: adding a Redis tool on a Web platform, and creating a preset task queue through the Redis tool, wherein the Web platform is a webpage platform developed based on an infrastructure and a Celery framework; acquiring tasks to be executed in a Celery queue in the Web platform; saving the task to be executed to the preset task queue; acquiring a task to be executed which meets a preset execution condition in the preset task queue, wherein the preset execution condition corresponds to a preset processing mode; and executing the tasks to be executed meeting the preset execution conditions according to the preset processing mode. The method accelerates the processing speed of the task to be executed, avoids task accumulation, further improves the response speed of the Web platform, and improves user experience.
Description
Technical Field
The present application relates to the field of internet technologies, and in particular, to a task processing method and apparatus, a computer device, and a storage medium.
Background
At present, in the internet industry, in order to facilitate operation and maintenance management, a Web platform is usually developed by combining an enterprise framework and a Celery framework, but tasks in a Celery queue can only be executed in a single process during execution, if the tasks are too many, the task consumption is slow due to the execution of the single process, and further a performance bottleneck is generated in the execution on a remote server, so that a great number of tasks are accumulated in the Celery queue. Therefore, it is desirable to provide a task processing method to solve the above problems.
Disclosure of Invention
The application provides a task processing method, a task processing device, computer equipment and a storage medium, and aims to accelerate the response speed of a Web platform.
In a first aspect, the present application provides a task processing method, including:
adding a Redis tool on a Web platform, and creating a preset task queue through the Redis tool, wherein the Web platform is a webpage platform developed based on an infrastructure and a Celery framework;
acquiring tasks to be executed in a Celery queue in the Web platform;
saving the task to be executed to the preset task queue;
acquiring a task to be executed which meets a preset execution condition in the preset task queue, wherein the preset execution condition corresponds to a preset processing mode; and
and executing the tasks to be executed meeting the preset execution conditions according to the preset processing mode.
In a second aspect, the present application provides a task processing device, comprising:
the system comprises an adding and creating unit, a task queue creating unit and a task queue creating unit, wherein the adding and creating unit is used for adding a Redis tool on a Web platform and creating a preset task queue through the Redis tool, and the Web platform is a webpage platform developed based on an infrastructure and a Celery framework;
the first acquisition unit is used for acquiring tasks to be executed in a Celery queue in the Web platform;
the task storage unit is used for storing the task to be executed to the preset task queue;
a second obtaining unit, configured to obtain a to-be-executed task that meets a preset execution condition in the preset task queue, where the preset execution condition corresponds to a preset processing manner; and
and the task execution unit is used for executing the tasks to be executed which meet the preset execution conditions according to the preset processing mode. .
In a third aspect, the present application further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the processor implements the steps of the task processing method provided in any one of the applications.
In a fourth aspect, the present application further provides a storage medium, wherein the storage medium stores a computer program, the computer program comprises program instructions, which when executed by a processor, cause the processor to execute the steps of the task processing method provided in any one of the applications.
According to the embodiment of the application, the preset task queue can be created by means of the Redis tool, and the tasks of the Celery queue can be quickly consumed by selecting different preset processing modes through the preset task queue, so that a large amount of accumulation of the tasks can be avoided, and the problem that performance bottleneck is caused when the remote server executes the tasks is solved. The processing speed of the task to be executed is increased, the response speed of the Web platform is increased, and the user experience is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic flow chart diagram of a task processing method according to an embodiment of the present application;
FIG. 2 is a schematic flow diagram of sub-steps of the task processing method of FIG. 1;
FIG. 3 is a schematic flow diagram of sub-steps of the task processing method of FIG. 2;
FIG. 4 is a schematic flow chart diagram of a task processing method according to another embodiment of the present application;
FIG. 5 is a schematic block diagram of a task processing device according to an embodiment of the present application;
FIG. 6 is a schematic block diagram of a task processing device according to another embodiment of the present application;
fig. 7 is a schematic block diagram of a computer device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Referring to fig. 1, fig. 1 is a schematic flow chart of a task processing method according to an embodiment of the present application. As shown in fig. 1, the task processing method includes steps S101 to S105.
S101, adding a Redis tool on a Web platform, and creating a preset task queue through the Redis tool, wherein the Web platform is a webpage platform developed based on an infrastructure and a Celery framework.
In this embodiment, the Web platform is a Web platform developed based on an anchor and a Celery framework, the anchor is an automated operation and maintenance tool developed based on Python, and the Celery is a distributed asynchronous message task queue developed based on Python, through which asynchronous processing of tasks can be easily implemented. The Celery can only execute tasks in a single thread, so that a lot of tasks can be accumulated in the Celery queue, and the response speed of the Web platform is influenced.
The Redis tool is a high-performance key-value database and is also an internal memory database, the Redis tool is added into a Web platform, a preset task queue is created through the Redis tool, and the preset task queue is used for recording tasks. Specifically, the preset task queue may be in the form of a queue table.
S102, obtaining the tasks to be executed in the Celery queue in the Web platform.
In this embodiment, since the Celery can only execute tasks by a single thread, the Celery queue may accumulate many tasks to be executed. Therefore, the acquiring the to-be-executed tasks in the query queue in the Web platform includes: and polling the Celery queue regularly to acquire the tasks to be executed in the Celery queue in the Web platform. The duration of the timed polling may be set according to actual needs, and is not limited herein.
In an embodiment, the acquiring the to-be-executed task in the query queue in the Web platform includes: and acquiring the Web application corresponding to the task to be executed. The Web application corresponding to the task to be executed specifically refers to a task to be executed issued by a certain Web application.
S103, storing the task to be executed to the preset task queue.
In this embodiment, the preset task queue is a task queue created in the memory database by a Redis tool, so that the content of the preset task queue is conveniently and quickly stored or read, and the processing speed of the task to be executed is increased.
S104, obtaining the to-be-executed tasks meeting preset execution conditions in the preset task queue, wherein the preset execution conditions correspond to a preset processing mode.
In this embodiment, the preset execution condition is used to classify the tasks to be executed in the preset task queue, so as to select different preset processing modes for execution according to different tasks to be executed. For example, the preset processing mode is a multi-thread processing mode, so that the processing speed of the task to be executed is increased.
In an embodiment, the preset execution condition is, for example, that the Web application corresponding to the task to be executed is in a running state. Specifically, as shown in fig. 2, the step S104 includes sub-steps S104a and S104 b. S104a, monitoring whether the Web application corresponding to the task to be executed in the preset task queue is in a running state; s104b, if the Web application corresponding to the task to be executed is in the running state, judging that the task to be executed meets the preset execution condition.
And monitoring whether the Web application corresponding to the task to be executed in the preset task queue is in a running state, specifically by monitoring the corresponding main thread of the Web application. As shown in fig. 3, step S104a includes sub-steps S104a1 and S104a 2. S104a1, monitoring the state of the running loop of the main thread corresponding to the Web application corresponding to the task to be executed; s104a2, judging whether the Web application is in the running state according to the running loop state of the main thread.
Specifically, whether the Web application is in a running state is judged by monitoring the state of a running loop of a main thread corresponding to the Web application, and if the state of the running loop of the main thread is in a sleep state, the Web application is in an idle state; this indicates instead that the Web application is in a running state.
In an embodiment, the obtaining of the to-be-executed task that satisfies the preset execution condition in the preset task queue includes: determining a preset quantity range corresponding to the quantity of the tasks to be executed in the preset task queue, wherein different preset quantity ranges correspond to different preset processing modes; and acquiring a preset processing mode corresponding to the preset quantity range according to the determined preset quantity range.
Specifically, the preset number range includes a first number range or a second number range, and may also include other number ranges, where the first number range and the second number range correspond to different numbers of tasks to be executed, and correspond to different preset processing manners, for example, multithreading processing manners are both adopted, but the multithreading processing manners include a first multithreading processing manner and a second multithreading processing manner, where the two multithreading processing manners have different numbers of open threads, the first number range corresponds to the first multithreading processing manner, and the second number range corresponds to the second multithreading processing manner. Therefore, the number range corresponding to different numbers of tasks to be executed can be determined, and the corresponding multithreading processing mode is selected for execution. Not only can the processing speed be accelerated, but also the resources can be saved.
And S105, executing the task to be executed meeting the preset execution condition according to the preset processing mode.
In this embodiment, the to-be-executed task meeting the preset execution condition is specifically executed according to a preset processing manner corresponding to the preset execution condition to complete the to-be-executed task in the preset task queue.
For example, a task to be executed corresponding to a running Web application is executed in a multi-thread processing manner to complete processing of the task to be executed.
For another example, the tasks to be executed in the preset task queue, which satisfy the preset number range, are executed according to a preset processing mode corresponding to the preset number range.
In the embodiment, a Redis tool is added on a Web platform, and a preset task queue is created through the Redis tool; acquiring tasks to be executed in a Celery queue in the Web platform; saving the task to be executed to the preset task queue; acquiring a task to be executed which meets a preset execution condition in the preset task queue, wherein the preset execution condition corresponds to a preset processing mode; and executing the tasks to be executed meeting the preset execution conditions according to the preset processing mode. According to the method, the tasks of the Celery queue can be quickly consumed by means of the preset task queue created by the Redis tool and different preset processing modes selected by the preset task queue, so that a large amount of accumulated tasks can be avoided, the processing speed of the tasks to be executed is increased, the response speed of the Web platform is increased, and the user experience is improved.
Referring to fig. 4, fig. 4 is a schematic flowchart of a task processing method according to another embodiment of the present application. As shown in fig. 4, the task processing method includes steps S201 to S208.
S201, adding a Redis tool on a Web platform, and creating a preset task queue through the Redis tool, wherein the Web platform is a webpage platform developed based on an android and a Celery framework.
Specifically, the preset task queue is created in the memory database by a Redis tool and is used for recording the to-be-executed tasks, where the to-be-executed tasks are to-be-processed tasks corresponding to the respective Web applications, and may also include to-be-processed tasks corresponding to other applications related to the Web applications. Specifically, the preset task queue may be in the form of a queue table.
S202, obtaining the tasks to be executed in the Celery queue in the Web platform and the Web application corresponding to the tasks to be executed.
Specifically, the query is performed on the query queue to obtain the task to be executed in the query queue in the Web platform, and meanwhile, the Web application corresponding to the task to be executed is also obtained.
S203, judging whether the number of the tasks to be executed in the Celery queue is larger than the preset number of the tasks.
In this embodiment, the preset number of tasks may be an empirical value, and it may be determined whether the number of tasks to be executed in the queue will form a pile through the preset number of tasks, where it indicates that a task pile may be formed when the number of tasks to be executed in the queue is greater than the preset number of tasks.
And S204, monitoring whether the Web application corresponding to the task to be executed in the preset task queue is in a running state.
Specifically, if the number of the tasks to be executed in the query queue is greater than the preset number of tasks, monitoring whether the Web application corresponding to the tasks to be executed in the preset task queue is in a running state.
When the Web application corresponding to the task to be executed is monitored to be in the running state, the fact that the user is using the Web application is indicated, and then the following steps are executed, so that the task processing speed is accelerated through the task processing method provided by the application, and the response speed of the running Web application is improved.
Specifically, the monitoring whether the Web application corresponding to the task to be executed is in the running state specifically adopts: monitoring the state of a running loop of a main thread corresponding to the Web application corresponding to the task to be executed; and judging whether the Web application is in the running state or not according to the state of the running loop of the main thread.
S205, storing the task to be executed to the preset task queue.
Specifically, if the Web application corresponding to the task to be executed is in the running state, the task to be executed is saved to the preset task queue. The preset task queue is a task queue created in the memory database through a Redis tool, so that the content of the preset task queue is conveniently and quickly stored or read, and the processing speed of the task to be executed is increased.
S206, determining a preset quantity range corresponding to the quantity of the tasks to be executed in the preset task queue, wherein different preset quantity ranges correspond to different preset processing modes.
In this embodiment, the preset number range includes a first number range and a second number range, and may also include other number ranges, where the first number range and the second number range correspond to different numbers of tasks to be executed.
Specifically, the first number range corresponds to the first multithread processing mode, and the second number range corresponds to the second multithread processing mode, where the number of open threads in the first multithread processing mode is different from that in the second multithread processing mode, for example, the first multithread processing mode opens 8 threads for processing, and the second multithread processing mode opens 16 threads for processing.
And S207, acquiring a preset processing mode corresponding to the preset quantity range according to the determined preset quantity range.
Specifically, a number range corresponding to the number of the tasks to be executed may be determined, for example, if the number range corresponding to the number of the tasks to be executed corresponds to a first number range, it may be determined that the tasks to be executed are processed in a first multithread processing manner. And selecting a corresponding multithreading processing mode to execute according to the quantity range corresponding to the tasks to be executed, so that the processing speed can be increased, and resources can be saved.
And S208, executing the tasks to be executed in the preset task queue, which meet the preset quantity range, according to a preset processing mode corresponding to the preset quantity range.
For example, if it is determined that the number of the to-be-executed tasks in the preset task queue satisfies a first number range, the to-be-executed tasks in the preset task queue are executed according to a first multithreading processing mode corresponding to the first number range to complete processing of the to-be-executed tasks.
In the embodiment, the Redis tool is added on the Web platform, and the preset task queue is established through the Redis tool; storing the tasks to be executed in the Celery queue, which meet the conditions, in the preset task queue; acquiring a task to be executed which meets a preset execution condition in the preset task queue, wherein the preset execution condition corresponds to a preset processing mode; so as to execute the tasks to be executed corresponding to different preset execution conditions by adopting different preset processing modes. Therefore, the processing speed of the task to be executed can be increased, and the response speed of the Web platform is further increased so as to improve the user experience.
Referring to fig. 5, fig. 5 is a schematic block diagram of a task processing device according to an embodiment of the present application. As shown in fig. 5, the task processing device 300 includes: an addition creation unit 301, a first acquisition unit 302, a task saving unit 303, a second acquisition unit 304, and a task execution unit 305.
The adding and creating unit 301 is used for adding a Redis tool on a Web platform and creating a preset task queue through the Redis tool, wherein the Web platform is a webpage platform developed based on an infrastructure and a Celery framework.
A first obtaining unit 302, configured to obtain a task to be executed in a query queue in the Web platform.
In an embodiment, the first obtaining unit 302 is further configured to obtain a Web application corresponding to the task to be executed.
A task saving unit 303, configured to save the task to be executed to the preset task queue.
A second obtaining unit 304, configured to obtain a to-be-executed task that meets a preset execution condition in the preset task queue, where the preset execution condition corresponds to a preset processing manner.
Wherein the second obtaining unit 304 includes: a task monitoring subunit 3041 and a conditional decision subunit 3042; a task monitoring subunit 3041, configured to monitor whether a Web application corresponding to a task to be executed in the preset task queue is in an operating state; a condition determining subunit 3042, configured to determine that the task to be executed satisfies the preset execution condition if the Web application corresponding to the task to be executed is in an operating state.
Specifically, the task monitoring subunit 3041 is specifically configured to: monitoring the state of an operation loop of a main thread corresponding to the Web application corresponding to the task to be executed; and judging whether the Web application is in the running state or not according to the state of the running loop of the main thread.
The task execution unit 305 is configured to execute the to-be-executed task meeting the preset execution condition according to the preset processing manner.
Referring to fig. 6, fig. 6 is a schematic block diagram of a task processing device according to an embodiment of the present application. As shown in fig. 6, the task processing device 500 includes: an addition creation unit 501, a first acquisition unit 502, a first judgment unit 503, a state monitoring unit 504, a task saving unit 505, a range determination unit 506, a manner acquisition unit 507, and a task execution unit 508.
The adding and creating unit 501 is used for adding a Redis tool on a Web platform and creating a preset task queue through the Redis tool, wherein the Web platform is a webpage platform developed based on an infrastructure and a Celery frame.
A first obtaining unit 502, configured to obtain a task to be executed in a query queue in the Web platform and a Web application corresponding to the task to be executed.
A first determining unit 503, configured to determine whether the number of to-be-executed tasks in the query queue is greater than a preset number of tasks.
A state monitoring unit 504, configured to monitor whether the Web application corresponding to the to-be-executed task in the preset task queue is in an operating state if the number of to-be-executed tasks in the query queue is greater than the preset task number.
Specifically, the state monitoring unit 504 is specifically configured to: monitoring the state of an operation loop of a main thread corresponding to the Web application corresponding to the task to be executed; and judging whether the Web application is in the running state or not according to the state of the running loop of the main thread.
And the task saving unit 505 is configured to save the to-be-executed task to the preset task queue if the Web application corresponding to the to-be-executed task is in the running state.
A range determining unit 506, configured to determine a preset number range corresponding to the number of to-be-executed tasks in the preset task queue, where different preset number ranges correspond to different preset processing manners;
a manner obtaining unit 507, configured to obtain, according to the determined preset number range, a preset processing manner corresponding to the preset number range.
And the task execution unit 508 is configured to execute the to-be-executed tasks in the preset task queue, which satisfy the preset number range, according to a preset processing manner corresponding to the preset number range.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the task processing apparatus and the unit described above may refer to corresponding processes in the foregoing method embodiments, and are not described herein again.
The above-described apparatus may be implemented in the form of a computer program which is executable on a computer device as shown in fig. 7.
Referring to fig. 7, fig. 7 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 700 may be a server.
Referring to fig. 7, the computer device 700 includes a processor 720, a memory, which may include a non-volatile storage medium 730 and an internal memory 740, and a network interface 750, which are connected by a system bus 710.
The non-volatile storage medium 730 may store an operating system 731 and computer programs 732. The computer programs 732, when executed, enable the processor 720 to perform any of a variety of task processing methods.
The processor 720 is used to provide computing and control capabilities, supporting the operation of the overall computer device 700.
The internal memory 740 provides an environment for the execution of the computer program 732 in the non-volatile storage medium 730, and when the computer program 732 is executed by the processor 720, the processor 720 can be caused to execute any one of the task processing methods.
The network interface 750 is used for network communication such as sending assigned tasks and the like. Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing device 700 to which the disclosed aspects apply, as a particular computing device 700 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components. Wherein the processor 720 is configured to execute the program code stored in the memory to perform the following steps:
adding a Redis tool on a Web platform, and creating a preset task queue through the Redis tool, wherein the Web platform is a webpage platform developed based on an infrastructure and a Celery framework;
acquiring tasks to be executed in a Celery queue in the Web platform;
saving the task to be executed to the preset task queue;
acquiring a task to be executed which meets a preset execution condition in the preset task queue, wherein the preset execution condition corresponds to a preset processing mode; and
and executing the tasks to be executed meeting the preset execution conditions according to the preset processing mode.
In an embodiment, the processor 720 is configured to execute the program code stored in the memory to implement the following steps before saving the task to be executed to the preset task queue:
judging whether the number of the tasks to be executed in the Celery queue is larger than the preset number of the tasks or not;
and if the number of the tasks to be executed in the Celery queue is greater than the preset number of the tasks, storing the tasks to be executed to the preset task queue.
In an embodiment, when the processor 720 is configured to run the program code stored in the memory to achieve the task to be executed in the query queue in the Web platform, the following steps are further implemented:
acquiring a Web application corresponding to the task to be executed;
monitoring whether the Web application corresponding to the task to be executed in the preset task queue is in a running state or not;
and if the Web application corresponding to the task to be executed is in the running state, judging that the task to be executed meets the preset execution condition.
In an embodiment, the processor 720 is configured to run a program code stored in a memory to monitor whether the Web application corresponding to the task to be executed is in a running state, and specifically implement the following steps:
monitoring the state of an operation loop of a main thread corresponding to the Web application corresponding to the task to be executed;
and judging whether the Web application is in the running state or not according to the state of the running loop of the main thread.
In an embodiment, when the processor 720 is configured to run the program code stored in the memory to obtain the to-be-executed task satisfying the preset execution condition in the preset task queue, the following steps are further implemented:
determining a preset quantity range corresponding to the quantity of the tasks to be executed in the preset task queue, wherein different preset quantity ranges correspond to different preset processing modes;
acquiring a preset processing mode corresponding to the preset quantity range according to the determined preset quantity range;
and executing the tasks to be executed in the preset task queue, which meet the preset quantity range, according to a preset processing mode corresponding to the preset quantity range.
It should be understood that, in the embodiment of the present Application, the Processor 720 may be a Central Processing Unit (CPU), and the Processor 720 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Those skilled in the art will appreciate that the configuration of computer device 700 depicted in FIG. 7 is not intended to be limiting of computer device 700 and may include more or less components than those shown, or some components in combination, or a different arrangement of components.
It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments may be implemented by hardware related to instructions of a computer program, and the computer program may be stored in a storage medium, which is a computer-readable storage medium. In the embodiment of the present invention, the computer program may be stored in a storage medium of a computer system and executed by at least one processor in the computer system to implement the flow steps of the embodiments including the methods as described above.
The computer readable storage medium may be a magnetic disk, an optical disk, a usb disk, a removable hard disk, a Read-only memory (ROM), a magnetic disk, or an optical disk, and various media capable of storing program codes.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the several embodiments provided in the present application, it should be understood that the disclosed task processing device and method can be implemented in other ways. For example, the task processing device embodiments described above are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.
The steps in the method of the embodiment of the application can be sequentially adjusted, combined and deleted according to actual needs.
The units in the device of the embodiment of the application can be combined, divided and deleted according to actual needs.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially or partially implemented in the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application.
While the invention has been described with reference to specific embodiments, the scope of the invention is not limited thereto, and those skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the invention. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (7)
1. A task processing method, comprising:
adding a Redis tool on a Web platform, and creating a preset task queue through the Redis tool, wherein the Web platform is a webpage platform developed based on an infrastructure and a Celery framework;
acquiring tasks to be executed in a Celery queue in the Web platform;
saving the task to be executed to the preset task queue;
acquiring a task to be executed which meets a preset execution condition in the preset task queue, wherein the preset execution condition corresponds to a preset processing mode; and
executing the tasks to be executed meeting the preset execution condition according to the preset processing mode;
the acquiring of the to-be-executed task meeting the preset execution condition in the preset task queue includes:
determining a preset quantity range corresponding to the quantity of the tasks to be executed in the preset task queue, wherein different preset quantity ranges correspond to different preset processing modes, the preset processing modes comprise multithreading processing modes, the multithreading processing modes comprise a first multithreading processing mode and a second multithreading processing mode, and the quantity of open threads of the first multithreading processing mode is different from that of open threads of the second multithreading processing mode;
acquiring a preset processing mode corresponding to the preset quantity range according to the determined preset quantity range;
executing the task to be executed meeting the preset execution condition according to the preset processing mode, wherein the executing mode comprises the following steps:
executing the tasks to be executed in the preset task queue, which meet the preset quantity range, according to a preset processing mode corresponding to the preset quantity range;
before the task to be executed is stored in the preset task queue, the method further includes:
judging whether the number of the tasks to be executed in the Celery queue is larger than the preset number of the tasks or not;
and if the number of the tasks to be executed in the Celery queue is larger than the preset number of the tasks, executing the step of storing the tasks to be executed to the preset task queue.
2. The task processing method according to claim 1, wherein the acquiring the to-be-executed task in the query queue in the Web platform includes: acquiring a Web application corresponding to the task to be executed;
the acquiring of the to-be-executed task meeting the preset execution condition in the preset task queue includes:
monitoring whether the Web application corresponding to the task to be executed in the preset task queue is in a running state or not;
and if the Web application corresponding to the task to be executed is in the running state, judging that the task to be executed meets the preset execution condition.
3. The task processing method according to claim 2, wherein the monitoring whether the Web application corresponding to the task to be executed is in a running state includes:
monitoring the state of a running loop of a main thread corresponding to the Web application corresponding to the task to be executed;
and judging whether the Web application is in the running state or not according to the state of the running loop of the main thread.
4. The task processing method according to claim 1, wherein the acquiring the to-be-executed task in the query queue in the Web platform includes:
and polling the Celery queue regularly to acquire the tasks to be executed in the Celery queue in the Web platform.
5. A task processing apparatus, comprising:
the system comprises an adding and creating unit, a task queue creating unit and a task queue creating unit, wherein the adding and creating unit is used for adding a Redis tool on a Web platform and creating a preset task queue through the Redis tool, and the Web platform is a webpage platform developed based on an infrastructure and a Celery framework;
the first acquisition unit is used for acquiring tasks to be executed in a Celery queue in the Web platform;
the task storage unit is used for storing the task to be executed to the preset task queue;
a second obtaining unit, configured to obtain a to-be-executed task that meets a preset execution condition in the preset task queue, where the preset execution condition corresponds to a preset processing manner; and
the task execution unit is used for executing the tasks to be executed which meet the preset execution conditions according to the preset processing mode;
the range determining unit is used for determining a preset number range corresponding to the number of the tasks to be executed in the preset task queue, wherein different preset number ranges correspond to different preset processing modes, the preset processing modes comprise multi-thread processing modes, the multi-thread processing modes comprise a first multi-thread processing mode and a second multi-thread processing mode, and the number of open threads of the first multi-thread processing mode is different from that of open threads of the second multi-thread processing mode;
the mode acquisition unit is used for acquiring a preset processing mode corresponding to the preset quantity range according to the determined preset quantity range;
the task execution unit is used for executing the tasks to be executed in the preset task queue, which meet the preset quantity range, according to a preset processing mode corresponding to the preset quantity range;
the task processing device further includes:
the first judging unit is used for judging whether the number of the tasks to be executed in the Celery queue is larger than the preset number of the tasks;
and the state monitoring unit is used for monitoring whether the Web application corresponding to the tasks to be executed in the preset task queue is in the running state or not if the number of the tasks to be executed in the Celery queue is greater than the preset task number.
6. A computer arrangement comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method according to any one of claims 1 to 4 when executing the computer program.
7. A storage medium, characterized in that the storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the steps of the method according to any one of claims 1 to 4.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810609360.XA CN108829521B (en) | 2018-06-13 | 2018-06-13 | Task processing method and device, computer equipment and storage medium |
| PCT/CN2018/109449 WO2019237593A1 (en) | 2018-06-13 | 2018-10-09 | Task processing method and apparatus, and computer device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810609360.XA CN108829521B (en) | 2018-06-13 | 2018-06-13 | Task processing method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108829521A CN108829521A (en) | 2018-11-16 |
| CN108829521B true CN108829521B (en) | 2022-05-13 |
Family
ID=64141353
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810609360.XA Active CN108829521B (en) | 2018-06-13 | 2018-06-13 | Task processing method and device, computer equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108829521B (en) |
| WO (1) | WO2019237593A1 (en) |
Families Citing this family (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109739529A (en) * | 2018-12-04 | 2019-05-10 | 贵阳朗玛信息技术股份有限公司 | A kind of method and device that program is issued automatically |
| CN111182061B (en) * | 2019-12-30 | 2023-04-14 | 苏宁云计算有限公司 | Task distribution processing method, system, computer device and storage medium |
| CN111240834B (en) * | 2020-01-02 | 2024-02-02 | 北京字节跳动网络技术有限公司 | Task execution method, device, electronic equipment and storage medium |
| CN111290846B (en) * | 2020-02-26 | 2023-08-18 | 杭州涂鸦信息技术有限公司 | Distributed task scheduling method and system |
| CN113411226A (en) * | 2020-03-16 | 2021-09-17 | 北京奇虎科技有限公司 | Router distributed automatic test method, system, storage medium and computer equipment thereof |
| CN111427694A (en) * | 2020-03-26 | 2020-07-17 | 北京金山云网络技术有限公司 | Task execution method, device, system and server |
| CN111831420B (en) * | 2020-07-20 | 2023-08-08 | 北京百度网讯科技有限公司 | Method for task scheduling, related device and computer program product |
| CN112181625A (en) * | 2020-10-12 | 2021-01-05 | 联通智网科技有限公司 | Task scheduling method, system, device, storage medium and computer equipment |
| CN112231092A (en) * | 2020-11-24 | 2021-01-15 | 政采云有限公司 | Timed task management method, device and equipment and readable storage medium |
| CN112667385A (en) * | 2021-01-15 | 2021-04-16 | 北京金和网络股份有限公司 | Cloud service system, task execution method and device thereof, and server |
| CN113722068A (en) * | 2021-08-31 | 2021-11-30 | 平安普惠企业管理有限公司 | Batch task processing method, device and equipment based on Azkaban and storage medium |
| CN113806058A (en) * | 2021-10-09 | 2021-12-17 | 京东科技控股股份有限公司 | Task management method and device, storage medium and electronic equipment |
| CN114253683B (en) | 2021-11-26 | 2022-11-01 | 北京百度网讯科技有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN114978929B (en) * | 2022-04-29 | 2023-08-18 | 苏州浪潮智能科技有限公司 | Network scheduling device and method |
| CN115601195B (en) * | 2022-10-17 | 2023-09-08 | 桂林电子科技大学 | Transaction bidirectional recommendation system and method based on real-time label of power user |
| CN117194549B (en) * | 2023-11-07 | 2024-01-26 | 上海柯林布瑞信息技术有限公司 | Data transmission method and device based on task data configuration |
Family Cites Families (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102402459B (en) * | 2010-09-10 | 2013-09-11 | 中兴通讯股份有限公司 | Method and device for summarizing performance data of network management system |
| CN102591721A (en) * | 2011-12-30 | 2012-07-18 | 北京新媒传信科技有限公司 | Method and system for distributing thread execution task |
| CN102902573B (en) * | 2012-09-20 | 2014-12-17 | 北京搜狐新媒体信息技术有限公司 | Task processing method and device based on shared resources |
| CN103902646B (en) * | 2013-12-27 | 2017-04-19 | 北京天融信软件有限公司 | Distributed task managing system and method |
| EP3262815B1 (en) * | 2015-02-24 | 2020-10-14 | Cisco Technology, Inc. | System and method for securing an enterprise computing environment |
| CN107239328B (en) * | 2016-03-29 | 2020-06-09 | 上海普兰金融服务有限公司 | Task allocation method and device |
| CN107918572A (en) * | 2016-10-08 | 2018-04-17 | 中兴通讯股份有限公司 | A kind of high-volume hard disk detection method, device and platform |
| WO2018068011A1 (en) * | 2016-10-09 | 2018-04-12 | The Charles Stark Draper Laboratory, Inc. | System for co-adaptive human-computer interaction |
| CN108090731B (en) * | 2016-11-23 | 2021-10-15 | 腾讯科技（深圳）有限公司 | Information processing method and device |
| CN107423120B (en) * | 2017-04-13 | 2020-06-30 | 阿里巴巴集团控股有限公司 | Task scheduling method and device |
| CN107220033A (en) * | 2017-07-05 | 2017-09-29 | 百度在线网络技术（北京）有限公司 | Method and apparatus for controlling thread pool thread quantity |
| CN107463439A (en) * | 2017-08-21 | 2017-12-12 | 山东浪潮通软信息科技有限公司 | A kind of thread pool implementation method and device |
| CN107748695B (en) * | 2017-10-24 | 2020-11-24 | 平安科技（深圳）有限公司 | Timed task processing method and device, storage medium and computer equipment |
| CN107704398A (en) * | 2017-11-01 | 2018-02-16 | 网易（杭州）网络有限公司 | Automated testing method and device, storage medium, electronic equipment |
| CN108011931B (en) * | 2017-11-22 | 2021-06-11 | 用友金融信息技术股份有限公司 | Web data acquisition method and Web data acquisition system |
-
2018
- 2018-06-13 CN CN201810609360.XA patent/CN108829521B/en active Active
- 2018-10-09 WO PCT/CN2018/109449 patent/WO2019237593A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| CN108829521A (en) | 2018-11-16 |
| WO2019237593A1 (en) | 2019-12-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108829521B (en) | Task processing method and device, computer equipment and storage medium | |
| CN109298990B (en) | Log storage method and device, computer equipment and storage medium | |
| US9448839B2 (en) | Backoff job queue polling mechanism | |
| CN107797853B (en) | Task scheduling method and device and multi-core processor | |
| US8881164B2 (en) | Computer process with utilization reduction | |
| US9875141B2 (en) | Managing pools of dynamic resources | |
| WO2017080273A1 (en) | Task management methods and system, and computer storage medium | |
| US10467054B2 (en) | Resource management method and system, and computer storage medium | |
| CN106598740B (en) | System and method for limiting CPU utilization rate occupied by multithreading program | |
| WO2014014944A2 (en) | Managing multi-threaded operations in a multimedia authoring environment | |
| CN103268247A (en) | Method and device for executing task and adjusting number of remaining threads in thread pool | |
| WO2021253851A1 (en) | Cluster distributed resource scheduling method, apparatus and device, and storage medium | |
| CN108762905B (en) | Method and device for processing multitask events | |
| CN109271290B (en) | Method and device for monitoring thread utilization rate and storage device | |
| CN111240864A (en) | Asynchronous task processing method, device, equipment and computer readable storage medium | |
| US20170160778A1 (en) | Method and device for monitoring power-consuming applications, user terminal, computer program and storage medium | |
| US9690626B2 (en) | Processing workloads in single-threaded environments | |
| CN111045825A (en) | Batch processing performance optimization method and device, computer equipment and storage medium | |
| CN110704206B (en) | Real-time computing method, computer storage medium and electronic equipment | |
| CN112860401A (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN107958414B (en) | Method and system for eliminating long transactions of CICS (common integrated circuit chip) system | |
| US9466042B2 (en) | Facilitating the design of information technology solutions | |
| CN103631571A (en) | Method and device for controlling progress bar | |
| CN110737320B (en) | Power consumption detection method and device, electronic equipment and storage medium | |
| CN113535354A (en) | Method and device for adjusting parallelism of Flink SQL operator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |