CN113448734B - SQL interrupt system and method under Spark SQL interactive scene - Google Patents

SQL interrupt system and method under Spark SQL interactive scene Download PDF

Info

Publication number
CN113448734B
CN113448734B CN202110799045.XA CN202110799045A CN113448734B CN 113448734 B CN113448734 B CN 113448734B CN 202110799045 A CN202110799045 A CN 202110799045A CN 113448734 B CN113448734 B CN 113448734B
Authority
CN
China
Prior art keywords
sql
task
subtask
interrupt
state
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110799045.XA
Other languages
Chinese (zh)
Other versions
CN113448734A (en
Inventor
田林
石慧彪
朱阿龙
王博瑞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bank of China Ltd
Original Assignee
Bank of China Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bank of China Ltd filed Critical Bank of China Ltd
Priority to CN202110799045.XA priority Critical patent/CN113448734B/en
Publication of CN113448734A publication Critical patent/CN113448734A/en
Application granted granted Critical
Publication of CN113448734B publication Critical patent/CN113448734B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention provides a SQL interrupt system and a method under Spark SQL interactive scene, which can be used in big data field, the system comprises: the instruction execution module is used for receiving the SQL task interrupt request; receiving an interrupt completion message and sending the interrupt completion message to a user; the task management module is used for acquiring SQL task identifiers from the SQL task interrupt requests; inquiring the state of the SQL task; when the state of the SQL task is the executing state, acquiring all SQL subtask identifiers under the SQL task; receiving a subtask interrupt completion message and sending the interrupt completion message to the instruction execution module; the subtask management module is used for receiving the SQL subtask identification; processing the subtasks; and after all SQL subtask processing is completed, sending a subtask interrupt completion message to the task management module. The invention can realize SQL interrupt in Spark SQL interactive scene.

Description

SQL interrupt system and method under Spark SQL interactive scene
Technical Field
The invention relates to the technical field of big data, in particular to an SQL interrupt system and method under Spark SQL interactive scene.
Background
Under Spark SQL interactive analysis scene, the SQL task submitted to the program can not be stopped freely, and the running Sql task continuously occupies server resources, so that the newly submitted task can not be distributed to the resource running. Therefore, there is a need for a method for SQL interrupt in Spark SQL interactive scenarios.
Disclosure of Invention
The embodiment of the invention provides an SQL interrupt system under Spark SQL interactive scene, which is used for realizing SQL interrupt under Spark SQL interactive scene, and comprises the following steps:
the instruction execution module is used for receiving the SQL task interrupt request sent by the user and sending the SQL task interrupt request to the task management module; receiving an interrupt completion message returned by the task management module and sending the interrupt completion message to a user;
the task management module is used for acquiring the SQL task identifier from the SQL task interrupt request after receiving the SQL task interrupt request; querying the state of the SQL task based on the SQL task identification; when the state of the SQL task is the executing state, acquiring all SQL subtask identifiers under the SQL task and sending the SQL subtask identifiers to a subtask management module; receiving a subtask interrupt completion message sent by a subtask management module, and sending the interrupt completion message to an instruction execution module;
the subtask management module is used for receiving the SQL subtask identification; executing task removing operation when the SQL subtasks are stored in a task pool to be executed, wherein the SQL subtasks submitted but needing to wait for the completion of the preconditions are stored in the task pool to be executed; executing task suspension operation when the SQL subtask is stored in an active task queue, releasing resources obtained by the SQL subtask from a resource thread pool, wherein the active task queue stores the SQL subtask being executed; executing a task rollback operation when the SQL subtasks are stored in a completed task pool, wherein the executed SQL subtasks under the SQL tasks are stored in the completed task pool; and after all SQL subtask processing is completed, sending a subtask interrupt completion message to the task management module.
The embodiment of the invention provides an SQL interrupt method under Spark SQL interactive scene, which is used for realizing SQL interrupt under Spark SQL interactive scene, and comprises the following steps:
receiving an SQL task interrupt request sent by a user;
acquiring an SQL task identifier from the SQL task interrupt request;
querying the state of the SQL task based on the SQL task identification;
when the state of the SQL task is the executing state, acquiring all SQL subtask identifiers under the SQL task;
executing task removing operation when the SQL subtasks are stored in a task pool to be executed, wherein the SQL subtasks submitted but needing to wait for the completion of the preconditions are stored in the task pool to be executed; executing task suspension operation when the SQL subtask is stored in an active task queue, releasing resources obtained by the SQL subtask from a resource thread pool, wherein the active task queue stores the SQL subtask being executed; executing a task rollback operation when the SQL subtasks are stored in a completed task pool, wherein the executed SQL subtasks under the SQL tasks are stored in the completed task pool;
after all SQL sub-tasks are processed, an interrupt completion message is sent to the user.
The embodiment of the invention also provides computer equipment, which comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the SQL interrupt method under the Spark SQL interactive scene is realized when the processor executes the computer program.
The embodiment of the invention also provides a computer readable storage medium, which stores a computer program for executing the SQL interrupt method under the Spark SQL interactive scene.
In the embodiment of the invention, the instruction execution module is used for receiving an SQL task interrupt request sent by a user and sending the SQL task interrupt request to the task management module; receiving an interrupt completion message returned by the task management module and sending the interrupt completion message to a user; the task management module is used for acquiring the SQL task identifier from the SQL task interrupt request after receiving the SQL task interrupt request; querying the state of the SQL task based on the SQL task identification; when the state of the SQL task is the executing state, acquiring all SQL subtask identifiers under the SQL task and sending the SQL subtask identifiers to a subtask management module; receiving a subtask interrupt completion message sent by a subtask management module, and sending the interrupt completion message to an instruction execution module; the subtask management module is used for receiving the SQL subtask identification; executing task removing operation when the SQL subtasks are stored in a task pool to be executed, wherein the SQL subtasks submitted but needing to wait for the completion of the preconditions are stored in the task pool to be executed; executing task suspension operation when the SQL subtask is stored in an active task queue, releasing resources obtained by the SQL subtask from a resource thread pool, wherein the active task queue stores the SQL subtask being executed; executing a task rollback operation when the SQL subtasks are stored in a completed task pool, wherein the executed SQL subtasks under the SQL tasks are stored in the completed task pool; and after all SQL subtask processing is completed, sending a subtask interrupt completion message to the task management module. In the process, the on-demand interruption of SQL tasks with different granularities can be effectively realized, so that server resources are released.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. In the drawings:
FIG. 1 is a schematic diagram of an SQL interrupt system in a Spark SQL interactive scenario in an embodiment of the invention;
FIG. 2 is a detailed flow chart of SQL interrupt in Spark SQL interactive scenario in accordance with an embodiment of the invention;
FIG. 3 is a flowchart of an SQL interrupt method in Spark SQL interactive scenario according to an embodiment of the invention;
FIG. 4 is a flow chart of SQL state queries in a Spark SQL interactive analysis scenario in accordance with an embodiment of the invention;
fig. 5 is a schematic diagram of a computer device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings. The exemplary embodiments of the present invention and their descriptions herein are for the purpose of explaining the present invention, but are not to be construed as limiting the invention.
In the description of the present specification, the terms "comprising," "including," "having," "containing," and the like are open-ended terms, meaning including, but not limited to. Reference to the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," etc., means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The sequence of steps involved in the embodiments is used to schematically illustrate the practice of the present application, and is not limited thereto and may be appropriately adjusted as desired.
FIG. 1 is a flowchart of an SQL interrupt system in Spark SQL interactive scenario according to an embodiment of the invention, as shown in FIG. 1, the system comprises:
the instruction execution module 101 is configured to receive an SQL task interrupt request sent by a user and send the SQL task interrupt request to the task management module; receiving an interrupt completion message returned by the task management module and sending the interrupt completion message to a user;
the task management module 102 is configured to obtain an SQL task identifier from the SQL task interrupt request after receiving the SQL task interrupt request; querying the state of the SQL task based on the SQL task identification; when the state of the SQL task is the executing state, acquiring all SQL subtask identifiers under the SQL task and sending the SQL subtask identifiers to a subtask management module; receiving a subtask interrupt completion message sent by a subtask management module, and sending the interrupt completion message to an instruction execution module;
a subtask management module 103, configured to receive an SQL subtask identifier; executing task removing operation when the SQL subtasks are stored in a task pool to be executed, wherein the SQL subtasks submitted but needing to wait for the completion of the preconditions are stored in the task pool to be executed; executing task suspension operation when the SQL subtask is stored in an active task queue, releasing resources obtained by the SQL subtask from a resource thread pool, wherein the active task queue stores the SQL subtask being executed; executing a task rollback operation when the SQL subtask is stored in a completed task pool, wherein the executed SQL subtask under the SQL task is stored in the completed task pool; and after all SQL subtask processing is completed, sending a subtask interrupt completion message to the task management module.
In the embodiment of the invention, the on-demand interruption of SQL tasks with different granularities can be effectively realized, so that server resources are released.
When the method is implemented, the instruction execution module is mainly interacted with a user, and the task management module is mainly used for processing the business with SQL task granularity; the subtask management module mainly processes SQL subtask granularity transactions.
For example, job is an SQL task, task1.1, task1.2, task1.3 are all SQL subtasks of job1, when task1.1 is executed and task1.2 is executing, task1.3 waits for task1.2 to be executed and then can be executed, task1.3 is stored in a task pool to be executed (task pool), task1.2 is stored in an active task queue (active task queue), and task1.1 is stored in a task pool to be completed (finish task pool).
In one embodiment, the task management module is specifically configured to:
and acquiring all SQL subtask identifiers under the SQL tasks from a cache server according to the SQL task identifiers, wherein the relationship between the SQL tasks and the SQL subtasks and the relationship between the SQL subtasks are stored in the cache server.
The task management module is further configured to: upon receiving the subtask interrupt completion message, the SQL task is removed from the cache server.
In an embodiment, the task management module is further configured to: when the state of the SQL task is the completed state, sending an SQL task completed message to the instruction execution module; when the state of the SQL task is the unexecuted state, all SQL subtask identifiers under the SQL task are obtained and sent to a subtask management module;
the instruction execution module is further to: and receiving the SQL task completed message and sending the SQL task completed message to the user.
In one embodiment, the instruction execution module is further to: receiving an SQL task query request sent by a user and sending the SQL task query request to a task manager; receiving the state of the SQL task sent by the task manager and sending the state to a user;
the task manager is further configured to: after receiving the SQL task query request, acquiring an SQL task identifier from the SQL task query request; querying the state of the SQL task based on the SQL task identification; and sending the state of the SQL task to the instruction execution module.
Based on the above embodiments, the present invention proposes the following embodiment to describe a detailed flow of the SQL interrupt in the Spark SQL interactive scenario, and fig. 2 is a detailed flow chart of the SQL interrupt in the Spark SQL interactive scenario in the embodiment of the present invention, as shown in fig. 2, including:
step 201, an instruction execution module receives an SQL task interrupt request sent by a user;
step 202, an instruction execution module sends an SQL task interrupt request to a task management module;
step 203, the task management module acquires an SQL task identifier from the SQL task interrupt request;
step 204, the task management module queries the state of the SQL task based on the SQL task identification;
step 205, when the state of the SQL task is the executing state, the task management module acquires all SQL subtask identifiers under the SQL task; turning to step 206, step 212 and step 214;
step 206, the task management module sends all subtask identifications under the SQL task to the subtask management module;
step 207, the subtask management module executes a task removal operation when the SQL subtask is stored in the task pool to be executed; executing task suspension operation when the SQL subtask is stored in the active task queue, and releasing resources obtained by the SQL subtask from the resource thread pool; executing task rollback operation when the SQL subtasks are stored in the completed task pool;
step 208, the subtask management module sends a subtask interrupt completion message to the task management module;
step 209, the task management module removes the SQL task from the cache server;
step 210, the task management module sends an interrupt completion message to the instruction execution module;
step 211, the instruction execution module sends an interrupt completion message to the user;
step 212, when the state of the SQL task is the completed state, the task management module sends an SQL task completed message to the instruction execution module;
step 213, the instruction execution module sends the SQL task completed message to the user;
step 214, the task management module obtains all SQL subtask identifiers under the SQL task when the state of the SQL task is the unexecuted state;
in step 215, the task management module sends all the SQL subtask identifiers under the SQL task to the subtask management module, and goes to step 207.
Of course, it is to be understood that other variations of the above detailed procedures are also possible, and all related variations should fall within the protection scope of the present invention.
In summary, in the system provided by the embodiment of the present invention, the instruction execution module is configured to receive an SQL task interrupt request sent by a user and send the SQL task interrupt request to the task management module; receiving an interrupt completion message returned by the task management module and sending the interrupt completion message to a user; the task management module is used for acquiring the SQL task identifier from the SQL task interrupt request after receiving the SQL task interrupt request; querying the state of the SQL task based on the SQL task identification; when the state of the SQL task is the executing state, acquiring all SQL subtask identifiers under the SQL task and sending the SQL subtask identifiers to a subtask management module; receiving a subtask interrupt completion message sent by a subtask management module, and sending the interrupt completion message to an instruction execution module; the subtask management module is used for receiving the SQL subtask identification; executing task removing operation when the SQL subtasks are stored in a task pool to be executed, wherein the SQL subtasks submitted but needing to wait for the completion of the preconditions are stored in the task pool to be executed; executing task suspension operation when the SQL subtask is stored in an active task queue, releasing resources obtained by the SQL subtask from a resource thread pool, wherein the active task queue stores the SQL subtask being executed; executing a task rollback operation when the SQL subtasks are stored in a completed task pool, wherein the executed SQL subtasks under the SQL tasks are stored in the completed task pool; and after all SQL subtask processing is completed, sending a subtask interrupt completion message to the task management module. In the process, the on-demand interruption of SQL tasks with different granularities can be effectively realized, so that server resources are released.
The embodiment of the invention also provides a Spark SQL interactive scene SQL interrupt method, the principle of which is similar to that of a Spark SQL interactive scene SQL interrupt system, and the description is omitted here.
FIG. 3 is a flowchart of an SQL interrupt method in Spark SQL interactive scenario according to an embodiment of the invention, as shown in FIG. 3, the method comprises:
step 301, receiving an SQL task interrupt request sent by a user;
step 302, acquiring an SQL task identifier from an SQL task interrupt request;
step 303, querying the state of the SQL task based on the SQL task identification;
step 304, when the state of the SQL task is the executing state, acquiring all SQL subtask identifiers under the SQL task;
step 305, executing a task removing operation when the SQL subtask is stored in a task pool to be executed, wherein the task pool to be executed stores the SQL subtask submitted but needs to wait for the completion of the precondition; executing task suspension operation when the SQL subtask is stored in an active task queue, releasing resources obtained by the SQL subtask from a resource thread pool, wherein the active task queue stores the SQL subtask being executed; executing a task rollback operation when the SQL subtasks are stored in a completed task pool, wherein the executed SQL subtasks under the SQL tasks are stored in the completed task pool;
and step 306, after all SQL sub-task processing is completed, sending an interrupt completion message to the user.
In one embodiment, obtaining all SQL sub-task identifications under an SQL task includes:
and acquiring all SQL subtask identifiers under the SQL tasks from a cache server according to the SQL task identifiers, wherein the relationship between the SQL tasks and the SQL subtasks and the relationship between the SQL subtasks are stored in the cache server.
In an embodiment, the method further comprises:
after all SQL sub-task processing is completed, the SQL task is removed from the cache server.
In an embodiment, the method further comprises:
when the state of the SQL task is the completed state, sending an SQL task completed message to the user;
when the state of the SQL task is the unexecuted state, all SQL subtask identifications under the SQL task are obtained, the task removing operation is executed on all SQL subtasks stored in the task pool to be executed, and an interrupt completion message is sent to a user.
FIG. 4 is a flowchart of SQL state query in Spark SQL interactive analysis scenario according to the embodiment of the invention, as shown in FIG. 4, comprising:
step 401, receiving an SQL task query request sent by a user;
step 402, acquiring an SQL task identifier from an SQL task query request;
step 403, querying the state of the SQL task based on the SQL task identification;
step 404, send the state of the SQL task to the user.
In summary, in the method provided by the embodiment of the present invention, the SQL task interrupt request sent by the user is received; acquiring an SQL task identifier from the SQL task interrupt request; querying the state of the SQL task based on the SQL task identification; when the state of the SQL task is the executing state, acquiring all SQL subtask identifiers under the SQL task; executing task removing operation when the SQL subtasks are stored in a task pool to be executed, wherein the SQL subtasks submitted but needing to wait for the completion of the preconditions are stored in the task pool to be executed; executing task suspension operation when the SQL subtask is stored in an active task queue, releasing resources obtained by the SQL subtask from a resource thread pool, wherein the active task queue stores the SQL subtask being executed; executing a task rollback operation when the SQL subtasks are stored in a completed task pool, wherein the executed SQL subtasks under the SQL tasks are stored in the completed task pool; after all SQL sub-tasks are processed, an interrupt completion message is sent to the user. In the process, the on-demand interruption of SQL tasks with different granularities can be effectively realized, so that server resources are released.
An embodiment of the present application further provides a computer device, and fig. 5 is a schematic diagram of a computer device in the embodiment of the present invention, where the computer device can implement all steps in the SQL interrupt method in the Spark SQL interactive scenario in the foregoing embodiment, and the computer device specifically includes the following contents:
a processor (processor) 501, a memory (memory) 502, a communication interface (Communications Interface) 503, and a communication bus 504;
wherein the processor 501, the memory 502, and the communication interface 503 perform communication with each other through the communication bus 504; the communication interface 503 is configured to implement information transmission between related devices such as a server device, a detection device, and a user device;
the processor 501 is configured to invoke a computer program in the memory 502, where the processor executes the computer program to implement all the steps in the SQL interrupt method in the Spark SQL interactive scenario in the above embodiment.
The embodiment of the application also provides a computer readable storage medium, which can realize all the steps of the SQL interrupt method in the Spark SQL interactive scene in the embodiment, and the computer readable storage medium stores a computer program which is executed by a processor to realize all the steps of the SQL interrupt method in the Spark SQL interactive scene in the embodiment.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing description of the embodiments has been provided for the purpose of illustrating the general principles of the invention, and is not meant to limit the scope of the invention, but to limit the invention to the particular embodiments, and any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the invention are intended to be included within the scope of the invention.

Claims (12)

1. A Spark SQL interrupt system in a Spark SQL interactive scenario, comprising:
the instruction execution module is used for receiving the SQL task interrupt request sent by the user and sending the SQL task interrupt request to the task management module; receiving an interrupt completion message returned by the task management module and sending the interrupt completion message to a user;
the task management module is used for acquiring the SQL task identifier from the SQL task interrupt request after receiving the SQL task interrupt request; querying the state of the SQL task based on the SQL task identification; when the state of the SQL task is the executing state, acquiring all SQL subtask identifiers under the SQL task and sending the SQL subtask identifiers to a subtask management module; receiving a subtask interrupt completion message sent by a subtask management module, and sending the interrupt completion message to an instruction execution module;
the subtask management module is used for receiving the SQL subtask identification; executing task removing operation when the SQL subtasks are stored in a task pool to be executed, wherein the SQL subtasks submitted but needing to wait for the completion of the preconditions are stored in the task pool to be executed; executing task suspension operation when the SQL subtask is stored in an active task queue, releasing resources obtained by the SQL subtask from a resource thread pool, wherein the active task queue stores the SQL subtask being executed; executing a task rollback operation when the SQL subtasks are stored in a completed task pool, wherein the executed SQL subtasks under the SQL tasks are stored in the completed task pool; and after all SQL subtask processing is completed, sending a subtask interrupt completion message to the task management module.
2. The Spark SQL interactive scene down SQL interrupt system according to claim 1, wherein the task management module is specifically configured to:
and acquiring all SQL subtask identifiers under the SQL tasks from a cache server according to the SQL task identifiers, wherein the relationship between the SQL tasks and the SQL subtasks and the relationship between the SQL subtasks are stored in the cache server.
3. The Spark SQL interactive under-scenario SQL interrupt system of claim 2, wherein the task management module is further to: upon receiving the subtask interrupt completion message, the SQL task is removed from the cache server.
4. The Spark SQL interactive under-scenario SQL interrupt system of claim 1, wherein the task management module is further to: when the state of the SQL task is the completed state, sending an SQL task completed message to the instruction execution module; when the state of the SQL task is the unexecuted state, all SQL subtask identifiers under the SQL task are obtained and sent to a subtask management module;
the instruction execution module is further to: and receiving the SQL task completed message and sending the SQL task completed message to the user.
5. The Spark SQL interactive under-scenario SQL interrupt system of claim 1, wherein the instruction execution module is further to: receiving an SQL task query request sent by a user and sending the SQL task query request to a task manager; receiving the state of the SQL task sent by the task manager and sending the state to a user;
the task manager is further configured to: after receiving the SQL task query request, acquiring an SQL task identifier from the SQL task query request; querying the state of the SQL task based on the SQL task identification; and sending the state of the SQL task to the instruction execution module.
6. The SQL interrupt method in the Spark SQL interactive scene is characterized by comprising the following steps:
receiving an SQL task interrupt request sent by a user;
acquiring an SQL task identifier from the SQL task interrupt request;
querying the state of the SQL task based on the SQL task identification;
when the state of the SQL task is the executing state, acquiring all SQL subtask identifiers under the SQL task;
executing task removing operation when the SQL subtasks are stored in a task pool to be executed, wherein the SQL subtasks submitted but needing to wait for the completion of the preconditions are stored in the task pool to be executed; executing task suspension operation when the SQL subtask is stored in an active task queue, releasing resources obtained by the SQL subtask from a resource thread pool, wherein the active task queue stores the SQL subtask being executed; executing a task rollback operation when the SQL subtasks are stored in a completed task pool, wherein the executed SQL subtasks under the SQL tasks are stored in the completed task pool;
after all SQL sub-tasks are processed, an interrupt completion message is sent to the user.
7. The method for the SQL interrupt in the Spark SQL interactive scene according to claim 6, wherein obtaining all SQL sub-task identifications in the SQL task comprises:
and acquiring all SQL subtask identifiers under the SQL tasks from a cache server according to the SQL task identifiers, wherein the relationship between the SQL tasks and the SQL subtasks and the relationship between the SQL subtasks are stored in the cache server.
8. The Spark SQL interactive under-scenario SQL interrupt method of claim 7, further comprising:
after all SQL sub-task processing is completed, the SQL task is removed from the cache server.
9. The Spark SQL interactive under-scenario SQL interrupt method of claim 6, further comprising:
when the state of the SQL task is the completed state, sending an SQL task completed message to the user;
when the state of the SQL task is the unexecuted state, all SQL subtask identifications under the SQL task are obtained, the task removing operation is executed on all SQL subtasks stored in the task pool to be executed, and an interrupt completion message is sent to a user.
10. The Spark SQL interactive under-scenario SQL interrupt method of claim 6, further comprising:
receiving an SQL task query request sent by a user;
acquiring an SQL task identifier from an SQL task query request;
querying the state of the SQL task based on the SQL task identification;
the state of the SQL task is sent to the user.
11. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 6 to 10 when executing the computer program.
12. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program for executing the method of any one of claims 6 to 10.
CN202110799045.XA 2021-07-15 2021-07-15 SQL interrupt system and method under Spark SQL interactive scene Active CN113448734B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110799045.XA CN113448734B (en) 2021-07-15 2021-07-15 SQL interrupt system and method under Spark SQL interactive scene

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110799045.XA CN113448734B (en) 2021-07-15 2021-07-15 SQL interrupt system and method under Spark SQL interactive scene

Publications (2)

Publication Number Publication Date
CN113448734A CN113448734A (en) 2021-09-28
CN113448734B true CN113448734B (en) 2024-02-23

Family

ID=77816349

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110799045.XA Active CN113448734B (en) 2021-07-15 2021-07-15 SQL interrupt system and method under Spark SQL interactive scene

Country Status (1)

Country Link
CN (1) CN113448734B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105989137A (en) * 2015-02-27 2016-10-05 中国移动通信集团河北有限公司 Structured query language performance optimization method and system
WO2018103318A1 (en) * 2016-12-06 2018-06-14 上海壹账通金融科技有限公司 Distributed transaction handling method and system
CN109189580A (en) * 2018-09-17 2019-01-11 武汉虹旭信息技术有限责任公司 A kind of multitask development model and its method based on multi-core platform
CN109558237A (en) * 2017-09-27 2019-04-02 北京国双科技有限公司 A kind of task status management method and device
CN111309372A (en) * 2020-01-15 2020-06-19 中国平安财产保险股份有限公司 Timed task execution method and device, computer equipment and storage medium
CN112486912A (en) * 2020-12-03 2021-03-12 北京普睿德利科技有限公司 File conversion system, method, electronic equipment and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105989137A (en) * 2015-02-27 2016-10-05 中国移动通信集团河北有限公司 Structured query language performance optimization method and system
WO2018103318A1 (en) * 2016-12-06 2018-06-14 上海壹账通金融科技有限公司 Distributed transaction handling method and system
CN109558237A (en) * 2017-09-27 2019-04-02 北京国双科技有限公司 A kind of task status management method and device
CN109189580A (en) * 2018-09-17 2019-01-11 武汉虹旭信息技术有限责任公司 A kind of multitask development model and its method based on multi-core platform
CN111309372A (en) * 2020-01-15 2020-06-19 中国平安财产保险股份有限公司 Timed task execution method and device, computer equipment and storage medium
CN112486912A (en) * 2020-12-03 2021-03-12 北京普睿德利科技有限公司 File conversion system, method, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN113448734A (en) 2021-09-28

Similar Documents

Publication Publication Date Title
CN107729139B (en) Method and device for concurrently acquiring resources
EP3567829B1 (en) Resource management method and apparatus
US9753783B2 (en) System and method for accelerating mapreduce operation
WO2017193737A1 (en) Software testing method and system
CN111367630A (en) Multi-user multi-priority distributed cooperative processing method based on cloud computing
CN111026541B (en) Rendering resource scheduling method, device, equipment and storage medium
CN103365718A (en) Thread scheduling method, thread scheduling device and multi-core processor system
CN108762905B (en) Method and device for processing multitask events
US8806497B2 (en) System and method for altering the work definitions in an iterative parallel opportunistic work stealing scheduler
WO2015131542A1 (en) Data processing method, device and system
CN110659131A (en) Task processing method, electronic device, computer device, and storage medium
CN110633145B (en) Real-time communication method and device in distributed system and distributed system
CN116795647A (en) Method, device, equipment and medium for managing and scheduling heterogeneous resources of database
CN113626173B (en) Scheduling method, scheduling device and storage medium
WO2022257247A1 (en) Data processing method and apparatus, and computer-readable storage medium
WO2019000790A1 (en) Method and device for calling remote procedure using synchronous mode
CN104702534A (en) Method and device for processing data of multi-process sharing port
CN113448734B (en) SQL interrupt system and method under Spark SQL interactive scene
CN107819855B (en) Message distribution method and device
CN112395062A (en) Task processing method, device, equipment and computer readable storage medium
CN113010280A (en) Distributed task processing method, system, device, equipment and medium
JP7122299B2 (en) Methods, apparatus, devices and storage media for performing processing tasks
CN113377360B (en) Task execution method, device, electronic equipment, storage medium and program product
CN112395063B (en) Dynamic multithreading scheduling method and system
CN114153573A (en) Multitasking method based on single thread pool and related device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant