CN113722198A - Script job submission control method and device, storage medium and electronic equipment - Google Patents
Script job submission control method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN113722198A CN113722198A CN202111027924.7A CN202111027924A CN113722198A CN 113722198 A CN113722198 A CN 113722198A CN 202111027924 A CN202111027924 A CN 202111027924A CN 113722198 A CN113722198 A CN 113722198A
- Authority
- CN
- China
- Prior art keywords
- script
- job
- running time
- sas system
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000004458 analytical method Methods 0.000 claims description 8
- 238000012544 monitoring process Methods 0.000 claims description 7
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 238000010801 machine learning Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3447—Performance evaluation by modeling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides a script job submission control method and device, a storage medium and an electronic device, and the method comprises the following steps: processing script operation to be submitted to the SAS system by utilizing a pre-constructed pre-estimation model to obtain the pre-estimated running time and operation type of the script operation, judging whether the script operation is low-efficiency operation or not based on the pre-estimated running time and the preset running time corresponding to the operation type, and if the script operation is low-efficiency operation, not submitting the script operation to the SAS system; and if the script operation is not the inefficient operation, submitting the script operation to the SAS system. Therefore, according to the scheme of the application, the script operation to be submitted to the SAS system is judged, so that the script operation is not submitted to the SAS system under the condition that the script operation is low-efficiency operation, the low-efficiency script operation is prevented from running in the SAS system, the memory utilization rate of the SAS system is reduced, and the system performance of the SAS system is improved.
Description
Technical Field
The present application relates to the field of SAS system application technologies, and in particular, to a script job submission control method and apparatus, a storage medium, and an electronic device.
Background
The SAS (Statistics Analysis System) is a large application software System that is modularized and integrated. The method has strong analysis and calculation functions, and is widely applied.
The SAS system has a large number of users, script operations written by business personnel are directly submitted to the SAS system, the script operation writing capability and quality of the business personnel are different, and when low-efficiency script operations occur, the business personnel can run in the SAS system for a long time, so that a large amount of computing resources are occupied, the memory utilization rate is increased, even the SAS system has low memory efficiency and cannot be forcibly stopped until the whole system becomes an unavailable state due to memory consumption, and the system performance is reduced.
Disclosure of Invention
The application provides a script job submission control method and device, a storage medium and electronic equipment, and aims to solve the problems that the memory utilization rate of an SAS (serial attached small computer system) is increased and the system performance is reduced due to the fact that script jobs are directly submitted to the SAS.
In order to achieve the above object, the present application provides the following technical solutions:
a script job submission control method, comprising:
acquiring script operation to be submitted to an SAS system;
processing the script operation by utilizing a pre-constructed pre-estimation model to obtain the pre-estimated running time and the operation category of the script operation; the pre-estimation model is pre-constructed based on a machine learning algorithm;
judging whether the script operation is inefficient operation or not based on the estimated operation time length and the target operation time length; the target operation duration is a preset operation duration corresponding to the operation type;
if the script operation is an inefficient operation, the script operation is not submitted to the SAS system; and if the script operation is not the inefficient operation, submitting the script operation to the SAS system.
Optionally, the method for acquiring the script job to be submitted to the SAS system includes:
monitoring whether script operation to be submitted to an SAS system exists in real time;
and when the script operation to be submitted to the SAS system is monitored, intercepting the script operation to obtain the script operation to be submitted to the SAS system.
Optionally, the method for processing the script job by using the pre-established pre-estimation model to obtain the pre-estimated running time and the job category of the script job includes:
performing keyword analysis on the script operation to obtain each keyword included in the script operation;
determining a target keyword from each keyword included in the script job; the target keyword is used for representing the operation category of the script operation;
determining a job category of the script job based on the target keyword;
and estimating the running time of the script operation based on each keyword included in the script operation to obtain the estimated running time of the script operation.
Optionally, the above method, where the estimating the running time of the script job based on each keyword included in the script job to obtain the estimated running time of the script job, includes:
acquiring preset duration corresponding to each keyword included in the script operation;
and calculating the estimated running time of the script operation based on the time corresponding to each keyword.
Optionally, the above method, wherein the determining whether the script job is an inefficient job based on the estimated running time and the target running time includes:
judging whether the estimated running time is longer than the target running time or not;
if the estimated running time is not greater than the target running time, determining that the script operation is not inefficient;
if the estimated running time length is longer than the target running time length, judging whether the difference value between the estimated running time length and the target running time length is larger than a preset threshold value or not;
if the difference value between the estimated running time length and the target running time length is larger than a preset threshold value, determining the script operation to be inefficient operation;
and if the difference value between the estimated running time length and the target running time length is not greater than a preset threshold value, determining that the script operation is not inefficient.
A script job submission control apparatus comprising:
the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring script jobs to be submitted to the SAS system;
the processing unit is used for processing the script operation by utilizing a pre-established pre-estimation model to obtain the pre-estimated running time and the operation type of the script operation; the pre-estimation model is pre-constructed based on a machine learning algorithm;
the judging unit is used for judging whether the script operation is inefficient operation or not based on the estimated operation time length and the target operation time length; the target operation duration is a preset operation duration corresponding to the operation type;
a submitting unit, configured to not submit the script job to the SAS system if the script job is an inefficient job; and if the script operation is not the inefficient operation, submitting the script operation to the SAS system.
Optionally, the above apparatus, wherein the obtaining unit is specifically configured to:
monitoring whether script operation to be submitted to an SAS system exists in real time;
and when the script operation to be submitted to the SAS system is monitored, intercepting the script operation to obtain the script operation to be submitted to the SAS system.
Optionally, the above apparatus, wherein the processing unit is specifically configured to:
performing keyword analysis on the script operation to obtain each keyword included in the script operation;
determining a target keyword from each keyword included in the script job; the target keyword is used for representing the operation category of the script operation;
determining a job category of the script job based on the target keyword;
and estimating the running time of the script operation based on each keyword included in the script operation to obtain the estimated running time of the script operation.
A storage medium storing a set of instructions which, when executed by a processor, implement a script job submission control method as described above.
An electronic device, comprising:
a memory for storing at least one set of instructions;
and the processor is used for executing the instruction set stored in the memory and realizing the script job submission control method by executing the instruction set.
Compared with the prior art, the method has the following advantages:
the application provides a script job submission control method and device, a storage medium and an electronic device, and the method comprises the following steps: processing script operation to be submitted to the SAS system by utilizing a pre-constructed pre-estimation model to obtain the pre-estimated running time and operation type of the script operation, judging whether the script operation is low-efficiency operation or not based on the pre-estimated running time and the preset running time corresponding to the operation type, and if the script operation is low-efficiency operation, not submitting the script operation to the SAS system; and if the script operation is not the inefficient operation, submitting the script operation to the SAS system. Therefore, according to the scheme of the application, the script operation to be submitted to the SAS system is judged, so that the script operation is not submitted to the SAS system under the condition that the script operation is low-efficiency operation, the low-efficiency script operation is prevented from running in the SAS system, the memory utilization rate of the SAS system is reduced, and the system performance of the SAS system is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flowchart of a method for controlling submission of a script job provided herein;
FIG. 2 is a flowchart of another method of controlling submission of a script job provided herein;
FIG. 3 is a flowchart of another method of controlling script job submission according to the present application;
FIG. 4 is a flowchart of another method of controlling submission of a script job provided herein;
FIG. 5 is a diagram illustrating an example of a script job submission control method provided herein;
fig. 6 is a schematic structural diagram of a script job submission control device provided in the present application;
fig. 7 is a schematic structural diagram of an electronic device provided in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the disclosure of the present application are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in the disclosure herein are exemplary rather than limiting, and those skilled in the art will understand that "one or more" will be understood unless the context clearly dictates otherwise.
In the existing script operation submitting control scheme, script operations compiled by business personnel are directly submitted to an SAS system, the script operation compiling capacity and quality of the business personnel are different, and when low-efficiency script operations occur, the business personnel can run in the SAS system for a long time, so that a large amount of computing resources are occupied, the memory utilization rate is increased, even the SAS system is caused to have low memory efficiency and cannot be forcibly stopped until the whole system becomes an unavailable state due to memory consumption, and the system performance is reduced. For example, when a user submits a simple SQL query statement to perform a full table search and a specified table search, the required operation time will be very different, especially for SAS systems with a large number of users, the data size of connected GP (greenplus, data warehouse) is very large, the number scale of tables is very large, the full table search without restriction will occupy a large amount of computing resources, if some closed loop statements are inadvertently submitted to a cluster to operate, the system memory will be greatly occupied, which seriously results in a monitoring alarm, and the whole system is in an unavailable state.
Therefore, the application provides a script job submission control method and device, a storage medium and an electronic device, which can solve the problems that the memory utilization rate of an SAS system is increased and the system performance is reduced due to the fact that script jobs are directly submitted to the SAS system.
An embodiment of the present application provides a script job submission control method, where a method flowchart of the script job submission control method is shown in fig. 1, and specifically includes:
s101, obtaining script operation to be submitted to the SAS system.
In this embodiment, a script job to be submitted to the SAS system is obtained, and specifically, the script job to be submitted to the SAS system is intercepted, so that the script job to be submitted to the SAS system is obtained.
Referring to fig. 2, the process of obtaining the script job to be submitted to the SAS system specifically includes the following steps:
s201, monitoring whether script operation to be submitted to the SAS system exists in real time.
In this embodiment, whether the script job to be submitted to the SAS system exists is monitored in real time, specifically, a monitoring program is preset, and an interface of the SAS system is monitored by the monitoring program to determine whether the script job to be submitted to the SAS system exists.
In this embodiment, when a script file sent to an interface of the SAS system is monitored, it is determined that a script job to be submitted to the SAS system exists, and otherwise, it is determined that a script job submitted to the SAS system does not exist.
S202, when the script operation to be submitted to the SAS system is monitored to exist, the script operation is intercepted, and the script operation to be submitted to the SAS system is obtained.
In this embodiment, when it is monitored that the script job to be submitted to the SAS system exists, the script job is intercepted outside the interface of the SAS system, so that the script job to be submitted to the SAS system is obtained.
S102, processing the script operation by using the pre-established pre-estimation model to obtain the pre-estimated running time and the operation type of the script operation.
In this embodiment, the estimation model is pre-constructed, and the estimation model is pre-constructed based on a machine learning algorithm.
In this embodiment, based on the pre-built pre-estimation model, the script job is processed to obtain the pre-estimated running time and job category of the script job, that is, the script job is input into the pre-built pre-estimation model to obtain the pre-estimated running time and job category of the script job output by the pre-estimation model.
The job type is used to describe a function realized by the script job.
Referring to fig. 3, the process of processing script jobs by using a pre-established pre-estimation model to obtain the pre-estimated running time and job category of the script jobs specifically includes the following steps:
s301, carrying out keyword analysis on the script operation to obtain each keyword included in the script operation.
In this embodiment, according to a preset parsing rule, keyword parsing is performed on the script job to obtain each keyword included in the script job. Keywords include, but are not limited to, select, and/or delete, and/or insert.
S302, determining target keywords from the keywords included in the script operation; the target keyword is used to characterize a job category of the script job.
In this embodiment, a preset keyword representing each type of job category is obtained.
In this embodiment, a target keyword is determined from keywords included in the script job, specifically, for each preset keyword, each keyword included in the script job is matched with the preset keyword, and if there is a keyword matched with the preset keyword, the keyword matched with the preset keyword is determined as the target keyword.
S303, based on the target keyword, determining the job type of the script job.
In this embodiment, the job type of the script job is determined based on the target keyword, that is, the target keyword is used as the job type of the script job.
S304, estimating the running time of the script operation based on each keyword included in the script operation to obtain the estimated running time of the script operation.
In this embodiment, the duration corresponding to the keyword is preset, and the running duration of the script operation is estimated based on each keyword included in the script operation and the duration corresponding to the preset keyword, so as to obtain the estimated running duration of the script operation.
Specifically, the process of estimating the running time of the script operation to obtain the estimated running time of the script operation based on each keyword included in the script operation specifically includes the following steps:
acquiring preset duration corresponding to each keyword included in the script operation;
and calculating the estimated running time of the script operation based on the time corresponding to each keyword.
In this embodiment, the preset duration corresponding to each keyword included in the script operation is obtained based on each keyword included in the script operation, and the estimated running duration of the script operation is calculated based on the duration corresponding to each keyword.
S103, judging whether the script operation is an inefficient operation or not based on the estimated operation time and the target operation time, if so, executing S104, and if not, executing S105.
In this embodiment, the preset operation duration corresponding to each type of job category is preset, where the specific setting process includes: the method comprises the steps of obtaining the running time of a historical job script corresponding to each job category, calculating the average value of the running time of the historical script corresponding to the job category aiming at each job category, and taking the calculated average value as the preset running time corresponding to the job category.
In this embodiment, whether the script operation is an inefficient operation is determined based on the estimated operation time and the target operation time, where the target operation time is a preset operation time corresponding to the operation type.
Referring to fig. 4, the process of determining whether the script job is an inefficient job based on the estimated running time and the target running time includes the following steps:
s401, judging whether the estimated running time is longer than the target running time, if not, executing S402, and if so, executing S403.
And judging whether the estimated running time is longer than the target running time, and specifically, comparing the estimated running time with the target running time.
S402, determining that the script operation is not an inefficient operation.
In this embodiment, if the estimated running time is not greater than the target running time, it is determined that the script job is not an inefficient job.
In this embodiment, if the estimated running time is longer than the target running time, but the difference between the estimated running time and the target running time is not greater than the preset threshold, it is determined that the script operation is not an inefficient operation.
And S403, judging whether the difference value between the estimated running time length and the target running time length is larger than a preset threshold value, if so, executing S404, and if not, executing S402.
In this embodiment, if the estimated running time is longer than the target running time, it is further determined whether a difference between the estimated running time and the target running time is greater than a preset threshold. It should be noted that the preset threshold may be adjusted according to the requirement.
S404, determining the script operation as an inefficient operation.
In this embodiment, if the estimated running time is longer than the target running time, and the difference between the estimated running time and the target running time is greater than the preset threshold, it is determined that the estimated running time of the script operation is too long, and the script operation is determined to be an inefficient operation.
And S104, not submitting the script operation to the SAS system.
In this embodiment, if it is determined that the script job is an inefficient job, the script job is limited to be submitted to the SAS system, so as to avoid the inefficient job from running in the SAS system.
And S105, submitting the script operation to the SAS system.
In this embodiment, if it is determined that the script job is not an inefficient job, the script job is submitted to the SAS system.
In this embodiment, the SAS system includes a management node and a plurality of computing nodes, where a management node in the SAS system receives a script job, randomly distributes the script job to any one of the computing nodes, and the computing node runs the script job.
It should be noted that one compute node can run multiple script jobs, and one script job can only be run by one compute node.
The script job submission control method provided by the embodiment of the application processes script jobs to be submitted to an SAS system by utilizing a pre-established pre-estimation model to obtain pre-estimated running time and job types of the script jobs, judges whether the script jobs are inefficient jobs or not based on the pre-estimated running time and preset running time corresponding to the job types, and does not submit the script jobs to the SAS system if the script jobs are inefficient jobs; and if the script operation is not the inefficient operation, submitting the script operation to the SAS system. By applying the script job submission control method provided by the embodiment of the application, the script job to be submitted to the SAS system is judged, so that the script job is not submitted to the SAS system under the condition that the script job is an inefficient job, thereby avoiding the inefficient script job from running in the SAS system, reducing the memory utilization rate of the SAS system and further improving the system performance of the SAS system.
Referring to fig. 5, the script job submission control method product mentioned in the embodiment of the present application is illustrated as follows:
after compiling a job (namely a script job), each user submits the compiled job to an SAS system, namely the user 1 submits the job 1 after compiling the job 1, the user 2 submits the job 2 after compiling the job 2, the user 3 submits the job 3 after compiling the job 3, a job time pre-estimating device intercepts the job under the condition that the job sent to the SAS system is monitored, estimates the running time of the job aiming at each job, obtains the estimated running time of the job and determines the job type of the job, judges whether the job is an inefficient job or not based on the estimated running time of the job and the preset running time corresponding to the job type, if the job is an inefficient job, the job is not submitted to the SAS system, if the job is not an inefficient job, the job is submitted to the SAS system, a management node in the SAS system receives the job, and randomly distributing the job to any one computing node, and running the job by the computing node.
It should be noted that while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous.
It should be understood that the various steps recited in the method embodiments disclosed herein may be performed in a different order and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the disclosure is not limited in this respect.
Corresponding to the method described in fig. 1, an embodiment of the present application further provides a script job submission control device, which is used for implementing the method in fig. 1 specifically, and a schematic structural diagram of the script job submission control device is shown in fig. 6, and specifically includes:
an obtaining unit 601, configured to obtain a script job to be submitted to an SAS system;
the processing unit 602 is configured to process the script job by using a pre-established pre-estimation model to obtain a pre-estimated running time and a job category of the script job; the pre-estimation model is pre-constructed based on a machine learning algorithm;
a determining unit 603, configured to determine whether the script job is an inefficient job based on the estimated running time and the target running time; the target operation duration is a preset operation duration corresponding to the operation type;
a submitting unit 604, configured to not submit the script job to the SAS system if the script job is an inefficient job; and if the script operation is not the inefficient operation, submitting the script operation to the SAS system.
The script job submission control device provided by the embodiment of the application processes script jobs to be submitted to an SAS system by using a pre-established pre-estimation model to obtain pre-estimated running time and job types of the script jobs, judges whether the script jobs are inefficient jobs or not based on the pre-estimated running time and preset running time corresponding to the job types, and does not submit the script jobs to the SAS system if the script jobs are inefficient jobs; and if the script operation is not the inefficient operation, submitting the script operation to the SAS system. By applying the script job submission control device provided by the embodiment of the application, the script job to be submitted to the SAS system is judged, so that the script job is not submitted to the SAS system under the condition that the script job is an inefficient job, thereby avoiding the inefficient script job from running in the SAS system, reducing the memory utilization rate of the SAS system and further improving the system performance of the SAS system.
In an embodiment of the present application, based on the foregoing scheme, the obtaining unit 601 is specifically configured to:
monitoring whether script operation to be submitted to an SAS system exists in real time;
and when the script operation to be submitted to the SAS system is monitored, intercepting the script operation to obtain the script operation to be submitted to the SAS system.
In an embodiment of the present application, based on the foregoing scheme, the processing unit 602 is specifically configured to:
performing keyword analysis on the script operation to obtain each keyword included in the script operation;
determining a target keyword from each keyword included in the script job; the target keyword is used for representing the operation category of the script operation;
determining a job category of the script job based on the target keyword;
and estimating the running time of the script operation based on each keyword included in the script operation to obtain the estimated running time of the script operation.
In an embodiment of the present application, based on the foregoing solution, the processing unit 602, when predicting the running duration of the script job based on each keyword included in the script job, and obtaining the predicted running duration of the script job, is specifically configured to:
acquiring preset duration corresponding to each keyword included in the script operation;
and calculating the estimated running time of the script operation based on the time corresponding to each keyword.
In an embodiment of the present application, based on the foregoing scheme, the determining unit 603 is specifically configured to:
judging whether the estimated running time is longer than the target running time or not;
if the estimated running time is not greater than the target running time, determining that the script operation is not inefficient;
if the estimated running time length is longer than the target running time length, judging whether the difference value between the estimated running time length and the target running time length is larger than a preset threshold value or not;
if the difference value between the estimated running time length and the target running time length is larger than a preset threshold value, determining the script operation to be inefficient operation;
and if the difference value between the estimated running time length and the target running time length is not greater than a preset threshold value, determining that the script operation is not inefficient.
The embodiment of the present application further provides a storage medium, where an instruction set is stored in the storage medium, where the script job submission control method disclosed in any of the above embodiments is executed when the instruction set is executed.
An electronic device is further provided in the embodiments of the present application, and a schematic structural diagram of the electronic device is shown in fig. 7, and specifically includes a memory 701 configured to store at least one set of instruction sets; a processor 702 for executing the instruction set stored in the memory, and implementing the script job submission control method as disclosed in any of the above embodiments by executing the instruction set.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
While several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
The foregoing description is only exemplary of the preferred embodiments disclosed herein and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the disclosure. For example, the above features and (but not limited to) technical features having similar functions disclosed in the present disclosure are mutually replaced to form the technical solution.
Claims (10)
1. A script job submission control method, comprising:
acquiring script operation to be submitted to an SAS system;
processing the script operation by utilizing a pre-constructed pre-estimation model to obtain the pre-estimated running time and the operation category of the script operation; the pre-estimation model is pre-constructed based on a machine learning algorithm;
judging whether the script operation is inefficient operation or not based on the estimated operation time length and the target operation time length; the target operation duration is a preset operation duration corresponding to the operation type;
if the script operation is an inefficient operation, the script operation is not submitted to the SAS system; and if the script operation is not the inefficient operation, submitting the script operation to the SAS system.
2. The method of claim 1, wherein obtaining the script job to be submitted to the SAS system comprises:
monitoring whether script operation to be submitted to an SAS system exists in real time;
and when the script operation to be submitted to the SAS system is monitored, intercepting the script operation to obtain the script operation to be submitted to the SAS system.
3. The method of claim 1, wherein processing the script job using a pre-built pre-estimated model to obtain an estimated run-time and job category of the script job comprises:
performing keyword analysis on the script operation to obtain each keyword included in the script operation;
determining a target keyword from each keyword included in the script job; the target keyword is used for representing the operation category of the script operation;
determining a job category of the script job based on the target keyword;
and estimating the running time of the script operation based on each keyword included in the script operation to obtain the estimated running time of the script operation.
4. The method according to claim 3, wherein the estimating the running time of the script job based on each keyword included in the script job to obtain the estimated running time of the script job comprises:
acquiring preset duration corresponding to each keyword included in the script operation;
and calculating the estimated running time of the script operation based on the time corresponding to each keyword.
5. The method of claim 1, wherein determining whether the scripted job is an inefficient job based on the estimated run length and a target run length comprises:
judging whether the estimated running time is longer than the target running time or not;
if the estimated running time is not greater than the target running time, determining that the script operation is not inefficient;
if the estimated running time length is longer than the target running time length, judging whether the difference value between the estimated running time length and the target running time length is larger than a preset threshold value or not;
if the difference value between the estimated running time length and the target running time length is larger than a preset threshold value, determining the script operation to be inefficient operation;
and if the difference value between the estimated running time length and the target running time length is not greater than a preset threshold value, determining that the script operation is not inefficient.
6. A script job submission control apparatus, comprising:
the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring script jobs to be submitted to the SAS system;
the processing unit is used for processing the script operation by utilizing a pre-established pre-estimation model to obtain the pre-estimated running time and the operation type of the script operation; the pre-estimation model is pre-constructed based on a machine learning algorithm;
the judging unit is used for judging whether the script operation is inefficient operation or not based on the estimated operation time length and the target operation time length; the target operation duration is a preset operation duration corresponding to the operation type;
a submitting unit, configured to not submit the script job to the SAS system if the script job is an inefficient job; and if the script operation is not the inefficient operation, submitting the script operation to the SAS system.
7. The apparatus according to claim 6, wherein the obtaining unit is specifically configured to:
monitoring whether script operation to be submitted to an SAS system exists in real time;
and when the script operation to be submitted to the SAS system is monitored, intercepting the script operation to obtain the script operation to be submitted to the SAS system.
8. The apparatus according to claim 6, wherein the processing unit is specifically configured to:
performing keyword analysis on the script operation to obtain each keyword included in the script operation;
determining a target keyword from each keyword included in the script job; the target keyword is used for representing the operation category of the script operation;
determining a job category of the script job based on the target keyword;
and estimating the running time of the script operation based on each keyword included in the script operation to obtain the estimated running time of the script operation.
9. A storage medium storing a set of instructions, wherein the set of instructions, when executed by a processor, implement the script job submission control method of any one of claims 1-5.
10. An electronic device, comprising:
a memory for storing at least one set of instructions;
a processor for executing the instruction set stored in the memory, the script job submission control method according to any one of claims 1 to 5 being implemented by executing the instruction set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111027924.7A CN113722198A (en) | 2021-09-02 | 2021-09-02 | Script job submission control method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111027924.7A CN113722198A (en) | 2021-09-02 | 2021-09-02 | Script job submission control method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113722198A true CN113722198A (en) | 2021-11-30 |
Family
ID=78681107
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111027924.7A Pending CN113722198A (en) | 2021-09-02 | 2021-09-02 | Script job submission control method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113722198A (en) |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080072230A1 (en) * | 2004-11-08 | 2008-03-20 | Cluster Resources, Inc. | System and Method of Providing System Jobs Within a Compute Environment |
| CN106844040A (en) * | 2016-12-20 | 2017-06-13 | 北京并行科技股份有限公司 | A kind of job submission method, system and server |
| CN108628727A (en) * | 2018-04-19 | 2018-10-09 | 山东省计算中心（国家超级计算济南中心） | It is a kind of based on mode operation when feature mode operations running state analysis method |
| CN110297701A (en) * | 2019-05-16 | 2019-10-01 | 平安科技（深圳）有限公司 | Data processing operation dispatching method, device, computer equipment and storage medium |
| US10452441B1 (en) * | 2018-10-15 | 2019-10-22 | Accenture Global Solutions Limited | Determining an allocation of computing resources for a job |
| CN110377519A (en) * | 2019-07-22 | 2019-10-25 | 中国工商银行股份有限公司 | Performance capability test method, device, equipment and the storage medium of big data system |
| CN110784374A (en) * | 2019-10-25 | 2020-02-11 | 上海中通吉网络技术有限公司 | Method, device, equipment and system for monitoring operation state of service system |
| US20200310874A1 (en) * | 2019-03-25 | 2020-10-01 | Fujitsu Limited | Job scheduler, job schedule control method, and storage medium |
| CN111797157A (en) * | 2020-07-21 | 2020-10-20 | 政采云有限公司 | Data processing method and system, electronic equipment and storage medium |
| CN111897707A (en) * | 2020-07-16 | 2020-11-06 | 中国工商银行股份有限公司 | Method and device for optimizing business system, computer system and storage medium |
| CN112052133A (en) * | 2019-06-06 | 2020-12-08 | 北京京东尚科信息技术有限公司 | Service system monitoring method and device based on Kubernetes |
| CN112181498A (en) * | 2020-10-09 | 2021-01-05 | 中国工商银行股份有限公司 | Concurrency control method, device and equipment |
| WO2021051531A1 (en) * | 2019-09-19 | 2021-03-25 | 平安科技（深圳）有限公司 | Method and apparatus for processing multi-cluster job record, and device and storage medium |
| CN112965820A (en) * | 2021-03-05 | 2021-06-15 | 中国建设银行股份有限公司 | Cluster job scheduling method, device, equipment and storage medium |
-
2021
- 2021-09-02 CN CN202111027924.7A patent/CN113722198A/en active Pending
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080072230A1 (en) * | 2004-11-08 | 2008-03-20 | Cluster Resources, Inc. | System and Method of Providing System Jobs Within a Compute Environment |
| CN106844040A (en) * | 2016-12-20 | 2017-06-13 | 北京并行科技股份有限公司 | A kind of job submission method, system and server |
| CN108628727A (en) * | 2018-04-19 | 2018-10-09 | 山东省计算中心（国家超级计算济南中心） | It is a kind of based on mode operation when feature mode operations running state analysis method |
| US10452441B1 (en) * | 2018-10-15 | 2019-10-22 | Accenture Global Solutions Limited | Determining an allocation of computing resources for a job |
| US20200310874A1 (en) * | 2019-03-25 | 2020-10-01 | Fujitsu Limited | Job scheduler, job schedule control method, and storage medium |
| CN110297701A (en) * | 2019-05-16 | 2019-10-01 | 平安科技（深圳）有限公司 | Data processing operation dispatching method, device, computer equipment and storage medium |
| CN112052133A (en) * | 2019-06-06 | 2020-12-08 | 北京京东尚科信息技术有限公司 | Service system monitoring method and device based on Kubernetes |
| CN110377519A (en) * | 2019-07-22 | 2019-10-25 | 中国工商银行股份有限公司 | Performance capability test method, device, equipment and the storage medium of big data system |
| WO2021051531A1 (en) * | 2019-09-19 | 2021-03-25 | 平安科技（深圳）有限公司 | Method and apparatus for processing multi-cluster job record, and device and storage medium |
| CN110784374A (en) * | 2019-10-25 | 2020-02-11 | 上海中通吉网络技术有限公司 | Method, device, equipment and system for monitoring operation state of service system |
| CN111897707A (en) * | 2020-07-16 | 2020-11-06 | 中国工商银行股份有限公司 | Method and device for optimizing business system, computer system and storage medium |
| CN111797157A (en) * | 2020-07-21 | 2020-10-20 | 政采云有限公司 | Data processing method and system, electronic equipment and storage medium |
| CN112181498A (en) * | 2020-10-09 | 2021-01-05 | 中国工商银行股份有限公司 | Concurrency control method, device and equipment |
| CN112965820A (en) * | 2021-03-05 | 2021-06-15 | 中国建设银行股份有限公司 | Cluster job scheduling method, device, equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 卢慧;高弘博;张丰满;王梅;肖震;: "Hadoop云平台下基于资源预估的作业调度算法", 计算机应用研究, no. 08, 31 August 2016 (2016-08-31), pages 2311 - 2314 * |
| 李观金;: "基于Web的在线作业管理系统研究与开发", 电脑知识与技术, no. 01, 5 January 2018 (2018-01-05) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230359516A1 (en) | Query watchdog | |
| CN108089921B (en) | Server for cloud big data operation architecture and operation resource optimization method thereof | |
| CN106528266B (en) | Method and device for dynamically adjusting resources in cloud computing system | |
| US7137115B2 (en) | Method for controlling multithreading | |
| CN108205469B (en) | MapReduce-based resource allocation method and server | |
| CN105701257A (en) | Data processing method and device | |
| CN115373835A (en) | Task resource adjusting method and device for Flink cluster and electronic equipment | |
| CN111209077A (en) | Deep learning framework design method | |
| CN103942197A (en) | Data monitoring processing method and device | |
| CN106020984B (en) | Method and device for creating process in electronic equipment | |
| Li et al. | A task allocation method for stream processing with recovery latency constraint | |
| CN111885184A (en) | Method and device for processing hot spot access keywords in high concurrency scene | |
| US8819239B2 (en) | Distributed resource management systems and methods for resource management thereof | |
| CN110955390B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN116089477B (en) | Distributed training method and system | |
| CN113722198A (en) | Script job submission control method and device, storage medium and electronic equipment | |
| CN110728372B (en) | Cluster design method and cluster system for dynamic loading of artificial intelligent model | |
| CN112615795A (en) | Flow control method and device, electronic equipment, storage medium and product | |
| Beltrán et al. | How to balance the load on heterogeneous clusters | |
| KR20230063015A (en) | Apparatus and method for managing virtual machine cpu resource in virtualization server | |
| CN113742059A (en) | Task allocation method and device, computer equipment and storage medium | |
| CN114443262A (en) | Computing resource management method, device, equipment and system | |
| CN115904640A (en) | Distributed task processing system and method | |
| Wang et al. | An adaptive elasticity policy for staging based in-situ processing | |
| US20190138931A1 (en) | Apparatus and method of introducing probability and uncertainty via order statistics to unsupervised data classification via clustering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |