CN117520631A - Z+ operation optimization method and device based on big data Z+ platform - Google Patents
Z+ operation optimization method and device based on big data Z+ platform Download PDFInfo
- Publication number
- CN117520631A CN117520631A CN202311484065.3A CN202311484065A CN117520631A CN 117520631 A CN117520631 A CN 117520631A CN 202311484065 A CN202311484065 A CN 202311484065A CN 117520631 A CN117520631 A CN 117520631A
- Authority
- CN
- China
- Prior art keywords
- target
- job
- platform
- big data
- optimized
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005457 optimization Methods 0.000 title claims abstract description 76
- 238000000034 method Methods 0.000 title claims abstract description 70
- 238000012360 testing method Methods 0.000 claims abstract description 23
- 238000012795 verification Methods 0.000 claims abstract description 23
- 238000004458 analytical method Methods 0.000 claims description 26
- 238000012545 processing Methods 0.000 claims description 21
- 238000009826 distribution Methods 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 9
- 230000002787 reinforcement Effects 0.000 claims description 7
- 238000003860 storage Methods 0.000 claims description 7
- 238000012549 training Methods 0.000 claims description 7
- 230000000153 supplemental effect Effects 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 11
- 230000009471 action Effects 0.000 description 7
- 230000000694 effects Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000011161 development Methods 0.000 description 5
- 238000010276 construction Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000000047 product Substances 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000006866 deterioration Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008014 freezing Effects 0.000 description 1
- 238000007710 freezing Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9532—Query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application provides a Z+ operation optimization method and device based on a big data Z+ platform, wherein the method comprises the following steps: acquiring Z+ operation to be optimized; optimizing the Z+ operation to obtain an optimized target Z+ operation; judging whether the target Z+ operation passes the test verification or not; if so, the target Z+ job is configured to the big data Z+ platform. Therefore, the method and the device can automatically realize Z+ operation optimization without manual participation, improve the efficiency of Z+ operation performance optimization, simultaneously reduce the operation threshold, reduce the input resources of hardware and save precious human resources.
Description
Technical Field
The application relates to the technical field of data processing, in particular to a Z+ operation optimization method and device based on a big data Z+ platform.
Background
At present, in the financial field, a big data Z+ platform is generally adopted for data processing, and a large number of data processing operations are currently carried out on the platform, which reaches more than 35 ten thousand. In daily inspection, part of the jobs are very slow to execute, and the individual jobs need to be executed for more than 6 hours, so that not only is great pressure on the computing resources of the platform caused, but also the execution timeliness of other key jobs is seriously influenced, and the service experience is greatly reduced. The existing Z+ operation performance optimization processing method is usually to manually optimize the performance of the operation by a development responsible person. In practice, it is found that the existing method generally needs to be an optimization method for designing HQL by a development responsible person, which reduces experience of service usage data. Moreover, due to the professional property of HQL optimization, developers often invest time to optimize, and the ideal effect is not necessarily achieved. Therefore, the existing Z+ operation performance optimization efficiency is low, the experience of service use data is reduced, meanwhile, a development responsible person has a certain professional foundation, not only the input resources of hardware are occupied, but also precious human resources are occupied.
Disclosure of Invention
The embodiment of the application aims to provide a Z+ operation optimization method and device based on a big data Z+ platform, which can automatically realize Z+ operation optimization without manual participation, improve Z+ operation performance optimization efficiency, reduce operation threshold, reduce input resources of hardware and save precious human resources.
The first aspect of the application provides a Z+ job optimization method based on a big data Z+ platform, which comprises the following steps:
acquiring Z+ operation to be optimized;
optimizing the Z+ operation to obtain an optimized target Z+ operation;
judging whether the target Z+ operation passes test verification or not;
if yes, configuring the target Z+ operation to a big data Z+ platform.
In the implementation process, the method can be used for preferentially acquiring Z+ operation to be optimized; then, optimizing the Z+ operation to obtain an optimized target Z+ operation; judging whether the target Z+ operation passes the test verification or not; and finally, when the left and right of the target Z+ pass the test verification, configuring the target Z+ operation to a big data Z+ platform. Therefore, the Z+ operation optimization can be automatically realized without manual participation, the Z+ operation performance optimization efficiency is improved, the operation threshold is reduced, the input resources of hardware are reduced, and precious human resources are saved.
Further, the optimizing the z+ job to obtain an optimized target z+ job includes:
acquiring table information corresponding to the Z+ operation;
analyzing the data distribution of the table information to obtain data distribution statistical information;
acquiring supplementary information for the Z+ job;
generating target statistical information according to the data distribution statistical information and the supplementary information;
and carrying out HQL rewriting optimization processing based on cost on the Z+ operation according to the target statistical information to obtain an optimized target Z+ operation.
Further, the performing, according to the target statistical information, cost-based HQL rewrite optimization processing on the z+ job to obtain an optimized target z+ job, including:
performing cost-based HQL rewrite optimization processing on the Z+ operation according to a pre-constructed rewrite sequence decision model and the target statistical information to obtain a plurality of optimized Z+ operation to be selected;
performing performance analysis on each Z+ operation to be selected to obtain an analysis result;
and selecting a target Z+ operation from a plurality of optimized Z+ operations to be selected according to the analysis result.
Further, after configuring the target z+ job to the big data z+ platform, the method further comprises:
acquiring an optimized HQL code according to the target Z+ operation;
and training and optimizing the rewritten sequence decision model through the HQL code.
Further, the method further comprises:
acquiring original HQL sample information of the Z+ operation sample and statistical sample information of the Z+ operation sample;
analyzing the original HQL sample information to obtain an abstract syntax tree;
generating a rewritten HQL operation according to a preset relational algebra and the abstract syntax tree;
and constructing a rewritten sequence decision model according to a preset reinforcement learning model, the rewritten HQL operation and the statistical sample information.
The second aspect of the present application provides a z+ job optimization device based on a big data z+ platform, the z+ job optimization device based on the big data z+ platform includes:
the acquisition unit is used for acquiring Z+ operation to be optimized;
the optimizing unit is used for optimizing the Z+ operation to obtain an optimized target Z+ operation;
the judging unit is used for judging whether the target Z+ operation passes the test verification or not;
and the configuration unit is used for configuring the target Z+ operation to the big data Z+ platform when judging that the target Z+ operation passes the test verification.
In the implementation process, the device can acquire Z+ operation to be optimized through the acquisition unit; optimizing the Z+ operation through an optimizing unit to obtain an optimized target Z+ operation; judging whether the target Z+ operation passes the test verification or not by a judging unit; and then when the target Z+ operation passes the test verification, the configuration unit configures the target Z+ operation to the big data Z+ platform. Therefore, the Z+ operation optimization device can automatically realize Z+ operation optimization without manual participation, improves the Z+ operation performance optimization efficiency, reduces the operation threshold, reduces the input resources of hardware, and saves precious human resources.
Further, the optimizing unit includes:
the first acquisition subunit is used for acquiring the table information corresponding to the Z+ operation;
the analysis subunit is used for analyzing the data distribution of the table information to obtain data distribution statistical information;
a second acquisition subunit configured to acquire supplemental information for the z+ job;
a generating subunit, configured to generate target statistics according to the data distribution statistics and the supplemental information;
and the optimizing subunit is used for carrying out HQL rewriting optimization processing on the Z+ operation based on cost according to the target statistical information to obtain an optimized target Z+ operation.
Further, the optimizing subunit includes:
the optimization module is used for carrying out HQL rewriting optimization processing on the Z+ operation based on cost according to a pre-constructed rewriting sequence decision model and the target statistical information to obtain a plurality of optimized Z+ operation to be selected;
the analysis module is used for carrying out performance analysis on each Z+ operation to be selected to obtain an analysis result;
and the selection module is used for selecting a target Z+ job from a plurality of optimized Z+ jobs to be selected according to the analysis result.
Further, the acquiring unit is further configured to acquire an optimized HQL code according to the target z+ job after the configuring unit configures the target z+ job to the big data z+ platform;
the optimizing unit is further used for training and optimizing the rewritten sequence decision model through the HQL code.
Further, the z+ job optimizing device based on the big data z+ platform further includes:
the acquisition unit is further used for acquiring original HQL sample information of the Z+ operation sample and statistical sample information of the Z+ operation sample;
the analysis unit is used for analyzing the original HQL sample information to obtain an abstract syntax tree;
the generating unit is used for generating a rewritten HQL job according to a preset relational algebra and the abstract syntax tree;
and the construction unit is used for constructing a rewritten sequence decision model according to a preset reinforcement learning model, the rewritten HQL operation and the statistical sample information.
A third aspect of the present application provides an electronic device, comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the big data z+ platform based z+ job optimization method of any one of the first aspects of the present application.
A fourth aspect of the present application provides a computer readable storage medium storing computer program instructions which, when read and executed by a processor, perform the z+ job optimization method according to any one of the first aspects of the present application based on a big data z+ platform.
The beneficial effects of this application are: the method and the device can automatically realize Z+ operation optimization without manual participation, improve the efficiency of Z+ operation performance optimization, simultaneously reduce the operation threshold, reduce the input resources of hardware and save precious human resources.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a z+ job optimization method based on a big data z+ platform according to an embodiment of the present application;
fig. 2 is a schematic flow chart of another z+ job optimization method based on a big data z+ platform according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a z+ job optimization device based on a big data z+ platform according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of another z+ job optimization device based on a big data z+ platform according to an embodiment of the present application;
fig. 5 is a schematic diagram of a ppo algorithm provided in an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance.
Example 1
Referring to fig. 1, fig. 1 is a flow chart of a z+ operation optimization method based on a big data z+ platform according to the present embodiment. The Z+ operation optimization method based on the big data Z+ platform comprises the following steps:
s101, acquiring Z+ operation to be optimized.
In this embodiment, the method may first go to the background platform side to check the table information corresponding to the job, and semi-automatically analyze the statistical information of the data distribution of each table.
In this embodiment, statistical distribution information may be used for sql rewrites and as input for cost prediction for each rewrite.
S102, performing optimization processing on the Z+ operation to obtain an optimized target Z+ operation.
In this embodiment, the method may supplement information such as primary key uniqueness.
In this embodiment, the method may perform cost-based HQL overwrite optimization (cost-based-optimization) based on statistical information. The optimization logic is similar to the SQL optimization engine of oracle, and then returns to the best 10 rewrite Z+ jobs of the data development colleague.
S103, judging whether the target Z+ operation passes test verification, if so, executing a step S104; if not, the process is ended.
In this embodiment, the method may further analyze the obtained results, and pick and verify the accuracy, consistency and performance of the semi-automatically improved HQL. At this point, the possible errors can be corrected and the HQL of the final production line can then be saved to the system for further training analysis by the model.
In this embodiment, the process may be manually operated by a developer, or may be automatically operated by various methods, which is not limited in this embodiment.
S104, configuring the target Z+ operation to the big data Z+ platform.
In this embodiment, the method may further train the rewritten sequence decision model by the system through the final optimized HQL code on-line.
In banking systems in the financial field, the data processing system currently in common use is the big data z+ platform. Statistics is performed based on practical application conditions, and hundreds of thousands of data processing operations exist on the platform at present. Based on the above, in daily inspection, banking operators have found that part of the operations are very slow to execute, even the individual operations need to be executed for more than 6 hours, so that great pressure is caused on the computing resources of the platform, and meanwhile, the execution timeliness of other key operations is seriously influenced, so that the service experience is greatly reduced.
In particular, these inefficient z+ jobs often have relatively complex HQL processing logic. The platform manager can regularly grasp the inefficient operation, and the development responsible person can optimize the performance of the operation. At this time, the data developer has to temporarily put down the business requirement at hand to study the optimization method of HQL, thereby resulting in a deterioration of the experience of using the data by the business. Moreover, due to the professionality of HQL optimization, developers often invest time to optimize, and the ideal effect is not necessarily achieved.
Therefore, due to the existence of the HQL of the low-efficiency Z+ operation, not only the input resources of hardware are occupied, but also precious human resources are occupied.
In order to solve the technical problems, the application provides a method for automatically optimizing the Z+ operation performance. By implementing the method, after Z+ operation is input, a series of processing is carried out by the tool, a recommended optimized Z+ operation is automatically generated, test verification is carried out on the optimized Z+ operation manually or automatically, and then the operation is directly put into production after the test verification is passed, so that logic optimization of the whole operation is directly completed.
In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, which is not limited in this embodiment.
In this embodiment, the execution body of the method may be an intelligent device such as a smart phone or a tablet computer, which is not limited in this embodiment.
Therefore, when the Z+ operation performance is time-consuming, the Z+ operation can be submitted to the system so that the system can recommend a plurality of codes for optimizing and improving the Z+ operation; based on this, it is possible to further select a code in which the comparison is superior, and perform corresponding performance verification and data test.
Example 2
Referring to fig. 2, fig. 2 is a flow chart of a z+ operation optimization method based on a big data z+ platform according to the present embodiment. The Z+ operation optimization method based on the big data Z+ platform comprises the following steps:
s201, acquiring original HQL sample information of a Z+ operation sample and statistical sample information of the Z+ operation sample.
S202, analyzing the original HQL sample information to obtain an abstract syntax tree.
S203, generating a rewritten HQL job according to a preset relation algebra and an abstract syntax tree.
S204, constructing a rewritten sequence decision model according to a preset reinforcement learning model, rewritten HQL operation and statistical sample information.
S205, obtaining Z+ operation to be optimized.
S206, obtaining table information corresponding to the Z+ operation.
S207, analyzing the data distribution of the table information to obtain data distribution statistical information.
S208, acquiring the supplementary information for the Z+ job.
S209, generating target statistical information according to the data distribution statistical information and the supplementary information.
S210, carrying out HQL rewriting optimization processing on the Z+ operation based on cost according to a pre-constructed rewriting sequence decision model and target statistical information to obtain a plurality of optimized Z+ operation candidates.
S211, performing performance analysis on each Z+ operation to be selected to obtain an analysis result.
S212, selecting a target Z+ job from a plurality of optimized Z+ jobs to be selected according to the analysis result.
S213, judging whether the target Z+ operation passes the test verification, if so, executing a step S214; if not, the process is ended.
S214, configuring the target Z+ operation to the big data Z+ platform.
S215, acquiring the optimized HQL code according to the target Z+ job.
S216, training and optimizing the rewritten sequence decision model through the HQL code.
In this embodiment, the method uses a reinforcement learning ppo algorithm. Referring to FIG. 5, FIG. 5 shows a schematic diagram of a ppo algorithm. Specifically, the method looks for those actions that are likely to get more rewards based on the idea of parameterizing the policy so that their corresponding probabilities are greater, making it more likely that the policy will select those actions. Based on this, the maximization objective function of the method is as follows:
the summing represents all conditions which can be generated by interaction with the environment. The goal of this approach is to make the probability of track occurrences that get larger rewards higher by tuning.
Wherein the probability of a trajectory occurring under a policy is defined as:
the definition is expressed as the product of the state transition probability and the action selection probability, as the state and action determine the trajectory.
In this embodiment, the concept of deriving the policy gradient is the same as the optimization concept of the neural network, and in order to optimize the objective function by parameters, we need to calculate the derivative of the objective function pair:
since the expression is now fully computable, it is only necessary to give a well-defined policy to be computable, thus updating the rules with the policy gradient:
and thereby continuously optimizing the parameters.
In this example, chatGLM is a large language model trained using chinese-english bilingual pre-training, with 130B parameters (1300 billions), trained using 400 Btoken. The large-scale language model is a generated self-supervision model trained by a large amount of corpus. For rewriting the base model of sql.
In this embodiment, the method may implement the effect of migration learning by freezing the model parameter weight of Chatglm, adding a trainable parameter between layers, and fine tuning the parameter. The method uses a low-rank decomposition technology to finely tune the model of the existing transfer learning algorithm, so that the adjusted result can better realize the corresponding technical effect.
In this embodiment, the method may continuously make a rewrite decision according to the final reward, and rewrite the sequence decision model with the final reward, so as to achieve the effect of reinforcement learning.
In this embodiment, the execution subject of the method may be a computing device such as a computer or a server, which is not limited in this embodiment.
In this embodiment, the execution body of the method may be an intelligent device such as a smart phone or a tablet computer, which is not limited in this embodiment.
Therefore, when the Z+ operation performance is time-consuming, the Z+ operation can be submitted to the system so that the system can recommend a plurality of codes for optimizing and improving the Z+ operation; based on this, it is possible to further select a code in which the comparison is superior, and perform corresponding performance verification and data test. Meanwhile, the method can further feed the code of the last online Z+ operation back into the system, so that the sequence decision model is further rewritten, the optimization effect of the system is gradually improved, and the time consumed by developers for optimization is further reduced. Therefore, the method can convert the optimization experience of the developer into information data, and the performance of the model is continuously improved.
Example 3
Referring to fig. 3, fig. 3 is a schematic structural diagram of a z+ operation optimizing apparatus based on a big data z+ platform according to the present embodiment. As shown in fig. 3, the z+ job optimizing apparatus based on the big data z+ platform includes:
an acquiring unit 310, configured to acquire a z+ job to be optimized;
the optimizing unit 320 is configured to perform optimizing processing on the z+ job, so as to obtain an optimized target z+ job;
a judging unit 330 for judging whether the target z+ job passes the test verification;
and a configuration unit 340, configured to configure the target z+ job to the big data z+ platform when it is determined that the target z+ job passes the test verification.
In this embodiment, the explanation of the z+ job optimizing apparatus based on the big data z+ platform may refer to the description in embodiment 1 or embodiment 2, and the description is not repeated in this embodiment.
It can be seen that, by implementing the z+ job optimizing apparatus based on the big data z+ platform described in this embodiment, when z+ job performance is time-consuming, the z+ job can be submitted to the system, so that the system may recommend a plurality of codes for optimizing the improved z+ job; based on this, it is possible to further select a code in which the comparison is superior, and perform corresponding performance verification and data test.
Example 4
Referring to fig. 4, fig. 4 is a schematic structural diagram of a z+ operation optimizing apparatus based on a big data z+ platform according to the present embodiment. As shown in fig. 4, the z+ job optimizing apparatus based on the big data z+ platform includes:
an acquiring unit 310, configured to acquire a z+ job to be optimized;
the optimizing unit 320 is configured to perform optimizing processing on the z+ job, so as to obtain an optimized target z+ job;
a judging unit 330 for judging whether the target z+ job passes the test verification;
and a configuration unit 340, configured to configure the target z+ job to the big data z+ platform when it is determined that the target z+ job passes the test verification.
As an alternative embodiment, the optimizing unit 320 includes:
a first obtaining subunit 321, configured to obtain table information corresponding to the z+ operation;
an analysis subunit 322, configured to analyze the data distribution of the table information to obtain statistical information of the data distribution;
a second acquisition subunit 323 configured to acquire supplemental information for the z+ job;
a generating subunit 324, configured to generate target statistics according to the data distribution statistics and the supplemental information;
and the optimizing subunit 325 is configured to perform cost-based HQL rewrite optimization on the z+ job according to the target statistics information, so as to obtain an optimized target z+ job.
As an alternative embodiment, the optimization subunit 325 includes:
the optimizing module is used for carrying out HQL rewriting optimization processing on the Z+ operation based on cost according to a pre-constructed rewriting sequence decision model and target statistical information to obtain a plurality of optimized Z+ operation to be selected;
the analysis module is used for carrying out performance analysis on each Z+ operation to be selected to obtain an analysis result;
and the selection module is used for selecting a target Z+ job from a plurality of optimized Z+ jobs to be selected according to the analysis result.
As an optional implementation manner, the obtaining unit 310 is further configured to obtain, after the configuring unit 340 configures the target z+ job to the big data z+ platform, an optimized HQL code according to the target z+ job;
the optimizing unit 320 is further configured to perform training optimization on the rewritten sequence decision model through the HQL code.
As an alternative embodiment, the z+ job optimizing apparatus based on big data z+ platform further includes:
the acquiring unit 310 is further configured to acquire original HQL sample information of the z+ job sample and statistical sample information of the z+ job sample;
the parsing unit 350 is configured to parse the original HQL sample information to obtain an abstract syntax tree;
a generating unit 360, configured to generate a rewritten HQL job according to a preset relational algebra and abstract syntax tree;
and the construction unit 370 is configured to construct a rewritten sequence decision model according to the preset reinforcement learning model, the rewritten HQL job and the statistical sample information.
In this embodiment, the explanation of the z+ job optimizing apparatus based on the big data z+ platform may refer to the description in embodiment 1 or embodiment 2, and the description is not repeated in this embodiment.
It can be seen that, by implementing the z+ job optimizing apparatus based on the big data z+ platform described in this embodiment, when z+ job performance is time-consuming, the z+ job can be submitted to the system, so that the system may recommend a plurality of codes for optimizing the improved z+ job; based on this, it is possible to further select a code in which the comparison is superior, and perform corresponding performance verification and data test. Meanwhile, the method can further feed the code of the last online Z+ operation back into the system, so that the sequence decision model is further rewritten, the optimization effect of the system is gradually improved, and the time consumed by developers for optimization is further reduced. Therefore, the method can convert the optimization experience of the developer into information data, and the performance of the model is continuously improved.
The embodiment of the application provides electronic equipment, which comprises a memory and a processor, wherein the memory is used for storing a computer program, and the processor runs the computer program to enable the electronic equipment to execute the Z+ job optimization method based on the big data Z+ platform in the embodiment 1 or the embodiment 2 of the application.
The present embodiment provides a computer readable storage medium storing computer program instructions that, when read and executed by a processor, perform the z+ job optimization method based on the big data z+ platform in embodiment 1 or embodiment 2 of the present application.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners as well. The apparatus embodiments described above are merely illustrative, for example, flow diagrams and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, the functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely exemplary embodiments of the present application and is not intended to limit the scope of the present application, and various modifications and variations may be suggested to one skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application. It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. The Z+ operation optimization method based on the big data Z+ platform is characterized by comprising the following steps of:
acquiring Z+ operation to be optimized;
optimizing the Z+ operation to obtain an optimized target Z+ operation;
judging whether the target Z+ operation passes test verification or not;
if yes, configuring the target Z+ operation to a big data Z+ platform.
2. The z+ job optimization method based on the big data z+ platform according to claim 1, wherein the optimizing the z+ job to obtain an optimized target z+ job includes:
acquiring table information corresponding to the Z+ operation;
analyzing the data distribution of the table information to obtain data distribution statistical information;
acquiring supplementary information for the Z+ job;
generating target statistical information according to the data distribution statistical information and the supplementary information;
and carrying out HQL rewriting optimization processing based on cost on the Z+ operation according to the target statistical information to obtain an optimized target Z+ operation.
3. The z+ job optimization method based on the big data z+ platform according to claim 2, wherein the performing, according to the target statistics information, a cost-based HQL rewrite optimization process on the z+ job to obtain an optimized target z+ job includes:
performing cost-based HQL rewrite optimization processing on the Z+ operation according to a pre-constructed rewrite sequence decision model and the target statistical information to obtain a plurality of optimized Z+ operation to be selected;
performing performance analysis on each Z+ operation to be selected to obtain an analysis result;
and selecting a target Z+ operation from a plurality of optimized Z+ operations to be selected according to the analysis result.
4. The big data z+ platform based z+ job optimization method of claim 3, wherein after configuring the target z+ job to the big data z+ platform, the method further comprises:
acquiring an optimized HQL code according to the target Z+ operation;
and training and optimizing the rewritten sequence decision model through the HQL code.
5. The z+ job optimization method based on big data z+ platform according to claim 1, further comprising:
acquiring original HQL sample information of the Z+ operation sample and statistical sample information of the Z+ operation sample;
analyzing the original HQL sample information to obtain an abstract syntax tree;
generating a rewritten HQL operation according to a preset relational algebra and the abstract syntax tree;
and constructing a rewritten sequence decision model according to a preset reinforcement learning model, the rewritten HQL operation and the statistical sample information.
6. Z+ operation optimizing device based on big data Z+ platform, characterized in that, Z+ operation optimizing device based on big data Z+ platform includes:
the acquisition unit is used for acquiring Z+ operation to be optimized;
the optimizing unit is used for optimizing the Z+ operation to obtain an optimized target Z+ operation;
the judging unit is used for judging whether the target Z+ operation passes the test verification or not;
and the configuration unit is used for configuring the target Z+ operation to the big data Z+ platform when judging that the target Z+ operation passes the test verification.
7. The z+ job optimizing apparatus based on big data z+ platform according to claim 6, wherein the optimizing unit includes:
the first acquisition subunit is used for acquiring the table information corresponding to the Z+ operation;
the analysis subunit is used for analyzing the data distribution of the table information to obtain data distribution statistical information;
a second acquisition subunit configured to acquire supplemental information for the z+ job;
a generating subunit, configured to generate target statistics according to the data distribution statistics and the supplemental information;
and the optimizing subunit is used for carrying out HQL rewriting optimization processing on the Z+ operation based on cost according to the target statistical information to obtain an optimized target Z+ operation.
8. The z+ job optimization device based on big data z+ platform of claim 7, wherein the optimization subunit comprises:
the optimization module is used for carrying out HQL rewriting optimization processing on the Z+ operation based on cost according to a pre-constructed rewriting sequence decision model and the target statistical information to obtain a plurality of optimized Z+ operation to be selected;
the analysis module is used for carrying out performance analysis on each Z+ operation to be selected to obtain an analysis result;
and the selection module is used for selecting a target Z+ job from a plurality of optimized Z+ jobs to be selected according to the analysis result.
9. An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the big data z+ platform based z+ job optimization method of any one of claims 1 to 5.
10. A readable storage medium, wherein computer program instructions are stored in the readable storage medium, which when read and executed by a processor, perform the big data z+ platform based z+ job optimization method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311484065.3A CN117520631A (en) | 2023-11-08 | 2023-11-08 | Z+ operation optimization method and device based on big data Z+ platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311484065.3A CN117520631A (en) | 2023-11-08 | 2023-11-08 | Z+ operation optimization method and device based on big data Z+ platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117520631A true CN117520631A (en) | 2024-02-06 |
Family
ID=89745038
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311484065.3A Pending CN117520631A (en) | 2023-11-08 | 2023-11-08 | Z+ operation optimization method and device based on big data Z+ platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117520631A (en) |
-
2023
- 2023-11-08 CN CN202311484065.3A patent/CN117520631A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11790161B2 (en) | Machine learning selection and/or application of a data model defined in a spreadsheet | |
| US11861462B2 (en) | Preparing structured data sets for machine learning | |
| JP2021518024A (en) | How to generate data for machine learning algorithms, systems | |
| CN111400338B (en) | SQL optimization method, device, storage medium and computer equipment | |
| EP3872626B1 (en) | Utilizing artificial intelligence and machine learning models to reverse engineer an application from application artifacts | |
| Vassiliadis et al. | Gravitating to rigidity: Patterns of schema evolution–and its absence–in the lives of tables | |
| CN112199512B (en) | Scientific and technological service-oriented case map construction method, device, equipment and storage medium | |
| CN113254630B (en) | Domain knowledge map recommendation method for global comprehensive observation results | |
| CN112434024A (en) | Relational database-oriented data dictionary generation method, device, equipment and medium | |
| CN110310012B (en) | Data analysis method, device, equipment and computer readable storage medium | |
| CN116244367A (en) | Visual big data analysis platform based on multi-model custom algorithm | |
| CN117709435A (en) | Training method of large language model, code generation method, device and storage medium | |
| CN109408592B (en) | AI characteristic engineering knowledge base in decision-making type distributed database system and implementation method thereof | |
| Revindasari et al. | Traceability between business process and software component using Probabilistic Latent Semantic Analysis | |
| Zhu et al. | A data cleaning method for heterogeneous attribute fusion and record linkage | |
| CN116502648A (en) | Machine reading understanding semantic reasoning method based on multi-hop reasoning | |
| CN111638926A (en) | Method for realizing artificial intelligence in Django framework | |
| CN116151235A (en) | Article generating method, article generating model training method and related equipment | |
| CN117520631A (en) | Z+ operation optimization method and device based on big data Z+ platform | |
| CN114218580A (en) | Intelligent contract vulnerability detection method based on multi-task learning | |
| CN114138743A (en) | ETL task automatic configuration method and device based on machine learning | |
| CN113505117A (en) | Data quality evaluation method, device, equipment and medium based on data indexes | |
| CN113254612A (en) | Knowledge question-answering processing method, device, equipment and storage medium | |
| US11830081B2 (en) | Automated return evaluation with anomoly detection | |
| CN113886360B (en) | Data table partitioning method, device, computer readable medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |