CN112417014A - Dynamic modification execution plan method, system, and computer-readable storage medium - Google Patents
Dynamic modification execution plan method, system, and computer-readable storage medium Download PDFInfo
- Publication number
- CN112417014A CN112417014A CN202011277554.8A CN202011277554A CN112417014A CN 112417014 A CN112417014 A CN 112417014A CN 202011277554 A CN202011277554 A CN 202011277554A CN 112417014 A CN112417014 A CN 112417014A
- Authority
- CN
- China
- Prior art keywords
- updated
- configuration
- execution
- statement
- configuration statement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 230000004048 modification Effects 0.000 title claims abstract description 25
- 238000012986 modification Methods 0.000 title claims abstract description 24
- 238000012545 processing Methods 0.000 claims description 22
- 230000006870 function Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 11
- 238000004458 analytical method Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 abstract description 3
- 238000004364 calculation method Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000003672 processing method Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/65—Updates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Stored Programmes (AREA)
Abstract
The invention provides a method, a system and a computer readable storage medium for dynamically modifying an execution plan, wherein the method comprises the steps of receiving a configuration statement to be updated, and distributing and configuring the configuration statement to be updated to each execution node; when the general operator ProcessFunction of the execution node monitors the information of the configuration statement to be updated, triggering a configuration refreshing instruction; converting the configuration statement to be updated into executable program logic; and correspondingly modifying and replacing the code of the original execution function. The dynamic modification execution plan method provided by the invention extracts the configuration statement from the starting execution logic of the computing framework to form an independent module and supports dynamic compiling, calling execution and dynamic refreshing service configuration.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method, a system, and a computer-readable storage medium for dynamically modifying an execution plan.
Background
At present, a big data computing framework cannot dynamically modify an execution plan, and an executed streaming computing task, the execution plan DAG or data processing logic of which cannot be dynamically modified, is submitted and must be restarted after the execution plan is modified to be effective.
Specifically, for example, in a Flink computing framework, writing a data processing logic ETL (Extract-Transform-Load: extracting (Extract), converting (Transform), and loading (Load) data from a source end to a destination end) through SQL cannot be dynamically modified, and a program must be restarted after the processing logic is modified to be applied to a flow.
Disclosure of Invention
Based on this, the present invention provides a method, system and computer readable storage medium for dynamically modifying an execution plan, which extracts a configuration statement from a computing framework boot execution logic to form an independent module, and supports dynamic compiling, call execution and dynamic refresh service configuration.
The invention provides a method for dynamically modifying an execution plan, wherein the method comprises the following steps:
receiving a configuration statement to be updated, and distributing and configuring the configuration statement to be updated to each execution node;
when the common operator ProcessFunction of the execution node monitors the configuration statement information to be updated, triggering a configuration refreshing instruction;
converting the configuration statement to be updated into executable program logic;
and correspondingly modifying and replacing the code of the original execution function.
When receiving a configuration statement to be updated, distributing the configuration statement to be updated to each execution node, monitoring whether the configuration statement to be updated is received or not in real time by a general operator ProcessFunction of the execution node, triggering a refresh instruction when receiving the configuration statement to be updated, extracting the configuration statement from a calculation frame starting execution logic, converting the configuration statement into a code and executing the code to complete dynamic compiling and executing of program logic, and then correspondingly modifying and replacing the code of an original execution function to dynamically refresh service configuration.
Further, the step of distributing and configuring the configuration statement to be updated to each execution node further includes:
analyzing the configuration statement to be updated to obtain target codes and a replacement path corresponding to each target code;
and determining whether the current execution node has modified content according to the alternative path.
Further, the step of converting the configuration statement to be updated into executable program logic further comprises:
checking whether the configuration statement to be updated is legal or not;
if not, the configuration refresh command is ended.
Further, the method for distributing and configuring the configuration statement to be updated to each execution node includes:
and pushing the configuration statement to be updated to a Flink broadcast stream in an information queue mode through a publishing subscription information system, wherein the subscription information system is kafka.
Further, an execution calculation framework of the dynamic modification execution plan method is Flink, and the configuration statement to be updated is one of a JSON statement or an SQL statement.
The invention provides a dynamic modification execution plan processing system, which comprises:
the receiving and distributing module is used for receiving the configuration statement to be updated and distributing and configuring the configuration statement to be updated to each execution node;
the refresh triggering module is used for triggering a configuration refresh instruction when the general operator ProcessFunction of the execution node monitors the configuration statement information to be updated;
the execution module is used for converting the configuration statement to be updated into executable program logic;
and the replacing module is used for correspondingly modifying and replacing the code of the original execution function.
And the analysis module is used for analyzing the configuration statement to be updated.
And the checking module is used for checking whether the configuration statement to be updated is legal or not, and if not, finishing the configuration refreshing instruction.
The invention also proposes a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method.
The invention also proposes a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the program.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flowchart of a method for dynamically modifying an execution plan according to a first embodiment of the present invention;
FIG. 2 is a flowchart of a first implementation of the present invention after a configuration statement to be updated is distributed and configured to each execution node;
FIG. 3 is a flowchart illustrating the check before the step of converting the configuration statement to be updated into executable program logic according to the first embodiment of the present invention;
FIG. 4 is a flowchart illustrating a configuration update trigger according to a first embodiment of the present invention;
FIG. 5 is a diagram illustrating a system for dynamically modifying an execution plan according to a second embodiment of the present invention;
fig. 6 is a schematic diagram of a hardware configuration of the dynamic modification execution plan processing apparatus according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.
It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.
Referring to fig. 1 to fig. 5, a method for dynamically modifying an execution plan according to a first embodiment of the present invention includes the following steps:
and step S10, receiving the configuration statement to be updated, and distributing and configuring the configuration statement to be updated to each execution node.
In the embodiment of the present invention, the computing framework receives the written configuration statement to be updated, and the configuration statement to be updated is parsed, distributed, and configured to each execution node, where the configuration statement to be updated is generally an SQL statement or a JSON syntax.
The calculation to implement the configured SQL expression takes the following code as an example:
the above code formula describes the SQL computing logic by JSON configuration file, wherein the meaning of the part code is shown in the following table:
class is a static table (called dimension table) and student is a data stream, which can be understood as a table whose number is increasing. I.e. the data streams processed in the description below. For the above SQL configuration statement, refer to the following description:
table: a table name is defined for the currently processed stream data.
Field: indicating what field is selected from the incoming data, can be understood as the select field name, age, generator in SQL.
Script is a nested structure.
Condition denotes a filtering Condition, which can be understood as a where expression in SQL that selects from student where >14and generator 1.
Joins: the table is a character string array which is used for representing data related to other tables, the syntax is not different from the common SQL, the logic of the table is that the current table student is related to some dimension tables in external systems, and class name is taken for student related class and is set into a return result as described in the configuration file.
Based on the configuration file, the program will be parsed into corresponding program logics, and each calculation logic will be described by using the object in the program, and the calculation result can be obtained by calling.
In step S20, when the generic operator ProcessFunction of the execution node monitors the information of the configuration statement to be updated, a configuration refresh instruction is triggered.
In the embodiment of the invention, the generic operator ProcessFunction of the execution node monitors whether the information about the configuration statement to be updated is received in real time, and when the execution node receives the information about the configuration statement to be updated, a configuration refreshing instruction is triggered.
In step S30, the configuration statement to be updated is converted into executable program logic.
After receiving the configuration refresh command of step S20, the program converts the configuration statement to be updated into executable program logic. And real-time program modification operation is realized.
And step S40, correspondingly modifying and replacing the code of the original execution function.
In the embodiment of the invention, after the program completes the conversion of the configuration statement to be updated, the code of the original execution function is further modified and replaced by the code after the conversion of the configuration statement to be updated, so as to realize the dynamic refreshing service configuration.
When the execution plan is modified, the configuration statement (the execution plan configuration statement to be modified) for definition and the static compiling process are extracted from the starting execution logic of the computing framework to form an independent module, and the independent module supports dynamic compiling, dynamic call execution and dynamic refreshing of the configuration statement.
Referring to fig. 2, a process after the step of distributing and configuring the configuration statement to be updated to each execution node according to the present invention specifically includes:
step S11, parsing the configuration statement to be updated, and acquiring the target codes and the alternative paths corresponding to each target code.
In the embodiment of the invention, the configuration statement to be updated is analyzed to obtain the target codes which need to be modified and replaced and the replacement path corresponding to each target code.
Step S12, determining whether there is modified content in the current execution node according to the alternative path.
And further, comparing the replacement path with the execution node to determine whether the current execution node has program codes which need to be modified and replaced.
Referring to fig. 3, the step of converting the configuration statement to be updated into the executable program logic further includes:
step S21, checking whether the configuration statement to be updated is legal.
If so, go to step S30,
if not, the configuration refresh command is ended.
In the embodiment of the invention, the safety and confidentiality performance in the process of dynamically modifying and executing the plan is improved by checking the legality of the configuration statement to be updated, and the specific mode of verifying whether the configuration statement to be updated is legal can be carried out in a key encryption mode. It can be understood that, in other embodiments of the present invention, the configuration statement to be updated may also be verified in other authentication manners, so as to achieve the purpose of improving the security of the processing process of modifying the execution plan.
In the embodiment of the invention, the configuration statement to be updated is pushed to the Flink broadcast stream in an information queue mode through a publishing and subscribing information system, and the subscribing information system is kafka. Namely, a Flink computing framework is selected and a message queue is sent to the Flink broadcast stream by kafka, so that data (configuration statements to be updated) are distributed to each execution node in a broadcast stream mode.
Referring to fig. 4, a flow chart of a push update trigger according to the present invention is shown. In the embodiment of the invention, besides the original calculation data stream, a broadcast stream is additionally added, the broadcast stream has the function of dynamically pushing and configuring each node of the distributed calculation engine to achieve the purpose of notification, and the function is used for updating and triggering the pushing of the configuration.
The present embodiment further provides a system for dynamically modifying an execution plan, where the system is used to implement the foregoing embodiments and preferred embodiments, and details of which have been already described are omitted. As used hereinafter, the terms "module," "unit," "subunit," and the like may implement a combination of software and/or hardware for a predetermined function. While the system described in the embodiments below is preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.
Referring to fig. 5, based on the same inventive concept, a system for dynamically modifying an execution plan according to a second embodiment of the present invention includes:
the receiving and distributing module is used for receiving the configuration statement to be updated and distributing and configuring the configuration statement to be updated to each execution node;
the refresh triggering module is used for triggering a configuration refresh instruction when the general operator ProcessFunction of the execution node monitors the configuration statement information to be updated;
the execution module is used for converting the configuration statement to be updated into executable program logic;
and the replacing module is used for correspondingly modifying and replacing the code of the original execution function.
The system for processing a dynamic modification execution plan according to the second embodiment of the present invention further includes:
and the analysis module is used for analyzing the configuration statement to be updated.
And the checking module is used for checking whether the configuration statement to be updated is legal or not, and if not, ending the configuration refreshing instruction.
The above modules may be functional modules or program modules, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.
Furthermore, an embodiment of the present invention also proposes a storage medium, in particular a readable storage medium, on which a computer program is stored, which program, when being executed by a processor, realizes the steps of the method described in the first embodiment. The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any system that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic system) having one or more wires, a portable computer diskette (magnetic system), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber system, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
Furthermore, an embodiment of the present invention also provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the steps of the method in the first embodiment when executing the program.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents. It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.
In addition, the dynamic modification execution plan processing method described in the embodiment of the present application with reference to fig. 1 may be implemented by a dynamic modification execution plan processing apparatus. Fig. 6 is a hardware configuration diagram of a dynamic modification execution plan processing apparatus according to an embodiment of the present application.
The dynamic modified execution plan processing device may include a processor 61 and a memory 62 storing computer program instructions.
Specifically, the processor 61 may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.
The memory 62 may be used to store or cache various data files for processing and/or communication use, as well as possibly computer program instructions for execution by the processor 62.
The processor 61 implements any of the above-described embodiments by reading and executing computer program instructions stored in the memory 62 to implement a method of dynamically modifying an execution plan.
In some of these embodiments, the dynamic modification execution plan processing apparatus may further include a communication interface 63 and a bus 60. As shown in fig. 6, the processor 61, the memory 62, and the communication interface 63 are connected via a bus 60 to complete mutual communication.
The communication interface 63 is used for implementing communication between modules, systems, units and/or devices in the embodiments of the present application. The communication interface 63 may also enable communication with other components such as: the data communication is carried out among external equipment, image/data acquisition equipment, a database, external storage, an image/data processing workstation and the like.
The bus 60 includes hardware, software, or both that couple the components of the dynamically modified execution plan processing device to each other. Bus 60 includes, but is not limited to, at least one of the following: data Bus (Data Bus), Address Bus (Address Bus), Control Bus (Control Bus), Expansion Bus (Expansion Bus), and Local Bus (Local Bus). By way of example, and not limitation, Bus 60 may include an Accelerated Graphics Port (AGP) or other Graphics Bus, an Enhanced Industry Standard Architecture (EISA) Bus, a Front-Side Bus (FSB), a Hyper Transport (HT) Interconnect, an ISA (ISA) Bus, an InfiniBand (InfiniBand) Interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a microchannel Architecture (MCA) Bus, a PCI (Peripheral Component Interconnect) Bus, a PCI-Express (PCI-X) Bus, a Serial Advanced Technology Attachment (SATA) Bus, a Video Electronics Bus (audio Electronics Association), abbreviated VLB) bus or other suitable bus or a combination of two or more of these. Bus 60 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.
In summary, in the dynamic modification execution plan processing apparatus in the above embodiment of the present invention, when receiving the configuration statement to be updated, the configuration statement to be updated is distributed to each execution node, a generic operator ProcessFunction of the execution node monitors in real time whether the configuration statement to be updated is received, and when receiving the configuration statement to be updated, a refresh instruction is triggered, so that the configuration statement is extracted from the computing framework starting execution logic, and the configuration statement is converted into a code and executed, thereby completing dynamic compilation execution on program logic, and then the code of the original execution function is correspondingly modified and replaced, thereby dynamically refreshing service configuration. In addition, the configuration statement to be updated is analyzed to obtain the target codes to be modified and replaced and the replacement path corresponding to each target code, so that the dynamic real-time modification and replacement of the codes are realized. In addition, the security and confidentiality performance in the process of dynamically modifying and executing the plan is improved by checking whether the configuration statement to be updated is legal or not. In addition, besides the original calculation data stream, a broadcast stream is additionally added, the broadcast stream has the function of dynamically pushing and configuring each node of the distributed calculation engine to achieve the purpose of notification, and the function is used for updating and triggering the pushing of the configuration.
The dynamic modification execution plan processing system may execute the dynamic modification execution plan processing method in the embodiment of the present application based on the acquired dynamic modification execution plan processing, thereby implementing the dynamic modification execution plan processing method described in conjunction with fig. 1.
In addition, in combination with the dynamic modification execution plan processing method in the foregoing embodiment, the embodiment of the present application may provide a computer-readable storage medium to implement. The computer readable storage medium having stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any of the dynamic modification execution plan processing methods of the above embodiments.
Those of skill in the art will understand that the logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be viewed as implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for dynamically modifying an execution plan, the method comprising:
receiving a configuration statement to be updated, and distributing and configuring the configuration statement to be updated to each execution node;
when the common operator ProcessFunction of the execution node monitors the configuration statement information to be updated, triggering a configuration refreshing instruction;
converting the configuration statement to be updated into executable program logic;
and correspondingly modifying and replacing the code of the original execution function.
2. The method for dynamically modifying an execution plan according to claim 1, wherein said step of distributing and configuring said configuration statement to be updated to each execution node further comprises:
analyzing the configuration statement to be updated to obtain target codes and a replacement path corresponding to each target code;
and determining whether the current execution node has modified content according to the alternative path.
3. The method of dynamically modifying an execution plan according to claim 1, wherein the step of converting the configuration statement to be updated into executable program logic is preceded by the step of:
checking whether the configuration statement to be updated is legal or not;
if not, the configuration refresh command is ended.
4. The method for dynamically modifying execution plan according to claim 1, wherein the method for distributing and configuring the configuration statement to be updated to each execution node is:
and pushing the configuration statement to be updated to a Flink broadcast stream in an information queue mode through a publishing subscription information system, wherein the subscription information system is kafka.
5. The dynamic modification execution plan method according to claim 1, wherein an execution computation framework of the dynamic modification execution plan method is Flink, and the configuration statement to be updated is one of a JSON statement or an SQL statement.
6. A system for dynamically modifying an execution plan, comprising:
the receiving and distributing module is used for receiving the configuration statement to be updated and distributing and configuring the configuration statement to be updated to each execution node;
the refresh triggering module is used for triggering a configuration refresh instruction when the general operator ProcessFunction of the execution node monitors the configuration statement information to be updated;
the execution module is used for converting the configuration statement to be updated into executable program logic;
and the replacing module is used for correspondingly modifying and replacing the code of the original execution function.
7. The dynamically modified execution plan processing system of claim 6, further comprising:
and the analysis module is used for analyzing the configuration statement to be updated.
8. The dynamically modified execution plan processing system of claim 6, further comprising:
and the checking module is used for checking whether the configuration statement to be updated is legal or not, and if not, finishing the configuration refreshing instruction.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 5.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 5 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011277554.8A CN112417014A (en) | 2020-11-16 | 2020-11-16 | Dynamic modification execution plan method, system, and computer-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011277554.8A CN112417014A (en) | 2020-11-16 | 2020-11-16 | Dynamic modification execution plan method, system, and computer-readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112417014A true CN112417014A (en) | 2021-02-26 |
Family
ID=74832311
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011277554.8A Withdrawn CN112417014A (en) | 2020-11-16 | 2020-11-16 | Dynamic modification execution plan method, system, and computer-readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112417014A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377360A (en) * | 2021-06-28 | 2021-09-10 | 北京百度网讯科技有限公司 | Task execution method, device, electronic equipment, storage medium and program product |
| CN113672601A (en) * | 2021-07-22 | 2021-11-19 | 北京明略软件系统有限公司 | Streaming data supplementing method and system, electronic device and storage medium |
| CN114385137A (en) * | 2021-12-29 | 2022-04-22 | 武汉达梦数据库股份有限公司 | Flow method and device for running ETL (extract transform load) by Flink framework |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170052970A1 (en) * | 2015-08-20 | 2017-02-23 | International Business Machines Corporation | Transforming and loading data from a source data system to a target data system |
| CN107678790A (en) * | 2016-07-29 | 2018-02-09 | 华为技术有限公司 | Flow calculation methodologies, apparatus and system |
| CN111506603A (en) * | 2020-04-23 | 2020-08-07 | 上海达梦数据库有限公司 | Data processing method, device, equipment and storage medium |
| CN111666296A (en) * | 2020-04-28 | 2020-09-15 | 中国平安财产保险股份有限公司 | SQL data real-time processing method and device based on Flink, computer equipment and medium |
-
2020
- 2020-11-16 CN CN202011277554.8A patent/CN112417014A/en not_active Withdrawn
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170052970A1 (en) * | 2015-08-20 | 2017-02-23 | International Business Machines Corporation | Transforming and loading data from a source data system to a target data system |
| CN107678790A (en) * | 2016-07-29 | 2018-02-09 | 华为技术有限公司 | Flow calculation methodologies, apparatus and system |
| CN111506603A (en) * | 2020-04-23 | 2020-08-07 | 上海达梦数据库有限公司 | Data processing method, device, equipment and storage medium |
| CN111666296A (en) * | 2020-04-28 | 2020-09-15 | 中国平安财产保险股份有限公司 | SQL data real-time processing method and device based on Flink, computer equipment and medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377360A (en) * | 2021-06-28 | 2021-09-10 | 北京百度网讯科技有限公司 | Task execution method, device, electronic equipment, storage medium and program product |
| CN113377360B (en) * | 2021-06-28 | 2023-09-26 | 北京百度网讯科技有限公司 | Task execution method, device, electronic equipment, storage medium and program product |
| CN113672601A (en) * | 2021-07-22 | 2021-11-19 | 北京明略软件系统有限公司 | Streaming data supplementing method and system, electronic device and storage medium |
| CN114385137A (en) * | 2021-12-29 | 2022-04-22 | 武汉达梦数据库股份有限公司 | Flow method and device for running ETL (extract transform load) by Flink framework |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112417014A (en) | Dynamic modification execution plan method, system, and computer-readable storage medium | |
| US8626786B2 (en) | Dynamic language checking | |
| CN103034805B (en) | Multi engine checking and killing virus method and apparatus | |
| CN107563201B (en) | Associated sample searching method and device based on machine learning and server | |
| US8832666B2 (en) | Dynamic instrumentation | |
| CN107688538B (en) | Script execution method and device and computing equipment | |
| CN109684008B (en) | Card rendering method, device, terminal and computer readable storage medium | |
| US20150143348A1 (en) | Hybrid dynamic code compiling device, method, and service system thereof | |
| CN109831466B (en) | Micro-service business processing method and nginx server | |
| CN113961919B (en) | Malicious software detection method and device | |
| CN111586097A (en) | Network request processing method, computing device and storage medium | |
| US20120297173A1 (en) | Debugger system, method and computer program product for debugging instructions | |
| CN114337920B (en) | Code analysis method and device and electronic equipment | |
| US8959088B2 (en) | Log management method, log management system, and information processing apparatus | |
| CN113282347B (en) | Plug-in operation method, device, equipment and storage medium | |
| US11200109B2 (en) | Method, device and computer program product for processing data | |
| CN113656830A (en) | Database desensitization grammar parsing method, system, computer and readable storage medium | |
| CN114637969A (en) | Target object authentication method and device | |
| CN114035846A (en) | Instruction verification method and instruction verification device | |
| CN110244954B (en) | Compiling method and equipment for application program | |
| CN113220362A (en) | Log record obtaining method and device, computer equipment and storage medium | |
| CN110888686B (en) | Application program starting method, device and storage medium | |
| CN113590133A (en) | Android system injection detection method, device, equipment and computer storage medium | |
| CN114490010A (en) | Resource operation control method, electronic device, chip and readable storage medium | |
| US20160012039A1 (en) | Providing context in functional testing of web services |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210226 |