CN111125594A - Method and system for managing a large number of task input parameters in distributed system - Google Patents
Method and system for managing a large number of task input parameters in distributed system Download PDFInfo
- Publication number
- CN111125594A CN111125594A CN201911227366.1A CN201911227366A CN111125594A CN 111125594 A CN111125594 A CN 111125594A CN 201911227366 A CN201911227366 A CN 201911227366A CN 111125594 A CN111125594 A CN 111125594A
- Authority
- CN
- China
- Prior art keywords
- input
- input parameter
- client
- context
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a method and a system for managing a large number of task input parameters in a distributed system, which comprises the following steps: step A: inputting an input parameter template and a corresponding context rule description; b, selecting a pair of input parameter templates and context rule description; and C: automatically generating a form interface; filling context data in the form interface; step D: the client automatically verifies the context data input by the user according to the context rule description; step D1: if the verification is not passed, the client reminds the user of re-inputting; step D2: if the verification is passed, the client calls a template engine to render a final input parameter by combining an input parameter template and the context data; step E: and the client conducts the input parameter result obtained in the last step to a task submitting flow. The invention provides a set of system realization method for management, generation and verification of a large number of complex input parameters.
Description
Technical Field
The invention belongs to distributed computing, and particularly relates to a method and a system for managing a large number of task input parameters in a distributed system.
Background
In the prior art, systematic management of a large number of complex input parameters is not emphasized: or based on flexibility consideration, the whole input parameter is automatically and completely specified by a service operator when scheduling tasks each time; or based on the standardized consideration, a program developer separately develops and generates a corresponding input form for the input parameters of each task, and then a business operator controls the final input parameters through the form.
If a business operator designates complete input parameters autonomously, the operator can pay more attention to the parameter details of the bottom task execution unit, and the operator needs to pay more attention to the application scene, which obviously improves the learning cost and the use threshold of the whole system;
if the form code is developed by the developer according to the input parameters of all types of tasks one by one, the development period is prolonged obviously, and the daily change of the parameter details also introduces additional program development or adjustment, so that the business system is lack of flexibility.
Disclosure of Invention
In order to solve the above technical problems, the present invention provides a method for managing a large number of task input parameters in a distributed system, which comprises the following steps:
step A: inputting an input parameter template and a corresponding context rule description;
b, selecting a pair of input parameter templates and context rule description;
and C: automatically generating a form interface; filling context data in the form interface;
step D: the client automatically verifies the context data input by the user according to the context rule description;
step D1: if the verification is not passed, the client reminds the user of re-inputting;
step D2: if the verification is passed, the client calls a template engine to render a final input parameter by combining an input parameter template and the context data;
step E: and the client conducts the input parameter result obtained in the last step to a task submitting flow.
Preferably, a task can be represented as an "executable unit" + "input parameter" at the technical level, and the executable unit adopts a Docker mirror image or a cloud function. To ensure flexible reusability, the executable units are implemented to perform some type of computational logic, typically to solve one type of task and adapt to different application scenarios. Therefore, in a specific task submission, an exhaustive structured document, i.e., input parameters, needs to be provided to indicate the details of the task execution. The task input parameters have certain complexity, and the parameter formats are also diversified. All input parameters can be expressed and passed through textual objects (such as the results of JSON/YAML serialization) and parsed by the executable unit into internal objects upon receipt of the textual input parameters.
Preferably, in order to conveniently manage a plurality of types of input parameters, a specific input parameter template is summarized for a common operation scene, one input parameter template is a text object following a specific grammar format, such a template can define the overall structure of the input parameters, shield details which do not need to be concerned by an operator in the scene to which the template is directed, mark the parts which need to be dynamically specified in a placeholder form, and finally render the parts into the target input parameters in a text form by combining a template engine and context data provided by the operator. The "target input parameter" is the "input parameter" and takes the "target" definition to mean that it is rendered by the template plus context data in this case and is not directly entered.
The invention also provides a system for managing a large number of task input parameters in a distributed system, which is connected with a client and comprises:
the management device: providing functions of inputting, modifying and deleting an input parameter template and a corresponding context rule description;
the inquiry device comprises: providing a query function to enable selection of a pair of input parameter templates and context rule descriptions according to a query condition;
the generation device: configuring to an automatic generation form interface, and filling context data in the form interface;
the checking device comprises: the client automatically verifies the context data input by the user according to the context rule description, and if the verification is not passed, the client reminds the user to input again;
a rendering device: if the verification is passed, the client calls a template engine to render a final input parameter by combining an input parameter template and the context data;
the execution device: and conducting the input parameter result obtained in the last step to a task submitting flow. Preferably, the input parameters are correspondingly provided with executable units, and the executable units adopt Docker images or cloud functions.
Preferably, the input parameter is a structured document.
The invention is suitable for a business system which depends on a plurality of tasks which run in a distributed task scheduling platform and can be disassembled into a plurality of tasks which are matched with each other to finish a certain type of calculation targets (cases) together. In such a business system, to accomplish a certain kind of computation objective, a plurality of tasks of different types need to be created in parallel or in series, and each task may be set with specific input parameters manually according to the scene (such as the characteristics of a case, the feedback results of other tasks, etc.). The technical scheme provides a set of system realization method for the management, generation and verification of a large number of complex input parameters, and provides guarantee for the orderly and reliable operation of a service system.
Drawings
Fig. 1 is a principle of template rendering in embodiment 1.
FIG. 2 shows the construction of the parameter sprite in example 2.
Fig. 3 is a parameter wizard table structure in embodiment 2.
Fig. 4 is a screenshot of the query interface in example 2.
FIG. 5 is an automatically generated form and input verification effect of example 2.
FIG. 6 is a JSON representation of the input parameters for the simplified XRD Indexing task in example 3.
FIG. 7 is a screenshot of an entry process for an input parameter template in a parameter wizard entry form.
FIG. 8 is a screenshot of an entry process described by context rules in a parameter wizard entry form.
Detailed Description
Preferred embodiments of the present invention will be described in further detail below with reference to the accompanying drawings:
example 1
A method for managing a large number of task input parameters in a distributed system is disclosed, which comprises the following steps:
the input parameters of all the executable units set by the method can be expressed and transmitted through text objects (such as the serialization result of JSON/YAML), and the executable units automatically analyze the input parameters into internal objects after receiving the input parameters of the text.
In order to conveniently manage a plurality of types of input parameters, a targeted input parameter template is summarized aiming at a common operation scene, one input parameter template is a text object following a specific grammar format, the template can define the overall structure of the input parameters, shield details which do not need to be concerned by an operator in the scene aimed at by the template, mark parts which need to be dynamically specified in a placeholder form, and finally render the target input parameters into a text form by combining a template engine and context data provided by the operator. The general principle of stencil rendering is explained, as is the rendering principle shown in FIG. 1.
In order to define and verify the context data required to be provided by the operator, a context rule description is defined for each input parameter template, and a special grammar is also required to be established for the rule description, wherein the context rule description details variables required to be provided by the operator and information such as types of the variables, constraint rules, default values, help and the like, and the rule description can be used for realizing an input indicator and an input checker: the client can automatically generate a friendly context fill-in form interface based on the rule description and can verify the user input according to the rule.
Based on the above setting, to generate a specific input parameter for the support service operator, the following steps are required:
A) a background manager inputs an input parameter template and a corresponding context rule description;
B) a service operator selects a pair of input parameter templates and context rule description at a client;
C) the client automatically generates a form interface according to the context rule description;
D) filling context data in a form interface generated by a user through a client;
E) the client automatically verifies the context data input by the user according to the context rule description;
F) if the verification is not passed, the client reminds the user of re-inputting;
G) if the verification is passed, the client calls a template engine to render a final input parameter by combining an input parameter template and the context;
H) example 2 in which the client conducts the input parameter result obtained in the previous step to the task submission process
Based on the method in the embodiment 1, a set of system for managing a large number of task input parameters in a distributed system is realized:
in the system, an input parameter template and a context rule description corresponding to the input parameter template are defined as a parameter sprite, and a special table is used in a database to store all the parameter sprites. The composition and table structure design of the parameter puck is shown in fig. 2 and 3.
Wherein, two core fields in the table structure are:
template, representing input parameter template, following the grammar rule of Jinja2, stored in text form
context spec represents the description of the context rule, and the syntax is based on the extension of JSON Schema and is stored in JSON format.
The system provides the functions of adding, deleting and changing parameters of a wizard facing an administrator, and a screenshot is shown in a query interface in figure 4.
In an application scenario where input parameters need to be generated according to the parameter sprites, an operator may list available parameter sprites through condition query (such as label filtering), or a client program may automatically list parameter sprites meeting conditions according to the application scenario, or one parameter sprite may be bound to a certain application scenario and automatically selected by the system.
After the parameter wizard is selected, the client automatically generates a friendly context filling-in form according to the method described in statement 1 by using the context rule description, and verifies the context data input by the user during the user input process and before submitting the form, as shown in fig. 5, which shows an automatically generated form and the input verification effect. If the form passes the verification, the template engine is called to render the final input parameter by combining the input parameter template and the context data, and the final input parameter is applied to the current business scene (how to use the input parameter by the subsequent business process does not belong to the attention scope of the invention).
Example 3
The management and use of the parametric puck is explained below by "XRD Indexing" (i.e. XRD indexed analysis) as an example:
the "XRD Indexing" task is used to analytically calculate the unit cell parameters of the crystal from the crystal powder XRD data. There are two core algorithms to implement this process: PSO and Rec; and need to provide known properties about the crystal, such as molecular volume estimated from the number of atoms, etc.; in addition, the service operator can also specify the number of CPU cores, the size of the internal memory and the like utilized when the task is executed according to the performance requirement. To focus on the subject matter of the present invention, this example does not detail all XRDIrendering related parameter items, and as mentioned above, a simplified JSON representation of the input parameters for the XRD Indexing task is shown in FIG. 6, where the right area of FIG. 6 describes this parameter using tabulation for ease of understanding.
The JSON document as in figure 6 is a structured representation of XRD Indexing task input parameters, which expresses elements of how the task is performed, embodied as parameter entries in said JSON document, each parameter entry being associated with a parameter value. The most important parameter item in fig. 6 is "XRD _ data", i.e. XRD data of the crystal powder, in addition: "image" corresponds to the name of an executable unit; "cores" corresponds to the number of CPU cores; "memory" represents the memory size; "method" corresponds to the two alternative core algorithms mentioned above; "volume _ mol" corresponds to the molecular volume.
Note that the system does not let the user directly input the input parameters in the above form, but automatically generates the target input parameters according to the method described in embodiment 1. The administrator therefore needs to enter the specific parameter wizard: FIG. 7 is a screenshot of an entry process for an input parameter template in a parameter wizard entry form: the parameter template as entered in fig. 7 amounts to a framework being formulated for all possible input parameters so that the finally generated input parameters can meet the format requirements of the executable unit. Variables that require autonomous control by the business operator, such as "{ { volume _ mol } }" in the screenshot, are marked with placeholders in the template, indicating that values for the volume _ mol variable need to be inserted at the location where they are located, from the context data that the business operator will provide.
FIG. 8 is a screenshot of the entry process described by the context rules in the parameter wizard entry form: fig. 8 shows a process of entering a context rule description, where the context rule description is used to define and verify context data that an operator needs to provide, and a context rule description details variables that the operator needs to provide, and information about the types of the variables, constraint rules, default values, and help, such a rule description can be used to implement both an input indicator and an input verifier: the client can automatically generate a friendly context fill-in form interface based on the rule description and can verify the user input according to the rule. In connection with the present embodiment, the context rule description referred to in FIG. 8 is explained as follows:
1. designate "xrd _ data" and "volume _ mol" as the parameter items that must be filled by the user, note the lighted "+" mark in the drawing
2. All the parameter item types are defined, such as "cores", "memory" is number type, and "indicating _ method" is string type
3. The input value defining the "indicating _ method" parameter item must be either "PSO" or "Rec", noting the setting of the "enumerated value" in the figure
4. A default value "PSO" of "indicating _ method" is specified, which will be automatically selected by the business operator, but the operator can choose whether to override the default value
The invention can make a large amount of input parameters be managed and maintained in a centralized way by storing the parameter template and the context rule description; due to the introduction of the text template concept, an operator does not need to pay more attention to the detail requirements of input parameters of the underlying task, and only needs to pay attention to the use scene, so that the learning cost and the use threshold of the system are effectively reduced; the idea of describing the automatic generation of the form interface and the checker by using the context rule reduces the development and maintenance cost of the system and enhances the flexibility of the business system.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (5)
1. A method for managing a large number of task input parameters in a distributed system is characterized by comprising the following steps:
step A: inputting an input parameter template and a corresponding context rule description;
b, selecting a pair of input parameter templates and context rule description;
and C: automatically generating a form interface; filling context data in the form interface;
step D: the client automatically verifies the context data input by the user according to the context rule description;
step D1: if the verification is not passed, the client reminds the user of re-inputting;
step D2: if the verification is passed, the client calls a template engine to render a final input parameter by combining an input parameter template and the context data;
step E: and the client conducts the input parameter result obtained in the last step to a task submitting flow.
2. The distributed-oriented system-based method for managing a plurality of task input parameters of claim 1, wherein all input parameters are expressed and passed through textual objects, which are parsed by the executable unit into internal objects upon receiving the textual input parameters.
3. A system for managing a plurality of task input parameters in a distributed-oriented system as recited in claim 2, wherein the input parameters are in the form of structured documents.
4. A method for managing a large number of task input parameters in a distributed-oriented system as recited in claim 2, wherein the text objects are in the form of JSON or YAML serialization.
5. A system for managing a plurality of task input parameters in a distributed-oriented system, connected to a client, comprising:
the management device: providing functions of inputting, modifying and deleting an input parameter template and a corresponding context rule description;
the inquiry device comprises: providing a query function to enable selection of a pair of input parameter templates and context rule descriptions according to a query condition;
the generation device: configuring to an automatic generation form interface, and filling context data in the form interface;
the checking device comprises: the client automatically verifies the context data input by the user according to the context rule description, and if the verification is not passed, the client reminds the user to input again;
a rendering device: if the verification is passed, the client calls a template engine to render a final input parameter by combining an input parameter template and the context data;
the execution device: and conducting the input parameter result obtained in the last step to a task submitting flow.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911227366.1A CN111125594A (en) | 2019-12-04 | 2019-12-04 | Method and system for managing a large number of task input parameters in distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911227366.1A CN111125594A (en) | 2019-12-04 | 2019-12-04 | Method and system for managing a large number of task input parameters in distributed system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111125594A true CN111125594A (en) | 2020-05-08 |
Family
ID=70497665
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911227366.1A Pending CN111125594A (en) | 2019-12-04 | 2019-12-04 | Method and system for managing a large number of task input parameters in distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111125594A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112445595A (en) * | 2020-11-26 | 2021-03-05 | 深圳晶泰科技有限公司 | Multitask submission system based on slurm computing platform |
| CN115113947A (en) * | 2022-06-27 | 2022-09-27 | 浙江惠瀜网络科技有限公司 | Configuration method of process engine, terminal equipment and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1920837A (en) * | 2006-09-14 | 2007-02-28 | 东北大学 | Complex equipment faced multi-subject design software integrated parameter mapping method |
| CN101211175A (en) * | 2006-12-29 | 2008-07-02 | 中国科学院沈阳计算技术研究所有限公司 | Numerical control system graph-aided macro programming design method |
| CN102346458A (en) * | 2010-08-06 | 2012-02-08 | 上海开通数控有限公司 | Method for realizing parameterization programming on numerical control system |
| US10108592B1 (en) * | 2013-12-12 | 2018-10-23 | Google Llc | Methods and systems for chunking markup language documents containing style rules |
| CN109460223A (en) * | 2018-11-14 | 2019-03-12 | 沈阳林科信息技术有限公司 | A kind of API gateway management system and its method |
-
2019
- 2019-12-04 CN CN201911227366.1A patent/CN111125594A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1920837A (en) * | 2006-09-14 | 2007-02-28 | 东北大学 | Complex equipment faced multi-subject design software integrated parameter mapping method |
| CN101211175A (en) * | 2006-12-29 | 2008-07-02 | 中国科学院沈阳计算技术研究所有限公司 | Numerical control system graph-aided macro programming design method |
| CN102346458A (en) * | 2010-08-06 | 2012-02-08 | 上海开通数控有限公司 | Method for realizing parameterization programming on numerical control system |
| US10108592B1 (en) * | 2013-12-12 | 2018-10-23 | Google Llc | Methods and systems for chunking markup language documents containing style rules |
| CN109460223A (en) * | 2018-11-14 | 2019-03-12 | 沈阳林科信息技术有限公司 | A kind of API gateway management system and its method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112445595A (en) * | 2020-11-26 | 2021-03-05 | 深圳晶泰科技有限公司 | Multitask submission system based on slurm computing platform |
| CN115113947A (en) * | 2022-06-27 | 2022-09-27 | 浙江惠瀜网络科技有限公司 | Configuration method of process engine, terminal equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11561772B2 (en) | Low-code development platform | |
| US8930337B2 (en) | Mapping dataset elements | |
| Seifermann et al. | Data-driven software architecture for analyzing confidentiality | |
| US7805635B2 (en) | Constraint programming for reduction of system test-configuration-matrix complexity | |
| CN108492005B (en) | Project data processing method and device, computer equipment and storage medium | |
| CN109299913B (en) | Employee salary scheme generation method and device | |
| US10275234B2 (en) | Selective bypass of code flows in software program | |
| US20190096012A1 (en) | System and method for real estate development and construction cost and risk management | |
| CN111125594A (en) | Method and system for managing a large number of task input parameters in distributed system | |
| EP2199905A1 (en) | Lifecycle management and consistency checking of object models using application platform tools | |
| Sarjoughian et al. | Behavioral DEVS metamodeling | |
| CN116069300A (en) | Workflow control code generation method and device, electronic equipment and storage medium | |
| US8448143B2 (en) | System and method for message choreographies of services | |
| CN112395339B (en) | Intersystem data admission verification method, device, computer equipment and storage medium | |
| CN112130849B (en) | Code automatic generation method and device | |
| US20170199858A1 (en) | Form management system and method | |
| CN116541020A (en) | Code generation method, device, equipment, medium and product based on field model | |
| CN116957512A (en) | Payment platform message management method, device and computer readable storage medium | |
| CN115757023A (en) | Business process processing method and device and finite-state machine model | |
| CN115511644A (en) | Processing method for target policy, electronic device and readable storage medium | |
| KR20190006349A (en) | System and method for providing client-server intergrated development platform | |
| WO2021109027A1 (en) | Method and system for managing large number of task input parameters in distributed system | |
| US20180284712A1 (en) | Integrated services platform | |
| US11971804B1 (en) | Methods and systems for an intelligent technical debt helper bot | |
| US20240152781A1 (en) | Rules determination via knowledge graph |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 3 / F, Shunfeng industrial building, No.2 Hongliu Road, Fubao community, Fubao street, Futian District, Shenzhen City, Guangdong Province Applicant after: Shenzhen Jingtai Technology Co.,Ltd. Address before: 518000 4th floor, No.9 Hualian Industrial Zone, Xinshi community, Dalang street, Longhua District, Shenzhen City, Guangdong Province Applicant before: Shenzhen Jingtai Technology Co.,Ltd. |