CN115061741A - Target plug-in operation method and device, electronic equipment and storage medium - Google Patents
Target plug-in operation method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115061741A CN115061741A CN202210615020.4A CN202210615020A CN115061741A CN 115061741 A CN115061741 A CN 115061741A CN 202210615020 A CN202210615020 A CN 202210615020A CN 115061741 A CN115061741 A CN 115061741A
- Authority
- CN
- China
- Prior art keywords
- script
- file
- target plug
- content
- script content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000003860 storage Methods 0.000 title claims abstract description 42
- 238000013515 script Methods 0.000 claims abstract description 348
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 117
- 238000004891 communication Methods 0.000 claims description 15
- 230000001419 dependent effect Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 238000010276 construction Methods 0.000 claims description 8
- 238000010801 machine learning Methods 0.000 description 23
- 230000008569 process Effects 0.000 description 6
- 238000004519 manufacturing process Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
- G06F9/44526—Plug-ins; Add-ons
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The application relates to a target plug-in operation method, a target plug-in operation device, an electronic device and a storage medium, wherein the method comprises the following steps: acquiring a file packet corresponding to a target plug-in; storing the file package to an algorithm platform, and acquiring configuration information of the file package in the algorithm platform, wherein the configuration information comprises a storage path and a file name of the file package in the algorithm platform; acquiring script content corresponding to the target plug-in, and writing configuration information into a script content designated position to update the script content, wherein the script content is used for generating a script file for starting the target plug-in; the script content is written into the local path to generate the script file, the script file is executed to start the target plug-in, and the script content is changed according to the configuration information of the file package in the algorithm platform when the target plug-in runs, so that the position of the file package in the algorithm platform is prevented from being directly set in the script content, and the script content and the file package can correspond to each other when the target plug-in runs.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for operating a target plug-in, an electronic device, and a storage medium.
Background
With the development of the internet, related personnel package the algorithm in a machine learning algorithm platform for use, and all plugins on the machine learning algorithm platform are stored in a database, including script contents (or script file addresses), file package file addresses, version numbers and the like corresponding to the plugins, and the file package file addresses, the version numbers and the like are directly written in the script contents. When the plug-in of the machine learning algorithm platform needs to run, firstly copying script contents corresponding to the plug-in into a corresponding environment and instantiating parameters, then copying a file packet to a path of a corresponding algorithm platform, and starting the plug-in through a script; in the iterative process of the plug-in, the attribute of the plug-in is covered to the latest version, and the versions of the plug-in are not necessarily downward compatible, after the file packet corresponding to the plug-in is updated, if the script content is not updated in time, the environment of the offline mode is damaged, so that the offline task may be tampered with, the data is incorrect, or the task execution fails due to the inconsistency of the file packet version and the parameters provided by the scheduling script, and the production accident is caused, therefore, how to avoid the operation error of the plug-in, and the task execution failure becomes the problem which needs to be solved urgently.
Disclosure of Invention
The application provides a target plug-in running method and device, electronic equipment and a storage medium, and aims to solve the problem that task execution fails due to plug-in running errors in the related art.
In a first aspect, the present application provides a target plug-in operation method, where the target plug-in operation method includes: acquiring a file packet corresponding to a target plug-in; storing the file package to an algorithm platform, and acquiring configuration information of the file package in the algorithm platform, wherein the configuration information comprises a storage path and a file name of the file package in the algorithm platform; acquiring script content corresponding to the target plug-in, and writing the configuration information into a specified position of the script content to update the script content, wherein the script content is used for generating a script file for starting the target plug-in; writing the script content into a local path to generate the script file, and executing the script file to start the target plug-in.
Optionally, writing the script content into a local path to generate the script file, and executing the script file to start the target plug-in, including: during manual tasks, after updating the script content is completed, the script content is directly written into a local path to generate the script file, and the script file is executed to start the target plug-in.
Optionally, writing the script content into a local path to generate the script file, and executing the script file to start the target plug-in, including: during an offline task, after updating the script content, storing the script content into a snapshot table, wherein the snapshot table is used for generating a corresponding script identifier according to the script content; submitting the script identification to a workflow scheduling platform so that the workflow scheduling platform generates a new scheduling task; and when a scheduling task with the script identifier, which is initiated by the workflow scheduling platform, is received, writing the script content into a local path to generate the script file, and executing the script file to start the target plug-in.
Optionally, when a scheduling task with the script identifier, which is initiated by the workflow scheduling platform, is received, writing the script content into a local path to generate a script file, and executing the script file to start the target plug-in, where the method includes: acquiring the script content according to the script identifier; writing the script content into a local path to generate the script file, and executing the script file to start the target plug-in.
Optionally, after writing the script content into a local path to generate the script file and executing the script file to start the target plug-in, the method further includes: saving an execution record, wherein the execution record comprises the script content currently executed.
Optionally, the obtaining of the file package corresponding to the target plug-in includes: and acquiring the file package corresponding to the target plug-in from an automatic construction tool, wherein the automatic construction tool is used for automatically constructing the file package according to the logic file and the dependent file corresponding to the target plug-in.
Optionally, the script content corresponding to the target plug-in includes: and acquiring the script content corresponding to the target plug-in from a warehouse management system, wherein the warehouse management system is used for managing the plurality of script contents corresponding to the target plug-in.
In a second aspect, the present application provides a target plug-in execution device, including: the acquisition module is used for acquiring a file package corresponding to the target plug-in; the uploading module is used for uploading the file package to an algorithm platform and acquiring configuration information of the file package in the algorithm platform, wherein the configuration information comprises a storage path and a file name of the file package in the algorithm platform; the acquisition module is further configured to acquire script content corresponding to the target plug-in, and write the configuration information into a specified position of the script content to update the script content, where the script content is used to generate a script file for starting the target plug-in; and the execution module is used for writing the script content into a local path to generate the script file and executing the script file to start the target plug-in.
In a third aspect, an electronic device is provided, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing the steps of the target plug-in operation method in any embodiment of the first aspect when executing the program stored in the memory.
In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the target plug-in execution method according to any one of the embodiments of the first aspect.
Compared with the prior art, the technical scheme provided by the embodiment of the application has the following advantages:
the method provided by the embodiment of the application comprises the following steps: acquiring a file package corresponding to a target plug-in, wherein the file package is used for storing a logic file and a dependent file corresponding to the target plug-in; storing the file package to an algorithm platform, and acquiring configuration information of the file package in the algorithm platform, wherein the configuration information comprises a storage path and a file name of the file package in the algorithm platform; acquiring script content corresponding to the target plug-in, and writing the configuration information into a specified position of the script content to update the script content, wherein the script content is used for generating a script file for starting the target plug-in; writing the script content into a local path to generate the script file, executing the script file to start the target plug-in, acquiring the configuration information of a file package corresponding to the target plug-in the algorithm platform when the plug-in is operated, and writing the configuration information into the script content, so that when the target plug-in is operated, the script content is changed according to the configuration information of the file package in the algorithm platform, and the problem that the position of the file package in the algorithm platform is directly set in the script content to cause the script content to be unmatched with the file package is avoided; when the target plug-in unit is in operation, the script content can correspond to the file packet, and then the target plug-in unit corresponding to the operation can be started successfully, so that the problem that the task execution fails due to the operation error of the plug-in unit is avoided, and the user experience is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic flowchart of a target plug-in operation method according to an embodiment of the present application;
fig. 2 is a schematic diagram of a basic structure of a target plug-in running device according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Fig. 1 is a schematic flowchart of a target plug-in operation method according to an embodiment of the present application. As shown in fig. 1, the target plug-in operation method includes:
s101, acquiring a file package corresponding to a target plug-in;
s102, uploading the file package to an algorithm platform, and acquiring configuration information of the file package in the algorithm platform, wherein the configuration information comprises a storage path and a file name of the file package in the algorithm platform;
s103, acquiring script content corresponding to the target plug-in, and writing the configuration information into a specified position of the script content to update the script content, wherein the script content is used for generating a script file for starting the target plug-in;
s104, writing the script content into a local path to generate the script file, and executing the script file to start the target plug-in.
It should be understood that the target plug-in operation method is applied to a server, for example, the target plug-in operation method is applied to a machine learning algorithm platform running on the server, in some examples, one target plug-in corresponds to one algorithm, that is, the machine learning algorithm platform stores the algorithm in the form of plug-ins, and the plug-ins currently needed to be used by the object are target plug-ins; the file package is used to store a logic file and a dependent file corresponding to the target plug-in, for example, the file package is a JAR (Java ARchive) package, where the JAR package includes a plurality of JAR files, and the JAR files are in a file format unrelated to the platform and can combine the plurality of files into one file. The use of objects can bind multiple Java applets and their required components (. class files, images and sounds) into JAR files, thereby storing the logical files executed by the target plug-in, and the corresponding dependent files.
It is understood that the file package is stored in an algorithm platform, wherein the algorithm platform is a storage server of a remote distributed cluster, and the storage server of the remote distributed cluster can realize distributed storage, and the distributed storage is to store data on a plurality of independent devices in a scattered manner. The traditional network storage system adopts a centralized storage server to store all data, the storage server becomes the bottleneck of the system performance, is also the focus of reliability and safety, and cannot meet the requirement of large-scale storage application. The distributed network storage system adopts an expandable system structure, utilizes a plurality of storage servers to share the storage load, and utilizes the position server to position the storage information, thereby not only improving the reliability, the availability and the access efficiency of the system, but also being easy to expand; the algorithm platform may adopt an intermediate control node architecture, such as an hdfs (hadoop Distribution File system) algorithm platform; the algorithm platform can also adopt a completely centerless architecture-computing mode, such as a Ceph system; moreover, the algorithm platform can also adopt a completely centerless architecture-consistent hash, such as a swift algorithm platform.
After the file package is stored in the algorithm platform, the file package has a storage path and a unique file name under the storage path in the algorithm platform, and the file package can be determined according to the storage path and the unique file name under the storage path.
It should be understood that script content corresponding to the target plug-in is obtained, where the script content is used to generate a script file, the script file is a necessary file for starting the target plug-in, and is usually a shell file, the script content includes task-related information, such as a task name, a task ID, a user name, a Hadoop queue, a task parameter, and the like, and there is a wildcard in the script content, where the location of the wildcard is a specified location, and the wildcard is used to replace configuration information (a storage path and a file name) representing a file package in an algorithm platform; and when acquiring the script content corresponding to the target plug-in and the configuration information of the file package in the algorithm platform, replacing the wildcard with the configuration information, and updating the script content, so that the updated script content comprises the configuration information.
It should be understood that the script contents are written into a local path to generate the script file, and the script file is executed to launch the target plug-in, wherein the local path refers to a path of a terminal that is to execute the script file, and the terminal may be implemented in various forms. For example, the terminal described in the present invention may include a mobile terminal such as a mobile phone, a tablet computer, a notebook computer, a palmtop computer, a Personal Digital Assistant (PDA), a Portable Media Player (PMP), a navigation device, a wearable device, a smart band, a pedometer, and the like, and a fixed terminal such as a Digital TV, a desktop computer, and the like. After the script content is written into the local path to generate the script file, executing the script file can start running the corresponding target plug-in.
In some examples of this embodiment, writing the script content to a local path to generate a script file, and executing the script file to launch the target plug-in includes: during manual tasks, after updating the script content is completed, the script content is directly written into a local path to generate a script file, and the script file is executed to start the target plug-in. After script content corresponding to the target plug-in is obtained and configuration information is written into the script content, after the script content is updated, whether to immediately write the script content into a local path is required to be judged, and the script file is executed to run the target plug-in; specifically, before writing the script content into a script file of a local path and executing the script content in the script file to run the target plug-in, the method further includes: the method comprises the steps of obtaining the deployment type of the current deployment target plug-in operation, determining to write the script content into a script file of a local path by obtaining the current deployment type, and executing the script content in the script file to operate the target plug-in mode.
Bearing the above example, when the current deployment type is the manual task mode, after updating the script content is completed, the script content is directly written into the local path, and the script file is executed to run the target plug-in, wherein the usage scenario of the manual mode is that the relevant usage object performs experimental and one-time task execution through an interface, so that the target plug-in of the latest version, the corresponding file package and the script are directly used for the algorithm in the manual mode.
It can be understood that, when the script content is written into the local path to generate the script file, the filename of the script content may be named as a suffix according to the timestamp, thereby ensuring that script content of other versions is not covered; it can be understood that, in this embodiment, the file name naming mode of the script content is not limited when the script content is written into the local path to generate the script file, and may be flexibly set by the related object to be used, so as to ensure that the script content of other versions is not covered.
In some examples of this embodiment, writing the script content to a local path to generate a script file, and executing the script file to run the target plug-in includes: during an offline task, after updating the script content, storing the script content into a snapshot table, wherein the snapshot table is used for generating a corresponding script identifier according to the script content; submitting the script identification to a workflow scheduling platform so that the workflow scheduling platform generates a new scheduling task; and when a scheduling task with the script identifier, which is initiated by the workflow scheduling platform, is received, writing the script content into a local path to generate a script file, and executing the script file to start the target plug-in. Acquiring script content corresponding to the target plug-in, writing configuration information into the script content, and after updating the script content, judging whether to immediately write the script content into a local path; specifically, before writing the script content into a script file of a local path and executing the script content in the script file to run the target plug-in, the method further includes: the method comprises the steps of obtaining the deployment type of the current deployment target plug-in operation, determining to write the script content into a script file of a local path by obtaining the current deployment type, and executing the script content in the script file to operate the target plug-in mode.
In the above example, when the current deployment type is the offline task mode, after updating the script content is completed, the script content is stored in a snapshot table, and the snapshot table is used for generating a corresponding script identifier according to the script content; submitting the script identification to a workflow scheduling platform so that the workflow scheduling platform generates a new scheduling task; when a scheduling task with the script identification, which is initiated by the workflow scheduling platform, is received, the script content is written into a local path to generate a script file, and the script file is executed to start the target plug-in; the off-line mode is that the using object submits the using requirement to the workflow scheduling platform, the workflow scheduling platform carries out periodic scheduling execution according to the using requirement, and for the target plug-in the off-line mode, in order to keep production safety, automatic version updating is not carried out on the target plug-in, but the version when the task is submitted should be used.
It should be understood that, where the snapshot table is stored in a database, the database may be disposed on any storage device with a storage function, when the script content is stored in the snapshot table, the snapshot table will generate a unique script identifier for the script content, and then the script identifier can be used to find the content of the script; it should be understood that, where the workflow scheduling platform includes, but is not limited to, a web workflow scheduling platform, when the script identification is submitted to the workflow scheduling platform, the workflow scheduling platform can subsequently generate a new scheduling task according to the script identification.
In some examples of this embodiment, when receiving a scheduling task with the script identifier initiated by the workflow scheduling platform, writing the script content into a local path to generate a script file, and executing the script file to start running the target plug-in, includes: acquiring the script content according to the script identifier; and writing the script content into a local path to generate a script file, and executing the script file to start running the target plug-in. Specifically, when a scheduling task with a script identifier, which is initiated by the workflow platform, is received, script content corresponding to the script identifier is searched in a snapshot table according to the script identifier, the searched script content is written into a local path, and the script file is executed to start a running target plug-in; the method for searching the script content according to the script identifier includes, but is not limited to: traversal database lookup, hash (hash) algorithm lookup, and the like.
In some examples of this embodiment, after writing the script content to the local path to generate a script file and executing the script file to start running the target plug-in, the method further includes: saving an execution record, wherein the execution record comprises the script content currently executed. It can be understood that, when the deployment type is the manual mode or the offline mode, the script content is written into the script file of the local path, and after the script content in the script file is executed to run the target plug-in, an execution record needs to be saved, where the execution record includes the script content of the currently executed script file, and if a program error (bug) of the target plug-in is triggered in the execution process, a related usage object can obtain the currently executed script content according to the execution record, and can determine a corresponding file package according to configuration information in the script content, so as to determine a code failure cause according to the obtained script content and file package.
In some examples of this embodiment, obtaining the package of files corresponding to the target plug-in includes: and acquiring the file package corresponding to the target plug-in from an automatic construction tool, wherein the automatic construction tool is used for automatically constructing the file package according to the logic file and the dependent file corresponding to the target plug-in. It should be understood that automated build tools include, but are not limited to: a maven automated construction tool; wherein, adding corresponding configuration (logic file) in the file at the related object, the maven will automatically download the corresponding jar package (file package); the dependency file can be downloaded according to the jar package, and as long as other jar packages on which the configured jar package depends can be automatically downloaded by the dependency file, the example automatically constructs the file package according to the logic file and the dependency file corresponding to the target plug-in through an automatic construction tool, so that the automatic generation of the script content corresponding to the target plug-in is realized, and related personnel do not need to manually generate according to the logic file and the dependency file corresponding to the script content, so that the operation and maintenance efficiency is improved; meanwhile, the file package is automatically built through an automatic building tool according to the logic file and the dependent file corresponding to the target plug-in, the file packages of different versions can be subjected to version management, the situation that related personnel manually perform version management on the file packages of different versions is avoided, operation and maintenance efficiency is improved, and user experience is further improved.
In some examples of this embodiment, the script content corresponding to the target plug-in includes: and acquiring the script content corresponding to the target plug-in from a warehouse management system, wherein the warehouse management system is used for managing the plurality of script contents corresponding to the target plug-in. It should be understood that warehouse management systems include, but are not limited to: a gitlab warehouse management system; the Web service is built on the basis that Git is used as a code management tool, and public or private items can be accessed through a Web interface. It has the similar function as Github, and can browse source code, manage defect and comment, so as to make the related object for use to manage script content; for example, when the script content of the target plug-in is updated by the related using object, the updated script content is submitted to the Gitlab for management, and when the machine learning algorithm platform needs to acquire the script content, the script content corresponding to the target plug-in is acquired from the Gitlab.
The target plug-in operation method provided by the embodiment comprises the following steps: acquiring a file package corresponding to a target plug-in, wherein the file package is used for storing a logic file and a dependent file corresponding to the target plug-in; storing the file package to an algorithm platform, and acquiring configuration information of the file package in the algorithm platform, wherein the configuration information comprises a storage path and a file name of the file package in the algorithm platform; acquiring script content corresponding to the target plug-in, and writing the configuration information into the specified position of the script content to update the script content, wherein the script content is used for starting the target plug-in; the script content is written into the script file of the local path, the script content in the script file is executed to run the target plug-in, when the plug-in is run, the configuration information of the file package corresponding to the target plug-in the algorithm platform is obtained, the configuration information is written into the script content, and then when the target plug-in is run, the script content is changed according to the configuration information of the file package in the algorithm platform, so that the position of the file package in the algorithm platform is directly set in the script content, the script content and the file package can correspond when the target plug-in is run, the problem of task execution failure caused by the running error of the plug-in is avoided, and the user experience is improved.
For better understanding of the present invention, the present embodiment provides a more specific example to illustrate the present invention;
firstly, a target plug-in method runs on a machine learning algorithm platform which is arranged on a server, and a machine learning algorithm platform runs a target plug-in, which usually involves several preconditions, specifically as follows:
1. the target plug-in runs the necessary files: executing the starting script file and the file packet;
the script file is generated according to script content, the script file is a necessary entry for starting a task, and is usually a shell file, and the script content comprises task related information such as a task name, a task ID, a user name, a Hadoop queue, task parameters and the like;
the file package is a jar package which comprises: the method comprises one or more jar files, wherein the jar files comprise logic files executed by an algorithm corresponding to a current target plug-in and corresponding dependent files, and jar packages need to be stored in a designated directory (such as HDFS) in a storage server of a remote distributed cluster.
2. Machine learning algorithm platform: the machine learning algorithm platform stores each algorithm in the form of a plug-in, and loads the algorithm by loading the plug-in, namely, each algorithm on the machine learning algorithm platform is loaded in a mode of configuration in a management background. For a big data machine learning algorithm, it is usually performed in a cluster task, for example, a Spark algorithm task. And a key material for executing a Spark algorithm task is a corresponding jar packet stored in an algorithm platform (taking HDFS as an example) (in the following, tasks running on Spark engines are taken as examples, the jar packet stored in HDFS needs to be called, if the algorithm is executed by other distributed engines, specific execution codes also need to be stored in a corresponding remote storage supported in a compiling packet form), and an execution script for locally submitting a cluster task. Bearing the above example, for an algorithm platform, a jar package and a corresponding script of an algorithm need to be iterated continuously, so that jar packages and scripts of different versions may exist.
It can be understood that the machine learning algorithm platform runs a task, and comprises two running modes, namely a manual task mode: the use scenario is that the object is used to perform experimental and disposable tasks through the interface. For algorithms in manual mode, the latest version is generally used directly; an offline task mode: the use scene is that the use object is submitted to a workflow scheduling platform and the platform performs periodic scheduling execution. For algorithms in offline mode, in order to keep production safe, they are not typically automatically version-updated, but rather should use the version at the time of the submission of the task.
It can be understood that the information of the algorithm platform for the algorithm plug-in is stored in the database, and includes script content (or script file address), jar package file address, version number, and the like. When the algorithm needs to run, the script content is copied to the corresponding environment and the parameters are instantiated, the jar packet is copied to the corresponding HDFS path, and the script can know from the above preconditions. Each version of the algorithm is not necessarily downward compatible, so that an offline task is tampered with, data is incorrect, or the task is failed to execute due to inconsistency of the version and parameters provided by a scheduling script, and a production accident is caused; in addition, when the offline task triggers the bug to have a fault, if the effective version management is not available, developers can hardly locate which version of plug-in code has the fault; finally, after the user iterates the algorithm code, a series of operations are quite complicated and errors can occur until the jar packet and the version are stored to the corresponding address and the configuration of the algorithm plug-in is modified.
Therefore, to isolate the environment, the manual mode and the offline mode are stored separately. The algorithmic properties required for the two modes include: the method comprises the steps of script (a git address of the script can be saved in a manual mode, so that automatic deployment is facilitated after development of developers), jar packet (a maven address saved in the jar packet is saved in a manual mode, so that automatic deployment of the developers is facilitated), and file path of a specific jar packet version saved in hdfs during deployment in an offline mode.
3. A workflow scheduling platform: for offline mode timing scheduling jobs. When the algorithm needs to be called off-line, the workflow scheduling platform initiates an off-line task request to the algorithm platform, and the ID of the off-line task is attached. After the algorithm platform obtains the off-line task ID, the off-line algorithm component is pulled, the off-line version script and jar packet are obtained, and the Spark task is initiated by the machine learning algorithm platform.
4. A third party store comprising:
code storage, specifically used for storing script contents;
storage of binary files such as maven, code compilation jar package and the like
The distributed algorithm platform is used for reading and storing data by a distributed computing engine such as Spark and the like, and a jar packet required by the distributed algorithm platform needs to be read from an hdfs path;
the main working idea of the target plug-in operation method provided by the embodiment is as follows:
executing a task (target plug-in) in a manual mode, and acquiring corresponding script content of the latest version and a jar packet maven path from a model table by a machine learning algorithm platform; after script contents are stored under a local temporary folder, uploading a jar package to an hdfs specified path after downloading a latest version from maven, and covering files with the same name; after the paths of jar packets corresponding to the scripts are modified, the scripts are executed, and then Spark tasks can be started according to the algorithm of the latest version;
before the offline mode, a user needs to perform offline deployment operation through an algorithm platform. The operation aims to save the current latest version algorithm into an offline snapshot, so that the current version is prevented from being influenced by tampering. When an offline task is deployed, the jar packet is uploaded to hdfs and then uniquely renamed, the path of the jar packet is saved in script content, and the script content is saved in a snapshot table. And the snapshot ID is used as the ID of the current offline version and is provided for the workflow scheduling platform to be started and used at regular time.
And starting an offline task by the workflow scheduling platform, and displaying an offline id to the algorithm platform. And the algorithm platform queries the offline id active offline script through the database, exports the script content into a script file, places the script file locally, executes the local script file, and then starts a Spark task according to the corresponding jar packet file version in the script.
The method for operating the target plug-in provided by the embodiment specifically comprises the following steps:
developing and configuring the algorithm plug-in by the machine learning algorithm platform:
a. the machine learning algorithm platform is used for configuring algorithms, the machine learning algorithm platform stores each algorithm in a plug-in mode, and the configuration algorithms comprise git addresses of script contents of a plug-in startup script and maven addresses corresponding to jar packets;
b. after the developer updates the script content of the algorithm starting script, submitting the script content to git; the JAR hdfs PATH involved in the script content is replaced with a wildcard (e.g., "$ { JAR _ PATH }"), instantiated when the script is deployed;
c. when a developer updates the spark algorithm, the spark algorithm can be directly deployed to maven through IDE (such as idea), the maven automatically updates the file version and renames the file name (such as predict-1.0-20211225.033010-1.jar, the file name is named according to time, and uniqueness can be ensured)
Deployment process of machine learning algorithm platform (since the manual mode is bundled to be deployed and executed, the deployment steps are similar to the offline mode, so the deployment process can be described separately)
a. Inquiring a jar package file path of the latest version from the maven path;
b. downloading jar packets from maven to local;
c. uploading the jar package to a hdfs specified folder, and recording the name of the current jar file;
d. drawing script content corresponding to the target plug-in from git, and replacing the $ { JAR _ PATH } wildcard in the script content to be the actual hdfs PATH + file name of the current JAR;
e. if the deployment of the manual task is performed, the script code is written into the script file under the path of the local path execution server, and the file name can be named as a suffix according to the timestamp, so that other scripts can not be covered, and the completion is realized.
f. If the deployment is the offline task, the script instance with the wildcard replaced is saved in a snapshot table of the database, and the completion is finished
The method comprises the following steps of manually running tasks on a machine learning algorithm platform:
a. performing the manual task deployment procedure a-e of 2 above
b. Executing the script generated by the a-e steps
c. Saving an execution record in a database, wherein the execution record comprises the currently executed script content;
the step of deploying the task offline on the machine learning algorithm platform comprises the following steps:
a. performing the offline task deployment procedure a-d, f of 2 above
b. After the script instance obtained after deployment is saved in a snapshot table, the current snapshot id is obtained
c. Submitting the snapshot id to a workflow scheduling platform to generate a new scheduling task;
offline task execution
a. The workflow scheduling platform initiates a scheduling task to the machine learning algorithm platform, and the scheduling task is provided with a snapshot id
b. After receiving the off-line task execution request, the machine learning algorithm platform reads the snapshot script content and writes the snapshot script content into a script file under a local path, and the file name can be named as a suffix according to the timestamp so as to ensure that other scripts cannot be covered;
c. executing the generated script file
d. Saving an execution record in a database, the execution record including script content currently executed
According to the target plug-in operation method provided by the example, the automatic deployment function of git and maven is combined, the automatic deployment of the algorithm plug-in is realized, the manual operation is guaranteed to keep the latest version and the offline operation version to be safely isolated and not influenced mutually, so that the problem of algorithm version management of related objects in numerous algorithm development and iteration is solved, the abnormal version in numerous versions can be quickly positioned, and the operation and maintenance efficiency is improved.
Based on the same concept, the present embodiment provides a target plug-in execution device, as shown in fig. 2, the target plug-in execution device includes:
the system comprises an acquisition module 1, a storage module and a processing module, wherein the acquisition module 1 is used for acquiring a file package corresponding to a target plug-in;
the uploading module 2 is used for uploading the file package to an algorithm platform and acquiring configuration information of the file package in the algorithm platform, wherein the configuration information comprises a storage path and a file name of the file package in the algorithm platform;
the obtaining module 1 is further configured to obtain script content corresponding to the target plug-in, and write the configuration information into a specified position of the script content to update the script content, where the script content is used to generate a script file for starting the target plug-in;
and the execution module 3 is used for writing the script content into a local path to generate a script file, and executing the script file to start running the target plug-in.
It should be understood that, in the target plug-in running apparatus provided in this embodiment, each module can be combined to implement each step of the target plug-in running method, so as to achieve the same technical effect as the target plug-in running method, and therefore, no further description is provided herein.
As shown in fig. 3, an electronic device according to an embodiment of the present application includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114, where the processor 111, the communication interface 112, and the memory 113 complete mutual communication via the communication bus 114,
a memory 113 for storing a computer program;
in an embodiment of the present application, when the processor 111 is configured to execute the program stored in the memory 113, the method for implementing the target plug-in operation provided in any one of the foregoing method embodiments includes:
acquiring a file package corresponding to a target plug-in, wherein the file package is used for storing a logic file and a dependent file corresponding to the target plug-in;
storing the file package to an algorithm platform, and acquiring configuration information of the file package in the algorithm platform, wherein the configuration information comprises a storage path and a file name of the file package in the algorithm platform;
acquiring script content corresponding to the target plug-in, and writing the configuration information into a specified position of the script content to update the script content, wherein the script content is used for generating a script file for starting the target plug-in;
and writing the script content into a local path to generate the script file, and executing the script file to start running the target plug-in.
The present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the target plug-in execution method provided by any one of the foregoing method embodiments.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A target plug-in running method is characterized by comprising the following steps:
acquiring a file packet corresponding to a target plug-in;
storing the file package to an algorithm platform, and acquiring configuration information of the file package in the algorithm platform, wherein the configuration information comprises a storage path and a file name of the file package in the algorithm platform;
acquiring script content corresponding to the target plug-in, and writing the configuration information into a specified position of the script content to update the script content, wherein the script content is used for generating a script file for starting the target plug-in;
writing the script content into a local path to generate the script file, and executing the script file to start the target plug-in.
2. The method of claim 1, wherein writing the script content to a local path to generate the script file and executing the script file to launch the target plug-in comprises:
during manual tasks, after updating the script content is completed, the script content is directly written into a local path to generate the script file, and the script file is executed to start the target plug-in.
3. The method of claim 1, wherein writing the script content to a local path to generate the script file and executing the script file to launch the target plug-in comprises:
during an offline task, after updating the script content, storing the script content into a snapshot table, wherein the snapshot table is used for generating a corresponding script identifier according to the script content;
submitting the script identification to a workflow scheduling platform so that the workflow scheduling platform generates a new scheduling task;
and when a scheduling task with the script identifier, which is initiated by the workflow scheduling platform, is received, writing the script content into a local path to generate the script file, and executing the script file to start the target plug-in.
4. The method according to claim 3, wherein upon receiving a scheduling task with the script identifier initiated by the workflow scheduling platform, writing the script content to a local path to generate the script file, and executing the script file to start the target plug-in, comprises:
acquiring the script content according to the script identifier;
writing the script content into a local path to generate the script file, and executing the script file to start the target plug-in.
5. The method of any of claims 1-4, wherein after writing the script content to a local path to generate the script file and executing the script file to launch the target plug-in, the method further comprises:
saving an execution record, wherein the execution record comprises the script content currently executed.
6. The method according to claim 5, wherein obtaining the file package corresponding to the target plug-in comprises:
and acquiring the file package corresponding to the target plug-in from an automatic construction tool, wherein the automatic construction tool is used for automatically constructing the file package according to the logic file and the dependent file corresponding to the target plug-in.
7. The method of claim 5, wherein obtaining the script content corresponding to the target plug-in comprises:
and acquiring the script content corresponding to the target plug-in from a warehouse management system, wherein the warehouse management system is used for managing the plurality of script contents corresponding to the target plug-in.
8. A target plug-in execution apparatus, comprising:
the acquisition module is used for acquiring a file packet corresponding to the target plug-in;
the uploading module is used for uploading the file package to an algorithm platform and acquiring configuration information of the file package in the algorithm platform, wherein the configuration information comprises a storage path and a file name of the file package in the algorithm platform;
the acquisition module is further configured to acquire script content corresponding to the target plug-in, and write the configuration information into a specified position of the script content to update the script content, where the script content is used to generate a script file for starting the target plug-in;
and the execution module is used for writing the script content into a local path to generate the script file and executing the script file to start the target plug-in.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the steps of the target plug-in operation method of any one of claims 1 to 7 when executing the program stored in the memory.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the target plug-in execution method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210615020.4A CN115061741A (en) | 2022-05-31 | 2022-05-31 | Target plug-in operation method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210615020.4A CN115061741A (en) | 2022-05-31 | 2022-05-31 | Target plug-in operation method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115061741A true CN115061741A (en) | 2022-09-16 |
Family
ID=83199162
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210615020.4A Pending CN115061741A (en) | 2022-05-31 | 2022-05-31 | Target plug-in operation method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115061741A (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101694624A (en) * | 2009-10-19 | 2010-04-14 | 中兴通讯股份有限公司 | Method for processing compact disc image files of software installation package and device |
| CN102789415A (en) * | 2011-04-13 | 2012-11-21 | 埃森哲环球服务有限公司 | Modular script designer for next generation testing system |
| CN105607941A (en) * | 2015-11-25 | 2016-05-25 | 上海斐讯数据通信技术有限公司 | Apparatus and method for compiling configuration in android project |
| CN105786538A (en) * | 2014-12-24 | 2016-07-20 | 博雅网络游戏开发（深圳）有限公司 | Software upgrading method and device based on Android system |
| CN108280023A (en) * | 2017-01-04 | 2018-07-13 | 中兴通讯股份有限公司 | Task executing method, device and server |
| WO2019139911A1 (en) * | 2018-01-09 | 2019-07-18 | Justdo, Inc. | Scripting language computer program modification methodology, system and software |
| CN110968359A (en) * | 2018-09-30 | 2020-04-07 | 北京国双科技有限公司 | Method and device for starting browser plug-in |
| CN110968357A (en) * | 2018-09-30 | 2020-04-07 | 北京国双科技有限公司 | Method and device for packaging maven item, storage medium and processor |
| CN111897570A (en) * | 2020-07-15 | 2020-11-06 | 杭州安恒信息技术股份有限公司 | Multi-dependency file extraction method and device based on Maven plug-in |
| CN112379916A (en) * | 2020-11-12 | 2021-02-19 | 中国人寿保险股份有限公司 | Method and device for modifying version number of maven project |
| US20210117548A1 (en) * | 2019-10-21 | 2021-04-22 | Michael Gokhman | Package dependencies representation |
| CN113805965A (en) * | 2021-09-11 | 2021-12-17 | 济南浪潮数据技术有限公司 | Method, device and equipment for installing external plug-in and readable medium |
-
2022
- 2022-05-31 CN CN202210615020.4A patent/CN115061741A/en active Pending
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101694624A (en) * | 2009-10-19 | 2010-04-14 | 中兴通讯股份有限公司 | Method for processing compact disc image files of software installation package and device |
| CN102789415A (en) * | 2011-04-13 | 2012-11-21 | 埃森哲环球服务有限公司 | Modular script designer for next generation testing system |
| CN105786538A (en) * | 2014-12-24 | 2016-07-20 | 博雅网络游戏开发（深圳）有限公司 | Software upgrading method and device based on Android system |
| CN105607941A (en) * | 2015-11-25 | 2016-05-25 | 上海斐讯数据通信技术有限公司 | Apparatus and method for compiling configuration in android project |
| CN108280023A (en) * | 2017-01-04 | 2018-07-13 | 中兴通讯股份有限公司 | Task executing method, device and server |
| WO2019139911A1 (en) * | 2018-01-09 | 2019-07-18 | Justdo, Inc. | Scripting language computer program modification methodology, system and software |
| CN110968359A (en) * | 2018-09-30 | 2020-04-07 | 北京国双科技有限公司 | Method and device for starting browser plug-in |
| CN110968357A (en) * | 2018-09-30 | 2020-04-07 | 北京国双科技有限公司 | Method and device for packaging maven item, storage medium and processor |
| US20210117548A1 (en) * | 2019-10-21 | 2021-04-22 | Michael Gokhman | Package dependencies representation |
| CN111897570A (en) * | 2020-07-15 | 2020-11-06 | 杭州安恒信息技术股份有限公司 | Multi-dependency file extraction method and device based on Maven plug-in |
| CN112379916A (en) * | 2020-11-12 | 2021-02-19 | 中国人寿保险股份有限公司 | Method and device for modifying version number of maven project |
| CN113805965A (en) * | 2021-09-11 | 2021-12-17 | 济南浪潮数据技术有限公司 | Method, device and equipment for installing external plug-in and readable medium |
Non-Patent Citations (2)
| Title |
|---|
| 任志斌: "基于电信数据的用户内容偏好画像补全", 《中国优秀硕士学位论文全文数据库经济与管理科学辑》, no. 10, 15 October 2018 (2018-10-15), pages 155 - 44 * |
| 王艺颖: "基于Kubernetes的Tensorflow分布式模型训练平台的设计与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》, no. 2, 15 February 2020 (2020-02-15), pages 138 - 589 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10095499B2 (en) | Optimization for multi-project package manager | |
| US6202207B1 (en) | Method and a mechanism for synchronized updating of interoperating software | |
| US8577937B1 (en) | Repository including exclusion list | |
| US8001083B1 (en) | Repository including version management | |
| US20060020937A1 (en) | System and method for extraction and creation of application meta-information within a software application repository | |
| US20200272440A1 (en) | Container image building using dependency container images | |
| US11442830B2 (en) | Establishing and monitoring programming environments | |
| GB2333865A (en) | Synchronised updating of interoperating software | |
| JP2004158012A (en) | System and method for software application management | |
| US10305731B2 (en) | System and method for provisioning cloud services across heterogeneous environments using partitioned provisioning instructions stored on a configuration management server | |
| US9542173B2 (en) | Dependency handling for software extensions | |
| CN108563440B (en) | Automatic adding method and device of list controller codes and readable storage medium | |
| US9141385B2 (en) | Managing operating system components | |
| US9411618B2 (en) | Metadata-based class loading using a content repository | |
| CN113515303B (en) | Project transformation method, device and equipment | |
| CN112286543B (en) | Application service deployment method and device | |
| US20190147047A1 (en) | Object-level image query and retrieval | |
| CN115061741A (en) | Target plug-in operation method and device, electronic equipment and storage medium | |
| CN111400243B (en) | Development management system based on pipeline service and file storage method and device | |
| CN113448493B (en) | Method, electronic device and computer readable medium for backing up data | |
| CN112114871A (en) | Code sharing method, device, server, terminal and medium | |
| CN117435244B (en) | Front-end engineering-crossing public library system and construction method thereof | |
| US20240061667A1 (en) | Incremental Image Import Process for Supporting Multiple Upstream Image Repositories | |
| CN116089387A (en) | Resource allocation method, apparatus, electronic device, readable medium and program product | |
| CN114064151A (en) | Cross-platform high-performance cluster application software management and use method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |