WO2024001594A1

WO2024001594A1 - Operator development method and apparatus, operator processing method and apparatus, and electronic device, system and storage medium

Info

Publication number: WO2024001594A1
Application number: PCT/CN2023/094913
Authority: WO
Inventors: 刘鹏; 郭朕; 石光川
Original assignee: 第四范式（北京）技术有限公司
Priority date: 2022-06-29
Filing date: 2023-05-18
Publication date: 2024-01-04
Also published as: CN117348848A

Abstract

Disclosed in the embodiments of the present disclosure are an operator development method and apparatus, an operator processing method and apparatus, and an electronic device, a system and a storage medium. The operator development method comprises: determining a description file of a target operator package on the basis of first operation information of a user on a development interface, wherein the description file is configured to describe a basic capability and an operation capability of an operator, the operation capability at least comprises framework information and a development language, and the framework information is configured to indicate an operation framework supported by the operator; determining a standard operator template, which matches the description file, wherein the standard operator template comprises at least a description file of the operator and an operator source code; filling the standard operator template on the basis of second operation information of the user on the development interface, so as to obtain a target operator; and packaging the target operator, and storing same in a standard operator library.

Description

Operator development, processing methods, devices, electronic equipment, systems and storage media

This disclosure request was submitted to the China Patent Office on June 29, 2022, with the application number 202210762431.6, and the application title is the priority of the Chinese patent application titled "Operator Development, Processing Methods, Devices, Electronic Equipment, Systems and Storage Media", which The entire contents are incorporated by reference into this disclosure.

Technical field

The present disclosure relates to the field of artificial intelligence technology, and more specifically, to an operator development, processing method, device, electronic equipment, system and storage medium.

Background technique

In the field of machine learning, the generation of models is often only the first step in the application of machine learning. After the model is generated, it often needs to go through many complex processes such as service-oriented encapsulation, parameter tuning, data docking, orchestration framework docking, and running framework docking. Moreover, each process often requires repeated communication and assistance from engineers with relevant technical backgrounds to complete.

In order to facilitate collaboration, the concept of operator is currently abstracted in the field of machine learning, and the operator is regarded as the smallest unit running in the running framework. In related technologies, operators are developed through SDK (Software Development Kit), and the development of operators is often bound to a certain operating framework. When the operator needs to be bound to other operating frameworks, this This SDK development method is highly intrusive to the logic of the operator itself and involves a certain transformation cost for the original logic. Moreover, each time an operator is developed, a large amount of repetitive work is required, which increases development costs and time costs.

Contents of the invention

One purpose of the embodiments of the present disclosure is to provide a new technical solution for operator development.

According to a first aspect of the present disclosure, an operator development method is provided, which method includes:

Based on the user's first operation information on the development interface, the description file of the target operator package is determined. The description file is configured to describe the basic capabilities and operating capabilities of the operator. The operating capabilities at least include framework information and development language, so The above frame information is configured to indicate the operating frame supported by the operator;

Determine the operator standard template that matches the description file; wherein the operator standard template at least includes: the operator description file and the operator source code;

Based on the user's second operation information on the development interface, fill in the operator standard template to obtain the target operator;

The target operator is packaged and stored in the standard operator library.

According to a second aspect of the present disclosure, an operator processing method is provided, which method includes:

Obtain the target operator from the standard operator library; wherein the target operator at least includes the operator's description file and the operator source code; the description file is configured to describe the operator's basic capabilities and operating capabilities, and the operating capabilities At least including framework information and a development language, the framework information is configured to indicate the running framework supported by the operator;

Convert the description file of the target operator according to the protocol supported by the target operation framework to obtain the processed target operator corresponding to the target operation framework;

The processed target operator corresponding to the target execution frame is sent to the target execution frame to run the target operator in the target execution frame.

According to a third aspect of the present disclosure, an operator development device is provided, which device includes:

The first determination module is configured to determine the description file of the target operator package based on the user's first operation information on the development interface. The description file is configured to describe the basic capabilities and operating capabilities of the operator. The operating capabilities are at least Including framework information and development language, the framework information is configured to indicate the running framework supported by the operator;

The second determination module is configured to determine the operator standard template that matches the description file; wherein the operator standard template at least includes: the operator description file and the operator source code;

A filling module configured to fill in the operator standard template based on the user's second operation information on the development interface to obtain the target operator;

The packaging module is configured to package the target operator and store it in the standard operator library.

According to a fourth aspect of the present disclosure, an operator processing device is provided, which device includes:

The acquisition module is configured to obtain the target operator from the standard operator library; wherein the target operator at least includes the operator's description file and the operator source code; the description file is configured to describe the operator's basic capabilities and operation Capabilities, the running capabilities include at least framework information and development language, the framework information is configured to indicate the running framework supported by the operator;

A conversion module configured to convert the description file of the target operator according to the protocol supported by the target operation framework, and obtain the processed target operator corresponding to the target operation framework;

A sending module configured to send the processed target operator corresponding to the target running frame to the target running frame, so as to run the target operator in the target running frame.

According to a fifth aspect of the present disclosure, there is provided an electronic device including at least one computing device and at least one storage device, wherein the at least one storage device is configured to store instructions configured to control the at least one storage device. A computing device performs the method according to the first aspect above; or, the device implements the device according to the second aspect above through the computing device and the storage device.

According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided with a computer program stored thereon, and when executed by a processor, the computer program implements any one of the first aspect or the second aspect. Methods.

According to a seventh aspect of the present disclosure, an operator operating system is provided, characterized in that the system includes: an agent configured to act as an agent for interaction with other systems, and the other systems include: an operator warehouse, an operator Sub-run framework; status tracker, configured to track and record task status; task configurator, configured to monitor tasks and configure tasks.

One beneficial effect of this disclosure is that it develops target operators through the description file of the target operator package and the operator standard template, where the description file of the target operator is configured to describe the basic capabilities and operating capabilities of the operator, The running capability at least includes framework information and development language. The framework information is configured to indicate the running framework supported by the operator. The operator standard template includes the operator description file and the operator source code. When specifically developing a target operator, it is necessary to determine the description file of the target operator package, then determine the operator standard template that matches the description file, and fill in the operator standard template to obtain the target operator. That is, it develops target operators by configuring operator standard templates, which can reduce repeated work when developing operators, improve the adaptability of operators, greatly reduce the difficulty of machine learning implementation, and reduce development costs and Time costs.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.

Figure 1 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure;

Figure 2 is a schematic flowchart of an operator development method according to an embodiment of the present disclosure;

Figure 3 is a schematic flowchart of an operator development method according to an example of the present disclosure;

Figure 4 is a schematic flowchart of an operator processing method according to an embodiment of the present disclosure;

Figure 5 is a schematic structural diagram of an operator operating system according to an embodiment of the present disclosure;

Figure 6 is a schematic structural diagram of an operator operating system according to another embodiment of the present disclosure;

Figure 7 is a schematic structural diagram of an operator operating system according to another embodiment of the present disclosure;

Figure 8 is a schematic diagram of operator operation data flow according to an example of the present disclosure;

Figure 9 is a functional block diagram of an operator development device according to an embodiment of the present disclosure;

Figure 10 is a functional block diagram of an operator processing device according to an embodiment of the present disclosure;

FIG. 11 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions, and numerical values set forth in these examples do not limit the scope of the disclosure unless otherwise specifically stated.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application or uses.

Techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods and devices should be considered a part of the specification.

In all examples shown and discussed herein, any specific values are to be construed as illustrative only and not as limiting. Accordingly, other examples of the exemplary embodiments may have different values.

It should be noted that similar reference numerals and letters refer to similar items in the following figures, so that once an item is defined in one figure, it does not need further discussion in subsequent figures.

The method of the embodiment of the present disclosure may be implemented by at least one electronic device, that is, the device 900 for implementing the operator development method or the device 1000 for implementing the operator processing method may be arranged on the at least one electronic device. Figure 1 shows the hardware structure of any electronic device. The electronic device shown in Figure 1 may be a portable computer, a desktop computer, a workstation, a server, etc., or any other device having a computing device such as a processor and a storage device such as a memory, and is not limited here.

As shown in Figure 1, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and so on. Among them, the processor 1100 is configured to execute the computer program. The computer program can be written using instruction sets of architectures such as x86, Arm, RISC, MIPS, SSE, etc. The memory 1200 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1400 is capable of wired or wireless communication, for example, and may specifically include Wifi communication, Bluetooth communication, 2G/3G/4G/5G communication, etc. The display device 1500 is, for example, a liquid crystal display screen, a touch display screen, or the like. The input device 1600 may include, for example, a touch screen, a keyboard, somatosensory input, and the like. The electronic device 1000 can output voice information through the speaker 1700, and can collect voice information through the microphone 1800, etc.

The electronic device shown in Figure 1 is illustrative only and is in no way meant to limit the present disclosure, its application, or uses. In the embodiments of the present disclosure, the memory 1200 of the electronic device 1000 is configured to store instructions, and the instructions are configured to control the processor 1100 to operate to execute the operator development method or algorithm of the embodiments of the present disclosure. sub-processing method. Skilled personnel can design instructions based on the solutions disclosed in this disclosure. How instructions control the processor to operate is well known in the art and will not be described in detail here.

In one embodiment, an electronic device is provided that includes at least one computing device and at least one storage device, the at least one storage device configured to store instructions configured to control the at least one computing device to perform execution in accordance with the present disclosure. Methods of any embodiment.

The device may include at least one electronic device 1000 shown in FIG. 1 to provide at least one computing device such as a processor and at least one storage device such as a memory, which is not limited here.

Figure 2 is a schematic flowchart of an operator development method according to an embodiment of the present disclosure. The method is executed by the electronic device 1000. As shown in Figure 2, the method may include the following steps S2100 to S2400:

Step S2100: Determine the description file of the target operator package based on the user's first operation information on the development interface.

The operator can be a functional module developed by the developer according to the needs. The operator can include but is not limited to at least one of the following: model training operator, model prediction operator, feature extraction operator, and model evaluation operator. Of course, The operator can also be a custom operator developed by the developer according to the customer's needs. The functions and application scenarios of the custom operator can be customized and designed by the customer. This embodiment is not limited here.

The description file is configured to describe the basic capabilities and running capabilities of the operator. The running capabilities include at least framework information and development language, and the framework information is configured to indicate the running framework supported by the operator. The format of the description file is yaml. For example, the description file can be named Operator.yaml to distinguish it from other files.

Wherein, the framework information is configured to describe the operating framework supported by the operator. The operating framework involved in the embodiment of the present disclosure may include but is not limited to at least one of the following: spark, flink, horovod, dask, mars, mindspore, local, k8s, hadoop, SAAS. In this embodiment, different operating frameworks can be defined in the description file as needed, so that the developed target operator can be adapted to the defined operating framework. Furthermore, when more than two operating frameworks are defined in the description file, the developed target operator can run on the two or more operating frameworks at the same time, thereby enabling the target operator to run across operating frameworks. . In other words, the operator development method provided by the embodiments of the present disclosure lowers the threshold for operator use. The operator has high portability, can run in a variety of operating frameworks, and has a wide range of applications.

Among them, the development language involved in the embodiment of the present disclosure may include but is not limited to at least one of the following: Python language, golang language, c language, c++ language, and Java language. In this embodiment, different development languages can be defined in the description file as needed, and then operators can be developed based on the defined development languages, which can be compatible with operators developed in multiple languages.

The operating capabilities of the operator may also include but are not limited to at least one of the following: operating type, scale, load mode, underlying platform, management mode, dependency, output data set, status, and architecture. The operation type may include but is not limited to online type operators and batch type operators. The online type operator can usually provide online prediction services. The online type operator may include but is not limited to services and streaming methods. This batch type operator can usually provide batch prediction services.

The scale can include but is not limited to stand-alone operators and distributed operators. Load modes can include but are not limited to remote and local. The underlying platform can include but is not limited to at least one of the following: k8s (kubernetes), yarn, and hadoop. Management modes can include but are not limited to hosting (watch through admission hook, non-invasive, suitable for native k8s types), status reporting (non-native frameworks, third-party k8s CRD or other frameworks), etc. Architectures can include but are not limited to x86, arm.

Dependencies can include but are not limited to ray, dask, and horovod. The status may include but is not limited to synchronization status information syncStatus and running status runningPhase, where synchronization status information syncStatus may include but is not limited to unsynced, synchronized, synchronized, and unknown. The running status runningPhase can include but is not limited to pending, running, succeeded, skipped, failed, and error.

Among them, the basic capabilities of the operator include but are not limited to: input, output, and operating parameters. The input is configured to describe the input information when the operator is running. The input may include but is not limited to source type, configuration, and input data. The output is configured to describe the output information when the operator is running. The output may include but is not limited to output type, configuration, and output data. Running parameters are necessary business parameters required when the operator is running. The necessary business parameters can include but are not limited to the number of concurrency, the number of training rounds, and the model type. In this embodiment, by defining the necessary business parameters required when the operator is running in the description file, the operator can run stably, avoid operator failure, and thereby improve the operator's operation capability.

It should be understood that the aforementioned contents contained in the description file, as well as the specific contents supported by each content, are only exemplary. In actual scenarios, each item in the description file can also be modified based on the actual situation. The content is adaptively increased or decreased.

In one example, the user's first operation information on the development interface may be information related to the user's selection operation, determination operation, or editing operation for the description file output by the electronic device. For example, the description file output by the electronic device may be is an initial description file. The initial description file may be a file pre-stored in the electronic device. Here, Developers can directly perform a determination operation to determine the initial description file as the description file of the target operator package; or, developers can edit the initial description file according to their own needs to obtain the description file of the target operator package. For another example, the electronic device can output multiple description files, and the user can select on the interface to determine the description file of the target operator package. This method of obtaining the description file of the target operator package based on the description file output by the electronic device has high operator development efficiency.

In one example, the user's first operation information on the development interface can also be the description file of the target operator package manually written by the user. This method of manually writing the description file of the target operator package is more in line with user needs. It can also meet the user's custom design needs, and provide users with remedial measures to customize the design description file when the preset description file of the electronic device does not meet the user's needs.

After determining the description file of the target operator package based on the user's first operation information on the development interface, enter:

Step S2200: Determine the operator standard template that matches the description file.

The operator standard template includes at least: the operator description file and the operator source code. It can be understood that the operator standard template can be embodied as scaffolding, and the scaffolding can be called the code directory structure and entry function. In this embodiment, the scaffold plays the role of the operator standard template.

Among them, the operator source code is the code written to realize the operator function. For example, the feature extraction operator is configured to extract features from the input data. Correspondingly, the source code of the feature extraction operator is the code written to implement the feature extraction function from the input data. For another example, the model training operator is configured to perform model training based on the results of feature extraction to obtain a machine learning model. Correspondingly, the source code of the model training operator is to implement the function of performing model training based on the results of feature extraction to obtain a machine learning model. code written. For another example, the model evaluation operator is configured to evaluate the performance of a trained machine learning model. Correspondingly, the source code of the model evaluation operator is written to implement the function of evaluating the performance of a trained machine learning model. code.

In any of the above embodiments, in addition to the description file and the operator source code, the operator standard template also includes at least one of the following: an operator description document and an operator test file.

Among them, operator documentation is usually written to facilitate customers and ordinary people to use, run, and debug the operator normally, thereby lowering the threshold for operator application.

Among them, the operator's test files can include but are not limited to test cases and test report templates. A test case is a description of the test task for an operator, which embodies the test plan, method and strategy. Its content can include but is not limited to test goals, test environment, input data, test steps, expected results, and test scripts. The test report template is the test report obtained by executing the test case against the operator. In this embodiment, by specifying the test file of the operator in the operator standard template, the quality of the operator can be improved, so that after the target operator is developed, the operator rating can be performed through a combination of automatic testing and manual review.

In one embodiment, step S2200 of determining the operator standard template that matches the description file may further include: selecting from the preset operator standard templates that matches the framework information and the development language. Operator standard template.

In this embodiment, the preset operator standard template includes multiple operator standard templates, such as but not limited to operator standard template 1 and operator standard template 2 . It can be understood that any two different development languages have different categories of operator standard templates corresponding to them, and the operator standard templates of different categories are different. Specifically, the major items in the operator standard template (description file , operator source code, documentation, test files), the content contained in them or the content generation methods are different. For example, the development language Python corresponds to a category of operator standard templates, and the development language C++ corresponds to a category of operator standard templates, and the content or content generation methods of each major item in the operator standard templates of these two categories are different. .

In this embodiment, the framework information and development language have a mapping relationship with the operator standard template. That is to say, if the framework information and development language are different, the corresponding operator standard templates are different. In this way, when determining the description file of the target operator package In this case, the operator standard template corresponding to the framework information and development language can be determined from the preset operator standard template according to the framework information and development language in the description file, and the operator standard can be accurately determined template.

In one embodiment, step S2200 of determining the operator standard template that matches the description file may further include: generating an operator standard template that matches the framework information and the development language.

In this embodiment, when the description file of the target operator package is determined, the electronic device can also automatically generate an operator standard template that matches the framework information and development language directly based on the framework information and development language in the description file. , which can efficiently generate operator standard templates.

After executing to determine the operator standard template that matches the description file, enter:

Step S2300: Fill in the operator standard template based on the user's second operation information on the development interface to obtain the target operator.

In this embodiment, after the operator standard template matching the description file is determined, the operator standard template can be filled in to obtain the target operator based on the user's second operation information on the development interface.

After executing the second operation information based on the user on the development interface, filling the operator standard template, and obtaining the target operator, enter:

Step S2400: Package the target operator and store it in the standard operator library.

In this embodiment, packaging the target operator and storing it in the standard operator library in step S2400 may further include: verifying whether the target operator satisfies the operator admission mechanism; when the target operator satisfies the In the operator admission mechanism, the target operator is packaged and stored in the standard operator library.

For example, the operator admission mechanism can be an automatic testing mechanism. In a possible embodiment, the operator admission mechanism may be related to the operator's test file. In practical applications, the operator's test file can be used to test the operator. When it passes the test, it can be determined that the operator access mechanism is met.

Alternatively, the operator access mechanism can also be manually controlled by business personnel. In another possible embodiment, after the target operator is obtained, it can be determined through manual control whether the target operator satisfies the operator admission mechanism. That is, it is provided for business personnel to manually control the quality of the operator. a possibility.

Of course, in specific applications, the automatic test mechanism and the manual control mechanism can also be combined and applied. When both pass, the target operator satisfies the operator admission mechanism.

In addition, the operator access mechanism can also be set according to actual needs. This embodiment does not limit the specific content of the operator access mechanism. In other words, after obtaining the target operator, it does not directly incorporate the target operator into the standard operator library, but first verifies the target operator based on the operator access mechanism, and if the verification passes, Only then will the target operators be packaged and stored in the standard operator library to further improve the quality of the operators, ensure the consistency of the operator levels in the standard operator library, and accordingly reduce the cost of operator development.

Moreover, standardized operators can run both in the cluster and locally, and have the same functional experience, which reduces the learning cost for operator developers who need to learn multiple distributed clusters.

In addition, if the target operator does not meet the operator admission mechanism, it will be returned to the developer so that the developer can modify or adjust it. The target operator adjusted by the developer can still be used as shown in Figure 2. for development.

According to the embodiment of the present disclosure, the target operator is developed through the description file of the target operator package and the operator standard template, wherein the description file of the target operator is configured to describe the basic capabilities and operating capabilities of the operator, and the operating capabilities At least the framework information and development language are included. The framework information is configured to indicate the running framework supported by the operator. The operator standard template includes the operator description file and the operator source code. When specifically developing a target operator, it is necessary to determine the description file of the target operator package, then determine the operator standard template that matches the description file, and fill in the operator standard template to obtain the target operator. That is, it develops target operators by configuring operator standard templates, which can reduce repeated work when developing operators, improve the adaptability of operators, greatly reduce the difficulty of machine learning implementation, and reduce development costs and Time costs.

It should be noted that the operators involved in the embodiments of the present disclosure can also be used in scenarios such as privacy computing, multi-party secure computing, federated learning, invisible (hidden) query, secure (private) intersection, etc.

Next, a schematic flow chart of an example operator development method is shown. In this example, as shown in FIG. 3 , the operator development method may include the following steps S310 to S390:

Step S310: Generate a description text of the target operator package based on the first operation information of the operator developer on the development interface. Operator.yaml.

Step S320: Automatically generate or manually write scaffolding code based on the development language and running framework defined in the description file Operator.yaml.

Step S330: perform opctl local debugging on the scaffolding code.

Step S340: Fill in the scaffolding code to obtain the target operator.

Step S350: Package the target operator and upload it to the continuous integration tool (CI tool).

Step S360: Perform automated testing on the target operator in the CI tool based on the operator test file to obtain automated test results.

Step S370: The operator developer debugs the target operator in the OpHub system based on the automated test results, and writes test files for the debugged target operator.

Step S380: After verifying in the CI tool that the target operator satisfies the operator admission mechanism, quality assurance personnel (QA personnel) manually test the target operator based on the written test file, and if the test passes, The packaged target operator is released through the CI tool.

Step S390: Drop the target operator into the standard operator library, that is, the product library.

According to this example, the benefits brought by operator standardization to R&D during the R&D process include: First, it develops target operators based on generating scaffolding code, which can reduce duplication of work when developing operators and improve the efficiency of operators. The adaptability and scope of use greatly reduce the difficulty of machine learning implementation and reduce development costs and time costs. Moreover, full life cycle management improves operator development efficiency and reduces development time by 50% compared to traditional operator development. Secondly, standardized operators can run in clusters or locally, which reduces the learning cost for operator developers to learn multiple distributed clusters.

The benefits of operator standardization to product managers include but are not limited to: testing the operator through the operator's test file before dropping it into the product library, and designing easy-to-use operator access through access to automated testing. The mechanism improves operator quality.

Figure 4 is a schematic flowchart of an operator processing method according to an embodiment of the present disclosure. The method is executed by the electronic device 1000. Specifically, it may be executed by the OpHub system in the electronic device, or it may also be executed by other modules or systems in the electronic device. , wherein these modules or systems can schedule and apply operators according to the method shown in Figure 4. Specifically, the method can include the following steps S4100 to S4300:

Step S4100: Obtain the target operator from the standard operator library.

The target operator may be one or more operators in the standard operator library developed according to the first embodiment. In this embodiment, the required one or more target operators can be obtained from the standard operator library. Referring to Figure 5, the target operator can be developed based on the local integrated development environment (local IDE) or the network integrated development environment (web IDE) and uploaded to the standard operator library (git repo), so that the system (such as OpHub) can be downloaded from the git rego Get the target operator.

The target operator at least includes the operator's description file and the operator source code; the description file is configured to describe the operator's basic capabilities and operating capabilities, and the operating capabilities at least include framework information and development language. The framework information Configured to indicate the operating framework supported by the operator.

The running framework supported by the framework information may include but is not limited to at least one of the following: spark, flink, horovod, dask, mars, mindspore, local, k8s, hadoop, and SAAS. In this embodiment, when more than two operating frameworks are defined in the description file, the developed target operator can run on the two or more operating frameworks at the same time, thereby enabling the target operator to run across operating frameworks, improving The scope of application and adaptability improve the development experience of distributed operator development.

The operating capabilities of the operator may also include but are not limited to at least one of the following: operating type, scale, load mode, The underlying platform, management model, dependencies, output data dataset, status, and architecture. For details, please refer to the introduction of the operator development embodiment. This embodiment will not be described in detail here.

The basic capabilities of the operator include but are not limited to: input, output, and operating parameters. For details, please refer to the introduction of the operator development embodiment. This embodiment will not be described in detail here.

The operator source code is the code written to realize the operator function. For example, the feature extraction operator is configured to extract features from the input data. Correspondingly, the source code of the feature extraction operator is the code written to implement the feature extraction function from the input data. For another example, the model training operator is configured to perform model training based on the results of feature extraction to obtain a machine learning model. Correspondingly, the source code of the model training operator is to implement the function of performing model training based on the results of feature extraction to obtain a machine learning model. code written. For another example, the model evaluation operator is configured to evaluate the performance of a trained machine learning model. Correspondingly, the source code of the model evaluation operator is written to implement the function of evaluating the performance of a trained machine learning model. code.

After obtaining the target operator from the standard operator library, enter:

Step S4200: Convert the description file of the target operator according to the protocol supported by the target execution framework to obtain the processed target operator corresponding to the target execution framework.

The target running framework is the running framework supported by the operator indicated by the operator's framework information in the operator's description file. It can be understood that there can be only one target execution framework, or there can be multiple target execution frameworks. For details, please refer to the above introduction.

In this embodiment, in step S4200, converting the description file of the target operator according to the protocol supported by the target operation framework, and obtaining the processed target operator corresponding to the target operation framework may further include: according to The protocol supported by the target execution framework parses the description file of the target standard into a second file readable by the protocol; according to the second file, obtains the processed target corresponding to the target execution framework operator.

It is understandable that since different operating frameworks have supported protocols, in order to ensure that the target operating framework can run the target operator normally, it is necessary to first parse the target operator's description file into its protocol according to the protocols supported by the target operating framework. A readable file, and then the processed target operator corresponding to the target execution framework is obtained based on the file. Referring to Figure 5, the provided running frameworks can include local, k8s, hadoop, SAAS... (not exhaustive). Here, the target running frameworks can be hadoop and SAAS, and the features can be extracted according to the protocols supported by hadoop. The description file of the operator is parsed into a readable file, and the processed feature extraction operator corresponding to hadoop is obtained. According to the protocol supported by SAAS, the description file of the feature extraction operator is parsed into a readable file, and SAAS is obtained. The corresponding processed feature extraction operator. It can be understood that the task tracking in Figure 5 is configured to track the task running status during the operator's execution of the task. The task tracking framework can include but is not limited to: xxx Platform, tfBorad, ES, Grafana, log, tracing.

Step S4300: Send the processed target operator corresponding to the target execution frame to the corresponding execution frame, so as to run the target operator in the corresponding execution frame.

Continuing the above example, after obtaining the processed feature extraction operator corresponding to hadoop, the processed feature extraction operator can be sent to hadoop so that the processed feature extraction operator can be run in hadoop. And after obtaining the processed feature extraction operator corresponding to SAAS, the processed feature extraction operator can be sent to SAAS so that the processed feature extraction operator can be run in SAAS.

According to an embodiment of the present disclosure, the target operator obtained is obtained from a standard operator library. The target operator includes a description file of the operator. The description file is configured to describe the operator's operating capability, and the operating capability at least includes Framework information and development language, the framework information is configured to indicate the running framework supported by the operator. That is, the obtained target operator can configure the framework information according to the needs, which improves the adaptability of the operator. Moreover, when different operating frameworks are defined in the description file, the target operator can be made to have cross-operation capabilities. The ability of the framework, that is, the target operator can be adapted to different operating frameworks, and then run in different operating frameworks to achieve cross-operating frameworks.

In one embodiment, when the target operator includes multiple, after performing the above step S3200, the description file of the target operator is converted according to the protocol supported by the target execution framework, and the description file of the target operator is obtained according to the protocol supported by the target execution framework. Before the corresponding processed target operator, the operator processing method in the embodiment of the present disclosure may further include the following steps: S5100~step S5200:

Step S5100: Parse the description file of the target operator into a first file readable by the orchestration framework through the adapter corresponding to the orchestration framework.

The orchestration framework can include but is not limited to at least one of the following: workflow, argo workflow, airflow, and kubeflow.

In this embodiment, in a scenario involving multiple target operators for orchestration and scheduling, an adapter executor will be involved. The adapter corresponds to the orchestration framework one-to-one. The adapter is configured to realize the interaction between the orchestration framework and the system. The adapter corresponding to the orchestration framework Specifically, it is configured as follows: According to the requirements of the orchestration framework, the description file of each target operator is converted into a file readable by the orchestration framework. For example, referring to Figure 5, the provided orchestration frameworks include kubeflow, airflow, argo workflow, and xxx workflow. Here, operator developers can select the orchestration framework workflow. The target operators include feature extraction operators and model training algorithms. Here, you can convert the description file of the feature extraction operator and the description file of the model training operator according to the adapter corresponding to the workflow.

Step S5200: The arrangement framework arranges the target operator based on the first file corresponding to the target operator, and obtains the arranged target operator.

Continuing the above example, the orchestration framework workflow arranges the feature extraction operator and the model training operator based on the first file corresponding to the feature extraction operator and the first file corresponding to the model training operator, and extracts the arranged feature Operators and model training operators are protocol converted through the OpHub system and sent to the running framework for execution.

It can be understood that after arranging multiple target operators, the system can convert the description files of the target operators according to the protocol supported by the target operation framework to obtain the processed target corresponding to the target operation framework. operator, and sends the processed target operator to the corresponding running framework to run the target operator in the corresponding running framework.

According to the embodiments of the present disclosure, the operator runtime is decoupled from the orchestration and scheduling, different adapters are developed for different orchestration frameworks, and the target operators are submitted to the system through the adapters.

An embodiment of the present disclosure also provides an operator operating system. Referring to FIG. 6 , the operator operating system 600 includes an agent 610, a status tracker 620, and a task configurator 630. The operator operating system 600 may be an OpHub system.

In this embodiment, the agent (transmitter) 610 is configured to act as a proxy for interaction with other systems. The other systems include: an operator warehouse and an operator execution framework.

The operator warehouse is the standard operator library described in the above embodiment, and the operator warehouse is the nexus repository shown in Figure 7.

The operator running framework may include but is not limited to at least one of the following: spark, flink, horovod, dask, mars, and mindspore. Figure 7 shows the running framework kubernetes.

In any embodiment, in addition to the operator warehouse and the operator running framework, the other system may further include a service framework and a management framework. The service framework is the data framework that the operator needs to depend on. The service framework can include but is not limited to elasticsearch and MySQL as shown in Figure 7. The management framework is configured as a management task, and the management framework may be the distributed version control system git shown in Figure 7.

In this embodiment, a status tracker (watcher) 620 is configured to track and record task status. Referring to Figure 7, the status tracker 620 can monitor tasks in kubernetes and record them to elasticsearch and MySQL. Depending on the specific situation, the status tracker 620 includes a log monitoring collection component or can connect to a third party.

Task configurator 630 (addmission webhook) is configured to monitor tasks and configure tasks. Referring to Figure 7, the task configurator 630 can monitor tasks submitted to kubernetes and perform unified configuration at the cluster level and namespace level.

It can be understood that the above agent 610, status tracker 620 and task configurator 630 provide the ability to expose data, and the storage of specific data can be flexibly selected according to on-site dependencies.

Combined with Figure 8, a schematic diagram of the operator operation data flow is shown. In Figure 8, the operator execution task is first submitted, and the task is submitted to the operator operation framework platform, and the task is executed in the operator operation framework. During the execution of tasks by an operator, information can be obtained while the operator is running through meta/status, log, and trace in the status tracker 620. Collect and view operator runtime information.

The embodiment of the present disclosure also provides an operator development device 900. As shown in Figure 9, the operator development device 900 includes a first determination module 910, a second determination module 920, a filling module 930, and a packaging module 940.

The first determination module 910 is configured to determine the description file of the target operator package based on the user's first operation information on the development interface. The description file is configured to describe the basic capabilities and operating capabilities of the operator. The operating capabilities It includes at least framework information and a development language, and the framework information is configured to indicate a running framework supported by the operator.

The second determination module 920 is configured to determine the operator standard template that matches the description file; wherein the operator standard template at least includes: the operator description file and the operator source code.

The filling module 930 is configured to fill in the operator standard template based on the user's second operation information on the development interface to obtain the target operator.

The packaging module 940 is configured to package the target operator and store it in the standard operator library.

In one embodiment, the second determination module 920 is specifically configured as:

From the preset operator standard templates, select the operator standard template that matches the framework information and the development language;

Alternatively, based on the framework information and the development language, a matching operator standard template is generated.

In one embodiment, the operator standard template also includes at least one of the following: an operator description document and an operator test file.

In one embodiment, the basic capabilities of the operator include: input, output, and operating parameters.

In one embodiment, the operator operation capability also includes at least one of the following: operation type, scale, load mode, underlying platform, management mode, dependency, output data set, status, and architecture.

In one embodiment, the running framework supported by the framework information includes at least one of the following: spark, flink, horovod, dask, mars, and mindspore.

In one embodiment, the packaging module 940 is specifically configured to: verify whether the target operator satisfies the operator admission mechanism; when the target operator satisfies the operator admission mechanism, the The target operator is packaged and stored in the standard operator library.

An embodiment of the present disclosure also provides an operator processing device 1000. As shown in Figure 10, the operator processing device 1000 includes an acquisition module 1010, a conversion module 1020, and a sending module 1030.

The acquisition module 1010 is configured to obtain the target operator from the standard operator library; wherein the target operator at least includes the operator's description file and the operator source code; the description file is configured to describe the operator's basic capabilities and Running capabilities include at least framework information and development language, and the framework information is configured to indicate the running framework supported by the operator.

The conversion module 1020 is configured to convert the description file of the target operator according to the protocol supported by the target execution framework, and obtain the processed target operator corresponding to the target execution framework.

The sending module 1030 is configured to send the processed target operator corresponding to the target execution frame to the target execution frame, so as to run the target operator in the target execution frame.

In one embodiment, the device further includes an orchestration module (not shown in the figure), specifically configured to: The adapter corresponding to the arrangement framework parses the description file of the target operator into a first file readable by the arrangement framework; the arrangement framework, based on the first file corresponding to the target operator, analyzes the The target operator is arranged, and the arranged target operator is obtained.

In one embodiment, the conversion module 1020 is specifically configured to: parse the description file of the target standard into a second file readable by the protocol according to the protocol supported by the target execution framework; The second file is used to obtain the processed target operator corresponding to the target operation framework.

In one embodiment, the operator operation capability further includes at least one of the following: operation type, scale, load mode, underlying platform, management mode, and dependencies.

The embodiment of the present disclosure provides an electronic device 1100. The electronic device 1100 may be a server or a terminal device. The server may be an integrated server or a distributed server across multiple computers or computer data centers. Terminal devices can be mobile phones, tablets, laptops, etc.

As shown in Figure 11, the electronic device 1100 includes a processor 1110 and a memory 1120. The memory 1120 is configured to store executable instructions; the processor 1110 is configured to run the electronic device 1100 according to the control of the instructions to execute any implementation according to the present disclosure. Examples of operator development methods or operator processing methods.

The electronic device may be an electronic product having a processor and memory. For example, it can be a smartphone, laptop computer, desktop computer, tablet computer, etc.

Embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored. When executed by a processor, the computer program implements the operator development method or operator processing method provided in any of the foregoing embodiments.

The present disclosure may be a system, method, and/or computer program product. A computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement aspects of the present disclosure.

Computer-readable storage media may be tangible devices that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or Flash memory), Static Random Access Memory (SRAM), Compact Disk Read Only Memory (CD-ROM), Digital Versatile Disk (DVD), Memory Stick, Floppy Disk, Mechanical Coding Device, such as a printer with instructions stored on it. Protruding structures in hole cards or grooves, and any suitable combination of the above. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or through electrical wires. transmitted electrical signals.

Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to various computing/processing devices, or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. Networks can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, Gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium in the respective computing/processing device .

Computer program instructions for performing operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source code or object code written in any combination of object-oriented programming languages - such as Smalltalk, C++, etc., and conventional procedural programming languages - such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server implement. In situations involving remote computers, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider through the Internet). connect). In some embodiments, by utilizing state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can Computer readable program instructions are executed to implement various aspects of the disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, thereby producing a machine that, when executed by the processor of the computer or other programmable data processing apparatus, , resulting in an apparatus that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium. These instructions cause the computer, programmable data processing device and/or other equipment to work in a specific manner. Therefore, the computer-readable medium storing the instructions includes An article of manufacture that includes instructions that implement aspects of the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other equipment, causing a series of operating steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executed on a computer, other programmable data processing apparatus, or other equipment to implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions that embody one or more elements for implementing the specified logical function(s). Executable instructions. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two consecutive blocks may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts. , or can be implemented using a combination of specialized hardware and computer instructions. It is well known to those skilled in the art that implementation through hardware, implementation through software, and implementation through a combination of software and hardware are all equivalent.

The embodiments of the present disclosure have been described above. The above description is illustrative, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements in the market of the embodiments, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the disclosure is defined by the appended claims.

Industrial applicability

Through the embodiments of the present disclosure, the target operator is developed through the description file of the target operator package and the operator standard template, Among them, the description file of the target operator is configured to describe the basic capabilities and operating capabilities of the operator. The operating capabilities include at least framework information and development language. The framework information is configured to indicate the operating framework supported by the operator. Among them, the operator standard template Includes operator description files and operator source code. When specifically developing a target operator, it is necessary to determine the description file of the target operator package, then determine the operator standard template that matches the description file, and fill in the operator standard template to obtain the target operator. That is, it develops target operators by configuring operator standard templates, which can reduce repeated work when developing operators, improve the adaptability of operators, greatly reduce the difficulty of machine learning implementation, and reduce development costs and Time costs.

Claims

An operator development method, the method includes:

Based on the user's first operation information on the development interface, the description file of the target operator package is determined. The description file is configured to describe the basic capabilities and operating capabilities of the operator. The operating capabilities at least include framework information and development language, so The above frame information is configured to indicate the operating frame supported by the operator;

Determine the operator standard template that matches the description file; wherein the operator standard template at least includes: the operator description file and the operator source code;

Based on the user's second operation information on the development interface, fill in the operator standard template to obtain the target operator;

The target operator is packaged and stored in the standard operator library.
The method according to claim 1, wherein determining the operator standard template matching the description file includes:

From the preset operator standard templates, select the operator standard template that matches the framework information and the development language;

Alternatively, based on the framework information and the development language, a matching operator standard template is generated.
The method according to claim 2, wherein the operator standard template further includes at least one of the following: an operator description document and an operator test file.
The method according to any one of claims 1-3, wherein the basic capabilities of the operator include: input, output, and operating parameters.
The method according to any one of claims 1 to 3, wherein the operator operation capability further includes at least one of the following: operation type, scale, load mode, underlying platform, management mode, dependency, output data dataset, status, structure.
The method according to any one of claims 1 to 3, wherein the running framework supported by the framework information includes at least one of the following: spark, flink, horovod, dask, mars, mindspore, local, k8s, hadoop, SAAS .
The method according to any one of claims 1-3, wherein said packaging the target operator and storing it in a standard operator library includes:

Verify whether the target operator satisfies the operator admission mechanism;

When the target operator satisfies the operator admission mechanism, the target operator is packaged and stored in the standard operator library.
An operator processing method, the method includes:

Obtain the target operator from the standard operator library; wherein the target operator at least includes the operator's description file and the operator source code; the description file is configured to describe the operator's basic capabilities and operating capabilities, and the operating capabilities At least including framework information and a development language, the framework information is configured to indicate the running framework supported by the operator;

Convert the description file of the target operator according to the protocol supported by the target operation framework to obtain the processed target operator corresponding to the target operation framework;

The processed target operator corresponding to the target execution frame is sent to the target execution frame to run the target operator in the target execution frame.
The method according to claim 8, wherein the description file of the target operator is converted according to the protocol supported by the target execution framework to obtain the processed target operator corresponding to the target execution framework. Previously, this also included:

Through the adapter corresponding to the orchestration framework, parse the description file of the target operator into a first file readable by the orchestration framework;

Through the arrangement framework, the target operator is arranged based on the first file corresponding to the target operator, and the arranged target operator is obtained.
The method according to claim 9, wherein the description file of the target operator is converted according to the protocol supported by the target operation framework to obtain the processed target operator corresponding to the target operation framework, Bag include:

According to the protocol supported by the target execution framework, parse the description file of the target standard into a second file readable by the protocol;

According to the second file, the processed target operator corresponding to the target execution framework is obtained.
The method according to any one of claims 8 to 10, wherein the operator standard template further includes at least one of the following: an operator description document and an operator test file.
The method according to any one of claims 8-10, wherein the basic capabilities of the operator include: input, output, and operating parameters.
The method according to any one of claims 8-10, wherein the operator operation capability further includes at least one of the following: operation type, scale, load mode, underlying platform, management mode, dependency, output data dataset, status, structure.
The method according to any one of claims 8-10, wherein the running framework supported by the framework information includes at least one of the following: spark, flink, horovod, dask, mars, mindspore, local, k8s, hadoop, SAAS .
An operator development device, the device includes:

The first determination module is configured to determine the description file of the target operator package based on the user's first operation information on the development interface. The description file is configured to describe the basic capabilities and operating capabilities of the operator. The operating capabilities are at least Including framework information and development language, the framework information is configured to indicate the running framework supported by the operator;

The second determination module is configured to determine the operator standard template that matches the description file; wherein the operator standard template at least includes: the operator description file and the operator source code;

A filling module configured to fill in the operator standard template based on the user's second operation information on the development interface to obtain the target operator;

The packaging module is configured to package the target operator and store it in the standard operator library.
The device according to claim 15, wherein the second determination module is specifically configured to:

From the preset operator standard templates, select the operator standard template that matches the framework information and the development language;

Alternatively, based on the framework information and the development language, a matching operator standard template is generated.
The device according to claim 16, wherein the operator standard template further includes at least one of the following: an operator description document and an operator test file.
The device according to any one of claims 15 to 17, wherein the basic capabilities of the operator include: input, output, and operating parameters.
The device according to any one of claims 15 to 17, wherein the operator operation capability further includes at least one of the following: operation type, scale, load mode, underlying platform, management mode, dependency, output data data set, status, structure.
The device according to any one of claims 15 to 17, wherein the operating framework supported by the framework information includes at least one of the following: spark, flink, horovod, dask, mars, mindspore, local, k8s, hadoop, SAAS .
The device according to any one of claims 15-17, wherein the packaging module is specifically configured to:

Verify whether the target operator satisfies the operator admission mechanism;

When the target operator satisfies the operator admission mechanism, the target operator is packaged and stored in the standard operator library.
An operator processing device, the device includes:

The acquisition module is configured to obtain the target operator from the standard operator library; wherein the target operator at least includes the operator's description file and the operator source code; the description file is configured to describe the operator's basic capabilities and operation Capabilities, the running capabilities include at least framework information and development language, the framework information is configured to indicate the running framework supported by the operator;

A conversion module configured to convert the description file of the target operator according to the protocol supported by the target operation framework, and obtain the processed target operator corresponding to the target operation framework;

A sending module configured to send the processed target operator corresponding to the target running frame to the target running frame, so as to run the target operator in the target running frame.
The device according to claim 22, wherein the device further includes an orchestration module, specifically configured as:

Through the adapter corresponding to the orchestration framework, parse the description file of the target operator into a first file readable by the orchestration framework;

Through the arrangement framework, the target operator is arranged based on the first file corresponding to the target operator, and the arranged target operator is obtained.
The device according to claim 23, wherein the conversion module is specifically configured to:

According to the protocol supported by the target execution framework, parse the description file of the target standard into a second file readable by the protocol;

According to the second file, the processed target operator corresponding to the target operation framework is obtained.
The device according to any one of claims 22 to 24, wherein the operator standard template further includes at least one of the following: an operator description document and an operator test file.
The device according to any one of claims 22 to 24, wherein the basic capabilities of the operator include: input, output, and operating parameters.
The device according to any one of claims 22 to 24, wherein the operator operation capability further includes at least one of the following: operation type, scale, load mode, underlying platform, management mode, and dependency.
The device according to any one of claims 22 to 24, wherein the operating framework supported by the framework information includes at least one of the following: spark, flink, horovod, dask, mars, mindspore, local, k8s, hadoop, SAAS .
An electronic device comprising at least one computing device and at least one storage device, the at least one storage device being configured to store instructions configured to control the at least one computing device to perform any of claims 1 to 14 The method according to one of the claims; or, the device implements the device according to claims 15 to 28 through the computing device and the storage device.
A computer-readable storage medium having a computer program stored thereon, which implements the method according to any one of claims 1 to 14 when executed by a processor.
An operator operating system, the system includes:

The agent is configured to act as an agent for interaction with other systems. The other systems include: operator warehouse and operator execution framework;

Status tracker, configured to track and record task status;

The task configurator is configured to monitor tasks and configure tasks.