CN116208500A

CN116208500A - Python modifier-based non-perception local code cloud functionalization deployment calling method

Info

Publication number: CN116208500A
Application number: CN202310213598.1A
Authority: CN
Inventors: 陈奕池
Original assignee: Tianjin University
Current assignee: Tianjin University
Priority date: 2023-03-07
Filing date: 2023-03-07
Publication date: 2023-06-02
Anticipated expiration: 2043-03-07
Also published as: CN116208500B

Abstract

The invention discloses a non-perception local code cloud functionalization deployment and calling method based on a Python modifier, and belongs to the field of cloud computing. According to the method, the cloud computing is used for partial content of the local computing, so that the running efficiency is greatly improved; the cloud functionalization technology is utilized to transfer the work, so that a user does not need to care about the content of the server, and the main program runs locally, so that the user is more visual; by means of storing the common functions as a static Yun Hanshu library in a lasting mode, part of functions can be cloud-functionalized to the cloud end and can be called without repeated writing in the local area, reusability of the functions is improved, and meanwhile requirements of users on manual operation according to different commercial platforms are saved; by modifying each function code needing cloud functionalization and then carrying out the flow of interface operation once, the learning cost is reduced; by enabling the user to operate the whole application by only designating the call function which is functionalized by the cloud without modifying the call function and the statement related to the call function.

Description

Python modifier-based non-perception local code cloud functionalization deployment calling method

Technical Field

The invention relates to the field of non-service architecture calculation, in particular to a non-perception local code cloud functionalization deployment and calling method based on a Python modifier.

Background

The core idea of the serverless architecture is to let the user pay no attention to the computing resources required for service operation, but mainly to the construction of business logic, and the user does not need to spend time and resources for server configuration, maintenance, update, expansion and capacity planning. The serverless architecture computing has become a viable solution to distributed scientific computing due to the ability of serverless architecture to provide automatic scaling and concurrent scheduling, and the recent advances and developments in network bandwidth and distributed storage.

At present, most of operations are omitted, but many complicated steps are still needed for a user to perform scientific calculation by using a cloud computing platform. On a commercial cloud platform, novice users often face a series of potential decisions which are confusing, the types of examples, the sizes of clusters, a pricing model, a programming model and task granularity must be determined in advance, the users often can smoothly use the cloud platform only by deeply knowing the idea of server-free computing and learning the interface operation of the cloud computing platform, and more difficult, the users need to independently split an original program into codes with smaller granularity, respectively convert the codes into deployment functions of cloud functions, and then respectively use the operation interfaces of the cloud computing platform to configure, so that scientific computing cloud functional deployment and calling are greatly hindered.

In view of this, the invention proposes a deployment and call method based on the non-perception local code cloud functionalization of the Python modifier, which is based on the principle that if a function is modified by the modifier, the function itself and the parameters carried when the modifier is called are transmitted as integral parameters to the modifier.

Disclosure of Invention

Aiming at the technical problems existing in the prior art, the invention aims to provide a deployment and calling method of non-perception local code cloud functionalization based on a Python modifier.

In order to achieve the above purpose, the present invention provides the following technical solutions:

the method for deploying and calling the non-perception local code cloud functionalization based on the Python modifier comprises the following steps:

s1, creating a three-layer Python function on a user to serve as a main body of the Python modifier, and dividing the Python function into an outermost layer, a middle layer and an innermost layer;

s2, setting a dictionary for storing all modified Python functions for calling the Python modifier, judging whether the modified Python functions are called for the first time by checking stored information in the dictionary, and directly entering S9 if the modified Python functions are judged to be called for the non-first time;

s3, inquiring whether the information of the static Yun Hanshu library is contained in a parameter list of the outermost layer of the Python function, if the information of the existing static cloud function is inquired, indicating that the modified Python function is deployed, and directly entering S8;

s4, extracting modified Python function content through a middle layer of the Python function, and performing binary stream conversion on the modified Python function at the innermost layer;

s5, encoding the modified Python function with binary stream conversion completed, and storing and uploading the encoded codes;

s6, reading parameter configuration in the modifier from a parameter list of the outermost layer of the Python function, and checking the read parameter configuration proportion to serve as cloud function configuration information;

s7, creating a cloud function with the same name as the name of the modified Python function through the SDK of the cloud platform according to the naming rule and the acquired code and cloud function configuration information, and acquiring a call address;

s8, updating the creation record in the dictionary;

s9, performing SDK call according to the call address obtained under the naming rule, returning a response object at the moment, and then taking a return data segment from the response object;

s10, analyzing and deserializing the returned data segment obtained in the S9, and returning data consistent with the normal returned data structure of the modified Python function;

and S11, deleting the functions of all the dynamic cloud function libraries at the end of program operation.

Preferably, the method further comprises S12, if the static function is not required to be used continuously after being deployed, the static function can be deleted remotely at the last position of the program.

Preferably, in the step S1, the outermost layer of the three-layer Python functions is used for analyzing the configuration parameters and judging whether the original Python function of the user is deployed by inquiring whether the static cloud function library information is contained in the configuration parameters; the middle layer is used for packaging the modified function and issuing the modified function into a cloud function; the innermost layer is used for calling the cloud function and deserializing the returned data, and analyzing the returned data into a correct data structure so as to realize the original function of the modified function.

Preferably, in the step S4, the binary stream conversion is performed on the function itself through marshal or cloudickle. This approach allows the global variable usage of the function itself to be converted as a function dependent data structure without losing semantics of the literal content as it would be converted to JSON form.

Preferably, in the step S5, the binary stream describing the function is encoded, and the encoded code is saved and uploaded, which specifically includes: and carrying out serialization and deserialization on a data stream obtained by converting the binary stream of the function in marshal or cloudlickle by adopting a pickle, and then saving and uploading codes obtained by completing serialization and deserialization to an OSS.

Preferably, in the step S6, the parameter configuration includes: the configuration of the CPU, the configuration of the memory size, the configuration of the disk size, the configuration of the timeout time, the configuration of concurrency and the like are supported.

Preferably, in the step S6, the read parameter configuration proportion is checked, if the read parameter configuration violates the predetermined configuration parameter proportion, an error is timely reported, and the error is returned to the user; if the configuration parameters are not read, default values are used.

Preferably, in the step S7, a cloud function with the same name as the function name is created through the SDK of the cloud platform according to the naming rule according to the acquired code and function configuration information, and the call address is obtained, specifically: and creating a cloud function with the same name as the function name under the runtime library of the user through the SDK of the cloud platform, and obtaining the call address under the naming rule.

Compared with the prior art, the invention has the following beneficial effects:

1. according to the method, the function codes are modified according to the sequence of the segmentation function granularity, the cloud platform creates the function, the cloud platform configures the function, the cloud platform deploys the function, the cloud platform tests the function, the interface is called locally, and the eight steps of modifying the original function call are integrated into the addition of the primary modifier, so that the reliability and simplicity of the method are ensured.

2. According to the invention, cloud computing is used for partial content of local computing, and the performance bottleneck is broken through by means of a super-large-scale machine, so that the running efficiency is greatly improved; meanwhile, the cloud functionalization technology is utilized to transfer the work, so that a user does not need to care about the content of a server, and the main program runs locally, so that the operation is more visual; by means of storing the common functions in a static Yun Hanshu library in a lasting mode, the functions can be cloud-functionalized to the cloud end and can be called without repeated writing in the local area, reusability of the functions is improved, and meanwhile requirements of users on manual operation according to different commercial platforms are saved; through modifying each function code needing cloud functionalization, then carrying out interface operation once to create, debug and run, finally obtaining an interface and then modifying the flow of a local program of the interface, thereby reducing the learning cost; by enabling the user to not need to modify the calling function and the sentences involved in the calling function, the whole application can be operated by only designating the calling function which is functionalized by the cloud.

Drawings

FIG. 1 is a schematic flow chart of a non-perception local code cloud functionalized deployment calling method based on a Python modifier;

FIG. 2 is a partial flow diagram of a Python modifier-based, non-aware native code cloud functionalized deployment and invocation method;

FIG. 3 is a schematic diagram comparing the process of the present invention with the original process of the prior art user;

FIG. 4 is a schematic diagram of an example of the modification of the original program according to the present invention;

FIG. 5 is a schematic diagram of an example of the present invention deployed to a static cloud function library code;

fig. 6 is a schematic diagram of an example of the multiplexing static Yun Hanshu library code of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Example 1

Referring to fig. 1 and 2, the method for deploying and calling the non-perception local code cloud functionalization based on the Python modifier comprises the following steps:

the outermost layer is used for analyzing the configuration parameters and judging whether the Python function is deployed by inquiring whether the configuration parameters contain static cloud function library information or not; the middle layer is used for packaging the modified function; the innermost layer is used for the main flow of the function.

specifically, a global dictionary is set to store all functions using the Python modifier, whether three layers of Python functions are called for the first time is judged by detecting and checking stored information in the dictionary, and if yes, the next step is carried out; otherwise, the call is not the first call, and at the moment, a runtime service name is created on the cloud platform and is used as a temporary cloud function library, and if persistence is required, the cloud function library name is created.

S3, inquiring whether the outermost layer of the Python function contains the information of the static Yun Hanshu library, if so, indicating that the Python function is deployed, and directly entering S8 only by calling so as to prevent subsequent repeated inspection;

s4, extracting function content through an intermediate layer of the Python function, and performing binary stream conversion on the function per se through marshal or cloudickle at the innermost layer. This way, the global variable usage and function dependent data structure of the method are converted, and the semantics of the text content is not lost as the text content is converted into a JSON form;

s5, coding a cloud function obtained by completing binary stream conversion, and storing and uploading the coded codes;

since cloud functions require the importation of a JSON as input data and a handler as the only master function, the function that completes the binary stream conversion needs to be encoded. In order to ensure the subsequent data receiving and decoding operation, the serialization and the deserialization of the data stream are performed by using a pick. Specifically, a data stream obtained by converting a binary stream of a function in marshal or cloudickle is serialized and deserialized by a pickle, a handler is added to the coded code, and then the code obtained by serialization and deserialization is saved and uploaded to an OSS.

For the input data content, as the consistency of data transfer is considered in the code conversion, the uploaded data can be directly packaged by using a seal, the data is initiated to a calling interface through the SDK of the platform so as to acquire return data, and then the data flow in the return data is decoded by the seal so as to obtain the original return value.

S6, reading parameter configuration in the modifier from a parameter list of the outermost layer of the Python function, wherein the parameter configuration comprises configuration supporting a CPU, a memory size, a disk size, a timeout time, concurrency and the like, checking the read parameter configuration proportion, for example, the disk size is required to be 512MB or 10240MB only, the proportion of the CPU and the memory size is 1-4 and the like, if the read parameter configuration violates the set configuration parameter proportion, reporting errors in time and returning to a user; if the configuration parameters are not read, using default values as function configuration information;

s7, creating a cloud function with the same name as the function name through the SDK of the cloud platform according to the naming rule and the acquired code and function configuration information to acquire a calling address;

specifically, a cloud function with the same name as the function name is newly built under the run time library of the user, and the call address (run time. Function_name) under the naming rule is obtained by creating through the SDK of the cloud platform. If the user does not have the runtime library, automatically creating the runtime library, and then creating a cloud function with the same name as the function name under the created runtime library.

S8, updating the creation record in the dictionary;

s9, performing SDK call according to the call address obtained under the naming rule;

whether the first call or the subsequent call is performed, the function creation according to the naming rule is finished only by entering the eighth step, at this time, the SDK call is directly performed according to the call address under the naming rule, at this time, a response object is returned, and then the returned data segment is taken from the response object;

since the return is a response object, the content of the data segment needs to be fetched first, and since the function return deployed by the invention is based on the pointer, the data consistent with the normal return data structure of the modified Python function can be obtained through direct deserialization.

S12, if the static function is not needed to be used continuously after being deployed, remote deletion can be carried out at the last position of the program.

Referring to fig. 3, the present invention integrates the existing user to modify the granularity of the function, create the function for the cloud platform, configure the function for the cloud platform, deploy the function for the cloud platform, test the function for the cloud platform, call the interface locally, modify the original function and call eight steps into a primary modifier, and make low modification to the original code, and meanwhile, automatically and rapidly deploy and call the cloud function, so that the present user can improve the execution efficiency by means of the cloud computing capability, further simplify the working flow of writing, debugging, deploying and calling the present user to self-cloud function for the cloud function, reduce the learning cost of the cloud function by performing non-perceptibility on the operation of the cloud computing platform, and enhance the reusability by adopting the design modes of a static cloud function library and a temporary cloud function library, thereby providing a practical way for the cloud function of scientific computing and distributed computing.

Example 2

Taking a parallelization calculation program for calculating Pi by monte carlo method as an example: 1e8 simulations were performed, 50 thread calculations. The present invention is configured as Intel (R) Core (TM) i5-9300H CPU@2.40GHz,RAM 16.0G, requiring 27.13 seconds, while each function configuration 0.35vCPU,RAM 256MB of the program using this technique requires only 1.66 seconds, resulting in an improvement in efficiency of 1634.33%, compared to the original cloud function editing process, or the prior art requires pre-specifying a configuration file and pre-compiling, the present invention changes and adds only one line (line 17) on the original code, and cloud functionalization is performed in operation (see fig. 4). The invention can easily functionalize the program cloud and can perform configuration specification even under the condition of not deeply learning the use of the server-free computing and the cloud platform.

The static Yun Hanshu library name is declared internally (e.g., fig. 5 designates the static function library name as "pi"), which may be persisted for deployment to the static Yun Hanshu library (see fig. 5), and may be reused directly in subsequent use. Multiplexing the static Yun Hanshu library code (see fig. 6) does not require re-implementation, thus greatly reducing the amount of code written. The invention can be deployed to the static Yun Hanshu library through the code with high reusability, thereby not only facilitating the coding process, but also being capable of better resorting to the function of no-service calculation.

The invention realizes the following technical effects by adding the Python modifier:

(1) Realizes the breakthrough of scientific calculation performance: since existing researchers mostly use personal workstations or small clusters that are not configured differently, their computing power is often limited. Because cloud function calculation has the advantages of automatic expansion and contraction capacity and concurrent scheduling, and because of the development of networks and equipment, part of functions of scientific calculation are suitable for cloud functionalization.

(2) The background cluster is not serviced: because the existing large-scale cluster is used for calculation, the cluster management software is required to be configured and installed by oneself, maintenance is required in the use process, and meanwhile, the cluster management software is required to be submitted to a job scheduling system and then is waited, so that the process is not as simple and clear as the local operation. The cloud function technology is utilized to transfer the tasks, the user does not need to care about the content of the server, and the main program runs locally, so that the cloud function server is more visual.

(3) The reusability of the function is improved: there is much content in the computing field that is reused in multiple modules or multiple programs, often requiring repeated writing and recall during normal encoding, which increases the development effort for users. According to the method, the common function is stored as the static Yun Hanshu library in a lasting manner, so that the part of the function can be cloud-functionalized to the cloud end, and the function is called without repeated writing in the local area, so that the reusability of the function is improved.

(4) Realizing the non-perceptibility of the cloud platform: because different commercial platforms use different interfaces for operation, users need to manually modify each function code needing cloud functionalization according to the requirements of different commercial platforms, then interface operation is performed once to create, debug and run, finally an interface is obtained, and then the local program of the user is modified. The invention omits the automatic process, and reduces the learning cost.

(5) Realizing the unaware of cloud function call: because the user needs to send a data request to the interface of the deployed cloud function to receive response data, the user needs to know the interface, the data format and the sending mode of the sent data, and the user needs to know the data format when using the data, so that the user needs to know the received data format. The invention automates the process, so that the user can run the whole application by only designating the call function which is functionalized by the cloud without modifying the call function and the statement related to the call function.

Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims

1. The method for deploying and calling the non-perception local code cloud functionalization based on the Python modifier is characterized by comprising the following steps of:

s5, encoding the function for completing binary stream conversion, and storing and uploading the encoded codes;

s8, updating the creation record in the dictionary;

s11, for all cloud functions of the dynamic function library, remotely deleting the cloud functions at the last position of the program through function names stored in a dictionary.

2. The method for deploying and calling the non-perception local code cloud based on the Python modifier according to claim 1, wherein the outermost layer is used for analyzing configuration parameters and judging whether the original Python function of the user is deployed by inquiring whether the static cloud function library information is contained in the configuration parameters; the middle layer is used for packaging the modified function and issuing the modified function into a cloud function; the innermost layer is used for calling the cloud function and deserializing the returned data, and analyzing the returned data into a correct data structure so as to realize the original function of the modified function.

3. The method for deploying and calling the non-perception local code cloud functionalization based on the Python modifier according to claim 1, wherein in S4, binary stream conversion is performed on the function itself through marshal or cloudickle.

4. The method for deploying and calling the non-perception local code cloud functionalization based on the Python modifier according to claim 1, wherein in S5, the binary stream of the description function is encoded, and the encoded code is saved and uploaded, specifically: and carrying out serialization and deserialization on a data stream obtained by converting a binary stream of a marshal or a cloudlickle completion function by adopting a pickle, and then saving and uploading codes obtained by completing serialization and deserialization to an OSS.

5. The method for deploying and calling the non-perception local code cloud based on the Python modifier according to claim 1, wherein in the step S6, the read parameter configuration proportion is checked, if the read parameter configuration violates the set configuration parameter proportion, an error is timely reported, and the error is returned to a user; if the configuration parameters are not read, default values are used.

6. The method for deployment and invocation of non-aware native code cloud functionalization based on a Python modifier according to claim 1, wherein in S6, the parameter configuration comprises: the configuration of the CPU, the configuration of the memory size, the configuration of the disk size, the configuration of the timeout time and the configuration of concurrency are supported.

7. The method for deploying and calling the non-perception local code cloud based on the Python modifier according to claim 1, wherein in S7, according to the acquired code and function configuration information, a cloud function with the same name as the function name is created through the SDK of the cloud platform according to a naming rule to acquire a call address, specifically: and creating a cloud function with the same name as the function name under the runtime library of the user through the SDK of the cloud platform, and obtaining the call address under the naming rule.

8. The method for deploying and calling the non-aware local code cloud functionalization based on the Python modifier according to claim 1, further comprising the step of remotely deleting the static function based on the Python modifier at the last bit of the program if the static function is not required to be used after being deployed.