CN117933889A - Data task processing method and device - Google Patents

Abstract

The embodiment of the application provides a data task processing method and device. The method comprises the following steps: when creating a data task, the device may create the same data task in the present environment based on a configuration file corresponding to the data task that has already been created by another environment. Therefore, the automatic creation of the data task under multiple environments is realized, the efficiency of the creation of the data task is effectively improved, and the consistency of the data characters under the multiple environments is ensured.

Description

Data task processing method and device

Technical Field

The embodiment of the application relates to the field of cloud service, in particular to a data task processing method and device.

Background

Currently, in a data system based on a cloud server, creation of a data service generally requires manual operation, and a creation process includes, but is not limited to: creating jobs, creating data compute nodes, creating transmission tasks, configuring job schedules, and the like. Each sub-step takes a minute or so, and when facing the creation task of a large number of data services, it requires a lot of manpower and takes too much time.

Disclosure of Invention

The application provides a data task processing method. The method can realize automatic creation of the data task.

In a first aspect, an embodiment of the present application provides a data task processing method. The method comprises the following steps: acquiring a configuration file of a data task; the configuration file of the data task is generated when the data task is created in another environment. Based on the configuration file, a data task is created in the present environment. In this way, in the embodiment of the application, the cloud server can automatically create the data task in the environment based on the configuration file of the created data task in other environments, thereby effectively saving the labor investment, reducing the time consumption of the system for creating the data task and improving the overall performance of the system.

By way of example, the data task may be a big data calculation task as described in embodiments of the present application.

By way of example, an environment may refer to a functional environment or a site.

In one possible implementation, obtaining a configuration file of a data task includes: acquiring a configuration file of a data task from a shared storage; wherein the configuration file is created for another environment and stored in the shared storage. In this way, by sharing the configuration file of the store data task, multiple environments can share the configuration file. That is, each environment may read a configuration file from the shared storage and create a data task based on the configuration file.

Illustratively, the shared storage is a code bin as described in embodiments of the present application.

In one possible implementation, after obtaining the configuration file of the data task, the method further includes: acquiring an environment configuration file corresponding to the environment from the shared storage; the environment configuration file comprises at least one environment configuration parameter; and carrying out environment configuration on the environment based on the environment configuration file. In this way, in the embodiment of the application, the environment configuration information corresponding to each environment can be preconfigured in the shared storage, so that when the environment creates the data task, the environment configuration required by the data task can be executed based on the environment configuration information corresponding to the environment.

Exemplary environment configuration parameters include, but are not limited to, connection parameters, cloud server identification information (e.g., name), and the like.

In one possible implementation, the configuration file includes a plurality of configuration information, and based on the configuration file, creating a data task in the present environment includes: based on the association relation of a plurality of configuration information, data tasks are sequentially created in the environment. Thus, when the data task is automatically created, the data task which is completely the same as another environment can be created based on the association relation so as to ensure the consistency of the data task.

In one possible implementation, the configuration information is included in both the data task created in the other environment and the data task created in the present environment. In this way, through sharing the configuration file of the data task in the embodiment of the application, the data task with the same configuration information can be created in a plurality of environments, and the consistency of the data task is ensured while the manual participation is reduced.

In one possible implementation, the number of data tasks is multiple. In this way, in the embodiment of the application, under the scene that a large number of data tasks need to be created, the creation efficiency is obviously improved, and the consistency of the data tasks is ensured.

In one possible implementation, the method further includes: acquiring an update configuration file of a data task; the update configuration file is generated when the data task is updated in another environment; based on the update profile, the data task is updated in the present environment. In this way, the embodiment of the application can provide an automatic maintenance mode, so that a user can maintain data tasks in any environment and update other environments synchronously, thereby reducing the manual maintenance cost and ensuring the consistency of the data tasks during maintenance.

In a second aspect, an embodiment of the present application provides a data task processing device. The device comprises an acquisition module and an execution module. The acquisition module is used for acquiring the configuration file of the data task; the configuration file of the data task is generated when the data task is created in another environment. And the execution module is used for creating data tasks in the environment based on the configuration file.

In one possible implementation manner, the acquiring module is specifically configured to: acquiring a configuration file of a data task from a shared storage; wherein the configuration file is created for another environment and stored in the shared storage.

In one possible implementation manner, the obtaining module is further configured to obtain an environment configuration file corresponding to the present environment from the shared storage; the environment configuration file comprises at least one environment configuration parameter; and the execution module is also used for carrying out environment configuration on the environment based on the environment configuration file.

In one possible implementation manner, the configuration file includes a plurality of configuration information, and the execution module is specifically configured to: based on the association relation of a plurality of configuration information, data tasks are sequentially created in the environment.

In one possible implementation, the configuration information is included in both the data task created in the other environment and the data task created in the present environment.

In one possible implementation, the number of data tasks is multiple.

In one possible implementation, the obtaining module is further configured to obtain an update configuration file of the data task; the update configuration file is generated when the data task is updated in another environment; and the execution module is also used for updating the data task in the environment based on the updating configuration file.

In a third aspect, embodiments of the present application provide a computer readable medium storing a computer program comprising instructions for performing the method of the first aspect or any possible implementation of the first aspect.

In a fourth aspect, embodiments of the present application provide a computer program comprising instructions for performing the method of the first aspect or any possible implementation of the first aspect.

In a fifth aspect, an embodiment of the present application provides a chip, where the chip includes a processing circuit and a transceiver pin. Wherein the transceiver pin and the processing circuit communicate with each other via an internal connection path, the processing circuit performing the method of the first aspect or any one of the possible implementation manners of the first aspect to control the receiving pin to receive signals and to control the transmitting pin to transmit signals.

In a sixth aspect, an embodiment of the present application provides a cloud service system, where the system includes a plurality of environments related to the first aspect above.

Drawings

FIG. 1 is a schematic diagram of an exemplary system architecture;

Fig. 2 is a flow chart illustrating an exemplary data traffic processing method;

FIG. 3 is a schematic diagram of an exemplary illustrated modular interaction flow;

FIG. 4 is a schematic diagram of an exemplary user interface;

FIG. 5 is a schematic diagram of an exemplary user interface;

FIG. 6 is a schematic diagram of an exemplary user interface;

FIG. 7 is a schematic diagram of an exemplary device configuration;

fig. 8 is a schematic view of an exemplary device configuration.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone.

The terms first and second and the like in the description and in the claims of embodiments of the application, are used for distinguishing between different objects and not necessarily for describing a particular sequential order of objects. For example, the first target object and the second target object, etc., are used to distinguish between different target objects, and are not used to describe a particular order of target objects.

In embodiments of the application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g." in an embodiment should not be taken as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

In the description of the embodiments of the present application, unless otherwise indicated, the meaning of "a plurality" means two or more. For example, the plurality of processing units refers to two or more processing units; the plurality of systems means two or more systems.

Before describing the technical scheme of the embodiment of the present application, a communication system of the embodiment of the present application is first described with reference to the accompanying drawings. Fig. 1 is a schematic diagram of a possible system architecture to which the embodiment of the present application is applicable, referring to fig. 1, the system includes an environment 1, an environment 2, and an environment 3.

In one possible implementation, the environment described in the embodiments of the present application may be divided according to data traffic types, and may include, but is not limited to: the open self-test environment, the business test environment, the automatic test environment, the production environment and the like can be set according to actual requirements, and the application is not limited.

In another possible implementation manner, the environment described in the embodiment of the present application may also be with granularity of a site. For example, the environment 1 may be a chinese station in an open self-test environment, the environment 2 may be a european station in an open self-test environment, the environment 3 may be a chinese station in an automatic test environment, etc., and may be set according to actual requirements, and the present application is not limited thereto.

In yet another possible implementation manner, the environment described in the embodiment of the present application may be a set of multiple environments, or a set of multiple sites, which is not limited by the present application.

That is, the data service processing method executed in the embodiment of the present application is executed with an environment as granularity, for example, a plurality of big data computing tasks are created and managed in the environment 1, where the environment may be a minimum granularity with respect to a specific environment such as an open self-test environment, a minimum granularity with respect to a site, or a minimum granularity with respect to a plurality of sites or a plurality of environments, and the present application is not limited thereto.

Illustratively, each environment includes a cloud server. The cloud server in the embodiment of the present application may refer to a single server or may refer to a server cluster, which is not limited by the present application. In addition, the "cloud server" in the embodiment of the present application may be replaced by "cloud server provider" or "cloud server control end", which is not limited by the present application.

Illustratively, the environments in the embodiments of the present application are isolated from each other. That is, the different environments cannot communicate with each other without authorization.

Still referring to FIG. 1, each environment may provide a large data platform, by way of example. For example, environment 1 provides a big data platform 1, environment 2 provides a big data platform 2, and environment 3 provides a big data platform 3. Illustratively, the big data platform refers to a set of infrastructure mainly used for processing the scenes of mass data storage, calculation, continuous stream data real-time calculation and the like. Typically comprising clusters of Hadoop series, spark, storm, flink, flume/Kafka, etc.

For example, the big data platform may provide a user interface for a user to perform operations on the user interface provided by the big data platform, such as creating big data computing tasks or updating big data computing tasks, and the like. The big data platform may perform operations to create and manage big data computing tasks in the corresponding environment in response to received user operations. For example, big data platform 1 is used to create and manage big data computing tasks in environment 1.

In the embodiment of the application, the big data calculation task can also be replaced by a big data task, a data service, a calculation task of a data job or a data service, and the like, and the application is not limited.

Currently, in the prior art, a user may manually create a big data computing task through a user interface provided by a big data platform. Big data computing task creation flows include, but are not limited to: creating big data computing tasks, creating configuration information related to big data computing tasks, creating subtasks (e.g., transmission tasks, etc.) in big data computing tasks, etc.

In the prior art, the initial creation of a big data calculation task takes about 18 minutes, and taking other jobs with multiple calculation nodes and multiple transmission tasks into account, it takes 20 minutes to evaluate the creation of one big data calculation task. The subsequent maintenance time is as follows: in addition to the first creation, there is a continuing maintenance cost to follow, such as replacing logical jar reports in each big data computing task, assuming an average of 5 minutes, and an average of about 20% of the jobs need to be modified per month.

Taking Hua as a cloud big data team as an example, the same operation needs to be deployed in a 7-ring environment from research and development to current network. Each environment needs to deploy more than 100 big data calculation tasks, and the manual maintenance workload statistics:

The work load is created:

7 (collar) ×118 (task number) ×20 (min) = 16,520 min=275 hours

Maintenance workload (monthly):

7 (collar) ×118 (task number) ×20% (modification range) ×5 (min) =826 min=13 hours

In summary, if the big data calculation task under multiple environments is created and managed manually, the maintenance cost is extremely high, the probability of misoperation or missing operation is very high, and the configuration consistency among the environments is difficult to ensure.

In order to solve the problems in the prior art, the embodiment of the application provides a data service processing method. In the method, the cloud can automatically create and maintain the big data calculation task so as to reduce the maintenance cost and ensure the consistency of the big data calculation task under multiple environments.

Fig. 2 is a flow chart illustrating an exemplary data traffic processing method. Referring to fig. 2, the method specifically includes, but is not limited to, the following steps:

S201, acquiring a configuration file of a big data computing task in the environment 1.

Illustratively, a user may create at least one big data computing task in the environment 1 through the big data platform 1 in the environment 1. Each created task comprises configuration information required by the task.

For example, FIG. 3 is a schematic diagram of an exemplary module interaction flow. Referring to fig. 3, an exemplary embodiment of the present application is described by taking a user instruction to create a big data computing task 1, a big data computing task 2, and a big data computing task 3 in an environment 1 as an example, and in other embodiments, more or fewer big data computing tasks may be included, which is not limited by the present application.

Illustratively, in creating big data computing tasks, a user may create configuration information for each big data computing task through big data platform 1. Configuration information may also be referred to as configuration objects, which are used to describe the task content of a big data computing task.

For example, the configuration information 1 of the big data calculation task 1 includes, but is not limited to, configuration information 11 (may also be referred to as a configuration object), configuration information 12, configuration information 13, and the like. Configuration information 2 for big data computing task 2 includes, but is not limited to: configuration information 21, configuration information 22, configuration information 23, and the like. Configuration information 3 for big data computing task 3 includes, but is not limited to: configuration information 31, configuration information 32, configuration information 33, and the like. Wherein, the configuration information 11, the configuration information 21 and the configuration information 31 may be hive tables, the configuration information 12, the configuration information 22 and the configuration information 32 may be transmission tasks, and the configuration information 13, the configuration information 23 and the configuration information 33 may be SQL scripts.

It should be noted that the types of configuration information included in each big data calculation task may be the same or different.

It should be further noted that each big data calculation task may include more or less configuration information, and may be set according to actual requirements, which is not limited by the present application.

It should be further noted that the specific content of the configuration information of each big data calculation task may be the same or different. For example, the hive table of big data computing task 1 and the hive table of big data computing task 2 may be the same or different, and the present application is not limited thereto.

Fig. 4 is a schematic diagram of an exemplary illustrated user interface. Referring to fig. 4, for example, a user may create task information for big data computing task 1 in a user interface provided by big data platform 1 of environment 1. For example, the task information includes, but is not limited to: the environment in which the big data calculation task is located, the name of the big data calculation task, and the like. In the embodiment of the present application, the name of the big data computing task 1 is merely taken as an example for illustration, and the application is not limited and can be set according to actual requirements.

Illustratively, the big data platform 1 creates the big data calculation task 1 under the environment 1 based on the acquired task information in response to the received user operation, and displays the user interface 401 of the big data calculation task 1. Including but not limited to: a configuration interface 402, and some control options.

Exemplary control options include, but are not limited to: save option, submit option, export option, refresh option, etc. It should be noted that the number and names of the control options shown in fig. 4 are merely illustrative examples, and may be set according to actual requirements, and the present application is not limited thereto.

Illustratively, a user may create configuration information 1 for big data computing task 1 in user interface 401 for big data computing task 1. The configuration information 1 optionally includes: configuration information 11 (e.g., a hive table), configuration information 12 (e.g., a transport file), and configuration information 13 (e.g., an SQL script). The specific creation mode and the interface view can refer to the embodiment of the prior art, and the application is not repeated.

Illustratively, the big data platform 1 creates configuration information 1 in the environment 1 in response to a received user operation, including configuration information 11, configuration information 12, and configuration information 13. And, the big data platform 1 displays the configuration information 11, the configuration information 12, and the configuration information 13 in the configuration interface 402. For example, a graphical icon and/or a text icon corresponding to the configuration information may be displayed for indicating that the corresponding configuration information has been created.

Illustratively, similar to big data computing task 1, a user may create big data computing task 2 and its corresponding configuration information, and big data computing task 3 and its corresponding configuration information in environment 1. For specific details reference is made to the creation of configuration information in big data computing task 2, which is not illustrated here one by one.

For example, the user may instruct the big data platform to upload configuration information of the big data computing task into the code bin after creating one big data computing task or after creating a plurality of big data computing tasks. By way of example, code bins may also be referred to as auto-deployment micro-service code bins, and the application is not limited. The code bin is used for providing a code storage function for the cloud server, and the specific description can refer to the embodiment of the prior art, so that the application is not limited.

By way of illustration, still taking big data computing task 1 as an example, the user may click on the "export" option in user interface 401. The big data platform generates a configuration file 1 based on configuration information 1 (i.e., including configuration information 11, configuration information 12, and configuration information 13) of big data computing task 1 in response to the received user operation. Wherein the configuration file has a specified format, for example, the configuration file may be a json file.

In the embodiment of the application, the configuration file can also be understood as a template, so that the configuration file is taken as the template in other subsequent environments to create the same configuration information.

Illustratively, based on the same steps, the user may click on the export options of big data computing task 2 and big data computing task 3, respectively. The big data platform generates a configuration file 2 based on the configuration information 2 of the big data calculation task 2 and generates a configuration file 3 based on the configuration information 3 of the big data calculation task 3 in response to the received user operation.

Still referring to FIG. 3, exemplary big data platform 1 archives the generated configuration files (e.g., including configuration file 1, configuration file 2, and configuration file 3) into a code repository.

Illustratively, in the process of archiving the configuration file to the code bin by the big data platform 1, the configuration file may be archived (i.e. stored) according to a preset rule. Optionally, the preset rule is used for indicating the storage position of the configuration file, and the specific rule can be set according to actual requirements, which is not limited by the application.

It should be noted that, in the embodiment of the present application, only the case where the user manually creates the big data calculation task in the environment 1 and extends it to other environments is taken as an example for explanation. In other embodiments, the user may create tasks in environment 2 or any other environment and extend to other environments, and the application is not limited.

S202, creating big data calculation tasks under other environments based on the configuration file.

In an embodiment of the application, the code bin is a storage device shared by multiple environments. That is, each environment may read the data in the code bins. For example, the big data platform 1 stores configuration files 1-3 in the code compartment, and both the environment 2 and the environment 3 can read configuration files 1-3 from the code compartment. And, as described above, the configuration file of the big data computing task in the environment 1 may be regarded as a template, and in the embodiment of the present application, other environments may acquire the configuration file 1 of the big data computing task 1, the configuration file 2 of the big data computing task 2, and the configuration file 3 of the big data computing task 3 from the code bin, and create the corresponding big data computing task in the present environment based on the acquired configuration files.

Specifically, referring to fig. 3, for example, the code bin may have environment configuration information of each environment stored therein. For example, configuration information of environment 1 (simply referred to as environment 1 configuration), configuration information of environment 2, and configuration information of environment 3 (simply referred to as environment 3 configuration).

Exemplary, configuration information for an environment is used to describe the configuration of the environment, including, for example, but not limited to: database connection information, cloud server name, etc. For example, the database connection mode of the chinese station is different from the database connection mode of the european station, and the configuration information of the environment is mainly used to describe the configuration under different environments, and may also be understood as the configuration of cloud servers in different sites. The category and specific parameters included in the configuration information of the environment can be set according to actual requirements, and the application is not limited.

Alternatively, the environment configuration information may be configured by the user before S201 is performed. Alternatively, the user may set the environment configuration information through a big data platform. Alternatively, the user may update the already stored environment information at any time, or add or delete the environment configuration information, which is not limited by the present application.

Illustratively, the creation of big data computing tasks in environment 2 is illustrated. The execution module in the environment 2 acquires, from the code bin, environment configuration information of the environment 2 (simply referred to as environment 2 configuration), logic codes, and configuration files of big data computing tasks to be created (including, for example, big data computing task 1, big data computing task 2, and big data computing task 3).

The logic code is a preset code, which can be understood as a program code for executing a subsequent creation data service. Alternatively, the logic code may be stored in jar packets, which is not limiting of the application. Illustratively, similar to the environment configuration, the logic code is also pre-stored by the user to the code bin. In the embodiment of the present application, only the logic code and the environment configuration are taken as examples for explanation, in other embodiments, if other configurations or codes are needed in the creation process of the big data calculation task, the user can store the corresponding configurations or codes in the code bin as well, and in the execution process, the execution module can read the corresponding configurations or codes from the code bin, so that the present application is not illustrated one by one.

Illustratively, the execution modules in environment 2 may build a deployment file based on the configuration file of the environment 2 configuration, the logic code, and the big data computing tasks. It is also understood that the above information or files are packaged to generate a deployment package.

The execution module can execute a packet transfer flow, namely, a deployment file packet in the framework is transmitted to the micro-service module. The micro-service module may perform micro-service deployment based on the configuration of the environment 2, the logic code, and the configuration file of the big data computing task in the deployment package. For example, the micro-service module may configure a task environment that deploys big data computing tasks based on the environment 2. For example, configuring the transport interface, etc., in accordance with the connection indicated in the configuration of environment 2.

For example, the micro-service deployment may further include deployment based on logic code and deployment of micro-service based on configuration files, and the specific deployment procedure may refer to the embodiment of the prior art, which is not described in detail herein.

Illustratively, after the micro-service deployment process is finished, the execution module may create the big data computing task 1 to the big data computing task 3 under the present environment (i.e., the environment 2).

For example, taking big data computing task 1 as an example, the execution module may upload a logical jar package (i.e., the logical code described above) to the object store service (Object Storage Service, OBS). And, the execution module calculates the configuration file of task 1 based on big data, creates the corresponding configuration information. For example, create hive tables, create transport tasks, create SQL scripts, etc.

Alternatively, in the embodiment of the present application, an association relationship may exist between configuration information. Wherein the association relationship may be used to indicate a dependency relationship or may be used to indicate a sequential relationship. For example, in environment 1, a user creates big data computing task 1 by first creating configuration information 11 (e.g., hive tables), then creating configuration information 12 (e.g., transport tasks), and then creating configuration information 13 (e.g., SQL scripts). Correspondingly, in the environment 2, the execution module of the environment 2 can create each configuration information according to the association relation. That is, the respective configuration information is also created in the order of the configuration information 11, the configuration information 12, and the configuration information 13. Optionally, the configuration file in the embodiment of the present application may include association relationship information, which is used to indicate an association relationship between configuration information. Optionally, the corresponding association relationship may be determined by the sequence of the configuration information in the configuration file, which is not limited in the present application.

For example, the execution module may create multiple big data computing tasks sequentially or synchronously, such as shown in fig. 3, and the execution module may also create big data computing task 2 and big data computing task 3, and the specific creation process is similar to big data computing task 1, and is not illustrated one by one.

Fig. 5 is a schematic diagram of an exemplary user interface. Referring to fig. 5, a user interface of the big data platform 2 of the environment 2 is taken as an example. Illustratively, after the execution module of the environment 2 creates and completes the big data computing task 1 through the big data computing task 2, the user may click on the graphical representation of the big data computing task 1. The big data platform 2 displays a user interface 501 of the big data calculation task 1 in response to the received user operation. The task is a big data calculation task created in the environment 2. Among these, the user interface 501 of big data computing task 2 includes, but is not limited to, a configuration interface 502 and some control options. Wherein configuration interface 502 displays a configuration file for big data computing task 1. Wherein the displayed configuration information is the same as the configuration information of the big data calculation task 1 in the environment 1. It can be understood that the big data computing task in the environment 2 is used as a template, and the corresponding big data computing task is created in the present environment (i.e. the environment 2) according to the self environment configuration. That is, big data computing tasks 1-3 in environment 2 are meeting the environmental requirements of environment 2, i.e., created based on the environmental configuration of environment 2 during the creation process. In addition, since the big data computing tasks 1 to 3 in the environment 2 are created by taking the configuration files of the big data computing tasks 1 to 3 in the environment 1 as templates, the accuracy of the configuration information of the big data computing tasks created in the environment 2 can be ensured.

By way of example, environment 3 may create big data computing task 1 through big data computing task 3 in environment 3 as well, following the execution flow of environment 2.

In the embodiment of the present application, the creation of the environment 2 and the environment 3 is taken as an example. In other embodiments, a user-specified environment is also possible. For example, environment 2 creates big data computing task 1 through big data computing task 3, and environment 3 need not be created.

In the embodiment of the present application, the creation of the big data calculation task 1 to the big data calculation task 3 by both the environment 2 and the environment 3 is exemplified. In other embodiments, the big data calculation task to be created may be indicated by the user in the big data platform corresponding to the environment. For example, environments 2 and 3 may create only big data computing task 2.

In one possible implementation, the deployment environment packages built by the respective environments may be stored in a local cloud server. For example, a user may invoke a pre-built deployment environment package through a big data platform at any time and create a corresponding big data computing task in the environment. Optionally, for other environments, the user may instruct the present environment to extract a deployment environment package constructed in any environment, and update the environment configuration in the deployment environment package to the environment configuration of the present environment. And creating corresponding big data computing tasks in the environment based on the deployment environment package.

In another possible implementation, each environment may also maintain big data computing tasks in the present environment according to the flows shown in fig. 2 and 3. For example, FIG. 6 is a schematic diagram of an exemplary user interface. Referring to fig. 6, taking the user interface of the big data platform 1 as an example, a user may modify the configuration information 11 of the big data computing task 1 into configuration information 11' in the configuration interface 602 of the user interface 601 of the big data computing task 1. The user may click on the export option. The big data platform 1 generates a configuration file 1' in response to the received user operation and files the configuration file to a code bin. Wherein configuration file 1 'comprises configuration information 11', configuration file 12 and configuration file 13. Accordingly, taking environment 2 as an example, environment 2 may read configuration file 1'. The environment 2 may determine that the big data computing task 1 is updated based on the configuration file 1'. In one example, environment 2 may delete big data computing task 1and recreate big data computing task 1 as per the steps in flow 3. In another example, environment 2 may update only a portion of the content, e.g., environment 2 detects that configuration information 11 is updated, environment 2 may delete configuration information 11 and create configuration information 11'.

The foregoing details of the method according to the embodiments of the present application and the apparatus according to the embodiments of the present application are provided below.

Fig. 7 is a schematic structural diagram of a data task processing device according to an embodiment of the present application. The device comprises: an acquisition module 701 and an execution module 702. The acquiring module 701 is configured to acquire a configuration file of a data task. The configuration file of the data task is generated when the data task is created in another environment. An execution module 702 is configured to create a data task in the present environment based on the configuration file.

In one possible implementation manner, the obtaining module 701 is specifically configured to: acquiring a configuration file of a data task from a shared storage; wherein the configuration file is created for another environment and stored in the shared storage.

In a possible implementation manner, the obtaining module 701 is further configured to obtain an environment configuration file corresponding to the present environment from the shared storage; the environment configuration file comprises at least one environment configuration parameter; the execution module 702 is further configured to perform environmental configuration on the present environment based on the environmental configuration file.

In one possible implementation, the configuration file includes a plurality of configuration information, and the execution module 702 is specifically configured to: based on the association relation of a plurality of configuration information, data tasks are sequentially created in the environment.

In one possible implementation, the configuration information is included in both the data task created in the other environment and the data task created in the present environment.

In one possible implementation, the number of data tasks is multiple.

In one possible implementation, the obtaining module 701 is further configured to obtain an update configuration file of the data task; the update profile is generated when the data task is updated in another environment. The execution module 702 is further configured to update the data task in the present environment based on the update configuration file.

In another example, FIG. 8 shows a schematic block diagram of an apparatus 800 of an embodiment of the application. The apparatus may include: the processor 801 and transceiver/transceiving pins 802, optionally, also include a memory 803. The processor 801 may be configured to perform the steps performed by the cloud server in the methods of the foregoing embodiments and control the receive pin to receive signals and the transmit pin to transmit signals.

The various components of the apparatus 800 are coupled together by a bus 804, where the bus system 804 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled in the drawing as bus system 804.

Alternatively, the memory 803 may be used for storing instructions in the foregoing method embodiments.

It should be understood that the apparatus 800 according to the embodiment of the present application may correspond to the cloud server in the foregoing methods of the embodiments, and that the foregoing and other management operations and/or functions of the respective elements in the apparatus 800 are respectively for implementing the corresponding steps of the foregoing methods, and are not repeated herein for brevity.

All relevant contents of each step related to the above method embodiment may be cited to the functional description of the corresponding functional module, which is not described herein.

Based on the same technical idea, the embodiments of the present application further provide a computer readable storage medium storing a computer program, where the computer program includes at least one piece of code, and the at least one piece of code is executable by an apparatus to control the apparatus to implement the above-mentioned method embodiments.

Based on the same technical idea, the embodiments of the present application also provide a computer program for implementing the above-mentioned method embodiments when the computer program is executed by an apparatus. In the embodiment of the application, the computer program can be stored on the target cloud server in the form of an SDK. Optionally, when any cloud server needs to implement the method flow in the embodiment of the present application, a computer program (for example, SDK) for executing the flow in the embodiment of the present application may be downloaded from the target cloud server. Any cloud server installs and runs the computer program, so that the cloud server can realize the flow in the embodiment of the application. Optionally, the computer program may also refer to an execution module and a micro-service deployment module in the embodiment of the present application, that is, a cloud server with a big data platform and a code bin, and after the computer program is downloaded and installed, the steps executed by the execution module and the micro-service deployment module in the embodiment of the present application may be implemented, for example, including steps of building, packaging, micro-service deployment, and creation of big data computing tasks.

The program may be stored in whole or in part on a storage medium that is packaged with the processor, or in part or in whole on a memory that is not packaged with the processor.

Based on the same technical concept, the embodiment of the application also provides a processor, which is used for realizing the embodiment of the method. The processor may be a chip.

The steps of a method or algorithm described in connection with the present disclosure may be embodied in hardware, or may be embodied in software instructions executed by a processor. The software instructions may be comprised of corresponding software modules that may be stored in random access Memory (Random Access Memory, RAM), flash Memory, read Only Memory (ROM), erasable programmable Read Only Memory (Erasable Programmable ROM), electrically Erasable Programmable Read Only Memory (EEPROM), registers, hard disk, a removable disk, a compact disk Read Only Memory (CD-ROM), or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC.

Those skilled in the art will appreciate that in one or more of the examples described above, the functions described in the embodiments of the present application may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, these functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are to be protected by the present application.

Claims (16)

1. A method of data task processing, comprising:

Acquiring a configuration file of a data task; the configuration file of the data task is generated when the data task is created in another environment;

and creating the data task in the environment based on the configuration file.

2. The method of claim 1, wherein the obtaining the profile of the data task comprises:

acquiring a configuration file of the data task from a shared storage; wherein the configuration file is created for the other environment and stored in the shared storage.

3. The method of claim 2, wherein after the obtaining the configuration file of the data task, the method further comprises:

acquiring an environment configuration file corresponding to the environment from the shared storage; wherein the environment configuration file comprises at least one environment configuration parameter;

and carrying out environment configuration on the environment based on the environment configuration file.

4. The method of claim 1, wherein the configuration file includes a plurality of configuration information, and wherein creating the data task in the present environment based on the configuration file includes:

and based on the association relation of the configuration information, the data tasks are sequentially created in the environment.

5. The method of claim 4, wherein the configuration information is included in both the data task created in the other environment and the data task created in the present environment.

6. A method according to any one of claims 1 to 4, wherein the number of data tasks is a plurality.

7. The method according to claim 1, wherein the method further comprises:

Acquiring an update configuration file of the data task; the update configuration file is generated when the data task is updated in the other environment;

And updating the data task in the environment based on the update configuration file.

8. A data task processing device, comprising:

the acquisition module is used for acquiring the configuration file of the data task; the configuration file of the data task is generated when the data task is created in another environment;

And the execution module is used for creating the data task in the environment based on the configuration file.

9. The apparatus of claim 8, wherein the obtaining module is specifically configured to:

acquiring a configuration file of the data task from a shared storage; wherein the configuration file is created for the other environment and stored in the shared storage.

10. The apparatus of claim 9, wherein the device comprises a plurality of sensors,

The acquisition module is further used for acquiring an environment configuration file corresponding to the environment from the shared storage; wherein the environment configuration file comprises at least one environment configuration parameter;

The execution module is further configured to perform environment configuration on the environment based on the environment configuration file.

11. The apparatus of claim 8, wherein the configuration file includes a plurality of configuration information, and the execution module is specifically configured to:

and based on the association relation of the configuration information, the data tasks are sequentially created in the environment.

12. The apparatus of claim 11, wherein the configuration information is included in both the data task created in the other environment and the data task created in the present environment.

13. The apparatus according to any one of claims 8 to 12, wherein the number of data tasks is a plurality.

14. The apparatus of claim 8, wherein the device comprises a plurality of sensors,

The acquisition module is also used for acquiring the update configuration file of the data task; the update configuration file is generated when the data task is updated in the other environment;

the execution module is further configured to update the data task in the present environment based on the update configuration file.

15. A computer-readable storage medium, comprising:

the computer readable storage medium is used for storing instructions or a computer program; the instructions or the computer program, when executed, cause the method of any one of claims 1 to 7 to be implemented.

16. A computer program product, comprising: instructions or computer programs;

The instructions or the computer program, when executed, cause the method of any one of claims 1 to 7 to be implemented.