CN115495225A - Data processing method and device, storage medium and electronic device - Google Patents
Data processing method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN115495225A CN115495225A CN202211452629.0A CN202211452629A CN115495225A CN 115495225 A CN115495225 A CN 115495225A CN 202211452629 A CN202211452629 A CN 202211452629A CN 115495225 A CN115495225 A CN 115495225A
- Authority
- CN
- China
- Prior art keywords
- data processing
- target
- container
- data
- initial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
- G06F8/63—Image based installation; Cloning; Build to order
Abstract
The application discloses a data processing method and device, a storage medium and an electronic device, wherein the data processing method comprises the following steps: acquiring one or more target toolkits corresponding to the received data processing service from a plurality of heterogeneous data processing toolkits; creating an initial data processing container corresponding to each target tool package to obtain a target tool package and an initial container set which have a corresponding relation; according to the target tool packages and the initial container sets with corresponding relations, each target tool package is installed in a corresponding initial data processing container to obtain one or more target container sets; the technical scheme is adopted to solve the problems of low data processing efficiency and the like in the related technology.
Description
Technical Field
The present application relates to the field of computers, and in particular, to a method and an apparatus for processing data, a storage medium, and an electronic apparatus.
Background
At present, in the field of automatic driving of automobiles, in order to serve iteration of algorithms, various tools are often required to perform operations such as desensitization, cleaning, unpacking, packaging and the like on a large amount of data, the tools are often distributed in each business department and are developed by using different languages, and heterogeneous tools developed by the different languages generally need to be called each other in the data processing process to realize distributed calculation on the data.
In the whole process of data closed loop (acquisition-storage-training-simulation-evaluation-deployment), a tool is very important for processing data, each link has a corresponding tool, a single computing node has insufficient computing power for processing the data, the data often needs to be loaded into the computing node in advance, the storage capacity of the node is limited, and finally the processing efficiency of the data is not high, and in some cases, function calling is needed between the tool and the tool, if the development languages of the tools are different, mutual calling has a great problem, the calling process is complicated, and the efficiency of data processing is influenced.
Aiming at the problems of low efficiency of data processing and the like in the related art, no effective solution is provided.
Disclosure of Invention
The embodiment of the application provides a data processing method and device, a storage medium and an electronic device, so as to at least solve the problems that the efficiency of data processing is low and the like in the related art.
According to an embodiment of the present application, there is provided a data processing method including:
acquiring one or more target tool packages corresponding to a received data processing service from a plurality of heterogeneous data processing tool packages, wherein the data processing service is used for requesting processing of target data, and the target tool packages are used for indicating data processing tools used for realizing the data processing service;
creating an initial data processing container corresponding to each target tool package to obtain a target tool package and an initial container set which have a corresponding relation;
according to the target toolkits and the initial container sets with corresponding relations, installing each target toolkit into a corresponding initial data processing container to obtain one or more target container sets, wherein each target container in the target container sets is provided with a data processing interface, and the data processing interfaces are used for converting a target format and a heterogeneous format corresponding to each target toolkit;
and scheduling one or more target container sets to process the target data through the data processing interface according to the data processing service.
Optionally, the creating an initial data processing container corresponding to each of the target toolkits, the method comprises the following steps:
creating a first configuration file corresponding to each target toolkit, wherein the first configuration file is used for indicating attribute information of the initial data processing container to be created for each target toolkit;
and executing the first configuration file in a cluster through a cluster control command to obtain the initial data processing container corresponding to each target tool package.
Optionally, the creating a first configuration file corresponding to each of the target toolkits includes:
and creating a first data serialization format yaml file corresponding to each target toolkit, wherein file parameters of the first data serialization format yaml file comprise a kid attribute of Deployment, the number of containers to be created, the name of the target toolkit and the address of the target toolkit.
Optionally, the installing each target tool package into a corresponding initial data processing container according to the target tool package and the initial container set having a corresponding relationship to obtain one or more target container sets, including:
creating a second configuration file corresponding to each initial data processing container, wherein the second configuration file is used for indicating the information of the target toolkit to be installed in the initial data processing container and the interface information of the data processing interface;
and executing the second configuration file in each initial data processing container through a cluster control command to obtain one or more target container sets.
Optionally, the creating a second configuration file corresponding to each of the initial data processing containers includes:
and creating a second xml file corresponding to each initial data processing container, wherein file parameters of the second xml file include a kind attribute as a service, a Selector as the target toolkit corresponding to each initial data processing container, and interface information of the data processing interface.
Optionally, the executing the second configuration file in each initial data processing container through the cluster control command includes:
installing the corresponding target toolkit in each initial data processing container in a service object constructing mode according to the kid attribute to obtain a reference container;
and deploying the data processing interface for the reference container according to the interface information of the data processing interface.
Optionally, the scheduling one or more target container sets to process the target data through the data processing interface according to the data processing service includes:
sequentially calling each target container set to perform data processing according to the operation sequence of the data processing service;
acquiring a data processing result returned by each target container set;
under the condition that the data processing result does not belong to the final processing result of the target data in the data processing service, sending the data processing result to the next target container set in the operation sequence;
and storing the data processing result under the condition that the data processing result belongs to the final processing result of the target data in the data processing service.
Optionally, the invoking each target container set to perform data processing includes:
creating a first call request in a target format, wherein the first call request carries a data link of data to be processed;
sending the first call request to the data processing interface on one target container in each target container set according to load balancing;
converting the first calling request from the target format into a corresponding second calling request in the heterogeneous format through the data processing interface;
acquiring the data link from the second calling request;
and loading the data to be processed into the target container for processing according to the data link.
Optionally, before the obtaining one or more target toolkits corresponding to the received data processing service from the multiple heterogeneous data processing toolkits, the method further includes:
generating a tool mirror image of each data processing tool in a plurality of heterogeneous data processing tools to obtain a plurality of heterogeneous data processing tool packages;
and creating a corresponding micro-service module for each data processing toolkit, wherein the micro-service module is used for calling each data processing toolkit.
Optionally, after the scheduling one or more target container sets to process the target data through the data processing interface according to the data processing service, the method further includes:
destroying one or more of the target container sets;
and releasing the storage resources and the computing resources occupied by one or more target container sets.
According to another embodiment of the present application, there is also provided a data processing apparatus including:
an obtaining module, configured to obtain one or more target toolkits corresponding to a received data processing service from a plurality of heterogeneous data processing toolkits, where the data processing service is used to request processing of target data, and the target toolkit is used to indicate a data processing tool used to implement the data processing service;
the first creating module is used for creating an initial data processing container corresponding to each target tool package to obtain a target tool package and an initial container set which have a corresponding relation;
the installation module is used for installing each target tool package into a corresponding initial data processing container according to the target tool package and the initial container set which have corresponding relations, so as to obtain one or more target container sets, wherein each target container in the target container sets is provided with a data processing interface, and the data processing interfaces are used for converting a target format and a heterogeneous format corresponding to each target tool package;
and the scheduling module is used for scheduling one or more target container sets to process the target data through the data processing interface according to the data processing service.
According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the above data processing method when running.
According to another aspect of the embodiments of the present application, there is also provided an electronic apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method for processing data through the computer program.
In the embodiment of the application, one or more target toolkits corresponding to a received data processing service are acquired from a plurality of heterogeneous data processing toolkits, wherein the data processing service is used for requesting to process target data, and the target toolkits are used for indicating data processing tools used for realizing the data processing service; creating an initial data processing container corresponding to each target tool package to obtain a target tool package and an initial container set which have a corresponding relation; according to the target toolkits and the initial container sets with the corresponding relations, each target toolkit is installed in the corresponding initial data processing container to obtain one or more target container sets, wherein each target container in the target container sets is provided with a data processing interface, and the data processing interfaces are used for converting a target format and a heterogeneous format corresponding to each target toolkit; the method comprises the steps of scheduling one or more target container sets to process target data through a data processing interface according to a data processing service, namely, firstly, obtaining one or more corresponding target tool packages from a plurality of heterogeneous data processing tool packages according to the received data processing service, wherein the data processing service is used for requesting to process the target data, the target tool packages are used for indicating data processing tools used for realizing the data processing service, then, creating corresponding initial data processing containers according to the target tool packages, obtaining the target tool packages and the initial container sets with corresponding relations, then, installing each target tool package into the corresponding initial data processing container according to the target tool packages and the initial container sets with corresponding relations, obtaining one or more target container sets, deploying the data processing interface on each target container in the target container sets, wherein the data processing interface is used for converting the target formats and the heterogeneous formats corresponding to the target tool packages, namely, realizing mutual calling among the heterogeneous tools through the data processing interface, then scheduling one or more target container sets to process the target data according to the data processing service, realizing convenient calling of the target container sets to process the target data, and increasing the target container processing capability of the target processing tools. By adopting the technical scheme, the problems of low data processing efficiency and the like in the related technology are solved, and the technical effect of improving the data processing efficiency is realized.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
FIG. 1 is a diagram of a hardware environment for a method of processing data according to an embodiment of the present application;
FIG. 2 is a flow chart of a method of processing data according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a destination toolkit acquisition according to an embodiment of the present application;
FIG. 4 is a schematic diagram of an initial container set creation according to an embodiment of the present application;
FIG. 5 is a schematic diagram of an initial data processing container creation according to an embodiment of the present application;
FIG. 6 is a schematic diagram of a set of target containers, according to an embodiment of the present application;
FIG. 7 is a schematic illustration of an alternative processing of data according to an embodiment of the application;
fig. 8 is a block diagram of a data processing apparatus according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The method provided by the embodiment of the application can be executed in a computer terminal, a device terminal or a similar operation device. Taking the example of being operated on a computer terminal, fig. 1 is a hardware environment diagram of a data processing method according to an embodiment of the present application. As shown in fig. 1, the computer terminal may include one or more (only one shown in fig. 1) processors 102 (the processors 102 may include, but are not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, and in an exemplary embodiment, may also include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration and is not intended to limit the structure of the computer terminal. For example, the computer terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration with equivalent functionality to that shown in FIG. 1 or with more functionality than that shown in FIG. 1.
The memory 104 may be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the data processing method in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, so as to implement the above-mentioned method. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 can further include memory located remotely from the processor 102, which can be connected to a computer terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the computer terminal. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
In this embodiment, a data processing method is provided, which is applied to the computer terminal, and fig. 2 is a flowchart of a data processing method according to an embodiment of the present application, and as shown in fig. 2, the flowchart includes the following steps:
step S202, one or more target tool packages corresponding to a received data processing service are obtained from a plurality of heterogeneous data processing tool packages, wherein the data processing service is used for requesting to process target data, and the target tool packages are used for indicating data processing tools used for realizing the data processing service;
step S204, an initial data processing container corresponding to each target tool package is created, and a target tool package and an initial container set with corresponding relations are obtained;
step S206, according to the target toolkits and the initial container sets with corresponding relations, installing each target toolkit into a corresponding initial data processing container to obtain one or more target container sets, wherein each target container in the target container sets is provided with a data processing interface, and the data processing interfaces are used for converting a target format and a heterogeneous format corresponding to each target toolkit;
step S208, scheduling one or more target container sets to process the target data through the data processing interface according to the data processing service.
Through the steps, firstly, one or more corresponding target toolkits are obtained from a plurality of heterogeneous data processing toolkits according to a received data processing service, the data processing service is used for requesting to process target data, the target toolkits are used for indicating data processing tools used for realizing the data processing service, then, corresponding initial data processing containers are created according to the target toolkits, target toolkits and initial container sets with corresponding relations are obtained, then, each target toolkit is installed into the corresponding initial data processing container according to the target toolkits and the initial container sets with corresponding relations, one or more target container sets are obtained, in addition, a data processing interface is deployed on each target container in the target container sets, the data processing interfaces are used for converting the target formats and the heterogeneous formats corresponding to the target toolkits, namely, mutual calling among the heterogeneous tools can be realized through the data processing interfaces, then, one or more target container sets are scheduled through the data processing interfaces according to the data processing service, the target toolkits are conveniently called, and the target toolkits are deployed in a plurality of containers, and the target toolkit capacity of the target toolkits is increased. By adopting the technical scheme, the problems of low data processing efficiency and the like in the related technology are solved, and the technical effect of improving the data processing efficiency is realized.
In the technical solution provided in step S202, the data processing service may be, but is not limited to, any processing service that may be required by data, and in the whole process of data closed loop (acquisition-storage-training-simulation-evaluation-deployment), the processing service that may be required by data may include, but is not limited to: desensitization services, cleaning services, unpacking services, packing services, and so on.
Optionally, in this embodiment, fig. 3 is a schematic diagram of obtaining a destination toolkit according to an embodiment of the present application, and as shown in fig. 3, a service type requested by destination data is determined according to a data processing service, for example, the data processing service requests to perform a processing service a and a processing service B on the destination data, and one or more destination toolkits (destination toolkit 1, … …, destination toolkit k) corresponding to the processing service a and the processing service B are determined from a plurality of heterogeneous data processing toolkits (data processing toolkit 1 to data processing toolkit n), where the destination toolkit may indicate a data processing tool used for implementing the data processing service, for example, the data processing tool 1 indicated by the data processing toolkit 1 may implement the processing service a requested by the data processing service, and the data processing tool k indicated by the data processing toolkit k may implement the processing service B requested by the data processing service.
In the technical solution provided in step S204 above, fig. 4 is a schematic diagram of creating an initial container set according to an embodiment of the present application, and as shown in fig. 4, taking a destination tool pack 1 in the one or more destination tool packs (destination tool pack 1, … …, destination tool pack k) as an example, an initial data processing container corresponding to each destination tool pack is created, so as to obtain a destination tool pack (destination tool pack 1) and an initial container set (initial data processing container 11 to initial data processing container 1 n) having a corresponding relationship, and similarly, a destination tool pack (destination tool pack k) and an initial container set (initial data processing container k1 to initial data processing container kn) having a corresponding relationship can be obtained.
In an exemplary embodiment, the initial data processing container corresponding to each of the target toolkits may be created, but is not limited to, by: creating a first configuration file corresponding to each target toolkit, wherein the first configuration file is used for indicating attribute information of the initial data processing container to be created for each target toolkit; and executing the first configuration file in a cluster through a cluster control command to obtain the initial data processing container corresponding to each target tool package.
Optionally, in this embodiment, taking the target toolkit 1 as an example, fig. 5 is a schematic diagram of creating an initial data processing container according to an embodiment of the present application, and as shown in fig. 5, a corresponding first configuration file may be created based on the target toolkit, where the first configuration file may include attribute information of the initial data processing container created by each target toolkit, the first configuration file is identified by a cluster, and the cluster control command is released, and the first configuration file is executed in the cluster, so as to obtain the initial data processing containers (initial data processing container 11 to initial data processing container 1 n) corresponding to each target toolkit (target toolkit 1).
In an exemplary embodiment, the first configuration file corresponding to each of the target toolkits may be created, but is not limited to, by: and creating a first data serialization format yaml file corresponding to each target toolkit, wherein file parameters of the first data serialization format yaml file comprise a kind attribute of Deployment of the Deployment, the number of containers to be created, the name of the target toolkit and the address of the target toolkit.
Optionally, in this embodiment, the first configuration file needs to be matched with a cluster, that is, the created first configuration file needs to be a file that can be identified by the cluster, and taking the cluster as a K8s cluster (kubernets, container cluster management system) as an example, the file format that can be identified by the K8s cluster is a first data serialization format yaml file, so that the first configuration file corresponding to the first created target toolkit is the first data serialization format yaml file, which is a format requirement of the first configuration file, and besides, the file content of the first configuration file, that is, the file parameter, also needs to include a kind attribute of Deployment of a deploymentom, the number of containers to be created, the name of the target toolkit, and the address of the target toolkit.
In the technical solution provided in step S206 above, fig. 6 is a schematic diagram of a target container set according to an embodiment of the present application, and as shown in fig. 6, each target kit is installed into a corresponding initial data processing container, and is installed into a corresponding initial data processing container (initial data processing container 11 to initial data processing container 1 n) by taking target kit 1 as an example, so as to obtain one target container set 1 (target container 11 to target container 1 n) corresponding to target kit 1, and similarly, the above steps are performed on each target kit, so as to obtain one or more target container sets (target container set 1 to target container set k).
Optionally, in this embodiment, a data processing interface is deployed on each target container, and as a target toolkit in different target container sets may be a heterogeneous tool, when data needs to call tools mutually, a data processing interface is used to convert a target format and a heterogeneous format corresponding to each target toolkit, for example, as shown in fig. 6, by taking an example of format conversion between a target container set 1 and a target container set k, data is processed in the target container set 1 to obtain initial data, the initial data needs to be processed in the target container set k continuously, but the initial data is directly transmitted to the target container set k, and as a heterogeneous tool is arranged between the target container set 1 and a tool deployed in the target container set k, an identification error may occur, and therefore, the data processing interface 1 may be used to convert the target format of the initial data into a heterogeneous format that is allowed to be identified by the tool deployed in the target container set k.
In an exemplary embodiment, each target tool package may be installed into a corresponding initial data processing container according to a target tool package and an initial container set having a corresponding relationship, but not limited to, by the following method, to obtain one or more target container sets: creating a second configuration file corresponding to each initial data processing container, wherein the second configuration file is used for indicating information of the target toolkit to be installed in the initial data processing container and interface information of the data processing interface; and executing the second configuration file in each initial data processing container through a cluster control command to obtain one or more target container sets.
Optionally, in this embodiment, the second configuration file may indicate information of a deployed target toolkit in the initial data processing container and interface information of a data processing interface corresponding to the initial data processing container, so as to obtain a target container corresponding to the initial data processing container.
In an exemplary embodiment, the second configuration file corresponding to each of the initial data processing containers may be created, but is not limited to, by: and creating a second xml file corresponding to each initial data processing container, wherein file parameters of the second xml file include a kind attribute as a service, a Selector as the target toolkit corresponding to each initial data processing container, and interface information of the data processing interface.
Optionally, in this embodiment, the second configuration file needs to be matched with a cluster, that is, the created second configuration file needs to be a file that can be identified by the cluster, and taking the cluster as a K8s cluster (kubernets, container cluster management system) as an example, the file format that can be identified by the K8s cluster is a first data serialization format yaml file, so that the second configuration file corresponding to the first created initial data processing container is a yaml file, which is a format requirement of the second configuration file, in addition to the above, the file content of the second configuration file, that is, the file parameter includes a kind attribute as a service, and the Selector is the target toolkit corresponding to each initial data processing container, and the interface information of the data processing interface.
In an exemplary embodiment, said second configuration file may be executed in each of said initial data processing containers by means of a cluster control command, but not limited to, by: installing the corresponding target toolkit in each initial data processing container in a service object constructing mode according to the kid attribute to obtain a reference container; and deploying the data processing interface for the reference container according to the interface information of the data processing interface.
Optionally, in this embodiment, the initial data processing container, the reference container, and the target container may be, but are not limited to, a pod for performing unified management on service objects in a K8s cluster, where the pod is a minimum atomic unit of the K8s cluster (kubernets), a storage resource and a computing resource are provided for an internally operating tool, and the pods in the same K8s cluster share the storage resource and the computing resource of the K8s cluster.
In the technical solution provided by step S208 above, in a case that creation in a target container set is completed, one or more target container sets may be called through a data processing interface to process the target data, that is, a data processing service corresponding to the target data may include multiple services, for example: desensitization service, cleaning service, unpacking service, packing service, etc.
In an exemplary embodiment, one or more of the target container sets may be scheduled to process the target data through the data processing interface according to the data processing service, but not limited to, by: sequentially calling each target container set to perform data processing according to the operation sequence of the data processing service; acquiring a data processing result returned by each target container set; under the condition that the data processing result does not belong to the final processing result of the target data in the data processing service, sending the data processing result to the next target container set in the operation sequence; and storing the data processing result under the condition that the data processing result belongs to the final processing result of the target data in the data processing service.
Optionally, in this embodiment, the data processing service corresponding to the target data may include multiple data processing services, such as: for example, the operation sequence of the data processing service of the target data is desensitization service-cleaning service-unpacking service, wherein each service is processed by a corresponding target container set, for example, the target container set 1 processes the desensitization service, the target container set 2 processes the cleaning service, and the target container set 3 processes the unpacking service, and it can be known from the operation sequence of the processing service that the final processing result of the target data should be returned for the target container set 3, so that the data processing result returned by each target container set is obtained, and in the case that the data processing result does not belong to the final processing result of the target data in the data processing service, for example, the target container set 2 returns the data processing result which does not belong to the final processing result of the target data in the data processing service, and the data processing result returned by the target container set 2 is sent to the next target container set (target container set 3) in the operation sequence.
In an exemplary embodiment, data processing in each of the target container sets may be invoked, but is not limited to, by: creating a first call request in a target format, wherein the first call request carries a data link of data to be processed; sending the first call request to the data processing interface on one target container in each target container set according to load balancing; converting the first calling request from the target format into a corresponding second calling request in the heterogeneous format through the data processing interface; acquiring the data link from the second calling request; and loading the data to be processed into the target container for processing according to the data link.
Optionally, in this embodiment, load balancing may be implemented, but not limited to, by using a target container pod that performs unified management on service objects in the K8s cluster, and the data to be processed is transmitted to the pod in the idle state for processing.
Optionally, in this embodiment, the processing manner of the data to be processed may be, but is not limited to, sending the data to the target container in a manner carried by the data link, and processing the data to be processed by using a tool deployed in the target container.
In an exemplary embodiment, before the obtaining one or more destination toolkits corresponding to the received data processing services from the plurality of heterogeneous data processing toolkits, the following manners may be further included, but are not limited to be included: generating a tool mirror image of each data processing tool in a plurality of heterogeneous data processing tools to obtain a plurality of heterogeneous data processing tool packages; and creating a corresponding micro-service module for each data processing toolkit, wherein the micro-service module is used for calling each data processing toolkit.
Optionally, in this embodiment, the tool mirror image of each data processing tool is consistent with the function of the data processing tool, and by deploying the data processing tool package to different target containers pod and performing unified management using a K8s cluster (kubernets), load balancing is achieved while increasing processing computing power and improving data processing efficiency.
In an exemplary embodiment, after said scheduling one or more target container sets to process the target data through the data processing interface according to the data processing service, the following manners may also be included, but are not limited to: destroying one or more of the target container sets; and releasing the storage resources and the computing resources occupied by one or more target container sets.
Optionally, in this embodiment, since the tool in the target container set may occupy the storage resource and the computing resource when operating, after one or more target container sets are scheduled to process the target data through the data processing interface according to the data processing service, the one or more target container sets may be destroyed, and the storage resource and the computing resource occupied by the one or more target container sets are released.
In order to better understand the process of the data processing, the following describes the data processing flow with reference to an alternative embodiment, but the present invention is not limited to the technical solutions of the embodiments of the present application.
In this embodiment, a data processing method is provided, and fig. 7 is a schematic diagram of an optional data processing according to an embodiment of the present application, as shown in fig. 7:
according to the requirement and Restful architecture specification, tool modules (C + + tools, python tools, java tools and the like) with certain functions are developed, wherein the Restful architecture specification can realize decoupling of tool development, is associated with the outside by using an interface and can be independently deployed, the tool modules can be heterogeneous tools developed by different development languages, and the tool modules and micro-services are in one-to-one correspondence, such as: c + + tools correspond to the micro-services 1, python tools correspond to the micro-services 2, and the like, so that tool chain integration is realized;
a container mirror is constructed for a single tool module. The specific mode can include: and creating a yaml file (which can also be deployed in a K8s cluster) by using the constructed tool mirror image, wherein the kind attribute is specified in file parameters as the information of the Deployment, the specified copy number, the specified mirror image name and address and the like. And then performing multi-pod deployment by using a kubecect create command through a master node. And creating a xml file (which can also be deployed in a K8s cluster) which can be deployed in a K8s cluster, wherein a kid attribute is designated as a service in a file parameter, a Selector is designated as an app tag value designated in the last step, and information such as a NodePort port is designated. And constructing the service object by using a kubecect create command through a master node from the node 1 to the node n. And request access is carried out by utilizing the IP of the host machine and the Nodeport port of the service, so that load balance is realized. In the above manner, a service object mechanism in K8s can be utilized to realize load balancing among the multiple pods (pod 1 to pod), the processing cluster and the computation cluster are shared among the multiple pods, and the platform achieves the effect of data parallel processing according to the continuous Restful request.
The platform acquires mass data to be processed through selection. And loading the links of the data, performing grouping sequencing according to the input requirement of the tool, and using the data links after grouping sequencing as request parameters of a Restful interface (REST API). And the platform loads data into the pod through the Restful interface to process the data, and the platform loads the data according to the number of the pods and processes the data in parallel. The steps can select the data to be processed by utilizing the advantages of the data platform, carry out grouping and sequencing, realize the separation of storage and calculation and exert the advantages of the platform.
And after each data is processed, returning success or failure parameters, if the data is successful, uploading processing result data to a storage center, adding processing result metadata, and marking the processing success. If the processing structure fails, processing result metadata is added, and the processing structure is marked to fail. The processing state of the tool is monitored through the tool log, the processing state of each datum can be recorded through the log of the monitoring tool by using the platform, the effect of unsupervised automatic processing is achieved, and if all processing is completed, the pod is destroyed, and storage and computing resources are released.
It should be noted that each tool module develops and deploys its own tool service function according to the Restful architecture, and decoupling is realized in development. Through the deployment of multiple containers, the parallel processing of data can be realized, and the load balance can be automatically realized. The development language is not strictly required, and the development language can be fused into the platform as long as the Restful architecture is met. The Restful architecture is an architecture unrelated to a tool development platform, different platforms can realize operations such as function calling, data input, data post-processing and the like through Restful style interfaces, multi-container deployment is carried out on a tool through a K8s cluster, load balance is realized through service, parallel processing of data can be realized, computing power of the cluster is fully utilized, and complete decoupling of development among modules can be realized through Restful architecture design.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method of the embodiments of the present application.
Fig. 8 is a block diagram of a data processing apparatus according to an embodiment of the present application; as shown in fig. 8, includes:
an obtaining module 802, configured to obtain one or more target toolkits corresponding to a received data processing service from a plurality of heterogeneous data processing toolkits, where the data processing service is used to request processing of target data, and the target toolkit is used to indicate a data processing tool used to implement the data processing service;
a first creating module 804, configured to create an initial data processing container corresponding to each target toolkit, so as to obtain a target toolkit and an initial container set having a corresponding relationship;
an installing module 806, configured to install each target toolkit into a corresponding initial data processing container according to a target toolkit and an initial container set that have a correspondence relationship, so as to obtain one or more target container sets, where a data processing interface is deployed on each target container in the target container sets, and the data processing interface is used to convert a target format and a heterogeneous format corresponding to each target toolkit;
a scheduling module 808, configured to schedule one or more target container sets to process the target data through the data processing interface according to the data processing service.
According to the embodiment, firstly, corresponding one or more target toolkits are obtained from a plurality of heterogeneous data processing toolkits according to a received data processing service, the data processing service is used for requesting to process target data, the target toolkits are used for indicating data processing tools used for realizing the data processing service, then corresponding initial data processing containers are created according to the target toolkits to obtain target toolkits and initial container sets with corresponding relations, then, each target toolkit is installed into the corresponding initial data processing container according to the target toolkits and the initial container sets with corresponding relations to obtain one or more target container sets, a data processing interface is deployed on each target container in the target container sets and used for converting a target format and a heterogeneous format corresponding to each target toolkit, namely, mutual calling among heterogeneous tools can be realized through the data processing interfaces, then, one or more target container sets are scheduled through the data processing interfaces according to the data processing service to process target data, tool processing target data is conveniently called, and multiple containers are deployed on the target toolkits, and the target toolkits are increased in target processing capacity. By adopting the technical scheme, the problems of low data processing efficiency and the like in the related technology are solved, and the technical effect of improving the data processing efficiency is realized.
In an exemplary embodiment, the first creating module includes:
a first creating unit, configured to create a first configuration file corresponding to each target toolkit, where the first configuration file is used to indicate attribute information of the initial data processing container to be created for each target toolkit;
and the first execution unit is used for executing the first configuration file in a cluster through a cluster control command to obtain the initial data processing container corresponding to each target tool package.
In an exemplary embodiment, the first creating unit is further configured to:
and creating a first data serialization format yaml file corresponding to each target toolkit, wherein file parameters of the first data serialization format yaml file comprise a kind attribute of Deployment of the Deployment, the number of containers to be created, the name of the target toolkit and the address of the target toolkit.
In an exemplary embodiment, the installation module includes:
a second creating unit, configured to create a second configuration file corresponding to each of the initial data processing containers, where the second configuration file is used to indicate information of the target toolkit to be installed in the initial data processing container and interface information of the data processing interface;
and the second execution unit is used for executing the second configuration file in each initial data processing container through a cluster control command to obtain one or more target container sets.
In an exemplary embodiment, the second creating unit is further configured to:
and creating a second xml file corresponding to each initial data processing container, wherein file parameters of the second xml file include a kind attribute as a service, a Selector as the target toolkit corresponding to each initial data processing container, and interface information of the data processing interface.
In an exemplary embodiment, the second execution unit is further configured to:
installing the corresponding target toolkit in each initial data processing container in a service object constructing mode according to the kid attribute to obtain a reference container;
and deploying the data processing interface for the reference container according to the interface information of the data processing interface.
In an exemplary embodiment, the scheduling module includes:
the calling unit is used for calling each target container set in sequence according to the operation sequence of the data processing service to process data;
the acquisition unit is used for acquiring a data processing result returned by each target container set;
a sending unit, configured to send the data processing result to a next target container set in the operation order when the data processing result does not belong to a final processing result of the target data in the data processing service;
and the storage unit is used for storing the data processing result under the condition that the data processing result belongs to the final processing result of the target data in the data processing service.
In an exemplary embodiment, the invoking unit is further configured to:
creating a first call request in a target format, wherein the first call request carries a data link of data to be processed;
sending the first call request to the data processing interface on one target container in each target container set according to load balancing;
converting the first calling request from the target format into a corresponding second calling request in the heterogeneous format through the data processing interface;
acquiring the data link from the second calling request;
and loading the data to be processed into the target container for processing according to the data link.
In one exemplary embodiment, the apparatus further comprises:
a generating module, configured to generate a tool mirror image of each data processing tool in the multiple heterogeneous data processing tools before the one or more target tool packages corresponding to the received data processing service are obtained from the multiple heterogeneous data processing tool packages, so as to obtain multiple heterogeneous data processing tool packages;
and the second creating module is used for creating a corresponding micro-service module for each data processing tool package, wherein the micro-service module is used for calling each data processing tool package.
In one exemplary embodiment, the apparatus further comprises:
a destruction module, configured to destroy one or more target container sets after the target data is processed by scheduling one or more target container sets through the data processing interface according to the data processing service;
and the releasing module is used for releasing the storage resources and the calculation resources occupied by one or more target container sets.
Embodiments of the present application also provide a storage medium including a stored program, where the program performs any one of the methods described above when executed.
Alternatively, in the present embodiment, the storage medium may be configured to store program codes for performing the following steps:
s1, one or more target toolkits corresponding to the received data processing service are obtained from a plurality of heterogeneous data processing toolkits, wherein, the data processing service is used for requesting to process target data, and the target tool package is used for indicating a data processing tool used for realizing the data processing service;
s2, establishing an initial data processing container corresponding to each target tool package to obtain a target tool package and an initial container set which have corresponding relations;
s3, according to the target toolkits and the initial container sets with the corresponding relations, installing each target toolkit into a corresponding initial data processing container to obtain one or more target container sets, wherein each target container in the target container sets is provided with a data processing interface, and the data processing interfaces are used for converting a target format and a heterogeneous format corresponding to each target toolkit;
and S4, scheduling one or more target container sets to process the target data through the data processing interface according to the data processing service.
Embodiments of the present application further provide an electronic device comprising a memory having a computer program stored therein and a processor configured to execute the computer program to perform the steps of any of the above method embodiments.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
the method comprises the steps that S1, one or more target tool packages corresponding to received data processing services are obtained from a plurality of heterogeneous data processing tool packages, wherein the data processing services are used for requesting processing of target data, and the target tool packages are used for indicating data processing tools used for realizing the data processing services;
s2, establishing an initial data processing container corresponding to each target tool package to obtain a target tool package and an initial container set which have corresponding relations;
s3, according to the target toolkits and the initial container sets with the corresponding relations, installing each target toolkit into a corresponding initial data processing container to obtain one or more target container sets, wherein each target container in the target container sets is provided with a data processing interface, and the data processing interfaces are used for converting a target format and a heterogeneous format corresponding to each target toolkit;
and S4, scheduling one or more target container sets to process the target data through the data processing interface according to the data processing service.
Optionally, in this embodiment, the storage medium may include, but is not limited to: various media capable of storing program codes, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and this embodiment is not described herein again.
It will be apparent to those skilled in the art that the modules or steps of the present application described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a memory device and executed by a computing device, and in some cases, the steps shown or described may be executed out of order, or separately as integrated circuit modules, or multiple modules or steps thereof may be implemented as a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.
The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.
Claims (13)
1. A method for processing data, comprising:
acquiring one or more target toolkits corresponding to a received data processing service from a plurality of heterogeneous data processing toolkits, wherein the data processing service is used for requesting processing of target data, and the target toolkits are used for indicating data processing tools used for realizing the data processing service;
creating an initial data processing container corresponding to each target tool package to obtain a target tool package and an initial container set with corresponding relations;
according to the target toolkits and the initial container sets with corresponding relations, installing each target toolkit into a corresponding initial data processing container to obtain one or more target container sets, wherein each target container in the target container sets is provided with a data processing interface, and the data processing interfaces are used for converting a target format and a heterogeneous format corresponding to each target toolkit;
and scheduling one or more target container sets to process the target data through the data processing interface according to the data processing service.
2. The method of claim 1, wherein said creating an initial data processing container for each of said target toolkits comprises:
creating a first configuration file corresponding to each target toolkit, wherein the first configuration file is used for indicating attribute information of the initial data processing container to be created for each target toolkit;
and executing the first configuration file in a cluster through a cluster control command to obtain the initial data processing container corresponding to each target tool package.
3. The method of claim 2, wherein creating the first configuration file corresponding to each of the target toolkits comprises:
and creating a first data serialization format yaml file corresponding to each target toolkit, wherein file parameters of the first data serialization format yaml file comprise a kind attribute of Deployment of the Deployment, the number of containers to be created, the name of the target toolkit and the address of the target toolkit.
4. The method according to claim 1, wherein the installing each target tool package into a corresponding initial data processing container according to the target tool package and the initial container set having a corresponding relationship, to obtain one or more target container sets, comprises:
creating a second configuration file corresponding to each initial data processing container, wherein the second configuration file is used for indicating the information of the target toolkit to be installed in the initial data processing container and the interface information of the data processing interface;
and executing the second configuration file in each initial data processing container through a cluster control command to obtain one or more target container sets.
5. The method of claim 4, wherein creating a second configuration file for each of the initial data processing containers comprises:
and creating a second xml file corresponding to each initial data processing container, wherein file parameters of the second xml file include a kind attribute as a service, a Selector as the target toolkit corresponding to each initial data processing container, and interface information of the data processing interface.
6. The method of claim 5, wherein executing said second configuration file in each of said initial data processing containers via a cluster control command comprises:
installing the corresponding target toolkit in each initial data processing container in a service object constructing mode according to the kid attribute to obtain a reference container;
and deploying the data processing interface for the reference container according to the interface information of the data processing interface.
7. The method of claim 1, wherein said scheduling one or more sets of the target containers to process the target data via the data processing interface in accordance with the data processing service comprises:
sequentially calling each target container set according to the operation sequence of the data processing service to process data;
acquiring a data processing result returned by each target container set;
under the condition that the data processing result does not belong to the final processing result of the target data in the data processing service, sending the data processing result to the next target container set in the operation sequence;
and storing the data processing result under the condition that the data processing result belongs to the final processing result of the target data in the data processing service.
8. The method of claim 7, wherein said invoking data processing in each of said target container sets comprises:
creating a first call request in a target format, wherein the first call request carries a data link of data to be processed;
sending the first call request to the data processing interface on one target container in each target container set according to load balancing;
converting the first calling request from the target format into a corresponding second calling request in the heterogeneous format through the data processing interface;
acquiring the data link from the second calling request;
and loading the data to be processed into the target container for processing according to the data link.
9. The method according to any one of claims 1 to 8, wherein before the obtaining one or more destination toolkits corresponding to the received data processing services from the plurality of heterogeneous data processing toolkits, the method further comprises:
generating a tool mirror image of each data processing tool in a plurality of heterogeneous data processing tools to obtain a plurality of heterogeneous data processing tool packages;
and creating a corresponding micro-service module for each data processing toolkit, wherein the micro-service module is used for calling each data processing toolkit.
10. The method according to any of claims 1 to 8, wherein after said scheduling one or more sets of said target containers to process said target data through said data processing interface in accordance with said data processing service, said method further comprises:
destroying one or more of the target container sets;
and releasing the storage resources and the computing resources occupied by one or more target container sets.
11. An apparatus for processing data, comprising:
an obtaining module, configured to obtain one or more target toolkits corresponding to a received data processing service from a plurality of heterogeneous data processing toolkits, where the data processing service is used to request processing of target data, and the target toolkit is used to indicate a data processing tool used to implement the data processing service;
the first creating module is used for creating an initial data processing container corresponding to each target tool package to obtain a target tool package and an initial container set which have a corresponding relation;
the installation module is used for installing each target tool package into a corresponding initial data processing container according to the target tool package and the initial container set which have corresponding relations, so as to obtain one or more target container sets, wherein each target container in the target container sets is provided with a data processing interface, and the data processing interfaces are used for converting a target format and a heterogeneous format corresponding to each target tool package;
and the scheduling module is used for scheduling one or more target container sets to process the target data through the data processing interface according to the data processing service.
12. A computer-readable storage medium, comprising a stored program, wherein the program when executed performs the method of any of claims 1 to 10.
13. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 10 by means of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211452629.0A CN115495225B (en) | 2022-11-21 | 2022-11-21 | Data processing method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211452629.0A CN115495225B (en) | 2022-11-21 | 2022-11-21 | Data processing method and device, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115495225A true CN115495225A (en) | 2022-12-20 |
| CN115495225B CN115495225B (en) | 2023-03-10 |
Family
ID=85116244
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211452629.0A Active CN115495225B (en) | 2022-11-21 | 2022-11-21 | Data processing method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115495225B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103970795A (en) * | 2013-01-31 | 2014-08-06 | 杭州勒卡斯广告策划有限公司 | Data processing method, device and system |
| CN104980468A (en) * | 2014-04-09 | 2015-10-14 | 深圳市腾讯计算机系统有限公司 | Method, device and system for processing service request |
| CN108921913A (en) * | 2018-06-29 | 2018-11-30 | 上海联影医疗科技有限公司 | The system and method for image reconstruction |
| US20190004725A1 (en) * | 2017-06-28 | 2019-01-03 | International Business Machines Corporation | Managing data container instances in a dispersed storage network |
| CN110008042A (en) * | 2019-03-28 | 2019-07-12 | 北京易华录信息技术股份有限公司 | A kind of algorithm Cascading Methods and system based on container |
| CN111488223A (en) * | 2020-03-24 | 2020-08-04 | 平安国际智慧城市科技股份有限公司 | Container-based data processing method, device, equipment and storage medium |
| CN112925759A (en) * | 2021-03-31 | 2021-06-08 | 北京金山云网络技术有限公司 | Data file processing method and device, storage medium and electronic device |
| CN113407522A (en) * | 2021-06-18 | 2021-09-17 | 上海市第十人民医院 | Data processing method and device, computer equipment and computer readable storage medium |
| CN114489957A (en) * | 2022-04-01 | 2022-05-13 | 国家卫星海洋应用中心 | Remote sensing satellite data processing method and device and electronic equipment |
| CN114510297A (en) * | 2022-03-31 | 2022-05-17 | 国家卫星海洋应用中心 | Satellite data reprocessing method and device and electronic equipment |
| CN114756555A (en) * | 2022-06-14 | 2022-07-15 | 浙江华东工程数字技术有限公司 | Multi-source heterogeneous three-dimensional model data processing method and system |
-
2022
- 2022-11-21 CN CN202211452629.0A patent/CN115495225B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103970795A (en) * | 2013-01-31 | 2014-08-06 | 杭州勒卡斯广告策划有限公司 | Data processing method, device and system |
| CN104980468A (en) * | 2014-04-09 | 2015-10-14 | 深圳市腾讯计算机系统有限公司 | Method, device and system for processing service request |
| US20190004725A1 (en) * | 2017-06-28 | 2019-01-03 | International Business Machines Corporation | Managing data container instances in a dispersed storage network |
| CN108921913A (en) * | 2018-06-29 | 2018-11-30 | 上海联影医疗科技有限公司 | The system and method for image reconstruction |
| CN110008042A (en) * | 2019-03-28 | 2019-07-12 | 北京易华录信息技术股份有限公司 | A kind of algorithm Cascading Methods and system based on container |
| CN111488223A (en) * | 2020-03-24 | 2020-08-04 | 平安国际智慧城市科技股份有限公司 | Container-based data processing method, device, equipment and storage medium |
| CN112925759A (en) * | 2021-03-31 | 2021-06-08 | 北京金山云网络技术有限公司 | Data file processing method and device, storage medium and electronic device |
| CN113407522A (en) * | 2021-06-18 | 2021-09-17 | 上海市第十人民医院 | Data processing method and device, computer equipment and computer readable storage medium |
| CN114510297A (en) * | 2022-03-31 | 2022-05-17 | 国家卫星海洋应用中心 | Satellite data reprocessing method and device and electronic equipment |
| CN114489957A (en) * | 2022-04-01 | 2022-05-13 | 国家卫星海洋应用中心 | Remote sensing satellite data processing method and device and electronic equipment |
| CN114756555A (en) * | 2022-06-14 | 2022-07-15 | 浙江华东工程数字技术有限公司 | Multi-source heterogeneous three-dimensional model data processing method and system |
Non-Patent Citations (3)
| Title |
|---|
| 刘立潮;: "数据质量管控智能化在运营支撑系统中的价值和实现" * |
| 赵智韬等: "基于容器云技术的典型遥感智能解译算法集成", 《大数据》 * |
| 金九平等: "基于kubernetes的海洋遥感数据产品服务平台设计与实现", 《海洋信息》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115495225B (en) | 2023-03-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115328663B (en) | Method, device, equipment and storage medium for scheduling resources based on PaaS platform | |
| CN107145380B (en) | Virtual resource arranging method and device | |
| CN108279892B (en) | Method, device and equipment for splitting large-scale application service into micro-service | |
| CN108959385B (en) | Database deployment method, device, computer equipment and storage medium | |
| CN109684054A (en) | Information processing method and device, electronic equipment and memory | |
| CN106371889B (en) | Method and device for realizing high-performance cluster system of scheduling mirror image | |
| CN112416353A (en) | Channel package packaging method and device and computer equipment | |
| CN111897539A (en) | Method and device for deploying applications according to service roles | |
| CN105045602A (en) | Method and device for constructing Hadoop application development framework and electronic device | |
| CN111552838A (en) | Data processing method and device, computer equipment and storage medium | |
| CN114168302A (en) | Task scheduling method, device, equipment and storage medium | |
| CN113867600A (en) | Development method and device for processing streaming data and computer equipment | |
| CN113918232B (en) | Algorithm service calling method, device, server and storage medium | |
| CN112882794B (en) | pod capacity expansion method, device, node and storage medium | |
| CN113467931B (en) | Processing method, device and system of calculation task | |
| CN111258742A (en) | Data synchronization method, system, computing device and storage medium | |
| CN109343970B (en) | Application program-based operation method and device, electronic equipment and computer medium | |
| CN112261125B (en) | Centralized unit cloud deployment method, device and system | |
| CN113434146A (en) | Code compiling method, device, storage medium and electronic device | |
| CN115495225B (en) | Data processing method and device, storage medium and electronic device | |
| CN111651169B (en) | Block chain intelligent contract operation method and system based on web container | |
| CN117435324A (en) | Task scheduling method based on containerization | |
| CN112202879A (en) | Middleware management method and device, electronic equipment and storage medium | |
| CN114546648A (en) | Task processing method and task processing platform | |
| CN115543491A (en) | Microservice processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |