CN107948254B - Big data processing framework arrangement system and method of hybrid cloud platform - Google Patents

Big data processing framework arrangement system and method of hybrid cloud platform Download PDF

Info

Publication number
CN107948254B
CN107948254B CN201711107600.8A CN201711107600A CN107948254B CN 107948254 B CN107948254 B CN 107948254B CN 201711107600 A CN201711107600 A CN 201711107600A CN 107948254 B CN107948254 B CN 107948254B
Authority
CN
China
Prior art keywords
module
data
data processing
big data
hybrid cloud
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201711107600.8A
Other languages
Chinese (zh)
Other versions
CN107948254A (en
Inventor
陈琪
陈劭力
王巍
谢朝阳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Eccom Network System Co ltd
Original Assignee
Eccom Network System Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Eccom Network System Co ltd filed Critical Eccom Network System Co ltd
Priority to CN201711107600.8A priority Critical patent/CN107948254B/en
Publication of CN107948254A publication Critical patent/CN107948254A/en
Application granted granted Critical
Publication of CN107948254B publication Critical patent/CN107948254B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/22—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/561—Adding application-functional data or data for application control, e.g. adding metadata

Abstract

The invention discloses a big data processing framework arrangement system and a method of a hybrid cloud platform, wherein the system comprises: the system comprises an arrangement service module, a hybrid cloud management module, an infrastructure resource module, a data analysis module, a data processing module, a data storage module and a data integration module, wherein the arrangement service module mainly comprises a visual interface module, a service directory module, a flow executor, a resource optimization module and the like. The invention can flexibly schedule the hybrid cloud management platform, supports a user to complete the arrangement and execution work of the whole frame and the subdivided tasks of big data processing through various visual operation forms such as a mobile terminal interface and the like, and the frame needs to meet the fast iteration requirement of the big data processing field of the user.

Description

Big data processing framework arrangement system and method of hybrid cloud platform
Technical Field
The invention relates to the technical field of big data processing, in particular to a big data processing framework arrangement system and method of a hybrid cloud platform.
Background
Currently, cloud platforms are widely deployed in various industries, and from the industry perspective, various industries build their own industry solution departments based on cloud computing technology. On the other hand, in combination with the technical development trend, the current trend of the big data platform infrastructure is modularization and cloud platform, and the cloud platform not only provides scheduling of physical bottom layer resources, but also needs and can provide various middleware PaaS services related to big data, and various services can be simply and flexibly combined to meet customization requirements of various application scenes.
Therefore, the whole scheme of the big data processing and the related flow framework at the present stage are combed, the key influence factors and practical experience in the present big data processing process are considered, the technical challenge of arranging the big data processing system on the hybrid cloud platform is settled, and the problem needs to be solved.
The hybrid cloud platform has flexible resource definition, and needs to cover various resource types in the big data processing process.
Different and targeted support of resources such as calculation, storage, network and the like should be provided in each link of big data processing, but no method for ensuring high availability and stability exists at present.
The whole process of big data processing needs to respond to user requirements more quickly, is used for realizing flow arrangement and realizing the transparentization and the servitization of big data processing facing to terminal users.
In summary, the conventional big data processing flow and the platform construction scheme thereof have been unable to satisfy the fast response requirement of the industry client in the big data analysis and processing, and the objective condition of the fast changing environment necessary for adapting to the actual data processing condition. There are many disadvantages and improvements are needed.
Disclosure of Invention
The invention aims to provide a big data processing framework arrangement system and a big data processing framework arrangement method of a hybrid cloud platform, so as to realize the flexibly-scheduled hybrid cloud management platform which is used for accessing various cloud computing resources of management customers and comprises a private cloud and a public cloud; the method can realize rapid definition and configuration of all managed resources, support different levels of services in the process of big data processing, provide resources required by meeting specific processing links, and meet the optimal resource utility of the cloud platform; through a management platform of a hybrid cloud, dynamic response to big data processing load adjustment can be realized, and a big data workload migration task across cloud stacks is completed; the method supports a user to complete the arrangement and execution work of the whole frame and the subdivision tasks of big data processing through various visual operation modes such as a mobile terminal interface and the like, and the frame needs to meet the fast iteration requirement of the big data processing field of the user.
The invention solves the technical problems through the following technical scheme: a big data processing framework arrangement system of a hybrid cloud platform is characterized by comprising:
the arrangement service module is used for calling a hybrid cloud management platform according to an input big data processing flow configuration strategy and applying for managing resources such as corresponding calculation, network and storage in a cloud stack;
the mixed cloud management module is connected with the arrangement service module and is used for completing scheduling distribution, authority management, migration and monitoring of various cloud resources according to a scheduling strategy required by a specific big data processing link and completing efficiency optimization of a big data overall processing process;
the infrastructure resource module is connected with the hybrid cloud management module and is used for collecting and storing public cloud, computing resources, storage resources and network resources so as to enable the hybrid cloud management module to complete management;
the data analysis module is connected with the arrangement service module and used for carrying out deep processing and analysis on data, carrying out multi-dimensional retrieval and analysis on stored information, communicating with each module of the system to obtain multi-dimensional monitoring information, comprehensively evaluating the operation efficiency of the whole system and finally displaying an analysis result and a prediction trend on a mobile terminal interface;
the data processing module is connected with the data analysis module, is used for the addition, deletion, reconstruction and background management of a programming model, is responsible for batch-oriented workload, schedules a specified batch processing engine to process the workload, is responsible for stream-oriented workload and schedules a specified stream processing engine to process the workload;
the data storage module is connected with the data processing module, is used for processing the data confidence in the loading cache, imports a corresponding subsequent module, is responsible for processing and preprocessing metadata transmitted from a lower layer, realizes various operations on database resources by calling relevant APIs (application programming interfaces) of the database, and completes the storage flow operation of data completion by cooperating with cache data management and metadata management;
and the data integration module is connected with the data storage module and is used for carrying out multi-dimensional screening on the data to be processed output by the lower-layer mixed cloud resource, transmitting the data to the data storage module, gathering and sorting various real-time messages in the data collection process, recording various error log alarm information, recording the data in a file form, and carrying out file type operations such as storage optimization on the stored file data.
Preferably, the orchestration service module mainly includes:
the visual interface module is used for displaying the whole flow of the big data processing framework, and information retrieval, data mining results and efficiency evaluation information which can be displayed;
the service directory module is used for packaging each link of the big data processing framework facing the end user, displaying a big data processing flow which can be directly called for the user, and scheduling the data processing module, the data storage module, the data integration module and the like in the system to complete a packaging task;
the flow executor is responsible for decomposing and executing a specific big data processing flow, and dynamically adjusts the workload of each cloud stack on the hybrid cloud platform through the communication with the resource optimization module to achieve the purpose of resource optimization;
and the resource optimization module is mainly responsible for cooperative work of the resource allocation modules of the south and hybrid cloud management subsystems and real-time allocation and specific scheduling of various computing, storage and network resources managed by the hybrid cloud platform.
Preferably, the data analysis module comprises:
the data mining module is responsible for deeply processing and analyzing data;
the information retrieval module is responsible for carrying out multi-dimensional retrieval on stored information;
and the efficiency evaluation module is communicated with each subsystem of the system to obtain multi-dimensional monitoring information, comprehensively evaluate the operation efficiency of the whole system and finally display the analysis result and the prediction trend on a visual interface.
Preferably, the data processing module includes:
the programming model management module is responsible for adding, deleting, reconstructing and background management of the programming model;
the batch processing module is responsible for facing batch processing workload and scheduling a specified batch processing engine to process the workload;
and the stream processing module is responsible for processing the workload in a stream-oriented mode and scheduling the specified stream processing engine to process the workload.
Preferably, the data storage module comprises:
the cache data management module is responsible for processing the data confidence in the loading cache and importing the data confidence into the corresponding follow-up module;
the metadata management module is used for processing and preprocessing metadata transmitted from a lower layer;
and the database management module is used for realizing various operations on database resources by calling the relevant API of the database, and finishing the storage flow operation finished by the data by cooperating with the cache data management module and the metadata management module.
Preferably, the data integration module includes:
the data synchronization module is responsible for performing multi-dimensional screening on the data to be processed output by the lower layer mixed cloud resource and transmitting the data to the data storage module;
the message transmission module is responsible for summarizing and sorting various real-time messages in the data collection process and recording various error log alarm information;
the file storage module is mainly responsible for recording data in a file form and can perform file type operations such as storage optimization on the stored file data.
Preferably, the hybrid cloud management module includes:
the resource allocation module is used for receiving a data calling instruction of the data integration module on the cloud resources and operating the managed hybrid cloud resources in combination with an optimization instruction of the hybrid cloud output by the resource optimization module; in the execution process, the verification of the operation authority with the authority management module is needed, and the specific lowering configuration of the operation command is carried out through a north API developed by cloud resources
The resource migration module is used for receiving an operation command of the resource allocation module on the cloud resources and is responsible for migration work of specific data processing workloads among the cloud stacks; when a specific execution command is too heavy, the operation of each step needs to be verified through the authority management module;
the authority management module is responsible for authority authentication of various operations of the mixed element management subsystem, and authorizes the resource distribution module, the resource migration module and the resource monitoring module to execute related operations through authority division of fine granularity;
and the resource monitoring module is responsible for monitoring the monitoring log information of each managed cloud stack in the hybrid cloud operation process, performing unified formatting processing, receiving the command scheduling of the authority management module, and making a corresponding cloud stack operation instruction in real time when the triggering condition of the monitoring threshold is met to complete the stable operation of the monitoring resources.
The invention also provides a big data processing framework arrangement method of the hybrid cloud platform, which is characterized by comprising the following steps:
firstly, a user operates the system through a visual interface module, inputs various links and configurations of a big data processing process to be deployed and executed, and the system inputs related applications to a background;
after the service directory module responds to the application, whether the stored service list and the corresponding big data processing flow list contain the applied processing flow or deployment template is searched;
step three, judging whether the system has a big data arrangement flow, if so, turning to step five, otherwise, turning to step four;
step four, the system requires the user to add corresponding big data processing flow and parameter configuration; continuing to execute the second step after the task is completed;
step five, determining a big data processing arrangement flow, and calling a big data flow executor by a system to complete scheme integration of configuration information such as configuration parameters, deployment flows, initialization parameters and the like required by the big data processing flow applied by a user; the step is also responsible for applying for corresponding resources to the governed hybrid cloud platform, and the step ten is executed;
step six, the data integration module receives the instruction, determines a specific operation strategy of data integration, and completes the process of extracting data from the hybrid cloud stack;
step seven, the data storage module receives the instruction, determines the specific operation strategy of data storage, and respectively calls the metadata management module and the database management module to finish the data storage operation according to different data types;
step eight, the data processing module receives the instruction, determines the specific operation strategy of data processing, and respectively calls the batch processing module and the stream processing module according to the execution condition output by the programming model management module;
step nine, the data analysis module receives the output result of the lower programming model management module, sends configuration execution parameters to the data mining module, the information retrieval module and the efficiency evaluation module, and determines a data analysis mode;
step ten, the resource allocation module executes resource allocation operation of cloud resources governed by the hybrid cloud to complete application of computing, network and storage resources related to the big data processing process;
eleventh, finishing abstract operation of data from a lower-layer module by a data synchronization module, a message transmission module and a file storage module in the data integration module according to different selected big data to-be-processed data sets;
step twelve, judging whether the execution data storage object is metadata, if so, turning to step thirteen, otherwise, turning to step fourteen;
step thirteen, if the data to be processed is metadata, the system calls a metadata management module to complete data storage;
step fourteen, if the data to be processed is relational data, the system calls a database management module to complete data storage;
step fifteen, the cache data management module receives lower layer processing data, reads the stored data into a cache resource pool, and performs screening processing optimization operation on the cache data set;
sixthly, loading a data processing model adapted to the required big data processing process through a programming model management module, judging whether the programming model is a batch processing model according to different processing scenes, if so, turning to the seventeenth step, otherwise, turning to the eighteenth step;
seventhly, if the big data processing scene to be processed is judged by the system to be suitable for adopting the batch processing mode, the system calls a batch processing module loading model to process the data
Eighteen, if the big data processing scene to be processed is judged by the system to be suitable for adopting a stream processing mode, the system calls a batch processing module loading model to process the data;
nineteenth, the data analysis module receives the data processing result and integrates and sends the result to the data mining module for operation; twenty one is converted;
twenty, the data mining module receives the integrated data confidence and completes deep analysis processing on information contained in the data in cooperation with the information retrieval module;
twenty-one, analyzing the operation efficiency of each link in the big data processing process by an efficiency evaluation module, and submitting an evaluation result to a system monitoring platform;
twenty-two, the hybrid cloud management module receives efficiency evaluation information of each link of the efficiency evaluation module for big data processing, and monitors and manages the operation condition of the whole system platform by combining the monitoring log output by the nano-management cloud resource;
twenty-third, the system keeps monitoring whether the operation efficiency of the managed hybrid cloud resources reaches the predefined triggering condition and alarm threshold of the resource optimization module, judges whether the predefined triggering condition is reached, if yes, turns to twenty-fourth, otherwise turns to twenty-twelve;
twenty-four steps, starting a big data workload migration process, and calling a resource allocation module and a resource migration module in the hybrid cloud management module through a resource optimization module to complete parameter configuration of the relevant big data processing workload migration process; the concrete process is transferred to the step four process.
The positive progress effects of the invention are as follows: the invention dynamically displays the working load state of big data processing on the system level through the real-time state and the working logic of each link of the real-time big data processing of the visual management system. The user can quickly and automatically arrange and combine the big data processing workflow through the service arrangement function provided by the system, and can customize each link of big data processing according to needs and quickly configure each sub-process parameter through a visual interface. The system provides the service capability of rapid deployment, debugging and execution of large data processing for the outside. According to the invention, through a hybrid cloud management subsystem of the system, according to a scheduling strategy required by a specific big data processing link, scheduling distribution of various cloud resources managed by a lower layer is completed, efficiency optimization of a big data overall processing process is completed, and optimal resource utility of a big data workload based on a hybrid cloud platform is met. According to the method and the device, the resource use condition of the big data workload in each cloud stack of the hybrid cloud is monitored, the migration process of the corresponding big data workload can be executed according to the pre-configuration strategy and manual triggering of a user, high-efficiency execution of the whole workflow of the big data is realized, and the resource migration condition is displayed in a multi-dimensional mode in a visual mode. And monitoring index parameters such as availability, migration progress, overall platform health state and the like of the application in the whole process.
Drawings
Fig. 1 is a schematic composition diagram of an embodiment of a big data processing framework orchestration system of a hybrid cloud platform according to the present invention.
FIG. 2 is a schematic diagram of the organization service module according to the present invention.
Fig. 3 is a schematic workflow diagram of a big data processing framework arrangement method of a hybrid cloud platform according to the present invention.
Detailed Description
The following provides a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings.
Fig. 1 is a schematic composition diagram of an embodiment of a big data processing framework arrangement system of a hybrid cloud platform according to the present invention. As shown in fig. 1, this embodiment mainly includes an orchestration service module 110, a hybrid cloud management module 120, an infrastructure resource module 130, a data analysis module 140, a data processing module 150, a data storage module 160, and a data integration module 170, where:
the arrangement service module 110 is used for calling a hybrid cloud management platform according to an input big data processing flow configuration strategy and applying for managing resources such as corresponding calculation, network and storage in a cloud stack;
the hybrid cloud management module 120 is connected to the orchestration service module 110, and is configured to complete scheduling allocation, authority management, migration, and monitoring of various cloud resources according to a scheduling policy required for a specific big data processing link, and complete efficiency optimization of a big data overall processing process;
an infrastructure resource module 130, connected to the hybrid cloud management module 120, for collecting and storing public cloud, computing resources, storage resources, and network resources for the hybrid cloud management module 120 to complete management;
the data analysis module 140 is connected with the arrangement service module 110, and is used for deeply processing and analyzing data, performing multi-dimensional retrieval and analysis on stored information, obtaining multi-dimensional monitoring information through communication with each module of the system, comprehensively evaluating the operation efficiency of the whole system, and finally displaying an analysis result and a prediction trend through a mobile terminal interface;
a data processing module 150, connected to the data analysis module 140, configured to add, delete, modify, and manage a background of the programming model, and is responsible for batch-oriented workload, scheduling a specified batch engine to process the workload, scheduling a stream-oriented workload, and scheduling a specified stream engine to process the workload;
a data storage module 160 connected to the data processing module 150, configured to process the data confidence in the load cache, and import the data confidence into a corresponding subsequent module, which is responsible for processing and preprocessing metadata transmitted from a lower layer, and implement various operations on database resources by calling a database-related API (Application Programming Interface), and cooperate with cache data management and metadata management to complete a storage flow operation completed by the data;
the data integration module 170 is connected to the data storage module 160, and is configured to perform multidimensional screening on data to be processed output by the lower layer hybrid cloud resource, transmit the data to the data storage module 160, collect and sort various real-time messages in the data collection process, record various error log alarm information, record data in a file form, and perform file type operations such as storage optimization on stored file data.
As shown in fig. 2, the orchestration service module 110 mainly includes:
the visualization interface module 211 is used for displaying the whole flow of the big data processing framework, and information retrieval, data mining results and efficiency evaluation information which can be displayed; the visual interface module provides an access and operation interface of the whole system. Compared with a common access interface, the difference is that an operator can complete the operations of configuration, deployment, arrangement, migration and the like of the application of each level on the large data processing framework arrangement system and the governed hybrid cloud platform through the interface, and the input mode is a method that a graphical interface is matched with mouse gesture motion as a main mode and manual input of individual parameters is assisted. In the running process of the system, the system continuously receives monitoring indexes and alarm logs which are output by the efficiency evaluation module and relate to various resources in the big data processing process, the monitoring indexes and the alarm logs are used as input of visual display, and the integrated data are output to the software service directory
The service directory module 212 is used for packaging each link of the big data processing framework facing the end user, displaying a big data processing flow which can be directly called for the user, and scheduling the data processing module, the data storage module, the data integration module and the like in the system to complete a packaging task; the service directory module is used for managing and configuring various service processes of big data processing which can be provided by the system externally, wherein the service specifically refers to a big data processing capability facing to a client formed by the cooperative operation of a relevant big data processing process set. In the specific operation process, the operator control of the visual interface module is used as input, the configuration items of the background big data processing process are called, and a series of confirmation of the configuration items of the big data processing module is executed through the flow executor. And the resource optimization module calls a resource allocation module of the lower hybrid cloud management layer subsystem to allocate and adjust corresponding cloud computing resources.
The flow executor 213 is responsible for decomposing and executing a specific big data processing flow, and dynamically adjusts the workload of each cloud stack on the hybrid cloud platform through communication with the resource optimization module to achieve the purpose of resource optimization; the flow executor is mainly used for determining the related configuration of a specific process group executed by a specific big data processing task and the like according to the action gesture input of a visual interface and the input of a software service directory module and the type and scale of the big data processing task to be executed, integrating the data into a script containing parameters required by the whole big data processing deployment scheme, and outputting the script to modules for data analysis, data processing, data storage, data integration and the like.
And the resource optimization module 214 is mainly responsible for cooperative work of the resource allocation modules in the south direction and the hybrid cloud management subsystem, and real-time allocation and specific scheduling of various computing, storage and network resources managed by the hybrid cloud platform. The resource optimization module receives the transmission parameters of the process executor, such as resource types, quantity, initialization parameters, security strategies and the like, and executes various cloud resources required by each link of scheduling big data processing through the resource allocation module. The method specifically operates resource deployment, configuration and initialization tasks on each cloud stack of the hybrid cloud platform, and is responsible for verifying the completed results.
The data analysis module 140 includes:
the data mining module is responsible for deeply processing and analyzing data; the data mining module comprehensively analyzes and processes the data processed by the generation by receiving the service directory which is the arranging requirement for the data processing, and outputs decision information which can be used by the terminal user by combining the screening information which is collected by the information retrieval module. Meanwhile, the efficiency evaluation module outputs log information in the execution process for analyzing the efficiency utilization level in the processing process.
The information retrieval module is responsible for carrying out multi-dimensional retrieval on stored information; the information retrieval module receives the information collection requirement provided by the service directory module, integrally searches and screens the existing information, and feeds the obtained result back to the data mining module;
and the efficiency evaluation module is communicated with each subsystem of the system to obtain multi-dimensional monitoring information, comprehensively evaluate the operation efficiency of the whole system and finally display the analysis result and the prediction trend on a visual interface. The efficiency evaluation module receives log information generated by the data mining module in the data processing process, is used for combining an efficiency evaluation model and a related algorithm of the efficiency evaluation module to carry out iterative calculation, analyzes and obtains efficiency information of each link in the whole big data processing process, and displays the efficiency information on the visual interface module in a visual and multi-dimensional mode;
the data processing module 150 includes:
the programming model management module is responsible for adding, deleting, reconstructing and background management of the programming model; and the programming model management module manages all the programming models in the big data processing subsystem, including the operations of adding, deleting, modifying and checking the models. In the specific data processing process, receiving cache data provided by a cache data management module of a lower data storage subsystem, and executing and calling a managed batch processing module and a managed stream processing module according to different data types;
the batch processing module is responsible for facing batch processing workload and scheduling a specified batch processing engine to process the workload; and receiving an instruction of the programming model management module according to different types of data to be processed, and processing the data set operation suitable for distributed processing.
And the stream processing module is responsible for processing the workload in a stream-oriented mode and scheduling the specified stream processing engine to process the workload. And the stream processing module receives an instruction of a programming model management C1 according to different types of data to be processed and is responsible for processing data set operation with large data magnitude.
The data storage module 160 includes:
the cache data management module is responsible for processing the data confidence in the loading cache and importing the data confidence into the corresponding follow-up module; the cache data management module is responsible for processing cache data in the computing resources, can receive metadata output by the metadata management module and relational data output by the database management module, selects a corresponding data optimization read-write mode according to different data storage types, and outputs a final result to a programming model management module in an upper system;
the metadata management module is used for processing and preprocessing metadata transmitted from a lower layer; the metadata management module is responsible for abstracting lower-layer original data into metadata for data storage, executing functional operations such as adding, deleting, modifying and checking the metadata and the like, and transmitting output results to cache data;
and the database management module is used for realizing various operations on database resources by calling the relevant API of the database, and finishing the storage flow operation finished by the data by cooperating with the cache data management module and the metadata management module. The database management module is responsible for managing the relational data in the data storage layer, executing operations including data adding, deleting, modifying, checking and the like, and receiving the data output by the file storage module and the data synchronization module of the lower-layer data integration subsystem.
The data integration module 170 includes:
the data synchronization module is responsible for performing multi-dimensional screening on the data to be processed output by the lower layer mixed cloud resource and transmitting the data to the data storage module; the data synchronization module receives the stored data output by the resource allocation module in the lower hybrid cloud management subsystem, integrates the data, sends the integrated data to the upper metadata management module, records the log in operation, and outputs the log to message transmission;
the message transmission module is responsible for summarizing and sorting various real-time messages in the data collection process and recording various error log alarm information; the method comprises the steps of recording log information of a resource allocation module on a hybrid cloud subsystem in the process of operating cloud resources, and receiving logs of a data synchronization module in the process of executing. And storing it in a file store;
the file storage module is mainly responsible for recording data in a file form and can perform file type operations such as storage optimization on the stored file data. The file storage module is used for storing and managing the related log information and other data in a file form in the data integration processing process.
The hybrid cloud management module 120 includes:
and the resource allocation module is used for receiving a data calling instruction of the data integration module on the cloud resources and operating the managed hybrid cloud resources in combination with the optimization instruction of the hybrid cloud output by the resource optimization module. In the execution process, the verification of the operation authority is required to be carried out with the authority management module, and the specific lowering configuration of the operation command is carried out through a northbound API developed by cloud resources;
and the resource migration module is used for receiving the operation command of the resource allocation module on the cloud resources and is responsible for the migration work of the specific data processing workload among the cloud stacks. When a specific execution command is too heavy, the operation of each step needs to be verified through the authority management module;
the authority management module is responsible for authority authentication of various operations of the mixed element management subsystem, and authorizes the resource distribution module, the resource migration module and the resource monitoring module to execute related operations through authority division of fine granularity;
and the resource monitoring module is responsible for monitoring the monitoring log information of each managed cloud stack in the hybrid cloud operation process, performing unified formatting processing, receiving the command scheduling of the authority management module, and making a corresponding cloud stack operation instruction in real time when the triggering condition of the monitoring threshold is met to complete the stable operation of the monitoring resources.
Fig. 3 is a schematic workflow diagram of a big data processing framework arrangement method of a hybrid cloud platform in an embodiment of the present invention in actual application. As shown in fig. 3, the actual workflow of the big data processing framework orchestration method for the hybrid cloud platform mainly includes the following steps:
firstly, a user operates the system through a visual interface module, inputs various links and configurations of a big data processing process to be deployed and executed, and the system inputs related applications to a background;
after the service directory module responds to the application, whether the stored service list and the corresponding big data processing flow list contain the applied processing flow or deployment template is searched;
step three, judging whether the system has a big data arrangement flow, if so, turning to step five, otherwise, turning to step four;
step four, the system requires the user to add corresponding big data processing flow and parameter configuration; continuing to execute the second step after the task is completed;
step five, determining a big data processing arrangement flow, and calling a big data flow executor by a system to complete scheme integration of configuration information such as configuration parameters, deployment flows, initialization parameters and the like required by the big data processing flow applied by a user; the step is also responsible for applying for corresponding resources to the governed hybrid cloud platform, and the step ten is executed;
step six, the data integration module receives the instruction, determines a specific operation strategy of data integration, and completes the process of extracting data from the hybrid cloud stack;
step seven, the data storage module receives the instruction, determines the specific operation strategy of data storage, and respectively calls the metadata management module and the database management module to finish the data storage operation according to different data types;
step eight, the data processing module receives the instruction, determines the specific operation strategy of data processing, and respectively calls the batch processing module and the stream processing module according to the execution condition output by the programming model management module;
step nine, the data analysis module receives the output result of the lower programming model management module, sends configuration execution parameters to the data mining module, the information retrieval module and the efficiency evaluation module, and determines a data analysis mode;
step ten, the resource allocation module executes resource allocation operation of cloud resources governed by the hybrid cloud to complete application of computing, network and storage resources related to the big data processing process;
eleventh, finishing abstract operation of data from a lower-layer module by a data synchronization module, a message transmission module and a file storage module in the data integration module according to different selected big data to-be-processed data sets;
step twelve, judging whether the execution data storage object is metadata, if so, turning to step thirteen, otherwise, turning to step fourteen;
step thirteen, if the data to be processed is metadata, the system calls a metadata management module to complete data storage;
step fourteen, if the data to be processed is relational data, the system calls a database management module to complete data storage;
step fifteen, the cache data management module receives lower layer processing data, reads the stored data into a cache resource pool, and performs screening processing optimization operation on the cache data set;
sixthly, loading a data processing model adapted to the required big data processing process through a programming model management module, judging whether the programming model is a batch processing model according to different processing scenes, if so, turning to the seventeenth step, otherwise, turning to the eighteenth step;
seventhly, if the big data processing scene to be processed is judged by the system to be suitable for adopting the batch processing mode, the system calls a batch processing module loading model to process the data
Eighteen, if the big data processing scene to be processed is judged by the system to be suitable for adopting a stream processing mode, the system calls a batch processing module loading model to process the data;
nineteenth, the data analysis module receives the data processing result and integrates and sends the result to the data mining module for operation; twenty one is converted;
twenty, the data mining module receives the integrated data confidence and completes deep analysis processing on information contained in the data in cooperation with the information retrieval module;
twenty-one, analyzing the operation efficiency of each link in the big data processing process by an efficiency evaluation module, and submitting an evaluation result to a system monitoring platform;
twenty-two, the hybrid cloud management module receives efficiency evaluation information of each link of the efficiency evaluation module for big data processing, and monitors and manages the operation condition of the whole system platform by combining the monitoring log output by the nano-management cloud resource;
twenty-third, the system keeps monitoring whether the operation efficiency of the managed hybrid cloud resources reaches the predefined triggering condition and alarm threshold of the resource optimization module, judges whether the predefined triggering condition is reached, if yes, turns to twenty-fourth, otherwise turns to twenty-twelve;
twenty-four steps, starting a big data workload migration process, and calling a resource allocation module and a resource migration module in the hybrid cloud management module through a resource optimization module to complete parameter configuration of the relevant big data processing workload migration process; the concrete process is transferred to the step four process.
The working principle of the invention is as follows: the visual interface module is used for providing an access and operation interface of the whole system; compared with a common access interface, the difference is that an operator can complete the operations of configuration, deployment, arrangement, migration and the like of the application of each level on the large data processing framework arrangement system and the governed hybrid cloud platform through the interface, and the input mode is a method that a graphical interface is matched with mouse gesture motion as a main mode and manual input of individual parameters is assisted. In the system operation process, the system continuously receives monitoring indexes and alarm logs of various resources in the big data processing process of efficiency evaluation, and the monitoring indexes and the alarm logs are used as input of mobile terminal display, and integrated data are output to a service directory module; then, various service processes for processing big data which can be provided externally by the system are managed and configured through the service directory module, wherein the service specifically refers to a big data processing capability facing to a client formed by the cooperative operation of a relevant big data processing process set. In the specific operation process, the operator control of the visual interface module is used as input, a large data processing process configuration item of a background is called, and a flow executor is used for executing the large data processing process configuration item; according to the action gesture input of the visual interface module and the input of the service directory module, according to the type and scale of the big data processing task to be executed, determining the relevant configuration of a specific process group and the like executed by the specific big data processing task, integrating the data into a script containing parameters required by the whole big data processing deployment scheme, and outputting the script to modules for data analysis, data processing, data storage, data integration and the like; the resource optimization module receives parameters transmitted by the process executor, such as resource types, quantity, initialization parameters, security strategies and the like, the resource allocation module executes various cloud resources required by scheduling each link of big data processing, specifically operates resource deployment, configuration and initialization tasks on each cloud stack of the hybrid cloud platform, and is responsible for checking the finished result; the data mining module comprehensively analyzes and processes the data processed by the generation through receiving the arrangement requirement of the service directory module on the data processing, and outputs decision information which can be used by a terminal user by combining the screening information which is provided by information retrieval and collected. Meanwhile, the system outputs log information in the execution process for the efficiency evaluation, and the log information is used for analyzing the efficiency use level in the processing process; the information retrieval module receives the information collection requirement provided by the service directory module, and integrally searches and screens the existing information to obtain a result which is fed back to the data mining module; receiving log information generated by a data mining module in the data processing process through an efficiency evaluation module, performing iterative computation by combining an efficiency evaluation model and a related algorithm of the efficiency evaluation module, analyzing to obtain efficiency information of each link in the whole big data processing process, and displaying the efficiency information on a visual interface module in a mobile terminal and multi-dimensional form; the programming model management module manages all programming models in the big data processing subsystem, and comprises the operations of adding, deleting, modifying and checking the models; in the specific data processing process, receiving cache data provided by a cache data management module of a lower data storage subsystem, and executing and calling a managed batch processing module and a managed stream processing module according to different data types; the batch processing module receives an instruction of the programming model management module according to different types of data to be processed and is responsible for processing data set operation suitable for distributed processing; the stream processing module receives an instruction of the programming model management module according to different types of data to be processed and is responsible for processing data set operation with a large data magnitude; the cache data management module is responsible for processing cache data in the computing resources. The system can receive metadata output by a metadata management module and relational data output by a database management module, select corresponding data optimization read-write modes according to different data storage types, and output a final result to a programming model management module in an upper system; the metadata base management module is responsible for abstracting lower-layer original data into metadata for data storage, and executing functional operations such as adding, deleting, modifying and checking the metadata. The output result is transmitted to the cache data; the database management module is responsible for managing the relational data in the data storage layer, executing operations including data adding, deleting, modifying, checking and the like, and receiving the data of the file storage module and the data synchronous output module of the lower data integration module; the data synchronization module receives the stored data output by the resource allocation module in the lower-layer hybrid cloud management subsystem, integrates the data and sends the integrated data to the upper-layer metadata management module; recording the log in the operation and outputting the log to a message transmission module; the message transmission module records log information of the resource allocation module on the hybrid cloud subsystem in the process of operating the cloud resources, and receives logs of the data synchronization module in the execution process. And store it in the storage module of the file; the file storage module is used for storing and managing related log information and other data in a file form in the data integration processing process; and the resource allocation module receives a data calling instruction of the upper layer data integration subsystem for the cloud resources and operates the managed hybrid cloud resources in combination with the optimization instruction of the hybrid cloud output by the resource optimization module. In the execution process, the verification of the operation authority is required to be carried out with the authority management module, and the specific lowering configuration of the operation command is carried out through a northbound API developed by cloud resources; and the resource migration module receives an operation command of the resource allocation module on the cloud resources and is responsible for the migration work of the specific data processing workload among the cloud stacks. When a specific execution command is too heavy, the operation of each step needs to be verified by an authority management module; the authority management module is responsible for authority authentication of various operations of the mixed element management subsystem, and authorizes the resource distribution module, the resource migration module and the resource monitoring module to execute related operations through authority division of fine granularity; the resource monitoring module is responsible for monitoring the monitoring log information of each managed cloud stack in the hybrid cloud operation process, carrying out unified formatting processing, receiving the command scheduling of the authority management module, and making a corresponding cloud stack operation instruction in real time when the triggering condition of the monitoring threshold is reached so as to complete the stable operation of the monitoring resources.
In summary, the hybrid cloud management platform capable of flexible scheduling of the present invention is used for accessing various cloud computing resources of a management client, including a private cloud and a public cloud; the method can realize rapid definition and configuration of all managed resources, support different levels of services in the process of big data processing, provide resources required by meeting specific processing links, and meet the optimal resource utility of the cloud platform; through a management platform of a hybrid cloud, dynamic response to big data processing load adjustment can be realized, and a big data workload migration task across cloud stacks is completed; the method supports a user to complete the arrangement and execution work of the whole frame and the subdivision tasks of big data processing through various visual operation modes such as a mobile terminal interface and the like, and the frame needs to meet the fast iteration requirement of the big data processing field of the user.
The above embodiments are described in further detail to solve the technical problems, technical solutions and advantages of the present invention, and it should be understood that the above embodiments are only examples of the present invention and are not intended to limit the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (8)

1. A big data processing framework arrangement system of a hybrid cloud platform is characterized by comprising:
the arrangement service module is used for calling a hybrid cloud management platform according to an input big data processing flow configuration strategy and applying for managing corresponding computing, network and storage resources in a cloud stack;
the mixed cloud management submodule is connected with the arrangement service module and is used for completing scheduling distribution, authority management, migration and monitoring of various cloud resources according to a scheduling strategy required by a specific big data processing link and completing efficiency optimization of a big data overall processing process;
the infrastructure resource module is connected with the hybrid cloud management submodule and used for collecting and storing public cloud, computing resources, storage resources and network resources so as to enable the hybrid cloud management submodule to complete management;
the data analysis module is connected with the arrangement service module and used for carrying out deep processing and analysis on data, carrying out multi-dimensional retrieval and analysis on stored information, communicating with each module of the system to obtain multi-dimensional monitoring information, comprehensively evaluating the operation efficiency of the whole system and finally displaying an analysis result and a prediction trend on a mobile terminal interface;
the data processing module is connected with the data analysis module, is used for the addition, deletion, reconstruction and background management of a programming model, is responsible for batch-oriented workload, schedules a specified batch processing engine to process the workload, is responsible for stream-oriented workload and schedules a specified stream processing engine to process the workload;
the data storage module is connected with the data processing module and used for processing the data information in the loading cache, importing the data information into a corresponding follow-up module, processing and preprocessing metadata transmitted from a lower layer, realizing various operations on database resources by calling relevant APIs (application program interfaces) of the database, and finishing the storage flow operation of data completion by cooperating with cache data management and metadata management;
and the data integration module is connected with the data storage module and is used for carrying out multi-dimensional screening on the data to be processed output by the lower-layer mixed cloud resource, transmitting the data to the data storage module, gathering and sorting various real-time messages in the data collection process, recording various error log alarm information, recording the data in a file form, and carrying out storage optimization file type operation on the stored file data.
2. The big data processing framework orchestration system of a hybrid cloud platform of claim 1, wherein the orchestration service module comprises:
the visual interface module is used for displaying the whole flow of the big data processing framework, and information retrieval, data mining results and efficiency evaluation information which can be displayed;
the service directory module is used for packaging each link of the big data processing framework facing the end user, displaying a big data processing flow which can be directly called for the user, and scheduling the data processing module, the data storage module and the data integration module in the system to complete a packaging task;
the flow executor is responsible for decomposing and executing a specific big data processing flow, and dynamically adjusts the workload of each cloud stack on the hybrid cloud management platform through the communication with the resource optimization module to achieve the purpose of resource optimization;
and the resource optimization module is responsible for cooperative work of the resource allocation modules in the south and hybrid cloud management submodules and real-time allocation and specific scheduling of various computing, storage and network resources managed by the hybrid cloud management platform.
3. The big data processing framework orchestration system of hybrid cloud platform of claim 1, wherein the data analysis module comprises:
the data mining module is responsible for deeply processing and analyzing data;
the information retrieval module is responsible for carrying out multi-dimensional retrieval on stored information;
and the efficiency evaluation module is communicated with each subsystem of the system to obtain multi-dimensional monitoring information, comprehensively evaluate the operation efficiency of the whole system and finally display the analysis result and the prediction trend on a visual interface.
4. The big data processing framework orchestration system of hybrid cloud platforms according to claim 1, wherein the data processing module comprises:
the programming model management module is responsible for adding, deleting, reconstructing and background management of the programming model;
the batch processing module is responsible for facing batch processing workload and scheduling a specified batch processing engine to process the workload;
and the stream processing module is responsible for processing the workload in a stream-oriented mode and scheduling the specified stream processing engine to process the workload.
5. The big data processing framework orchestration system of hybrid cloud platform of claim 1, wherein the data storage module comprises:
the cache data management module is responsible for processing and loading data information in the cache and importing the data information into a corresponding follow-up module;
the metadata management module is used for processing and preprocessing metadata transmitted from a lower layer;
and the database management module is used for realizing various operations on database resources by calling the relevant API of the database, and finishing the storage flow operation finished by the data by cooperating with the cache data management module and the metadata management module.
6. The big data processing framework orchestration system of hybrid cloud platform of claim 1, wherein the data integration module comprises:
the data synchronization module is responsible for performing multi-dimensional screening on the data to be processed output by the lower layer mixed cloud resource and transmitting the data to the data storage module;
the message transmission module is responsible for summarizing and sorting various real-time messages in the data collection process and recording various error log alarm information;
and the file storage module is responsible for recording data in a file form and can perform storage optimization file type operation on the stored file data.
7. The big data processing framework orchestration system of hybrid cloud platform of claim 1, wherein the hybrid cloud management sub-module comprises:
the resource allocation module is used for receiving a data calling instruction of the data integration module on the cloud resources and operating the managed hybrid cloud resources in combination with an optimization instruction of the hybrid cloud output by the resource optimization module; in the execution process, the verification of the operation authority is required to be carried out with the authority management module, and the specific lowering configuration of the operation command is carried out through a northbound API developed by cloud resources;
the resource migration module is used for receiving an operation command of the resource allocation module on the cloud resources and is responsible for migration work of specific data processing workloads among the cloud stacks; in the specific command execution process, the operation of each step needs to be verified through the authority management module;
the authority management module is responsible for authority authentication of various operations of the hybrid cloud management submodule and authorizes the resource allocation module, the resource migration module and the resource monitoring module to execute related operations through authority division of fine granularity;
and the resource monitoring module is responsible for monitoring the monitoring log information of each managed cloud stack in the hybrid cloud operation process, performing unified formatting processing, receiving the command scheduling of the authority management module, and making a corresponding cloud stack operation instruction in real time when the triggering condition of the monitoring threshold is met to complete the stable operation of the monitoring resources.
8. A big data processing framework arrangement method of a hybrid cloud platform is characterized by comprising the following steps:
firstly, a user operates a big data processing framework arrangement system of a hybrid cloud platform through a visual interface module, inputs various links and configurations of a big data processing process needing deployment and execution, and the big data processing framework arrangement system of the hybrid cloud platform inputs related applications to a background;
after the service directory module responds to the application, whether the stored service list and the corresponding big data processing flow list contain the applied processing flow or deployment template is searched;
step three, judging whether the system has a big data arrangement flow, if so, turning to step five, otherwise, turning to step four;
step four, requiring a user to add a corresponding big data processing flow and parameter configuration by a big data processing framework arrangement system of the hybrid cloud platform; after the big data processing framework arrangement system of the hybrid cloud platform is completed, the user is required to add a corresponding big data processing flow and a parameter configuration task, and then the second step is continuously executed;
step five, determining a big data processing arrangement flow, and calling a big data flow executor by a big data processing framework arrangement system of the hybrid cloud platform to complete scheme integration of configuration parameters, deployment flows and initialization parameter configuration information required by the big data processing flow applied by a user; the step is also responsible for applying for corresponding resources to the administered hybrid cloud management platform, and the step ten is executed;
step six, the data integration module receives the instruction, determines a specific operation strategy of data integration, and completes the process of extracting data from the hybrid cloud stack;
step seven, the data storage module receives the instruction, determines the specific operation strategy of data storage, and respectively calls the metadata management module and the database management module to finish the data storage operation according to different data types;
step eight, the data processing module receives the instruction, determines the specific operation strategy of data processing, and respectively calls the batch processing module and the stream processing module according to the execution condition output by the programming model management module;
step nine, the data analysis module receives the output result of the lower programming model management module, sends configuration execution parameters to the data mining module, the information retrieval module and the efficiency evaluation module, and determines a data analysis mode;
step ten, the resource allocation module executes resource allocation operation of cloud resources governed by the hybrid cloud to complete application of computing, network and storage resources related to the big data processing process;
eleventh, finishing abstract operation of data from a lower-layer module by a data synchronization module, a message transmission module and a file storage module in the data integration module according to different selected big data to-be-processed data sets;
step twelve, judging whether the execution data storage object is metadata, if so, turning to step thirteen, otherwise, turning to step fourteen;
step thirteen, if the data to be processed is metadata, the big data processing framework arrangement system of the hybrid cloud platform calls a metadata management module to finish data storage;
step fourteen, if the data to be processed is relational data, the big data processing framework arrangement system of the hybrid cloud platform calls a database management module to complete data storage;
step fifteen, the cache data management module receives lower layer processing data, reads the stored data into a cache resource pool, and performs screening processing optimization operation on the cache data set;
sixthly, loading a data processing model adapted to the required big data processing process through a programming model management module, judging whether the programming model is a batch processing model according to different processing scenes, if so, turning to the seventeenth step, otherwise, turning to the eighteenth step;
seventhly, if the big data processing scene to be processed is judged to be suitable for adopting the batch processing mode through the big data processing framework arrangement system of the hybrid cloud platform, the big data processing framework arrangement system of the hybrid cloud platform calls the batch processing module loading model to process data
Eighteen, if the big data processing scene to be processed is judged to be suitable for adopting the stream processing mode through the big data processing framework arrangement system of the hybrid cloud platform, the big data processing framework arrangement system of the hybrid cloud platform calls a batch processing module loading model to perform data processing;
nineteenth, the data analysis module receives the data processing result and integrates and sends the result to the data mining module for operation; twenty one is converted;
twenty, the data mining module receives the integrated data information and completes the deep analysis processing of the information contained in the data in cooperation with the information retrieval module;
twenty-one, analyzing the operation condition efficiency of each link in the big data processing process by an efficiency evaluation module, and submitting an evaluation result to a big data processing framework arrangement system monitoring platform of a hybrid cloud platform;
twenty-two, the hybrid cloud management submodule receives efficiency evaluation information of the efficiency evaluation module aiming at each link of big data processing, and monitors and manages the running condition of a big data processing framework arrangement system platform of the whole hybrid cloud platform by combining a monitoring log output by the nano-tube cloud resource;
twenty-third, the big data processing framework arrangement system of the hybrid cloud platform keeps monitoring whether the operation efficiency of the managed hybrid cloud resources reaches the predefined trigger condition of the resource optimization module and the alarm threshold value, judges whether the predefined trigger condition is reached, if yes, turns to twenty-fourth, otherwise turns to twenty-twelve;
twenty-four steps, starting a big data workload migration process, and calling a resource allocation module and a resource migration module in a hybrid cloud management submodule through a resource optimization module to complete parameter configuration of the relevant big data processing workload migration process; the concrete process is transferred to the step four process.
CN201711107600.8A 2017-11-10 2017-11-10 Big data processing framework arrangement system and method of hybrid cloud platform Active CN107948254B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711107600.8A CN107948254B (en) 2017-11-10 2017-11-10 Big data processing framework arrangement system and method of hybrid cloud platform

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711107600.8A CN107948254B (en) 2017-11-10 2017-11-10 Big data processing framework arrangement system and method of hybrid cloud platform

Publications (2)

Publication Number Publication Date
CN107948254A CN107948254A (en) 2018-04-20
CN107948254B true CN107948254B (en) 2020-09-22

Family

ID=61934761

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711107600.8A Active CN107948254B (en) 2017-11-10 2017-11-10 Big data processing framework arrangement system and method of hybrid cloud platform

Country Status (1)

Country Link
CN (1) CN107948254B (en)

Families Citing this family (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108920951A (en) * 2018-07-20 2018-11-30 山东浪潮通软信息科技有限公司 A kind of security audit frame based under cloud mode
CN109144724A (en) * 2018-07-27 2019-01-04 众安信息技术服务有限公司 A kind of micro services resource scheduling system and method
CN109446180B (en) * 2018-10-18 2021-07-02 郑州云海信息技术有限公司 Method and device for configuring cloud data platform
CN109347676A (en) * 2018-11-02 2019-02-15 杭州云霁科技有限公司 A kind of isomery, integrated mixed cloud resource management platform
CN109739845A (en) * 2018-12-26 2019-05-10 贵州商学院 A kind of big data service system based on intensified learning
CN111611221A (en) * 2019-02-26 2020-09-01 北京京东尚科信息技术有限公司 Hybrid computing system, data processing method and device
US11182697B1 (en) 2019-05-03 2021-11-23 State Farm Mutual Automobile Insurance Company GUI for interacting with analytics provided by machine-learning services
US11392855B1 (en) 2019-05-03 2022-07-19 State Farm Mutual Automobile Insurance Company GUI for configuring machine-learning services
CN110572276B (en) * 2019-08-13 2021-02-09 华云数据控股集团有限公司 Deployment method, device, equipment and storage medium
CN110750256A (en) * 2019-09-27 2020-02-04 北京远舢智能科技有限公司 Intelligent PaaS cloud service platform
CN111212115A (en) * 2019-12-23 2020-05-29 恩亿科(北京)数据科技有限公司 Method and device for realizing hybrid cloud management, computer storage medium and terminal
CN111580977B (en) * 2020-05-12 2023-08-29 中国民航信息网络股份有限公司 Resource adjustment method and related equipment
CN112035419A (en) * 2020-08-12 2020-12-04 湖北世纪创新科技有限公司 Novel data center visualization algorithm
CN112100217A (en) * 2020-09-18 2020-12-18 山东浪潮商用系统有限公司 Asset management system and method based on tax big data
CN112394701A (en) * 2020-12-10 2021-02-23 之江实验室 Multi-robot cloud control system based on cloud-edge-end hybrid computing environment
CN112596914B (en) * 2020-12-29 2024-03-15 贵州大学 IoT-oriented edge node system architecture, working method thereof and computing migration method
CN113485964A (en) * 2021-06-11 2021-10-08 国网内蒙古东部电力有限公司 Lightweight data management system oriented to energy big data ecology
CN113485650A (en) * 2021-07-26 2021-10-08 南京鹏云网络科技有限公司 Data arrangement system
CN114443295B (en) * 2022-01-21 2024-01-12 苏州浪潮智能科技有限公司 Heterogeneous cloud resource management scheduling method, device and system
CN114443025B (en) * 2022-01-28 2023-10-24 悦锦数字科技(上海)股份有限公司 Modularized ETL task processing system and ETL task processing method for data management platform
CN114648026B (en) * 2022-05-20 2022-08-09 广州嘉为科技有限公司 Resource delivery method, system and storage medium in multi-cloud environment

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104202416A (en) * 2014-09-16 2014-12-10 浪潮(北京)电子信息产业有限公司 Service orchestration system and method under cloud operating system
CN106506710A (en) * 2017-01-04 2017-03-15 成都华栖云科技有限公司 A kind of PaaS cloud platforms suitable for media business
CN107315776A (en) * 2017-05-27 2017-11-03 国网安徽省电力公司信息通信分公司 A kind of data management system based on cloud computing

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120221454A1 (en) * 2011-02-28 2012-08-30 Morgan Christopher Edwin Systems and methods for generating marketplace brokerage exchange of excess subscribed resources using dynamic subscription periods

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104202416A (en) * 2014-09-16 2014-12-10 浪潮(北京)电子信息产业有限公司 Service orchestration system and method under cloud operating system
CN106506710A (en) * 2017-01-04 2017-03-15 成都华栖云科技有限公司 A kind of PaaS cloud platforms suitable for media business
CN107315776A (en) * 2017-05-27 2017-11-03 国网安徽省电力公司信息通信分公司 A kind of data management system based on cloud computing

Also Published As

Publication number Publication date
CN107948254A (en) 2018-04-20

Similar Documents

Publication Publication Date Title
CN107948254B (en) Big data processing framework arrangement system and method of hybrid cloud platform
CN102375731B (en) Coding-free integrated application platform system
CN107612886B (en) Spark platform Shuffle process compression algorithm decision method
CN105893593B (en) A kind of method of data fusion
CN102508639B (en) Distributed parallel processing method based on satellite remote sensing data characteristics
CN109146081B (en) Method and device for creating model project in machine learning platform
US20140096188A1 (en) System and method for policy generation
WO2009082388A1 (en) Modelling computer based business process for customisation and delivery
CN110377595A (en) A kind of vehicle data management system
CN113298503A (en) Government affair-oriented workflow management system and database and table dividing method thereof
CN113741883B (en) RPA lightweight data middling station system
CN107402926A (en) A kind of querying method and query facility
CN110740079A (en) full link benchmark test system for distributed scheduling system
CN115934680A (en) One-stop big data analysis processing system
US9626156B2 (en) Application architecture design method, application architecture design system, and recording medium
US10785102B2 (en) Modifying distributed application based on cloud diagnostic data
Hajji et al. Optimizations of Distributed Computing Processes on Apache Spark Platform.
US8918410B2 (en) System and method for fast identification of variable roles during initial data exploration
CN115392501A (en) Data acquisition method and device, electronic equipment and storage medium
Campos et al. Engineering environment to support product-service design using value chain data
CN107018160B (en) Manufacturing resource and clouding method based on layering
CN114693103A (en) Method, apparatus and storage medium for generating device management model and managing device
CN114443293A (en) Deployment system and method for big data platform
Chalvantzis et al. BBQ: Elastic MapReduce over cloud platforms
CN114066110A (en) System for providing machine learning service for user

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant