CN108429768A - Cloud data analysis service manages system, method and cloud server - Google Patents

Cloud data analysis service manages system, method and cloud server Download PDF

Info

Publication number
CN108429768A
CN108429768A CN201810536629.6A CN201810536629A CN108429768A CN 108429768 A CN108429768 A CN 108429768A CN 201810536629 A CN201810536629 A CN 201810536629A CN 108429768 A CN108429768 A CN 108429768A
Authority
CN
China
Prior art keywords
resource
host
management module
data analysis
service
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201810536629.6A
Other languages
Chinese (zh)
Inventor
刘梦雯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
New H3C Cloud Technologies Co Ltd
Original Assignee
New H3C Cloud Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by New H3C Cloud Technologies Co Ltd filed Critical New H3C Cloud Technologies Co Ltd
Priority to CN201810536629.6A priority Critical patent/CN108429768A/en
Publication of CN108429768A publication Critical patent/CN108429768A/en
Pending legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/40Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using virtualisation of network functions or resources, e.g. SDN or NFV entities
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/50Network service management, e.g. ensuring proper service fulfilment according to agreements
    • H04L41/5041Network service management, e.g. ensuring proper service fulfilment according to agreements characterised by the time relationship between creation and deployment of a service
    • H04L41/5051Service on demand, e.g. definition and deployment of services in real time
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/12Network monitoring probes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/20Arrangements for monitoring or testing data switching networks the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/06Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1095Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/133Protocols for remote procedure calls [RPC]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/34Network arrangements or protocols for supporting network services or applications involving the movement of software or configuration parameters 
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/04Network management architectures or arrangements
    • H04L41/046Network management architectures or arrangements comprising network management agents or mobile agents therefor

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Debugging And Monitoring (AREA)

Abstract

Present disclose provides a kind of cloud data analysis service management system, method and cloud servers;Wherein, which includes:Service management module, the request to create for receiving cloud data analysis service example determine the corresponding host performance requirement of example according to request to create;Resource management module, for requiring the host resource to match, and the image file needed for the acquisition example for service management module scheduling and host performance;Wherein, host resource includes:Physical host resource and fictitious host computer resource;Mirror image management module, for providing the image file needed for example to resource management module;Service management module is additionally operable to install image file generation example on host resource, runs simultaneously admin instance.The disclosure can meet the high in the clouds deployment requirements of the data analysis service of different host performance requirements.

Description

Cloud data analysis service manages system, method and cloud server
Technical field
This disclosure relates to field of cloud computer technology, more particularly, to a kind of cloud data analysis service management system, method and Cloud server.
Background technology
Cloud computing is a kind of distributed computing model, is based on technologies such as distributed treatment, parallel processing and grid computings Shared architecture method.Cloud computing can be with the huge virtual computing resource of self and management, including calculates service Device, storage server, broadband resource etc., to provide to the user reliably, customized, maximum resource utilize IT clothes Business.
High in the clouds big data analysis service is sent out on the basis of SaaS (Software as a Service, software service) A kind of cloud computing technology of exhibition;High in the clouds big data analysis service is typically deployed in the virtual machine of cloud server, due to The limitation of virtual machine self performance, data analysis service solely dispose mode on a virtual machine and are difficult to meet high-performance data The demand of Analysis Service causes these high performance data analysis services that can only be realized by entity physical resource, cannot dispose Beyond the clouds.
Invention content
In view of this, the disclosure is designed to provide a kind of cloud data analysis service management system, method and high in the clouds clothes Business device, to meet the high in the clouds deployment requirements of the data analysis service of different host performance requirements.
To achieve the goals above, the technical solution that the disclosure uses is as follows:
In a first aspect, present disclose provides a kind of cloud data analysis services to manage system, system includes:Service management mould Block, the request to create for receiving cloud data analysis service example determine that the corresponding host performance of example is wanted according to request to create It asks;Resource management module, for requiring the host resource to match, and acquisition for service management module scheduling and host performance Image file needed for example;Wherein, host resource includes:Physical host resource and fictitious host computer resource;Mirror image management module, For providing the image file needed for example to resource management module;Service management module is additionally operable to install mirror on host resource As file generated example, simultaneously admin instance is run.
Second aspect, present disclose provides a kind of cloud data analysis service management method, method is applied to above-mentioned cloud data Analysis Service manages system;Method includes:The request to create for receiving cloud data analysis service example determines real according to request to create The corresponding host performance requirement of example;Scheduling requires the host resource to match with host performance;Wherein, host resource includes:Object Manage host resource and fictitious host computer resource;Obtain the image file needed for example;Image file is installed to generate on host resource Example runs simultaneously admin instance.
The third aspect, disclosure embodiment provide a kind of cloud server, including memory and processor, wherein Memory is for storing one or more computer instruction, and one or more computer instruction is executed by processor, in realization State cloud data analysis service management method.
Fourth aspect, disclosure embodiment provide a kind of machine readable storage medium, and machine readable storage medium is deposited Machine-executable instruction is contained, when being called and being executed by processor, machine-executable instruction promotes to locate machine-executable instruction It manages device and realizes above-mentioned cloud data analysis service management method.
Above-mentioned cloud data analysis service management system, method and cloud server, service management module can pass through resource Management module is scheduled physical host resource and fictitious host computer resource, so that host resource and the host needed for establishment example Performance requirement matches;In which, the deployment way of cloud data analysis service is more flexible, and high performance data analysis is made to take Business can also be realized by cloud computing, meet the high in the clouds deployment requirements of the data analysis service of different host performance requirements.
Other feature and advantage of the disclosure will illustrate in the following description, alternatively, Partial Feature and advantage can be with Deduce from specification or unambiguously determine, or by implement the disclosure above-mentioned technology it can be learnt that.
To enable the above objects, features and advantages of the present invention to be clearer and more comprehensible, better embodiment cited below particularly, and match Appended attached drawing is closed, is described in detail below.
Description of the drawings
It, below will be to specific in order to illustrate more clearly of disclosure specific implementation mode or technical solution in the prior art Embodiment or attached drawing needed to be used in the description of the prior art are briefly described, it should be apparent that, in being described below Attached drawing is some embodiments of the disclosure, for those of ordinary skill in the art, before not making the creative labor It puts, other drawings may also be obtained based on these drawings.
Fig. 1 is the structural schematic diagram that a kind of cloud data analysis service that disclosure embodiment provides manages system;
Fig. 2 is service management module in another cloud data analysis service management system that disclosure embodiment provides Concrete structure schematic diagram;
Fig. 3 is service management module in another cloud data analysis service management system that disclosure embodiment provides Concrete structure schematic diagram;
Fig. 4 is the structural schematic diagram that another cloud data analysis service that disclosure embodiment provides manages system;
Fig. 5 is that the framework that cloud data analysis service is disposed using multi-tenant form that disclosure embodiment provides is illustrated Figure;
Fig. 6 is a kind of flow chart for cloud data analysis service management method that disclosure embodiment provides;
Fig. 7 is a kind of structural schematic diagram for cloud server that disclosure embodiment provides.
Specific implementation mode
To keep the purpose, technical scheme and advantage of disclosure embodiment clearer, below in conjunction with attached drawing to this public affairs The technical solution opened is clearly and completely described, it is clear that and described embodiment is a part of embodiment of the disclosure, Rather than whole embodiment.Based on the embodiment in the disclosure, those of ordinary skill in the art are not making creation Property labour under the premise of the every other embodiment that is obtained, belong to the range of disclosure protection.
High in the clouds big data analysis is serviced with dynamic scalable, availability is high, utilization cost is low, ease for use is high, extensive The features such as parallel processing, is suitable for the storage and analysis of mass data.High in the clouds big data analysis service is usually deployed big beyond the clouds In the virtual machine of amount, this deployment way can enhance the storage capacity and computing capability of big data platform, be configured without repeating Personnel, hardware and software are saved platform building cost, and are safeguarded convenient for the subsequent upgrade of soft and hardware;At the same time it can also virtual Change the portion rear end function of big data platform;But since virtual machine is that have complete hardware system work(by what software was simulated Can computer system, the hardware instruction for simulating bottom is needed in operational process, leads to the application program speed of service, data input The performances such as output are affected;Which limits high performance big data analysis services (for example, high data processing performance, height are defeated Enter output performance etc.) it is difficult to dispose beyond the clouds, it can only be disposed by entity physical resource.
In order to make high performance big data analysis service that can also realize that data analysis service needs more clever by cloud computing Deployment way living, to meet the data analysis requirements of different performance;Based on this, disclosure embodiment provides a kind of cloud number According to Analysis Service management system, method and cloud server, it is specifically described below.
Disclosure embodiment provides firstly a kind of cloud data analysis service management system, which can be deployed in cloud In the server of platform, as shown in Figure 1, the system includes service management module 10, resource management module 11 and mirror image management mould Block 12.
Wherein, service management module 10 is used to receive the request to create of cloud data analysis service example, according to request to create Determine the corresponding host performance requirement of example;Resource management module 11 is used to want with host performance for the scheduling of service management module 10 The host resource to match is sought, and obtains the image file needed for example;The host resource includes:Physical host resource and void Quasi- host resource;Mirror image management module 12 is used to provide the image file needed for example to resource management module 11;Above-mentioned service Management module 10 is additionally operable to install image file generation example on host resource, runs simultaneously admin instance.
The service management module 10 can open an interface to user, for example, can be with login account and the website logged in; User can submit request to create by the interface;The association attributes of cloud data analysis service are generally comprised in the request to create, Including affiliated industry, data volume, analysis requirement of real-time, security requirement etc.;Service management module 10 receives request to create Afterwards, whether the available resources in usually first preliminary audit survey current system can substantially meet the request to create, if it is possible to meet, Relevant feedback then can be carried out to user.
Can directly include host performance requirement in the request to create that above-mentioned user submits, for example, needed for running example Memory, processor, data space, data input and output rate etc.;Service management module 10 can be directly according to establishment Host performance requirement in request, passes through 11 dispatching host machine resource of resource management module.However, certain customers are likely difficult to submit Accurate host performance requirement, the association attributes for the cloud data analysis service that service management module 10 can submit user at this time It is analyzed, to obtain host performance requirement;The mode or other intelligence of machine learning specifically may be used in the association attributes Mode, which automates, to be analyzed, and can also analyze to obtain the host performance requirement of the user using administrator's manual intervention.
After service management module 10 determines host performance requirement, scheduling of resource can be sent to resource management module 11 and referred to It enables and the host performance requires.Above-mentioned resource management module 11 is connect with each resource pool of cloud platform, and the resource pool is usual Including resources of virtual machine pond, physical machine resource pool, container resource pool etc.;Resource management module 11 monitors and manages these resource pools Operating status and use state;After receiving above-mentioned scheduling of resource instruction, transferred from these resource pools and above-mentioned host The resource that performance requirement matches, while the resource being deployed into is identified.
In general, if host performance requirement is overall general, fictitious host computer, which is only used only, to be met the requirements, in order to save into Originally and resource utilization is improved, can is preferably that user distributes fictitious host computer resource;If difficult using fictitious host computer merely Then can be that the user distributes physics host resource to meet above-mentioned host performance requirement;The data processing of physical host resource Ability, storage capacity and safety etc. are typically superior to fictitious host computer resource.Can be only the user when actually distributing Physics host resource is distributed, the demand that can also be run according to example, while distributing fictitious host computer resource and physics for the user Host resource.
The image file needed for various types or the cloud data analysis service of version has usually been pre-saved in cloud platform, it should Image file is it can be appreciated that Software Compression packet;Mirror image management module 12 is for managing these image files.Above-mentioned service pipe After reason module 10 receives request to create, corresponding image file can be sent to mirror image management module 12 and obtains request, the acquisition Can include the corresponding type information of current cloud data analysis service and version information in request;Mirror image management module 12 receives After obtaining request, according to type information and version information that the acquisition asks to include, extracts and correspond to from the image file of preservation Image file, and the preservation address of the image file extracted or image file is fed back into resource management module 11 so that Resource management module 11 gets above-mentioned image file, and image file is stored on the host resource of scheduling.
So far, service management module 10 can be found that the image file on the host resource and the host resource, in master Image file is decompressed in machine resource, is installed, the example of cloud data analysis service is generated, starts and run the example;Tool Body may be used cloud computing management platform openstack and complete these steps:First in installation operation frame, selection installation comes Source preserves the address of image file that is, on the host resource;And then determine the OS Type of current hosts resource;It sets The disk size of host resource, while multidomain treat-ment is carried out to disk;Again in installation operation frame, installation path is selected, i.e., in fact The storage location of example;And then the image file is decompressed, after the completion of decompression, starts installation steps, image file is installed to above-mentioned In storage location;After installation is complete, the primary host resource is usually restarted again, and after the completion of restarting, which starts and can be normal Operation.
The example during running, executing subsequent data analysis business, service management module 10 by batch processing, A variety of data analysis service works such as stream process, MPP (Massively Parallel Processor, MPP) Tool, manages, monitors the example operation conditions, to which data analysis service is managed and be controlled.
The above-mentioned cloud data analysis service that disclosure embodiment provides manages system, and service management module can pass through money Source control module 11 is scheduled physical host resource and fictitious host computer resource, so that needed for host resource and establishment example Host performance requires to match;In which, the deployment way of cloud data analysis service is more flexible, makes high performance data point Analysis service can also realize that meeting the high in the clouds deployment of the data analysis service of different host performance requirements needs by cloud computing It asks.
By the above embodiment it is found that service management module 10 is as opposite core in cloud data analysis service management system Module have the function of coordinating, other modules in commander's system in the establishment and operational process of example;In order to preferably Understand the above embodiment provide cloud data analysis service management system, below the function to the service management module 10 and The operational process of system is described in detail.
When creating example for user, the storage resource and computing resource in platform are occupied;Therefore, present embodiment In, user is referred to as the tenant on the platform.As shown in Fig. 2, above-mentioned service management module 10 includes API component 20, appoints Business component 21 and Agent components 22.
Wherein, API (Application Programming Interface, application programming interface) component 20 is used In providing interactive interface to tenant, the request to create of the cloud data analysis service example of tenant is received by interactive interface;Task Component 21 is used for through 11 dispatching host machine resource of resource management module, and controls resource management module to mirror image management module 12 Image file is obtained, image file is stored in host resource;Agent (agency) component 22 is for installing image file, to peace Image file after dress carries out initialization process, obtains the example of cloud data analysis service.
Above-mentioned API component 20 is mainly used for providing the external service towards tenant, can be with the terminal equipment in communication of tenant Connection;API component 20 in the system can specifically pass through WSGI (Web Server Gateway Interface, Web service Device gateway interface) it realizes, WSGI is the universal interface standard between Web server and web application or between frame;Together When, the realization of RESTFUL frameworks may be used in the design of the API component 20, which needs to meet REST (Representational State Transfer, declarative state transfer) principle.
When providing service to tenant, the application program installed on the user terminal of tenant may use to be removed API component 20 Other frameworks other than RESTFUL frameworks are realized;The application program and RESTFUL frameworks of various frameworks can be realized in above-mentioned WSGI Communication between the API component 20 of realization;API component 20 can ask access application by various RESTFUL, to obtain The relevant information data of user, while receiving the data of application program transmission;The process can also be described as in API component 20, RESTFUL asks the scheduling of resource between application program.
In creating example procedure, user sends out the request to create of example by application program, which can be RESTFUL API requests, include different HTTP request type in the RESTFUL API requests, and API component 20 receives the wound After building request, the preparations such as inspection quota, initialized cryptographic can be carried out, request to create is then passed through into RPC (Remote again Procedure Call, remote procedure call) or the modes such as message be sent to task component 21.
Task component 21 is as the core component in service management module 10, the work with other assemblies in scheduler module With the core work for completing cloud data analysis example;During creating example, task component 21 passes through API component After 20 receive the request to create of example, task component 21 sends the request to create and corresponding master to resource management module 11 Machine performance requirement;Between task component 21 and resource management module 11 can by the API component 20 of resource management module 11 into Row communication.
After resource management module 11 receives request, require to judge to create the specific host class needed for example according to host performance Type;In general, can run more fictitious host computers in a physical host, the data processing resources of the physical host and storage provide Source is shared by more fictitious host computers, and by the security isolation between each fictitious host computer of software realization;Therefore, physical host resource Data-handling capacity, storage capacity and safety etc. are typically superior to fictitious host computer resource.If the host that user submits Performance requirement needs a physical host or more physical hosts that could realize, for example, the storage capacity needed for service is two The summation of physical host;At this point, being directly that user distribution physical host is no longer transported by the direct running example of the physical host Row fictitious host computer.And if the host performance of user requires the resource that can be provided well below physical host, it is provided based on saving The considerations of source, then distributes fictitious host computer for the user.
If the example needs physical host, resource management module 11 to be found by internal scheduling component currently available Physical host, and the physical host is started by relevant computation module;If the example needs fictitious host computer, can pass through The Hypervisor (being referred to as virtual machine manager) of virtualization is created and is started corresponding fictitious host computer.
After physical host or fictitious host computer start completion, resource management module 11 obtains the section of physical host or fictitious host computer Point information carries out network of relation configuration, for example, distributing address for physical host or fictitious host computer according to the nodal information to host Mark;For not running fictitious host computer, the physical host of direct running example then distributes unique address label for the physical host Know;If operation has multiple fictitious host computers on physical host, corresponding address mark is distributed for each fictitious host computer, no longer for The physical host distributes address mark.Cloud management platform realizes physical host and virtual master by management address above mentioned mark The unified management and distribution of machine.Certainly, for other administrative purposes, for example, the management of pure hardware view, operation has fictitious host computer Physical host corresponding address mark can also be set.
The resource management module 11 is also responsible for downloading the image file needed for example from mirror image management module 12, by the mirror As some Agent information and configuration information of file and example are injected into the physical host or fictitious host computer of startup.Extremely This, creates host resource and relevant file needed for example and prepares successfully.
In addition, above-mentioned task component 21 is additionally operable to be managed the whole life cycle of example, including above-mentioned created Journey, further include example operation monitoring, example destruction, tenant's account management etc..
In data analysis service example operational process, if receiving the service request of example, pass through above-mentioned resource pipe Module 11 is managed it is also an option that the corresponding container computing resource of service request, big data is carried out using container computing resource to example Analyzing processing.The container computing resource can be selected from container resource pool, when carrying out analyzing processing, can be asked according to service The specific business asked further selects suitable big data processing frame to complete data analysis, and it is usual which handles frame Including Hadoop, Spark, Storm, Solr, MPP etc..
Before example establishment, the corresponding Agent components of example 22 are stored in the related image file of the example;It creates After host resource needed for example starts, above-mentioned Agent components 22 start automatically, to the image file being stored on host resource It is installed;After receiving the initialization requests of the transmission of task component 21, the image file after installation is carried out at initialization Reason, the example created at this time are updated to operating status.In general, Agent components 22 and the service type of example correspond, service Type is different, and Agent components 22 are also different;In example operational process, Agent components 22 are responsible for executing specific data analysis The inter-related task of service, for example, the data transmission between example and service management module 10, carries out data analysis service itself Operation.
In another embodiment, above-mentioned service management module 10 has more perfect functional structure, such as Fig. 3 institutes Show, on the basis of shown in Fig. 2, which further includes aft-end assembly 23, which can be logical by MQ Road (or bus) is connect with Agent assembly communications;Above-mentioned Agent components 22 are additionally operable in example operational process, acquire example Status information;The aft-end assembly 23 is used to status information preserving database corresponding to tenant;Specifically, the reality created After example is updated to operating status, which is sent to aft-end assembly 23 by Agent components 22, and aft-end assembly 23 corresponds to the state Conditional code be updated in database.
Aft-end assembly 23 can be understood as the interface module of Agent components 22 and database;It can be seen from the above, Agent groups Part 22 is corresponded with example, and therefore, also there are Agent components 22 itself multiple types, different types of Agent components 22 to visit Ask the mode of database, and there may be variations for the type for the database being able to access that;If again according to Agent components 22 The type or access mode of database are set, example can be caused to create cost and increased.Aft-end assembly 23 can be different Agent components 22 provide a unified interface for accessing database, even if there was only a kind of database, different Agent in platform Component 22 can also smoothly access database by the aft-end assembly 23, improve the convenience of system data storage.
In the process of running, the same time may handle the wound of multiple tenants to above-mentioned cloud data analysis service management system Build request, it is also possible to while the operating status of multiple examples is managed, at this point, the nucleus module as system, service management mould Block 10 is likely to the processing condition data for high concurrent occur, network blockage caused by being transmitted in order to avoid mass data instruction, Pass through MQ (Message Queue, message queue) tunneling traffic between various components in service management module 10.The channels MQ can To realize the asynchronous transmission of data between each component, to alleviate the data transmission pressure in service management module 10.
When cloud data analysis service management system manages, runs the example of multiple tenants at the same time, not only need to consider number According to the performance issue of Analysis Service, it is also necessary to consider the safety issue of data;Based on this, provided in disclosure embodiment In another cloud data analysis service management system, network management module 13 and authentication management module 14 are further comprised;Network pipe Module 13 and authentication management module 14 is managed to communicate to connect with above-mentioned service management module respectively;As shown in figure 4, the network management mould Block 13 is used for the multi-tenant mutually isolated corresponding host resource of way to manage admin instance, container computing resource and data Library, to realize the security isolation between different tenant's corresponding instances.Wherein, the mutually isolated way to manage of multi-tenant includes to rent Family configures the hardware or software resource exclusively enjoyed, avoids other tenants from using these resources, you can to accomplish the isolation between tenant.Together When, network management module 13 is also responsible for ensureing the connectivity of multi-tenant way to manage lower network, with the resource occupied to multi-tenant Carry out unified coordination arrangement.
The management mode of this multi-tenant can be that each tenant disposes and manage a set of relatively independent big data analysis Cluster;Fig. 5 show the configuration diagram that cloud data analysis service is disposed using multi-tenant form;In Fig. 5, based on cloud platform Virtualization pool and physical resource pool (physical resource pool is referred to as bare metal resource pool) can be that each tenant establishes One VPC (Virtual Private Cloud, virtual private cloud), in each VPC operation have respective big data processing frame (such as Hadoop), database (such as HBase) and MPP device.
Using VPC as the data analysis service service network of tenant, security isolation and the visit of multi-tenant can be effectively realized Ask limitation.In the VPC of each tenant, specific requirements unrestricted choice fictitious host computer that can be according to data analysis service or physics master Machine, to dispose the big data analysis cluster of the tenant, the configuration (such as CPU, memory, storage) of each host and node rule in cluster Lattice can also change according to the portfolio of subsequent data analysis service carries out dynamic expansion.
In general, the bottom of cloud platform in multiple hardwares resource is deployed in a distributed manner, these hardware resources are taken out by hardware As layer hides the various interface details of hardware resource, by hardware abstraction, it is convenient for the management of bottom hardware resource;Tenant is in platform When the example of upper establishment and operation cloud data analysis service, the hardware management mode of this abstract allows tenant to think that the example is transported Row can not perceive the example actual capabilities and operate on more physical hosts or fictitious host computer on single server.
The above-mentioned management mode with multi-tenant creates and management cloud data analysis service, can make the number of various performance requirements It is deployed in cloud platform according to Analysis Service, by the isomery cloud resource infrastructure of cloud platform, can effectively carry user's Data analysis and persistence task.
With continued reference to Fig. 4, above system further includes authentication management module 14, for carrying out authentication and access to tenant Rights management.Specifically, the authentication management module 14 to tenant carry out authentication when, can check the tenant account, Password and the identification card number etc. of submission;When tenant logs in, the tenant resource clothes that can be accessed can be provided on the page Business catalogue;System would generally create an access token, the SID (Security returned comprising logon process in the token Identifiers, secure identifier) and distribute to the privilege list of the tenant.Authentication management module 14 can be visited by managing It asks token, controls the data area that the tenant can access, and executable operation etc..The authentication management module 14 can be into One step improves the safety of system.
It should be noted that the embodiment of above-mentioned each system is described in a progressive manner, each embodiment weight Point explanation is all difference with other embodiment, and identical similar part is mutually referring to i.e. between each embodiment It can.
System is managed based on above-mentioned cloud data analysis service, embodiment further provides a kind of cloud data analyses to take for the disclosure The creation method of pragmatic example, this method comprises the following steps:
Step (1), API component 20 receive the request to create of cloud data analysis service, carry out inspection quota, initialized cryptographic Etc. preparations, then request to create is sent to task component 21 by RPC.
Step (2), task component 21 sends the request to create to resource management module 11 and corresponding host performance is wanted It asks.
Step (3) after resource management module 11 receives request, requires to judge to create the tool needed for example according to host performance Body Host Type;If the example needs physical host, currently available physical host is found by internal scheduling component, And the physical host is started by relevant computation module;If the example needs fictitious host computer, pass through virtualization Hypervisor is created and is started corresponding fictitious host computer.
After step (4), physical host or fictitious host computer start completion, resource management module 11 obtains physical host or virtual The nodal information of host carries out network of relation configuration according to the nodal information to host.
Step (5), resource management module 11 downloads the image file needed for example from mirror image management module 12, by the mirror As some Agent information and configuration information of file and example are injected into the physical host or fictitious host computer of startup.
Step (6), after host resource starts, above-mentioned Agent components 22 start automatically;Agent components 22 are to being stored in master Image file in machine resource is installed;After receiving the initialization requests of task component transmission, Agent components 22 are to peace Image file after dress carries out initialization process, and the example created at this time is updated to operating status.
The operating status is sent to aft-end assembly 23, rear end group by step (7), Agent components 22 by RPC message queues The corresponding conditional code of the operating status is updated in database by part 23, and cloud data analysis service example creates successfully.
In aforesaid way, system can dispose data analysis clothes automatically, in batches on fictitious host computer, physical host Business, and these data analysis services can be managed collectively, this data analysis service mixed deployment mode is more flexible.Meanwhile The operating status and performance indicator that user is serviced by user terminal real time inspection, it is more transparent in service management.
Embodiment further provides a kind of cloud data analysis service management method, this method to be applied to above-mentioned cloud number for the disclosure System is managed according to Analysis Service;As shown in fig. 6, this method comprises the following steps:
Step S602 receives the request to create of cloud data analysis service example, determines that example is corresponding according to request to create Host performance requirement;
Step S604, scheduling require the host resource to match with host performance;Wherein, which includes:Physics Host resource and fictitious host computer resource;
Step S606 obtains the image file needed for example;
Step S608 installs image file on host resource and generates example, operation and admin instance.
The above-mentioned cloud data analysis service management method that disclosure embodiment provides, service management module can pass through money Source control module is scheduled physical host resource and fictitious host computer resource, so that host resource and the master needed for establishment example Machine performance requirement matches;In which, the deployment way of cloud data analysis service is more flexible, makes high performance data analysis Service can also be realized by cloud computing, meet the high in the clouds deployment requirements of the data analysis service of different host performance requirements.
The above method further includes:In example operational process, the status information of example is acquired;Status information is preserved to rent The corresponding database in family.
The above method further includes:In example operational process, if receiving the service request of example, service request is selected Corresponding container computing resource carries out big data analysis processing using container computing resource to example.
The above method further includes:With the mutually isolated corresponding host resource of way to manage admin instance of multi-tenant, container Computing resource and database, to realize the security isolation between different tenant's corresponding instances.
The above method further includes:Authentication and access rights management are carried out to tenant.
Present embodiments provide for a kind of cloud servers corresponding with above system embodiment.Fig. 7 is the high in the clouds The structural schematic diagram of server, as shown in fig. 7, the cloud server includes processor 701 and memory 702;Wherein, memory 702 for storing one or more computer instruction, and one or more computer instruction is executed by processor, to realize above-mentioned cloud Data analysis service management method.
Cloud server shown in Fig. 7 further includes bus 703 and forwarding chip 704, processor 701,704 and of forwarding chip Memory 702 is connected by bus 703.The cloud server can be network edge device.
Wherein, memory 702 may include high-speed random access memory (RAM, Random Access Memory), May further include non-labile memory (non-volatile memory), for example, at least a magnetic disk storage.Bus 703 can be isa bus, pci bus or eisa bus etc..The bus can be divided into address bus, data/address bus, control always Line etc..For ease of indicating, only indicated with a four-headed arrow in Fig. 7, it is not intended that an only bus or a type of Bus.
Forwarding chip 704 will be sealed for being connect at least one user terminal and other network element by network interface The IPv4 messages or IPv6 messages installed is sent to the user terminal by network interface.
Processor 701 may be a kind of IC chip, the processing capacity with signal.It is above-mentioned during realization Each step of method can be completed by the integrated logic circuit of the hardware in processor 701 or the instruction of software form.On The processor 701 stated can be general processor, including central processing unit (Central Processing Unit, abbreviation CPU), network processing unit (Network Processor, abbreviation NP) etc.;It can also be digital signal processor (Digital Signal Processing, abbreviation DSP), application-specific integrated circuit (Application Specific Integrated Circuit, abbreviation ASIC), ready-made programmable gate array (Field-Programmable Gate Array, abbreviation FPGA) or Person other programmable logic device, discrete gate or transistor logic, discrete hardware components.It may be implemented or execute sheet Disclosed each method, step and logic diagram in invention embodiment.General processor can be microprocessor or this at It can also be any conventional processor etc. to manage device.The step of method in conjunction with disclosed in embodiment of the present invention, can direct body Be now that hardware decoding processor executes completion, or in decoding processor hardware and software module combine and execute completion.It is soft Part module can be located at random access memory, and flash memory, read-only memory, programmable read only memory or electrically erasable programmable are deposited In the storage medium of this fields such as reservoir, register maturation.The storage medium is located at memory 702, and processor 701 reads storage Information in device 702, in conjunction with its hardware complete aforementioned embodiments method the step of.
Embodiment of the present invention additionally provides a kind of machine readable storage medium, and machine readable storage medium storage is organic Device executable instruction, for the machine-executable instruction when being called and being executed by processor, machine-executable instruction promotes processor Realize that above-mentioned cloud data analysis service management method, specific implementation can be found in method embodiment, details are not described herein.
The cloud data analysis service management method and cloud server that embodiment of the present invention is provided, realization principle and The technique effect of generation is identical with aforementioned system embodiment, to briefly describe, method and cloud server embodiment part Place is not referred to, can refer to corresponding contents in aforementioned system embodiment.
In several embodiments provided herein, it should be understood that disclosed device and method can also lead to Other modes are crossed to realize.Device embodiments described above are only schematical, for example, the flow chart in attached drawing and Block diagram show the device of multiple embodiments according to the present invention, method and computer program product system in the cards Framework, function and operation.In this regard, each box in flowchart or block diagram can represent a module, program segment or generation A part for code, the part of the module, section or code include one or more for realizing defined logic function Executable instruction.It should also be noted that at some as in the realization method replaced, the function that is marked in box can also be with Occur different from the sequence marked in attached drawing.For example, two continuous boxes can essentially be basically executed in parallel, they Sometimes it can also execute in the opposite order, this is depended on the functions involved.It is also noted that block diagram and or flow chart In each box and the box in block diagram and or flow chart combination, the special of function or action as defined in executing can be used Hardware based system is realized, or can be realized using a combination of dedicated hardware and computer instructions.
In addition, each function module or unit in each embodiment of the present invention can integrate to form one solely Vertical part can also be modules individualism, can also two or more modules be integrated to form one it is independent Part.
It, can be with if the function is realized in the form of SFU software functional unit and when sold or used as an independent product It is stored in a computer read/write memory medium.Based on this understanding, the technical solution of the disclosure is substantially in other words The part of the part that contributes to existing technology or the technical solution can be expressed in the form of software products, the meter Calculation machine software product is stored in a storage medium, including some instructions are used so that a computer equipment (can be People's computer, server or network equipment etc.) execute all or part of step of each embodiment the method for the disclosure Suddenly.And storage medium above-mentioned includes:USB flash disk, read-only memory (ROM, Read-Only Memory), is deposited mobile hard disk at random The various media that can store program code such as access to memory (RAM, Random Access Memory), magnetic disc or CD.
Finally it should be noted that:Embodiment described above, the only specific implementation mode of the disclosure, to illustrate this public affairs The technical solution opened, rather than its limitations, the protection domain of the disclosure are not limited thereto, although with reference to aforementioned embodiments pair The disclosure is described in detail, it will be understood by those of ordinary skill in the art that:Any technology for being familiar with the art Personnel can still modify to the technical solution recorded in aforementioned embodiments in the technical scope that the disclosure discloses Or variation or equivalent replacement of some of the technical features can be readily occurred in;And these modifications, variation or replacement, The spirit and scope for disclosure embodiment technical solution that it does not separate the essence of the corresponding technical solution, should all cover in this public affairs Within the protection domain opened.Therefore, the protection domain of the disclosure shall be subject to the protection scope of the claim.

Claims (13)

1. a kind of cloud data analysis service manages system, which is characterized in that the system comprises:
Service management module, the request to create for receiving cloud data analysis service example determine institute according to the request to create State the corresponding host performance requirement of example;
Resource management module, for requiring the host to match to provide with the host performance for service management module scheduling Source, and obtain the image file needed for the example;Wherein, the host resource includes:Physical host resource and virtual master Machine resource;
Mirror image management module, for providing the image file needed for the example to the resource management module;
The service management module is additionally operable to install the image file generation example on the host resource, and operation is simultaneously Manage the example.
2. system according to claim 1, which is characterized in that the service management module includes:
API component, for providing interactive interface to tenant, the cloud data analysis that the tenant is received by the interactive interface takes The request to create of pragmatic example;
Task component, for dispatching the host resource, and the control resource management mould by the resource management module Block obtains the image file to mirror image management module, and the image file is stored in the host resource;
Agent components are all connected with the API component and task component, for installing the image file, to the institute after installation It states image file and carries out initialization process, obtain the example of the cloud data analysis service.
3. system according to claim 2, which is characterized in that the service management module further includes aft-end assembly;
The Agent components are additionally operable in the example operational process, acquire the status information of the example;
The aft-end assembly is connect with the Agent components, for the status information to be preserved number corresponding to the tenant According to library.
4. system according to claim 3, which is characterized in that pass through between the various components in the service management module MQ tunneling traffics.
5. system according to claim 1, which is characterized in that the service management module is additionally operable to:
In the example operational process, if receiving the service request of the example, selected by the resource management module The corresponding container computing resource of the service request is selected, the example is carried out at big data analysis using container computing resource Reason.
6. system according to claim 1, which is characterized in that the system also includes:Network management module, for more The mutually isolated way to manage of tenant manages the corresponding host resource of the example, container computing resource and database, to realize Security isolation between different tenant's corresponding instances.
7. system according to claim 1, which is characterized in that the system also includes:Authentication management module, for renting Family carries out authentication and access rights management.
8. a kind of cloud data analysis service management method, which is characterized in that the method is applied to any one of claim 1 to 7 The system;The method includes:
The request to create for receiving cloud data analysis service example, the corresponding host of the example is determined according to the request to create It can require;
Scheduling requires the host resource to match with the host performance;Wherein, the host resource includes:Physical host resource With fictitious host computer resource;
Obtain the image file needed for the example;
The image file is installed on the host resource and generates the example, runs and manages the example.
9. according to the method described in claim 8, it is characterized in that, the method further includes:
In the example operational process, the status information of the example is acquired;
The status information is preserved into database corresponding to the tenant.
10. according to the method described in claim 8, it is characterized in that, the method further includes:
In the example operational process, if receiving the service request of the example, select the service request corresponding Container computing resource carries out big data analysis processing using container computing resource to the example.
11. according to the method described in claim 8, it is characterized in that, the method further includes:
The corresponding host resource of the example, container computing resource and data are managed with the mutually isolated way to manage of multi-tenant Library, to realize the security isolation between different tenant's corresponding instances.
12. according to the method described in claim 8, it is characterized in that, the method further includes:To tenant carry out authentication and Access rights management.
13. a kind of cloud server, which is characterized in that including memory and processor, wherein the memory is for storing one Item or a plurality of computer instruction, one or more computer instruction are executed by the processor, with realize claim 8 to 12 any one of them methods.
CN201810536629.6A 2018-05-29 2018-05-29 Cloud data analysis service manages system, method and cloud server Pending CN108429768A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810536629.6A CN108429768A (en) 2018-05-29 2018-05-29 Cloud data analysis service manages system, method and cloud server

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810536629.6A CN108429768A (en) 2018-05-29 2018-05-29 Cloud data analysis service manages system, method and cloud server

Publications (1)

Publication Number Publication Date
CN108429768A true CN108429768A (en) 2018-08-21

Family

ID=63164590

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810536629.6A Pending CN108429768A (en) 2018-05-29 2018-05-29 Cloud data analysis service manages system, method and cloud server

Country Status (1)

Country Link
CN (1) CN108429768A (en)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109358965A (en) * 2018-09-25 2019-02-19 杭州朗和科技有限公司 Cloud computing cluster resource dispatching method, medium, device and calculating equipment
CN109731337A (en) * 2018-12-28 2019-05-10 北京金山安全软件有限公司 Method and device for creating special effect of particles in Unity, electronic equipment and storage medium
CN110740194A (en) * 2019-11-18 2020-01-31 南京航空航天大学 Micro-service combination method based on cloud edge fusion and application
CN110955492A (en) * 2019-11-11 2020-04-03 广州供电局有限公司 Service processing method and device based on container technology and computer equipment
CN110971391A (en) * 2018-09-30 2020-04-07 新华三技术有限公司合肥分公司 Message forwarding method and network equipment
CN111339564A (en) * 2020-03-27 2020-06-26 河北凯通信息技术服务有限公司 Cloud service analysis management system based on big data
CN112015698A (en) * 2020-08-26 2020-12-01 北京金山云网络技术有限公司 Method and device for cleaning mirror image data and electronic equipment
CN112187703A (en) * 2019-07-02 2021-01-05 北京百度网讯科技有限公司 Data processing method and device
CN113127137A (en) * 2019-12-30 2021-07-16 中标软件有限公司 Cloud computing management platform using self-hosting virtual machine and creation implementation method thereof
CN113900892A (en) * 2021-09-30 2022-01-07 北京青云科技股份有限公司 Cloud firewall system facing cloud platform and protection method
CN114780228A (en) * 2022-06-23 2022-07-22 浙江九州云信息科技有限公司 Hybrid cloud resource creation method and system
CN115499388A (en) * 2022-08-30 2022-12-20 阿里巴巴(中国)有限公司 Virtual host resource allocation method, device, equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103747107A (en) * 2014-01-27 2014-04-23 西安雷迪维护系统设备有限公司 Compatible cloud operating platform and realizing method thereof
CN104639594A (en) * 2013-11-15 2015-05-20 中国电信股份有限公司 System and method for allocating physical resources and virtual resources
CN108037978A (en) * 2017-12-22 2018-05-15 天津津航计算技术研究所 A kind of managing computing resources method based on virtualization technology

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104639594A (en) * 2013-11-15 2015-05-20 中国电信股份有限公司 System and method for allocating physical resources and virtual resources
CN103747107A (en) * 2014-01-27 2014-04-23 西安雷迪维护系统设备有限公司 Compatible cloud operating platform and realizing method thereof
CN108037978A (en) * 2017-12-22 2018-05-15 天津津航计算技术研究所 A kind of managing computing resources method based on virtualization technology

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
唐跃平 等: "《科技信息云服务及军事应用》", 31 January 2015 *
姚世军: "《Oracle 12c云数据库备份与恢复技术》", 31 January 2018 *

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109358965A (en) * 2018-09-25 2019-02-19 杭州朗和科技有限公司 Cloud computing cluster resource dispatching method, medium, device and calculating equipment
CN110971391A (en) * 2018-09-30 2020-04-07 新华三技术有限公司合肥分公司 Message forwarding method and network equipment
CN110971391B (en) * 2018-09-30 2022-03-11 新华三技术有限公司合肥分公司 Message forwarding method and network equipment
CN109731337A (en) * 2018-12-28 2019-05-10 北京金山安全软件有限公司 Method and device for creating special effect of particles in Unity, electronic equipment and storage medium
CN112187703A (en) * 2019-07-02 2021-01-05 北京百度网讯科技有限公司 Data processing method and device
CN110955492A (en) * 2019-11-11 2020-04-03 广州供电局有限公司 Service processing method and device based on container technology and computer equipment
CN110740194A (en) * 2019-11-18 2020-01-31 南京航空航天大学 Micro-service combination method based on cloud edge fusion and application
CN113127137A (en) * 2019-12-30 2021-07-16 中标软件有限公司 Cloud computing management platform using self-hosting virtual machine and creation implementation method thereof
CN111339564B (en) * 2020-03-27 2021-07-13 深圳市中投产业经济咨询有限公司 Cloud service analysis management system based on big data
CN111339564A (en) * 2020-03-27 2020-06-26 河北凯通信息技术服务有限公司 Cloud service analysis management system based on big data
CN112015698A (en) * 2020-08-26 2020-12-01 北京金山云网络技术有限公司 Method and device for cleaning mirror image data and electronic equipment
CN113900892A (en) * 2021-09-30 2022-01-07 北京青云科技股份有限公司 Cloud firewall system facing cloud platform and protection method
CN114780228A (en) * 2022-06-23 2022-07-22 浙江九州云信息科技有限公司 Hybrid cloud resource creation method and system
CN115499388A (en) * 2022-08-30 2022-12-20 阿里巴巴(中国)有限公司 Virtual host resource allocation method, device, equipment and storage medium

Similar Documents

Publication Publication Date Title
CN108429768A (en) Cloud data analysis service manages system, method and cloud server
US9450783B2 (en) Abstracting cloud management
US9602335B2 (en) Independent network interfaces for virtual network environments
US8800055B2 (en) Node controller for an endpoint in a cloud computing environment
US8769083B2 (en) Metering software infrastructure in a cloud computing environment
CN104813289B (en) Diagnose virtual machine
US9317319B2 (en) Monitoring and dynamically reconfiguring virtual machine patterns
CN110063051A (en) Reconfigurable server
US8595328B2 (en) Self-updating node controller for an endpoint in a cloud computing environment
US9038063B2 (en) Determining virtual machine image pattern distributions in a networked computing environment
US20100306767A1 (en) Methods and systems for automated scaling of cloud computing systems
US20110055396A1 (en) Methods and systems for abstracting cloud management to allow communication between independently controlled clouds
US9710292B2 (en) Allowing management of a virtual machine by multiple cloud providers
US9785475B2 (en) Managing a shared pool of configurable computing resources using a set of scaling factors and a set of workload resource data
WO2015034484A1 (en) Providing recursively-generated instantiated computing resource in a multi-tenant environment
US11520609B2 (en) Template-based software discovery and management in virtual desktop infrastructure (VDI) environments
US8825862B2 (en) Optimization of resource provisioning in a networked computing environment
US10176017B2 (en) Configuration management for a shared pool of configurable computing resources
CN111625346A (en) High performance computing infrastructure as a service
DE112021006372T5 (en) SECURE PROVISION OF A DATA PROCESSING RESOURCE USING HOMOMORPHOUS ENCRYPTION
Venâncio et al. Beyond VNFM: Filling the gaps of the ETSI VNF manager to fully support VNF life cycle operations
CN114489954A (en) Tenant creation method based on virtualization platform, tenant access method and equipment
Dittakavi IAAS CLOUD ARCHITECTURE DISTRIBUTED CLOUD INFRA STRUCTURES AND VIRTUALIZED DATA CENTERS
AU2020426828B2 (en) Reserving one or more security modules for secure guest
US10291488B1 (en) Workload management in multi cloud environment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20180821

RJ01 Rejection of invention patent application after publication