CN111258764A - Method and system for providing multi-tenant persistent task records for data center - Google Patents
Method and system for providing multi-tenant persistent task records for data center Download PDFInfo
- Publication number
- CN111258764A CN111258764A CN202010047960.9A CN202010047960A CN111258764A CN 111258764 A CN111258764 A CN 111258764A CN 202010047960 A CN202010047960 A CN 202010047960A CN 111258764 A CN111258764 A CN 111258764A
- Authority
- CN
- China
- Prior art keywords
- user
- request
- task
- cluster
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000002085 persistent effect Effects 0.000 title claims abstract description 48
- 238000000034 method Methods 0.000 title claims abstract description 18
- 238000013475 authorization Methods 0.000 claims description 8
- 235000014510 cooky Nutrition 0.000 claims description 4
- 230000001052 transient effect Effects 0.000 claims description 2
- 230000002688 persistence Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/604—Tools and structures for managing or administering access control systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2141—Access rights, e.g. capability lists, access control lists, access tables, access matrices
Abstract
The invention discloses a method and a system for providing multi-tenant persistent task records for a data center, and relates to the technical field of data access and storage; the cluster agent server receives a request of a user for information related to the task and judges whether the request is related to the task processed or operated on a terminated cluster; if so, the user's request is directed to a corresponding history server, which provides the user with task history and/or log file information of the information requested by the user about the task by accessing the persistent storage device.
Description
Technical Field
The invention discloses a method and a system, relates to the technical field of data access and storage, and particularly relates to a method and a system for providing a multi-tenant persistent task record for a data center.
Background
In general, the data processing center can Reduce the cost through automatic scaling, for example, the cluster can manage and run tasks such as Spark, Tez, Map-Reduce, Presto-Query, etc. through Yarn or Presto, and automatically shut down after a period of inactivity according to workload demand and cost considerations. The problem that errors cannot be easily checked according to the logs is generated while the cost is saved, for example, a running Hadoop cluster is closed because of non-working within a period of time, and the JobHistory Server may not run any more, so that a user cannot obtain logs which can help to debug the logs.
Disclosure of Invention
The invention provides a method and a system for providing multi-tenant persistent task records for a data center, aiming at the problems of the prior art, the method and the system can still inquire logs or historical records under the condition that the data center is automatically closed, and under some conditions, the system comprises persistent storage equipment (Persistence) which can be configured to store task records, configuration and cluster log files; a History Server (terminatejob History Server) configured to respond to a log or a task ending of a History request; and a Cluster Proxy server (Cluster Proxy) providing a Proxy layer that can forward log requests.
The specific scheme provided by the invention is as follows:
a system for providing multi-tenant persistent task records for a data center comprises a persistent storage device, a history record server and a cluster proxy server,
the persistent storage device persistently stores task history and/or log files associated with the ephemeral run cluster,
the cluster proxy server receives a request of a user for information related to a task, and judges whether the request is related to a task processed or operated on a terminated cluster; if so, the user's request is directed to the corresponding history server,
the history server provides the user with task history and/or log file information of the user-requested information related to the task by accessing the persistent storage device.
In the system, before the cluster proxy server directs the user request to the corresponding history record server, authorization operation is executed on the user request.
The cluster proxy server in the system completes the authorization of the user request by comparing the server host name information attached to the user request with the cluster node information.
The cluster proxy server in the system executes authentication operation to authorized user request, and then directs the user request to the corresponding history record server.
In the system, the cluster proxy server determines whether the request of the user comes from an authorized user through cookies.
The cluster proxy server in the system is provided with a graphical user interface.
The cluster proxy servers in the system provide an interface layer to direct requests of users related to tasks processed or run on the terminated cluster to the corresponding history server.
The cluster proxy server in the described system parses any links used by the historian server, confirms that the links are in a usable format and references reference data stored in the persistent storage device.
A method for providing multi-tenant persistent task records for a data center: the task history and/or log file associated with the ephemeral run cluster is saved in persistent storage,
the cluster proxy server receives a request of a user for information related to a task, and judges whether the request is related to a task processed or operated on a terminated cluster; if so, the user's request is directed to the corresponding history server,
the history server provides the user with task history and/or log file information of the user-requested information related to the task by accessing the persistent storage device.
In the method, a cluster proxy server is utilized to direct a user request to a corresponding history record server and execute authorization operation;
and performing an authentication operation on the authorized user request.
The invention has the advantages that:
the invention provides a system for providing multi-tenant persistent task records for a data center, which comprises persistent storage equipment, a history record server and a cluster proxy server, wherein task history and/or log files associated with a transient running cluster are persistently stored by the persistent storage equipment; if the log file is relevant to the large data processing center, the request of the user is directed to the corresponding history record server, the history record server provides the task history and/or the log file information of the information relevant to the task requested by the user to the user through accessing the persistent storage device, the log or the history record can still be inquired under the condition that the data center is closed, the problem that the log file and the task history record are easy to store untimely when the user effectively accesses the large data processing center with automatic zooming is solved, and the user can conveniently debug the jobs executed in the past.
Drawings
FIG. 1 is a schematic diagram of a system application framework of the present invention;
FIG. 2 is a schematic flow diagram of the process of the present invention.
Detailed Description
The present invention is further described below in conjunction with the following figures and specific examples so that those skilled in the art may better understand the present invention and practice it, but the examples are not intended to limit the present invention.
The invention provides a system for providing multi-tenant persistent task records for a data center, which comprises persistent storage equipment, a history record server and a cluster proxy server,
the persistent storage device persistently stores task history and/or log files associated with the ephemeral run cluster,
the cluster proxy server receives a request of a user for information related to a task, and judges whether the request is related to a task processed or operated on a terminated cluster; if so, the user's request is directed to the corresponding history server,
the history server provides the user with task history and/or log file information of the user-requested information related to the task by accessing the persistent storage device.
In specific implementation, when the system of the present invention is used to perform data center multi-tenant persistent task recording, the system may be directed to one or more running clusters, a persistent storage device (Persistence) of the system may be a cloud storage device, such as Amazon S3 or other devices capable of performing persistent storage, a history server (terminated jobhistoryserver) of the system is abbreviated as TJHS, a Cluster proxy server (Cluster proxy) of the system, there is a proxy layer and a corresponding interface layer, the interface layer may redirect a request to a correct server based on a specific job and a Cluster, and the Cluster proxy server may have a Graphical User Interface (GUI), and may parse any link used by the history server, confirm that the link is in an available format, and refer to reference data stored in the persistent storage device.
During operation, the Cluster Proxy receives requests for logs and task records from users, which may be sent by a web server,
the Cluster Proxy firstly authenticates and authorizes the request of the user by inquiring a corresponding database, and then sends the request to the Cluster after the authentication is passed, and judges whether the request is related to the task processed or operated on the terminated Cluster; if so, a request is sent to the TJHS to obtain the log and task records for stopping the Cluster, the TJHS retrieves the log and task records from the persistent storage device at the moment to respond to the request from the Cluster Proxy, and the running Cluster periodically persists the log and task records to the persistent storage device.
In the above process, the authorization of the request of the user by the Cluster Proxy is completed by comparing the host name information attached to the request with the node information stored in the database, and the authentication process is based on cookies to determine whether the request is from an authorized user, meanwhile, the database can also record the state of the deployed machines and the state of the cluster to which each machine belongs, if the host name information corresponds to an active Cluster, the Cluster Proxy may route the user's request to the corresponding storage device of the Cluster, if the hostname information corresponds to the terminated Cluster, the Cluster Proxy retrieves the TJHS from the persistent storage location and the required credential information, determines if the request is related to a task processed or run on the terminated Cluster, and routes the user's request to the TJHS to obtain a log of stopped clusters and task records.
In the system of the invention, the form of the persistent storage device is not limited, for example, JobHistoryServer of Map-Reduce stores the history records and the configuration files in the HDFS, and the history records and the configuration files are configured by the Map Reduce. Similarly, after the log integration function is started, Yarn stores the log of the container on the HDFS. In the case of other Yarn applications, such as Tez, the task records and associated configuration may be stored in an embedded database, such as leveldb, configured in Yarn. To enable these task logging, configuration, and availability even after cluster shutdown, this information may be stored in some storage device, such as Amazon S3. We can configure the above attributes as the user's storage device address.
The TJHS is also persistent, multi-tenant, and system-wide, and may maintain job records for different users/clients in various systems and clusters, and for each job there may be one Map-Reduce TJHS, one Spark TJHS, or one application-terminating timeline server, etc., responding to requests from different users with different storage addresses and credentials. For example, a standard Hadoop task record server can be used as the TJHS for Map-Reduce and can become multi-tenant for different values in response to different requests by expanding it to accept yarn. A same historian server, which can also act as a timeline server for an end application, can be made multi-tenant, e.g., leveldb, by extending it to accept similar parameters at different values. Thus, the TJHS can now run as an internal service at a large data processing center.
The invention also provides a method for providing the multi-tenant persistent task record for the data center, which corresponds to the system, and the method comprises the following steps: the task history and/or log file associated with the ephemeral run cluster is saved in persistent storage,
the cluster proxy server receives a request of a user for information related to a task, and judges whether the request is related to a task processed or operated on a terminated cluster; if so, the user's request is directed to the corresponding history server,
the history server provides the user with task history and/or log file information of the user-requested information related to the task by accessing the persistent storage device.
Data center multi-tenant persistent task recording is still performed by using a persistent storage device (Persistence), a history server (terminatejobhistoryserver), TJHS (short for TJHS), and a Cluster Proxy server (Cluster Proxy), wherein the Cluster Proxy receives a request for logs and task records from a user, the request can be sent by a web server,
the Cluster Proxy firstly authenticates and authorizes the request of the user by inquiring a corresponding database, and then sends the request to the Cluster after the authentication is passed, and judges whether the request is related to the task processed or operated on the terminated Cluster; if so, a request is sent to the TJHS to obtain the log and task records for stopping the cluster, the TJHS retrieves the log and task records from the persistent storage device at the moment to respond to the request from the Clusterproxy, and the running cluster periodically persists the log and task records to the persistent storage device.
In the above process, the authorization of the request of the user by the Cluster Proxy is completed by comparing the host name information attached to the request with the node information stored in the database, and the authentication process is based on cookies to determine whether the request is from an authorized user, meanwhile, the database can also record the state of the deployed machines and the state of the cluster to which each machine belongs, if the host name information corresponds to an active Cluster, the Cluster Proxy may route the user's request to the corresponding storage device of the Cluster, if the hostname information corresponds to the terminated Cluster, the Cluster Proxy retrieves the TJHS from the persistent storage location and the required credential information, determines if the request is related to a task processed or run on the terminated Cluster, and routes the user's request to the TJHS to obtain a log of stopped clusters and task records.
The above-mentioned embodiments are merely preferred embodiments for fully illustrating the present invention, and the scope of the present invention is not limited thereto. The equivalent substitution or change made by the technical personnel in the technical field on the basis of the invention is all within the protection scope of the invention. The protection scope of the invention is subject to the claims.
Claims (10)
1. A system for providing multi-tenant persistent task records for a data center is characterized by comprising persistent storage equipment, a history record server and a cluster proxy server,
the persistent storage device persistently stores task history and/or log files associated with the ephemeral run cluster,
the cluster proxy server receives a request of a user for information related to a task, and judges whether the request is related to a task processed or operated on a terminated cluster; if so, the user's request is directed to the corresponding history server,
the history server provides the user with task history and/or log file information of the user-requested information related to the task by accessing the persistent storage device.
2. The system of claim 1, wherein the cluster proxy server performs authorization operations on the user's request before directing the user's request to the corresponding history server.
3. The system of claim 2, wherein the cluster proxy server performs authorization of the user request by comparing server hostname information associated with the user request with cluster node information.
4. A system as claimed in claim 2 or 3, wherein the cluster proxy server performs authentication operations on authorised user requests and directs the user requests to the respective history server.
5. The system of claim 4, wherein the clustered proxy server determines whether the user's request is from an authorized user via cookies.
6. The system according to claim 1 or 5, characterized in that the cluster proxy server has a graphical user interface.
7. The system of claim 6, wherein the cluster proxy servers provide an interface layer to direct requests from users associated with tasks processed or run on the terminated cluster to the corresponding history server.
8. The system of claim 1 or 7, wherein the cluster proxy server parses any links used by the historian server, confirms that the links are in a usable format and references reference data stored in the persistent storage device.
9. A method for providing multi-tenant persistent task records for a data center is characterized in that task history and/or log files related to transient operation clusters are saved in a persistent storage device,
the cluster proxy server receives a request of a user for information related to a task, and judges whether the request is related to a task processed or operated on a terminated cluster; if so, the user's request is directed to the corresponding history server,
the history server provides the user with task history and/or log file information of the user-requested information related to the task by accessing the persistent storage device.
10. The method of claim 9, wherein the authorization is performed before the request of the user is directed to the corresponding history server using the cluster proxy server;
and performing an authentication operation on the authorized user request.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010047960.9A CN111258764A (en) | 2020-01-16 | 2020-01-16 | Method and system for providing multi-tenant persistent task records for data center |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010047960.9A CN111258764A (en) | 2020-01-16 | 2020-01-16 | Method and system for providing multi-tenant persistent task records for data center |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111258764A true CN111258764A (en) | 2020-06-09 |
Family
ID=70952172
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010047960.9A Pending CN111258764A (en) | 2020-01-16 | 2020-01-16 | Method and system for providing multi-tenant persistent task records for data center |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111258764A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114143321A (en) * | 2021-11-26 | 2022-03-04 | 中国电信集团系统集成有限责任公司 | Multi-tenant application configuration distribution system based on cross-IDC environment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170054590A1 (en) * | 2015-08-21 | 2017-02-23 | Rohit Agarwal | Multi-Tenant Persistent Job History Service for Data Processing Centers |
| CN109669821A (en) * | 2018-11-16 | 2019-04-23 | 深圳证券交易所 | Cluster partial fault restoration methods, server and the storage medium of message-oriented middleware |
-
2020
- 2020-01-16 CN CN202010047960.9A patent/CN111258764A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170054590A1 (en) * | 2015-08-21 | 2017-02-23 | Rohit Agarwal | Multi-Tenant Persistent Job History Service for Data Processing Centers |
| CN109669821A (en) * | 2018-11-16 | 2019-04-23 | 深圳证券交易所 | Cluster partial fault restoration methods, server and the storage medium of message-oriented middleware |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114143321A (en) * | 2021-11-26 | 2022-03-04 | 中国电信集团系统集成有限责任公司 | Multi-tenant application configuration distribution system based on cross-IDC environment |
| CN114143321B (en) * | 2021-11-26 | 2023-08-25 | 中电信数智科技有限公司 | Multi-tenant application configuration distribution system based on cross-IDC environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113169952B (en) | Container cloud management system based on block chain technology | |
| US10296440B2 (en) | Multi-tenant aware debugging methods and systems | |
| US8880694B2 (en) | Server system and control method for same | |
| CN110647580A (en) | Distributed container cluster mirror image management main node, slave node, system and method | |
| US9176829B2 (en) | Managing recovery virtual machines in clustered environment | |
| US10798218B2 (en) | Environment isolation method and device | |
| US20070244999A1 (en) | Method, apparatus, and computer product for updating software | |
| CN107005426B (en) | Method and device for managing life cycle of virtual network function | |
| JP2008517358A (en) | Apparatus, system, and method for facilitating storage management | |
| US8863278B2 (en) | Grid security intrusion detection configuration mechanism | |
| WO2019210580A1 (en) | Access request processing method, apparatus, computer device, and storage medium | |
| KR20090085058A (en) | Message forwarding backup manager in a distributed server system | |
| CN111625496A (en) | Method, device and equipment for deploying distributed file system in virtual machine environment | |
| US5857076A (en) | Program product for obtaining the state of network resources in A distributed computing environment | |
| CN111625497A (en) | Deployment method, device, equipment and storage medium of distributed file system | |
| US20170054590A1 (en) | Multi-Tenant Persistent Job History Service for Data Processing Centers | |
| CN113505027B (en) | Business system backup method, device, equipment and storage medium | |
| WO2020252724A1 (en) | Log processing method and device, and computer-readable storage medium | |
| CN113946408A (en) | Cloud native edge container control method and system and storage medium | |
| CN111258764A (en) | Method and system for providing multi-tenant persistent task records for data center | |
| WO2021121067A1 (en) | Task execution method and apparatus | |
| US20130007508A1 (en) | Managing storage providers in a clustered appliance environment | |
| US20060129685A1 (en) | Authenticating a node requesting another node to perform work on behalf of yet another node | |
| US20150244780A1 (en) | System, method and computing apparatus to manage process in cloud infrastructure | |
| CN111884837B (en) | Migration method and device of virtual encryption machine and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200609 |
|
| RJ01 | Rejection of invention patent application after publication |