CN111192123A - Account checking system based on hybrid cloud computing platform and account checking method thereof - Google Patents
Account checking system based on hybrid cloud computing platform and account checking method thereof Download PDFInfo
- Publication number
- CN111192123A CN111192123A CN201911359843.XA CN201911359843A CN111192123A CN 111192123 A CN111192123 A CN 111192123A CN 201911359843 A CN201911359843 A CN 201911359843A CN 111192123 A CN111192123 A CN 111192123A
- Authority
- CN
- China
- Prior art keywords
- task
- data
- bill
- computing
- reconciliation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/12—Accounting
- G06Q40/125—Finance or payroll
Abstract
The invention provides a reconciliation system based on a hybrid cloud computing platform and a reconciliation method thereof.A bill data processing module processes data from a public cloud bill API or a downloaded bill detailed file; the monitoring data processing module is used for processing data of the resource pool and the task resource request collected from the resource pool of the computing management system; the task data processing module is used for processing the time when the resource core is consumed by the statistical task in the task database of the computing scheduling system; and the statistical alarm module is used for obtaining the utilization rate and the difference range of the calculation task boxing by combining the resource data tables obtained by the three modules, and realizing account checking alarm and prediction during task checking. The method realizes the management and control of unit price of the global computing instance by integrating the bill API of each large public cloud manufacturer; checking and accounting aiming at the computing power cost of the high-performance computing task, and detecting and judging the utilization rate of the computing power; the purchase of future computing power can be guided by the computing power requirements of a large number of existing tasks.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to an account checking system based on a hybrid cloud computing platform and an account checking method thereof.
Background
Today, enterprise services are hosted on public clouds, and it is becoming more common to purchase a large amount of computing resources through cloud service providers instead of building computing servers locally by self-construction. When a single cloud service provider is directly connected, the monthly use cost can be directly obtained through a corresponding cloud service provider bill system, and the use amount and cost of each service and the use amount and cost of a computing pool can be calculated through manually downloading a bill detail file. This technique has the following problems:
(1) the account checking is strongly bound with the architecture of a single cloud service provider, and cross-cloud and local computing need to be separately and manually accounted, so that the cost evaluation difficulty of different cloud service providers is increased.
(2) Data structures and service names of bill products of different cloud service providers are not unified, so that a user is difficult to aggregate calculation results in unified accounting, and historical data of multi-cloud services cannot be analyzed and processed.
(3) Due to the problem of idle running caused by resource boxing and elastic expansion of the computing task scheduling system, certain loss exists in the use of actual computing resources, the computing resources required to be applied for the operation of the actual computing tasks cannot be obtained, and cost control corresponding to the actual tasks is lacked.
Disclosure of Invention
Aiming at the technical problems, the invention provides an account checking system based on a hybrid cloud computing platform and an account checking method thereof, which are suitable for the field of high-performance computing, and are combined with a monitoring and warning system to realize computing resource cost accounting and actual utilization rate statistics of a multi-cloud service provider.
The adopted technical scheme is as follows:
a reconciliation system based on a hybrid cloud computing platform, comprising:
the bill data processing module can directly obtain the bill through an API (application programming interface) interface, or manually download the bill detailed file, or obtain the detailed data through object storage stored in the bill file according to the difference of the cloud service provider and the support of the bill interface. Because the computing instance types of each cloud provider are different, the number of the virtual cores is various, the bill time zones are different, and the total cost, the cost of each instance, the cost of unit core time, the total core time and the like need to be counted according to the settlement period of each cloud service provider;
and the task data processing module and a scheduling system of the computing platform can start, run, complete and other state changes according to the actual task, and record data of parameters such as the life cycle of the task, CPU consumption, memory consumption and the like in a database. Through the analysis and processing of the task database, the occupation duration of the scheduled calculation example when the consumption core of each calculation task is obtained through statistics;
and a monitoring data processing module, wherein a monitoring system of the computing platform acquires CPU information of each computing cluster, the CPU information comprises the total CPU number, the CPU number used by the task request and the CPU number actually used by the task, and time-series CPU number data is formed in a database. Through the analysis and processing of the time sequence database, the comparison effect on the bill data and the task data can be obtained when the core is consumed by the computing cluster and when the core is actually used by the task.
The statistical warning module is used for analyzing and integrating the processing data of the three modules, acquiring a three-party periodic check comparison table, acquiring a corresponding relation between a bill data calculation example and a task data calculation example based on a calculation example id, and checking the account correctly when the utilization rate meets a standard and is consistent with a monitoring data error; otherwise, alarming the abnormal account checking place. And based on historical reconciliation data, predicting the consumption of the core time, the cost of the unit core time and the like of the next reconciliation period.
The reconciliation method of the reconciliation system based on the hybrid cloud computing platform is specifically characterized in that four modules work in a coordinated manner, and the following steps of each module are described:
(1) the bill data processing module is used for firstly counting the access of an interface layer to a cloud provider, and generally triggering the account checking system after monthly detailed bills of all cloud manufacturers are generated in order to keep uniformity; secondly, whether all the supported detailed bills of the cloud manufacturers are complete or not is checked, and data information such as the total cloud computing cost, the instance cost and the total consumed core time is obtained by using a uniform interface layer; finally, persisting to a database for storage;
(2) the task data processing module can run in real time and can also be synchronously carried out by triggering the account checking system. Firstly, checking whether the data of the period is recorded in a database, if not, acquiring data such as start-stop time of task operation, core number, operation calculation examples and the like from life cycle data of calculation tasks of a scheduling system, thereby acquiring consumption cores of each calculation task as cost information, acquiring a time consumption period of the calculation example where the calculation task is located, and comparing the time consumption period with bill processing data;
(3) the monitoring data processing module can run in real time and can also be synchronously operated by triggering the account checking system. Firstly, checking whether the cycle data is processed or not, if not, acquiring the CPU core number of the public cloud resource pool at the sequential time point and the CPU core number actually applied by the calculation task from an acquisition time sequence database of the alarm system, thereby acquiring the core time consumption of the cycle time as comparison data of the bill cycle core time and the task cycle core time;
(4) the statistical warning module can obtain a three-party comparison table of the data when the periodic account is checked and the cell core is monitored, the data when the task is checked and the data when the task is monitored, and the data when the account is checked and the data when the task is checked; based on the calculation instance id, obtaining a matched calculation instance id data table from a bill processing database and a task processing database; therefore, the utilization rate and the difference range of the calculation task boxing are efficiently and accurately obtained, and whether the data of which party has a problem is determined. And making an aggregated reconciliation view based on the data tables, wherein the consumption of the core time, the unit cost and the utilization efficiency of the core time at the cloud provider level can be seen, and the consumption of the core time and the budget of the next period can be estimated.
The account checking method of the account checking system based on the hybrid cloud computing platform has the following technical effects:
(1) by integrating the bill API and the bill detail data of each great public cloud manufacturer, the difference of each public cloud is further eliminated, and the bill data in a standard format is provided; the data view can be output, so that a user can better know the resource use condition of the computing task;
(2) through a series of data processing modules, the data can be analyzed and processed, the calculation power cost and the use condition of the calculation power utilization rate of an actual calculation task can be known, and the cost management of the calculation task can be guided;
(3) by introducing the monitoring data, any party of the three-party account checking is ensured to have data problems, and the data problems can be found in time.
Drawings
FIG. 1 is a schematic diagram of the reconciliation method of the present invention;
FIG. 2 is a schematic diagram of the reconciliation system architecture of the present invention;
FIG. 3 is a flow diagram of a reconciliation system of the present invention;
fig. 4 is a schematic view of the structure of the present invention in use, in connection with the outside.
Detailed Description
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
The account checking of computing resources is accurate and efficient, the account checking mode is flexible, the operation is simple and automatic, the account checking period can be flexibly set, different time zones are supported to be set, account checking can be carried out according to the month and season, the day and the day according to the support of cloud manufacturers, and the account checking can be carried out according to different resource types; the account checking result can have guiding significance on the application of the computing resources in the next period, the abnormal account checking is alarmed, and abnormal data records can be checked.
The first embodiment is as follows:
the invention provides an account checking method, and fig. 1 shows a schematic diagram of the account checking method of the invention:
specifically, after the statement bill is issued in the reconciliation period, the calculation amount core time can be calculated through the calculation example core number and the operation time interval of each public cloud platform; the number of the pool cores is calculated through the time sequence of the monitoring data, and the time of applying for calculating the quantity core in the period can be calculated; and determining whether the error range exceeds a threshold value or not by monitoring account checking between the time of applying for the core and the time of checking the account list. The actual calculation amount of the period can be calculated according to the task request core number and the operation life cycle of the task data; the actual used calculation amount core time of the period can be calculated by monitoring the time sequence task core number of the data; and determining whether the error range exceeds a threshold value or not by monitoring the reconciliation between the actual use of the cores and the task cores.
The monitoring data has certain bias relative to the real data, and approximately equal relation exists between the utilization rate calculated when the task cores are monitored and the application cores are monitored and the utilization rate calculated when the task cores are operated and the bill cores are monitored. And determining whether the difference range exceeds a threshold value or not through the difference of the utilization rates of the two.
And the account checking condition of the computing resources can be obtained relatively quickly through the three-party account checking.
Example two:
the invention also provides a reconciliation system, and fig. 2 shows a structural schematic diagram of a reconciliation system in the second embodiment of the invention, which comprises a bill data processing module, a task data processing module, a monitoring data processing module and a statistic warning module. In the high-performance hybrid cloud computing oriented technology, the bill data processing module has the following attributes after bill data processing:
according to the running interval and unit price of the calculation example, the cost of the calculation example can be calculated; according to the calculation example running time interval and the calculation example core number, the calculation amount consumption core time is the core number multiplied by the time (hour); namely, the cost of the computing resources of the starting time period and the ending time period and the computing amount in the checking time period can be counted; and calculating the instance ID as the reconciliation identification code of the use flow of the calculated amount.
The computing examples are various and comprise bidding examples, on-demand examples, annual fee examples and the like, the physical cores corresponding to the examples of different models are different, 1u of each public cloud is taken as a core, the core is a standard metering unit of the calculated amount, and the core unit price is the cost of the core. According to different bill data input forms, the bill processing module can call a bill API of a public cloud service provider periodically to process bill data; or periodically processed from a ledger file stored in an object store such as s 3; the corresponding monthly bill detail file can also be downloaded to manually trigger the bill processing, and the processed data is dumped in the database.
The task data processing module acquires relevant data of task operation from state time points fed back from task starting to task ending, and the processed and counted records have the following attributes:
and calculating the calculation amount during the checking through the task application checking number and the total running time. Calculating task cost range is estimated according to the unit price during the kernel, and the task cost range corresponds to a task submitting user; and calculating the instance ID as the reconciliation identification code of the use flow of the calculated amount.
The monitoring data processing module regularly obtains the time series core number of the real-time calculation resource pool and the time series core number of the real-time operation container from the monitoring acquisition database, and calculates the actual task core number of the container.
| Public cloud provider | Number of resource pool cores | Number of task cores | Point in time |
According to different time point data of the resource pool core number and the task core number, the calculation amount of the resource pool core time and the task core time can be estimated. And according to the historical fluctuation data of the resource pool, the total computation data required in a short time can be predicted, and the purchase limit for the public cloud computing instance in the next period can be conveniently predicted.
| Public cloud provider | Number of resource pool cores | Number of task cores | Point in time |
| Cloud A | 50000 | 49000 | 1572248260 |
| B cloud | 10000 | 9200 | 1572248250 |
| C cloud | 3000 | 2800 | 1572248240 |
Finally, as shown in fig. 3, a detail reconciliation table and a three-party core-time reconciliation table with the calculation instance ID as the reconciliation identifier are generated by the statistics alarm module. Due to the scheduling delay of the calculation tasks and the matching of the packing, the actual use calculation amount of the calculation tasks and the calculation amount of the applied calculation examples have certain difference, and the core time utilization rate of the calculation examples is taken as the calculation flow data.
Whether data are wrong when one core exists is judged according to whether the difference value between the bill core time and the resource pool monitoring time and the difference value between the task core time and the task container monitoring time exceeds a preset threshold value of 2%. If the difference value is too large, a detailed data comparison table needs to be generated, and abnormal time point data is compared; if the calculation amount is within the normal range, a calculation amount prediction table for the calculation instance purchase of the next period is generated. And then obtaining the utilization rate of task boxing according to the ratio of the task core time to the account list core time, judging whether the utilization rate is consistent with the core time utilization rate between the monitoring core times, and taking the core time utilization rate of the calculation example as a review account checking. Through the account checking system, the actual use cost and the actual utilization efficiency of the calculation task finally reach preset values and are counted in mind.
By using the method and the system, a host and a database can be applied on the cloud and relevant operation authority is attached, and the operation can be carried out, as shown in figure 4. When the user submits the account checking application, the account checking report can be formed through the database generated by the account checking system.
Claims (2)
1. A reconciliation system based on a hybrid cloud computing platform, comprising:
the bill data processing module is used for processing data from a public cloud bill API or a downloaded bill detailed file;
the monitoring data processing module is used for acquiring task request resources from a monitoring system of the computing platform and processing data;
the task data processing module is used for acquiring and processing task data from a scheduling system task database of the computing platform;
the statistic warning module is used for obtaining a data comparison table from bill monitoring and task data and warning a user according to the reconciliation difference threshold value; and to guide future cost management using historical reconciliation data.
2. The reconciliation method for the reconciliation system based on the hybrid cloud computing platform as claimed in claim 1, comprising the following steps:
(1) the bill data processing module performs data processing according to the detailed account of the public cloud service provider to generate bill calculation instance cost calculation force data and to-be-checked system subsequent flow processing;
(2) the task data processing module acquires the start-stop time of task operation and the application core number data from a life cycle database of the computing tasks stored in the computing platform scheduling system, so as to acquire the consumption core time of each computing task as cost information;
(3) the monitoring data processing module acquires data from a database acquired by a monitoring system of the computing platform, wherein the acquired data comprises the CPU core number of each time point of each public cloud resource pool and the CPU core number actually applied by the computing task at each time point; acquiring the public cloud computing cluster monitoring core and the task request monitoring core as comparison group data according to the task monitoring data;
(4) the statistical warning module obtains the utilization rate and the difference range of the calculation task boxing according to the obtained resource data table in the steps (1) to (3), and the three-party account checking during the task checking is realized; and alarming the abnormal reconciliation data, accumulating the historical reconciliation data, and predicting and guiding the management of the calculation cost.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911359843.XA CN111192123A (en) | 2019-12-25 | 2019-12-25 | Account checking system based on hybrid cloud computing platform and account checking method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911359843.XA CN111192123A (en) | 2019-12-25 | 2019-12-25 | Account checking system based on hybrid cloud computing platform and account checking method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111192123A true CN111192123A (en) | 2020-05-22 |
Family
ID=70707633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911359843.XA Pending CN111192123A (en) | 2019-12-25 | 2019-12-25 | Account checking system based on hybrid cloud computing platform and account checking method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111192123A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112422299A (en) * | 2020-11-18 | 2021-02-26 | 杭州飞致云信息科技有限公司 | Method and device for analyzing public cloud charging data and computer readable storage medium |
| CN113239243A (en) * | 2021-07-08 | 2021-08-10 | 湖南星汉数智科技有限公司 | Graph data analysis method and device based on multiple computing platforms and computer equipment |
| CN116205614A (en) * | 2023-05-04 | 2023-06-02 | 无锡锡商银行股份有限公司 | Business reconciliation management system and method for distributed scene |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050216396A1 (en) * | 2004-03-25 | 2005-09-29 | International Business Machines Corporation | Automatic billing event submission reconciliation for on demand systems |
| CN110262998A (en) * | 2019-05-14 | 2019-09-20 | 阿里巴巴集团控股有限公司 | A kind of reconciliation data processing method and device |
-
2019
- 2019-12-25 CN CN201911359843.XA patent/CN111192123A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050216396A1 (en) * | 2004-03-25 | 2005-09-29 | International Business Machines Corporation | Automatic billing event submission reconciliation for on demand systems |
| CN110262998A (en) * | 2019-05-14 | 2019-09-20 | 阿里巴巴集团控股有限公司 | A kind of reconciliation data processing method and device |
Non-Patent Citations (1)
| Title |
|---|
| 张平: "混合云平台 VM 系统的设计与实现" * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112422299A (en) * | 2020-11-18 | 2021-02-26 | 杭州飞致云信息科技有限公司 | Method and device for analyzing public cloud charging data and computer readable storage medium |
| CN113239243A (en) * | 2021-07-08 | 2021-08-10 | 湖南星汉数智科技有限公司 | Graph data analysis method and device based on multiple computing platforms and computer equipment |
| CN116205614A (en) * | 2023-05-04 | 2023-06-02 | 无锡锡商银行股份有限公司 | Business reconciliation management system and method for distributed scene |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107360006B (en) | Resource charging method and device | |
| CN111192123A (en) | Account checking system based on hybrid cloud computing platform and account checking method thereof | |
| US8832267B2 (en) | System and method for adaptive baseline calculation | |
| CN106156115B (en) | A kind of resource regulating method and device | |
| US20140188563A1 (en) | Customer demographic data change detection based on monitored utility consumption | |
| CN113962468A (en) | Energy consumption monitoring and statistics-based energy consumption carbon emission management method and system | |
| CN102056200A (en) | Service processing flow monitoring method and system | |
| US9020856B2 (en) | License compliance management | |
| CN116882804A (en) | Intelligent power monitoring method and system | |
| CN112488745A (en) | Intelligent charge control management method, device, equipment and storage medium | |
| CN111897706A (en) | Server performance prediction method, device, computer system and medium | |
| CN111756589A (en) | Cloud resource management method and device, cloud server and storage medium | |
| CN111800807A (en) | Method and device for alarming number of base station users | |
| US20150073878A1 (en) | Device to perform service contract analysis | |
| CN111667180A (en) | Power business management method and device and electronic equipment | |
| CN111222757A (en) | Statistical method and device for use condition of charging pile | |
| CN113780625B (en) | User electricity charge prediction method, system, terminal and storage medium | |
| AU2012346010B2 (en) | Configurable billing with subscriptions having conditional components | |
| US20210374814A1 (en) | A reconciliation system based on hybrid cloud computing platform and its reconciliation method | |
| CN115640167A (en) | Data backup method and device, electronic equipment and storage medium | |
| US8306911B1 (en) | Payment option analyzer based on measured utilization | |
| US20180108057A1 (en) | Prepaid utility metering with complex rates using advanced metering infrastructure (ami) smart meters | |
| CN111679973B (en) | Software test scheduling method, device, computer equipment and readable storage medium | |
| CN110825787A (en) | Data reminding method and device, electronic equipment and storage medium | |
| CN114546812B (en) | Energy consumption measuring method and device for application service, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 3 / F, Shunfeng industrial building, No.2 Hongliu Road, Fubao community, Fubao street, Futian District, Shenzhen City, Guangdong Province Applicant after: Shenzhen Jingtai Technology Co.,Ltd. Address before: 518000 4th floor, No.9 Hualian Industrial Zone, Xinshi community, Dalang street, Longhua District, Shenzhen City, Guangdong Province Applicant before: Shenzhen Jingtai Technology Co.,Ltd. |
|
| CB02 | Change of applicant information |