CN110069349A - A kind of resource consumption accounting system based on big data platform - Google Patents
A kind of resource consumption accounting system based on big data platform Download PDFInfo
- Publication number
- CN110069349A CN110069349A CN201910367998.1A CN201910367998A CN110069349A CN 110069349 A CN110069349 A CN 110069349A CN 201910367998 A CN201910367998 A CN 201910367998A CN 110069349 A CN110069349 A CN 110069349A
- Authority
- CN
- China
- Prior art keywords
- resource consumption
- data
- consumption
- resource
- cpu
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Economics (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- General Engineering & Computer Science (AREA)
- Entrepreneurship & Innovation (AREA)
- Strategic Management (AREA)
- Game Theory and Decision Science (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
Abstract
The present invention provides a kind of resource consumption accounting system based on big data platform, belongs to computer field.The resource consumption that the present invention lays particular emphasis on the self-built big data platform of enterprise is calculated, and compared with public cloud manufacturer is towards the accounting criteria of all users, more suits enterprises actual needs.Accounting criteria is easily understood simultaneously, and versatility is high, and statistical indicator is calculated according to actual use amount, has evaded unnecessary accounting resources consumption burden.
Description
Technical field
The present invention relates to field of computer technology, in particular to a kind of resource consumption core based on big data platform
Calculation system.
Background technique
With the development of science and technology and commodity economy, resource consumption structure gradually changes, resource consumption accounting pair
It can be based on playing key effect among increasingly fierce market competition environment in enterprise.However existing resource consumption is calculated
It cannot reflect the resource utilization in each production link, be unfavorable for increasing economic efficiency, also be unfavorable for improving environmental performance.
Each department of enterprise is independent from each other, and resource consumption data information cannot mutually be shared, and forms information island.
Since the design link of product, the after sale link last to product, each link generates a large amount of data information, if not
Promptly and accurately these information can be integrated and be applied, then the resource consumption control and management of enterprise just will receive very big limit
System, and the division of labor of each department, enterprise is fine, it is different using software, cause department that can not share with inter-sectional information, data letter
Breath is collected dependent on artificial statistics.
With the development of development of Mobile Internet technology, current many companies are in internal self-built big data platform, but big data is flat
The construction of platform and operation resource consumption are high, need to calculate resource consumption by way of science, and share actual consumption
The business department of platform resource.Current industry is mostly in a manner of resource isolation, according to the service for being disposably allocated to business side
Device memory, CPU quantity etc. calculate fixed resource consumption, the drawbacks of such mode be it is inflexible, for being drawn for business side
The resource divided often in idle state and is not used, and causes resource consumption is excessively high not meet actual conditions.In addition there is part public
You Yun manufacturer provides big data platform service, partially has by the way of dynamic charging, the money actually used according to user's single
Source charging, but the fixation of its charging unit price is opaque, and unit price rule is indefinite for a user there is a situation where higher, be unsatisfactory for using
Actual demand of the family to internal cost accounting.
Therefore, in long-term research and development, inventor is to the resource progress expense how realized according to user's actual use
It calculates and combines intra-company to calculate scene and carried out a large amount of research using the internal monovalent accounting method needed is more met,
A kind of big data platform resource consumption accounting method based on dynamic resource consumption charging is proposed, to solve the above technical problems
One of.
Summary of the invention
The purpose of the present invention is to provide a kind of resource consumption accounting system based on big data platform, is able to solve above-mentioned
At least one technical problem mentioned.Concrete scheme is as follows:
A kind of resource consumption accounting system based on big data platform, the resource consumption accounting system include user terminal with
And hadoop big data platform server, it is characterised in that:
The user terminal is communicated with the hadoop big data platform server by the foundation of wired or wireless network
Connection, and interaction data information；
The user terminal includes measuring and calculating unit, consumption data acquiring unit and consumption Accounting unit；
Measuring and calculating unit described in resource consumption obtains the practical branch of enterprises from the hadoop big data platform server
Data out, and preset hardware device service condition is combined, the metadata that the hardware device as unit of hour uses is calculated,
Wherein the preset hardware device service condition includes the capacity of the CPU of physical server, memory, disk space；
The consumption data acquiring unit obtains task run log from the hadoop big data platform server, from
Actual hardware device service condition is parsed and obtained in the task run log, and will parse the described actual hard of acquisition
Part equipment service condition is sent to the consumption Accounting unit, wherein the actual hardware device service condition includes physics
CPU, memory, the disk space consumption indicators of server；
The consumption Accounting unit receives the actual hardware device service condition, and obtains to measuring and calculating unit transmission
Request is taken, to obtain the metadata that hardware device uses from the measuring and calculating unit, the consumption Accounting unit is according to the hardware
Metadata that equipment uses, the actual hardware device service condition carry out resource consumption and carry out resource consumption accounting.
Further, the enterprises actual expenses data include server procurement data, computer room hosted data, network
Band data, for server procurement data with secondary for accounting unit, computer room hosted data and network bandwidth data are with year for accounting
Unit；
The preset hardware device service condition of combination, calculates first number that the hardware device as unit of hour uses
According to specifically including:
Default CPU, memory, disk space three resource consumption ratio be 1:2:100, the consumption of cpu resource every month is
F_CPU_M, the consumption of memory source every month are F_MEM_M, and the consumption of disk resource every month is F_DISK_M；
The metadata of cpu resource consumption is PF_CPU=F_CPU_M/A/30/24；
The metadata of memory source consumption is PF_MEM=F_MEM_M/B/30/24；
The metadata of disk resource consumption is PF_DISK=F_DISK_M/C/30/24.
Wherein, A is practical CPU total quantity, B is actual memory total quantity, C is disk total quantity.
Further, described to consume metadata, the actual hardware that Accounting unit is used according to the hardware device
Equipment service condition specifically includes to carry out resource consumption accounting:
The server procurement data is split according to preset first service life, the server after being split
Procurement data F_P_M；
Wherein, preset first service life is 5 years resource consumptions.
Further, described to consume metadata, the actual hardware that Accounting unit is used according to the hardware device
Equipment service condition specifically includes to carry out resource consumption accounting: by computer room hosted data and network bandwidth data respectively according to the
Two service life are split, computer room hosted data F_S_M and network bandwidth data the F_N_M money of the every month after being split
Source consumption.
Further, the enterprises actual expenses data are calculated by following formula:
F_M=F_P_M+F_S_M+F_N_M
Wherein, F_M is the enterprises actual expenses data, and F_P_M is the server procurement data after the fractionation,
F_S_M is the computer room hosted data after splitting, and F_N_M is the network bandwidth data after splitting
Further, task is obtained from the hadoop big data platform server in the consumption data acquiring unit
Before running log, specifically include:
Computing resource is distributed based on yarn Technique dynamic, after each single item task execution, by what is consumed in implementation procedure
Resource data is recorded in the task run log.
Further, actual hardware device service condition is parsed and obtained from the task run log, it is specific to wrap
It includes:
It is the unique identification generated when being executed according to calculating task based on yarn Technique dynamic distribution computing resource
Application_id, the historic task api interface for transferring server obtain log recording data, obtain log recording from described
In data extract vCore-second index, and using the vCore-second index extracted as task at runtime between
Interior consumption core cpu quantity obtains extracting Mb-second index in log recording data as task at runtime from described
The amount of memory of interior consumption.
Further, data are written into toward object table in the task of accounting, and the disk storage space that object table occupies is included in
Resource consumption calculates resource consumption.
Further, the disk storage space that object table is occupied is included in resource consumption and is calculated, and specifically includes:
Hive tables of data is obtained from the hadoop big data platform server, transfers metadata in the Hive tables of data
The information in library, the resource consumption for obtaining the disk storage space that the object table occupies is S_T；
It is calculated according to preset third service life, and calculates the resource of single calculating task using following formula
Consumption:
F_C=PF_CPU*USE_CPU+PF_MEM*USE_MEM
Wherein, for the third service life using day as unit of account, F_C is the resource consumption of the single calculating task,
PF_CPU is the metadata of cpu resource consumption, and USE_CPU is cpu resource quantity consumed, and PF_MEM is the member of memory source consumption
Data, USE_MEM are memory source quantity consumed；
The resource consumption of individual data table storage is calculated using following formula:
F_S=PF_DISK*S_T
Wherein, F_S is the resource consumption of individual data table storage, and S_T is the resource consumption of disk storage space, PF_
DISK is the metadata of disk resource consumption.
Further, described to consume metadata, the actual hardware that Accounting unit is used according to the hardware device
Equipment service condition carries out resource consumption accounting, specifically includes:
The resource consumption that the cumulative resource consumption F_C of all calculating tasks daily, tables of data store, and according to calculating task
Statistic of classification is carried out with the home subscriber of tables of data storage, relevant department, to obtain the resource consumption of business object.
The above scheme of the embodiment of the present invention compared with prior art, at least has the advantages that
The resource consumption that the big data platform based on hadoop can be calculated based on the present invention, carries out resource convenient for enterprises
Consumption control, reduces the wasting of resources.The resource consumption that the present invention lays particular emphasis on the self-built big data platform of enterprise is calculated, with public cloud factory
Quotient compares towards the accounting criteria of all users, more suits enterprises actual needs.Accounting criteria is easily understood general simultaneously
Property it is high, statistical indicator is calculated according to actual use amount, has evaded unnecessary accounting resources consumption burden.
Detailed description of the invention
A part that the drawings herein are incorporated into the specification, illustrates embodiments consistent with the invention, and and specification
Principle for explaining the present invention together.It should be evident that drawings in the following description are only some embodiments of the invention,
For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings
Other attached drawings.In the accompanying drawings:
Fig. 1 shows the structural block diagram of resource consumption accounting system of the invention.
Specific embodiment
To make the objectives, technical solutions, and advantages of the present invention clearer, below in conjunction with attached drawing to the present invention make into
It is described in detail to one step, it is clear that described embodiments are only a part of the embodiments of the present invention, rather than whole implementation
Example.Based on the embodiments of the present invention, obtained by those of ordinary skill in the art without making creative efforts
All other embodiment, shall fall within the protection scope of the present invention.
The term used in embodiments of the present invention is only to be not intended to be limiting merely for for the purpose of describing particular embodiments
The present invention.In the embodiment of the present invention and the "an" of singular used in the attached claims, " described " and "the"
It is also intended to including most forms, unless the context clearly indicates other meaning, " a variety of " generally comprise at least two.
It should be appreciated that term "and/or" used herein is only a kind of incidence relation for describing affiliated partner, indicate
There may be three kinds of relationships, for example, A and/or B, can indicate: individualism A, exist simultaneously A and B, individualism B these three
Situation.In addition, character "/" herein, typicallys represent the relationship that forward-backward correlation object is a kind of "or".
It will be appreciated that though may be described in embodiments of the present invention using term first, second, third, etc..,
But these ... it should not necessarily be limited by these terms.These terms be only used to by ... distinguish.For example, not departing from implementation of the present invention
In the case where example range, first ... can also be referred to as second ..., and similarly, second ... can also be referred to as the
One ....
Depending on context, word as used in this " if ", " if " can be construed to " ... when " or
" when ... " or " in response to determination " or " in response to detection ".Similarly, context is depended on, phrase " if it is determined that " or " such as
Fruit detection (condition or event of statement) " can be construed to " when determining " or " in response to determination " or " when detection (statement
Condition or event) when " or " in response to detection (condition or event of statement) ".
It should also be noted that, the terms "include", "comprise" or its any other variant are intended to nonexcludability
Include, so that commodity or device including a series of elements not only include those elements, but also including not clear
The other element listed, or further include for this commodity or the intrinsic element of device.In the feelings not limited more
Under condition, the element that is limited by sentence "including a ...", it is not excluded that in the commodity or device for including the element also
There are other identical elements.
The alternative embodiment that the invention will now be described in detail with reference to the accompanying drawings.
A kind of resource consumption accounting system based on big data platform, the resource consumption accounting system include user terminal with
And hadoop big data platform server, it is characterised in that:
The user terminal is communicated with the hadoop big data platform server by the foundation of wired or wireless network
Connection, and interaction data information；
The user terminal includes measuring and calculating unit, consumption data acquiring unit and consumption Accounting unit；
Measuring and calculating unit described in resource consumption obtains the practical branch of enterprises from the hadoop big data platform server
Data out, and preset hardware device service condition is combined, the metadata that the hardware device as unit of hour uses is calculated,
Wherein the preset hardware device service condition includes the capacity of the CPU of physical server, memory, disk space；
The consumption data acquiring unit obtains task run log from the hadoop big data platform server, from
Actual hardware device service condition is parsed and obtained in the task run log, and will parse the described actual hard of acquisition
Part equipment service condition is sent to the consumption Accounting unit, wherein the actual hardware device service condition includes physics
CPU, memory, the disk space consumption indicators of server；
The consumption Accounting unit receives the actual hardware device service condition, and obtains to measuring and calculating unit transmission
Request is taken, to obtain the metadata that hardware device uses from the measuring and calculating unit, the consumption Accounting unit is according to the hardware
Metadata that equipment uses, the actual hardware device service condition carry out resource consumption and carry out resource consumption accounting.
Further, the enterprises actual expenses data include server procurement data, computer room hosted data, network
Band data, for server procurement data with secondary for accounting unit, computer room hosted data and network bandwidth data are with year for accounting
Unit；
It is the physical resource of memory, CPU, disk space since big data platform mainly consumes, it is practical based on every month
The metadata of the consumption of total resources consumption conversion memory, CPU, disk space；
The preset hardware device service condition of combination, calculates first number that the hardware device as unit of hour uses
According to specifically including:
Default CPU, memory, disk space three resource consumption ratio be 1:2:100, the consumption of cpu resource every month is
F_CPU_M, the consumption of memory source every month are F_MEM_M, and the consumption of disk resource every month is F_DISK_M；
The metadata of cpu resource consumption is PF_CPU=F_CPU_M/A/30/24；
The metadata of memory source consumption is PF_MEM=F_MEM_M/B/30/24；
The metadata of disk resource consumption is PF_DISK=F_DISK_M/C/30/24.
Wherein, A is practical CPU total quantity, B is actual memory total quantity, C is disk total quantity.
Further, described to consume metadata, the actual hardware that Accounting unit is used according to the hardware device
Equipment service condition specifically includes to carry out resource consumption accounting:
The server procurement data is split according to preset first service life, the server after being split
Procurement data F_P_M；
Wherein, preset first service life is 5 years resource consumptions.
Further, described to consume metadata, the actual hardware that Accounting unit is used according to the hardware device
Equipment service condition specifically includes to carry out resource consumption accounting: by computer room hosted data and network bandwidth data respectively according to the
Two service life are split, computer room hosted data F_S_M and network bandwidth data the F_N_M money of the every month after being split
Source consumption.
Further, the enterprises actual expenses data are calculated by following formula:
F_M=F_P_M+F_S_M+F_N_M
Wherein, F_M is the enterprises actual expenses data, and F_P_M is the server procurement data after the fractionation,
F_S_M is the computer room hosted data after splitting, and F_N_M is the network bandwidth data after splitting
Further, task is obtained from the hadoop big data platform server in the consumption data acquiring unit
Before running log, specifically include:
Computing resource is distributed based on yarn Technique dynamic, after each single item task execution, by what is consumed in implementation procedure
Resource data is recorded in the task run log.
Further, actual hardware device service condition is parsed and obtained from the task run log, it is specific to wrap
It includes:
It is the unique identification generated when being executed according to calculating task based on yarn Technique dynamic distribution computing resource
Application_id, the historic task api interface for transferring server obtain log recording data, obtain log recording from described
In data extract vCore-second index, and using the vCore-second index extracted as task at runtime between
Interior consumption core cpu quantity obtains extracting Mb-second index in log recording data as task at runtime from described
The amount of memory of interior consumption.
Further, data are written into toward object table in the task of accounting, and the disk storage space that object table occupies is included in
Resource consumption calculates resource consumption.
Further, the disk storage space that object table is occupied is included in resource consumption and is calculated, and specifically includes:
Hive tables of data is obtained from the hadoop big data platform server, transfers metadata in the Hive tables of data
The information in library, the resource consumption for obtaining the disk storage space that the object table occupies is S_T；
It is calculated according to preset third service life, and calculates the resource of single calculating task using following formula
Consumption:
F_C=PF_CPU*USE_CPU+PF_MEM*USE_MEM
Wherein, for the third service life using day as unit of account, F_C is the resource consumption of the single calculating task,
PF_CPU is the metadata of cpu resource consumption, and USE_CPU is cpu resource quantity consumed, and PF_MEM is the member of memory source consumption
Data, USE_MEM are memory source quantity consumed；
The resource consumption of individual data table storage is calculated using following formula:
F_S=PF_DISK*S_T
Wherein, F_S is the resource consumption of individual data table storage, and S_T is the resource consumption of disk storage space, PF_
DISK is the metadata of disk resource consumption.
Further, described to consume metadata, the actual hardware that Accounting unit is used according to the hardware device
Equipment service condition carries out resource consumption accounting, specifically includes:
The resource consumption that the cumulative resource consumption F_C of all calculating tasks daily, tables of data store, and according to calculating task
Statistic of classification is carried out with the home subscriber of tables of data storage, relevant department, to obtain the resource consumption of business object.
In above-mentioned principle, the resource consumption accounting system described herein based on big data platform can be by existing
Some programming languages mention program means, are stored on computer-readable medium in the form of encapsulation or encapsulation, and according to
Documented logical construction or original part constitute figure as shown in figure 1, to realize that by executing equipment include but is not limited to electronic equipment
Execution and operation.Briefly, overall plan described herein mainly includes that user terminal and hadoop big data are flat
Platform server, the user terminal are communicated with the hadoop big data platform server by the foundation of wired or wireless network
Connection, and interaction data information, the user terminal include that measuring and calculating unit, consumption data acquiring unit and consumption accounting are single
Member.And specifically, in the case where the metadata of resource consumption is calculated, practical total expenditure is built by obtaining data platform, so
Data platform is obtained afterwards and builds single month expenditure, and data platform CPU, memory, disk are obtained under conditions of above-mentioned data basis again
Total capacity on the basis of, obtain data platform CPU, memory, disk resource consumption metadata.
In the case where obtaining the consumption of calculating task real resource, the resource consumption resource consumption core based on big data platform
The step of calculation system executes can be generalized following content: 1) obtaining the unique identification number that calculating task generates；2) root
Specific log acquisition interface is obtained according to mission number；3) log record file after task run is obtained；4) log note is obtained
CPU, memory consumption data in record file；5) CPU and memory of task consumption are converted for unit by the hour；6) it obtains to calculate and appoint
The target hive tables of data of business write-in；7) memory space of tables of data is obtained from hive metadatabase；8) it is practical to obtain task
The resource consumption total amount of CPU, memory, disk.
Those of ordinary skill in the art will appreciate that all or part of the steps in the above method can be instructed by program
Related hardware is completed, and described program can store in computer readable storage medium, such as read-only memory, disk or CD
Deng.Optionally, one or more integrated circuits also can be used to realize, accordingly in all or part of the steps of above-described embodiment
Ground, each module/unit in above-described embodiment can take the form of hardware realization, can also use the shape of software function module
Formula is realized.The present invention is not limited to the combinations of the hardware and software of any particular form.
It should be noted that the invention may also have other embodiments, without departing substantially from spirit of that invention and its essence
In the case of, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these are corresponding
Change and modification all should fall within the scope of protection of the appended claims of the present invention.
Claims (10)
1. a kind of resource consumption accounting system based on big data platform, the resource consumption accounting system include user terminal and
Hadoop big data platform server, it is characterised in that:
The user terminal and the hadoop big data platform server are established by wired or wireless network to be communicated to connect,
And interaction data information；
The user terminal includes measuring and calculating unit, consumption data acquiring unit and consumption Accounting unit；
Measuring and calculating unit described in resource consumption obtains enterprises actual expenses number from the hadoop big data platform server
According to, and preset hardware device service condition is combined, the metadata that the hardware device as unit of hour uses is calculated, wherein
The preset hardware device service condition includes the capacity of the CPU of physical server, memory, disk space；
The consumption data acquiring unit obtains task run log from the hadoop big data platform server, from described
Actual hardware device service condition is parsed and obtained in task run log, and the actual hardware that parsing obtains is set
Standby service condition is sent to the consumption Accounting unit, wherein the actual hardware device service condition includes physical services
CPU, memory, the disk space consumption indicators of device；
The consumption Accounting unit receives the actual hardware device service condition, and sends to obtain to the measuring and calculating unit and ask
It asks, to obtain the metadata that hardware device uses from the measuring and calculating unit, the consumption Accounting unit is according to the hardware device
Metadata, the actual hardware device service condition used carrys out resource consumption and carries out resource consumption accounting.
2. the resource consumption accounting system according to claim 1 based on big data platform, which is characterized in that the enterprise
Internal actual expenses data include server procurement data, computer room hosted data, network bandwidth data, server procurement data with
Secondary is accounting unit, and computer room hosted data and network bandwidth data are using year as accounting unit；
The preset hardware device service condition of combination, calculates the metadata that the hardware device as unit of hour uses,
It specifically includes:
Default CPU, memory, disk space three resource consumption ratio be 1:2:100, the consumption of cpu resource every month is F_
CPU_M, the consumption of memory source every month are F_MEM_M, and the consumption of disk resource every month is F_DISK_M；
The metadata of cpu resource consumption is PF_CPU=F_CPU_M/A/30/24；
The metadata of memory source consumption is PF_MEM=F_MEM_M/B/30/24；
The metadata of disk resource consumption is PF_DISK=F_DISK_M/C/30/24；
Wherein, A is practical CPU total quantity, B is actual memory total quantity, C is disk total quantity.
3. the resource consumption accounting system according to claim 2 based on big data platform, which is characterized in that the consumption
Metadata that Accounting unit is used according to the hardware device, the actual hardware device service condition carry out resource consumption
Accounting specifically includes:
The server procurement data is split according to preset first service life, the server buying after being split
Data F_P_M；
Wherein, preset first service life is 5 years resource consumptions.
4. the resource consumption accounting system according to one of Claims 2 or 3 based on big data platform, which is characterized in that
Metadata, the actual hardware device service condition that the consumption Accounting unit is used according to the hardware device carry out
Resource consumption accounting specifically includes: computer room hosted data and network bandwidth data are torn open according to the second service life respectively
Point, the computer room hosted data F_S_M and network bandwidth data F_N_M resource consumption of the every month after being split.
5. the resource consumption accounting system according to claim 4 based on big data platform, which is characterized in that the enterprise
Internal actual expenses data are calculated by following formula:
F_M=F_P_M+F_S_M+F_N_M
Wherein, F_M is the enterprises actual expenses data, and F_P_M is the server procurement data after the fractionation, F_S_
M is the computer room hosted data after splitting, and F_N_M is the network bandwidth data after splitting.
6. the resource consumption accounting system according to claim 1 based on big data platform, which is characterized in that disappear described
Data capture unit is consumed before obtaining task run log in the hadoop big data platform server, is specifically included:
Computing resource, after each single item task execution, the resource that will consume in implementation procedure are distributed based on yarn Technique dynamic
Data are recorded in the task run log.
7. the resource consumption accounting system according to claim 1 or described in one of 6 based on big data platform, which is characterized in that
Actual hardware device service condition is parsed and obtained from the task run log, is specifically included:
It is the unique identification generated when being executed according to calculating task based on yarn Technique dynamic distribution computing resource
Application_id, the historic task api interface for transferring server obtain log recording data, obtain log recording from described
In data extract vCore-second index, and using the vCore-second index extracted as task at runtime between
Interior consumption core cpu quantity obtains extracting Mb-second index in log recording data as task at runtime from described
The amount of memory of interior consumption.
8. the resource consumption accounting system according to claim 7 based on big data platform, which is characterized in that appoint calculating
It is engaged in that data are written toward object table, and the disk storage space that object table occupies is included in resource consumption and calculates resource consumption.
9. the resource consumption accounting system according to claim 8 based on big data platform, which is characterized in that described by mesh
The disk storage space that mark table occupies is included in resource consumption accounting, specifically includes:
Hive tables of data is obtained from the hadoop big data platform server, transfers metadatabase in the Hive tables of data
Information, the resource consumption for obtaining the disk storage space that the object table occupies is S_T；
It is calculated according to preset third service life, and the resource for being calculated using following formula single calculating task is disappeared
Consumption:
F_C=PF_CPU*USE_CPU+PF_MEM*USE_MEM
Wherein, for the third service life using day as unit of account, F_C is the resource consumption of the single calculating task, PF_
CPU is the metadata of cpu resource consumption, and USE_CPU is cpu resource quantity consumed, and PF_MEM is first number of memory source consumption
According to USE_MEM is memory source quantity consumed；
The resource consumption of individual data table storage is calculated using following formula:
F_S=PF_DISK*S_T
Wherein, F_S is the resource consumption of individual data table storage, and S_T is the resource consumption of disk storage space, and PF_DISK is
The metadata of disk resource consumption.
10. the resource consumption accounting system according to claim 9 based on big data platform, which is characterized in that described to disappear
Metadata, the actual hardware device service condition that consumption Accounting unit is used according to the hardware device disappear to carry out resource
Consumption is calculated, and is specifically included:
The resource consumption that the cumulative resource consumption F_C of all calculating tasks daily, tables of data store, and according to calculating task sum number
Statistic of classification is carried out according to the home subscriber of table storage, relevant department, to obtain the resource consumption of business object.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910367998.1A CN110069349A (en) | 2019-05-05 | 2019-05-05 | A kind of resource consumption accounting system based on big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910367998.1A CN110069349A (en) | 2019-05-05 | 2019-05-05 | A kind of resource consumption accounting system based on big data platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110069349A true CN110069349A (en) | 2019-07-30 |
Family
ID=67369858
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910367998.1A Pending CN110069349A (en) | 2019-05-05 | 2019-05-05 | A kind of resource consumption accounting system based on big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110069349A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111400033A (en) * | 2020-03-03 | 2020-07-10 | 京东数字科技控股有限公司 | Platform resource cost allocation method and device, storage medium and computer equipment |
| CN115600985A (en) * | 2022-10-27 | 2023-01-13 | 深圳标普云科技有限公司（Cn） | Online platform task management method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140245298A1 (en) * | 2013-02-27 | 2014-08-28 | Vmware, Inc. | Adaptive Task Scheduling of Hadoop in a Virtualized Environment |
| CN106527993A (en) * | 2016-11-09 | 2017-03-22 | 北京搜狐新媒体信息技术有限公司 | Mass file storage method and device for distributed type system |
| CN109063074A (en) * | 2018-07-24 | 2018-12-21 | 浪潮(北京)电子信息产业有限公司 | Data processing method, device, equipment and medium based on Hadoop |
| CN109088747A (en) * | 2018-07-10 | 2018-12-25 | 郑州云海信息技术有限公司 | The management method and device of resource in cloud computing system |
| CN109165045A (en) * | 2018-08-09 | 2019-01-08 | 网宿科技股份有限公司 | A kind of method and apparatus for the hardware configuration adjusting server |
| CN109375987A (en) * | 2018-10-30 | 2019-02-22 | 张家口浩扬科技有限公司 | A kind of method and system of virtual machine selection physical machine |
| CN109408225A (en) * | 2018-09-18 | 2019-03-01 | 平安科技（深圳）有限公司 | Resource capacity expansion method, apparatus, computer equipment and storage medium |
-
2019
- 2019-05-05 CN CN201910367998.1A patent/CN110069349A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140245298A1 (en) * | 2013-02-27 | 2014-08-28 | Vmware, Inc. | Adaptive Task Scheduling of Hadoop in a Virtualized Environment |
| CN106527993A (en) * | 2016-11-09 | 2017-03-22 | 北京搜狐新媒体信息技术有限公司 | Mass file storage method and device for distributed type system |
| CN109088747A (en) * | 2018-07-10 | 2018-12-25 | 郑州云海信息技术有限公司 | The management method and device of resource in cloud computing system |
| CN109063074A (en) * | 2018-07-24 | 2018-12-21 | 浪潮(北京)电子信息产业有限公司 | Data processing method, device, equipment and medium based on Hadoop |
| CN109165045A (en) * | 2018-08-09 | 2019-01-08 | 网宿科技股份有限公司 | A kind of method and apparatus for the hardware configuration adjusting server |
| CN109408225A (en) * | 2018-09-18 | 2019-03-01 | 平安科技（深圳）有限公司 | Resource capacity expansion method, apparatus, computer equipment and storage medium |
| CN109375987A (en) * | 2018-10-30 | 2019-02-22 | 张家口浩扬科技有限公司 | A kind of method and system of virtual machine selection physical machine |
Non-Patent Citations (1)
| Title |
|---|
| 吴凯亮: "基于Web的IT应用网管系统的研究与开发", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111400033A (en) * | 2020-03-03 | 2020-07-10 | 京东数字科技控股有限公司 | Platform resource cost allocation method and device, storage medium and computer equipment |
| CN111400033B (en) * | 2020-03-03 | 2024-04-19 | 京东科技控股股份有限公司 | Platform resource cost allocation method and device, storage medium and computer equipment |
| CN115600985A (en) * | 2022-10-27 | 2023-01-13 | 深圳标普云科技有限公司（Cn） | Online platform task management method and system |
| CN115600985B (en) * | 2022-10-27 | 2023-09-15 | 深圳标普云科技有限公司 | On-line platform task management method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109412829B (en) | Resource allocation prediction method and equipment | |
| Zhou et al. | The effect of artificial intelligence on China’s labor market | |
| CN107577805B (en) | Business service system for log big data analysis | |
| US10715849B2 (en) | Automatically generating a recommendation based on automatic aggregation and analysis of data | |
| JP6147757B2 (en) | Providing resource usage information for each application | |
| US10783002B1 (en) | Cost determination of a service call | |
| CN109034993A (en) | Account checking method, equipment, system and computer readable storage medium | |
| CN104657194B (en) | Method and system for calculating the influence acted in network | |
| CN109032801A (en) | A kind of request scheduling method, system and electronic equipment and storage medium | |
| CN102982489A (en) | Power customer online grouping method based on mass measurement data | |
| WO2021174945A1 (en) | Data cost calculation method, system, computer device, and storage medium | |
| CN111367989B (en) | Real-time data index calculation system and method | |
| US20130197863A1 (en) | Performance and capacity analysis of computing systems | |
| CN104615526A (en) | Monitoring system of large data platform | |
| KR20100092850A (en) | Apparatus for resource distribution in cloud computing and method thereof | |
| CN105049218A (en) | PhiCloud cloud charging method and system | |
| Bejeck | Kafka Streams in Action | |
| CN113010795B (en) | User dynamic image generation method, system, storage medium and electronic device | |
| CN108900619A (en) | A kind of independent Statistics of accessing population method and device | |
| CN110069349A (en) | A kind of resource consumption accounting system based on big data platform | |
| CN108011764A (en) | A kind of method for predicting more cloud platform storage resource increments | |
| US10691653B1 (en) | Intelligent data backfill and migration operations utilizing event processing architecture | |
| CN115408546A (en) | Time sequence data management method, device, equipment and storage medium | |
| CN114218291A (en) | Portrait generation method, apparatus, device and storage medium based on target object | |
| KR102292578B1 (en) | System and method for brokeringof energy data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190730 |
|
| RJ01 | Rejection of invention patent application after publication |