TW201909002A - Data set transaction and computing resource integration method with which the consumers may conveniently search desired data sets according to their needs, and purchase the desired data sets with payments - Google Patents

Abstract

The present invention discloses a data set transaction and computing resource integration method, which is established on a Hadoop platform. The present invention is based on the conventional resource renting mode, and additionally provides the users with a data set transaction mechanism, thereby allowing the user not only to use the general renting application, but also to upload data sets owned by the user to a data market of the present invention for sale. Furthermore, in combination with a data dictionary of the present invention to be inquired and purchased by other users, the consumers may conveniently search desired data sets according to their needs, and take the desired data sets by purchasing with payments.

Description

 Data set transaction and computing resource integration method  

The invention relates to a data set transaction and computing resource integration method, in particular to a data set transaction and computing resource integration method for a user to purchase a data set and seamlessly connect to the platform resources that are rented and operated.

At present, due to the rise of massive data analysis, the collision and cross-analysis between data extends the demand for supply and demand of data sets. However, the current usage patterns of the distributed Hadoop platform (hereinafter referred to as the Hadoop platform) are only limited to The way users rent space and computing resources, there is no platform to provide a program mechanism for data set transactions.

With the rise of massive data analysis, more and more governments, enterprises, and organizations have established open data platforms for the public to use, and hope to assist decision-making through trend analysis of data, but Open Data is often accompanied by certain analysis practices. Levels that need to be overcome, the content of the data is often chaotic, the quality is different, and it is not easy to download in a myriad of servers.

In addition to the inability to efficiently find the desired data set, analysts also need to deal with complicated procedures and procedures such as data access, unified format, data cleaning, and data verification, which often leads to delays and efficiency in overall analysis. Not at all.

On the other hand, because the current Hadoop platform does not provide a data set sales and purchase transaction mechanism for the supply and demand of data sets, the carefully prepared data set, except for its own use, seems to have only the option of free contribution. The lack of a trading pipeline allows dataset keepers to generate revenue by selling the datasets they own.

In view of the above-mentioned problems of the prior art, the purpose of the data set transaction and computing resource integration method of the present invention is to provide a transaction mechanism that additionally provides a user data set based on the traditional renting mode on the Hadoop platform. In addition to the general space and computing resources, the user can also put the prepared data set on the data collection market of the present invention for sale. On the one hand, the data requester can purchase a pipeline to facilitate legally obtaining the required data. The data set, on the other hand, also provides information to the pipeline to get a profitable pipeline.

The second object of the present invention is to provide convenience for data and analysis environment integration, to improve the characteristics of the privilege control design and the Hadoop platform, and to construct a convenient environment ready for use, so that users can immediately connect to the platform after purchasing the materials. Renting computing resources to perform computational analysis operations, eliminating the inconvenience of user data movement, greatly reducing the cost of data transmission and the cost of constructing an analysis environment.

Another object of the present invention is to provide a data set that the demander can query, browse, and purchase in a more efficient manner by the disclosed data dictionary mechanism. Data dictionary mechanism, systematically classify data sets through labeling information, and use the popularity analysis optimization program to provide accurate recommendation mechanism for data users to easily obtain the desired data set, so that data analysts stand well. Based on the source of the prepared data, it is more convenient and quick to carry out in-depth analysis and exploration to create more valuable data.

The data set transaction and computing resource integration method of the present invention comprises the following steps: allocating storage space and computing resources on the Hadoop platform according to the resource requesting request by the user; and running the data set trading platform on the Hadoop platform to provide the resource renting request The user performs data set transaction; and sets an authorization management agent module on the Hadoop platform, and performs permission integration according to the data set transaction request submitted by the user, and provides the user to directly access the data set corresponding to the transaction via the data set transaction on the Hadoop platform.

As described above, the data set transaction and computing resource integration method according to the present invention may have one or more of the following advantages:

1. Based on the Hadoop platform, an additional data set transaction method is provided. In addition to the application of the Hadoop platform as a data storage and computing function, users can also sell their original data for analysis and use by other users. Earnings can be derived from it, and the proceeds can also offset the operating mode of the subscription fee.

2. The invention provides a user-efficient data exploration mode in a data dictionary mode, and combines real-time recommendation analysis and labeling classification exploration to help users quickly find applicable data and reduce search time cost in the data market.

3. The invention provides a transaction mode of data collection and sale on the Hadoop platform, and improves the convenience, ease of use, updateability and readability of the source of the source data set, so as to facilitate the utilization of data consumption users.

4. The present invention provides a seamless integration environment. The data consumer user purchases the data set and is managed and integrated by the authorization mechanism of the present invention, and can immediately use the data set and directly connect to the resources and space of the platform rented on the Hadoop platform for analysis. Calculations eliminate the time and cost of additional erection and maintenance of the analysis environment.

S110~S140, S210~S224, S410~S430‧‧‧ steps

FIG. 1 is a flow chart of a method for integrating data set transaction and computing resources according to the present invention.

2 is another flow chart of the method for integrating data set transaction and computing resources according to the present invention.

FIG. 3 is still another flow chart of the method for integrating data set transaction and computing resources according to the present invention.

The specific embodiments are described below to illustrate the embodiments of the invention, but are not intended to limit the scope of the invention.

Please refer to FIG. 1 , which is a flowchart of a data set transaction and computing resource integration method according to the present invention. The steps of the present invention include: S110: Allocating storage space and computing resources on a Hadoop platform according to a resource request request submitted by a user; S120: The data set transaction platform is used in the Hadoop platform to provide data collection transactions for users corresponding to the resource subscription request; S130: setting an authorization management agent module on the Hadoop platform, and performing permission integration according to the data set transaction request submitted by the user, providing users Accessing the data set corresponding to the data set transaction directly on the Hadoop platform; S140: calculating the fee according to the user's data transaction and the resource subscription request.

The following is a detailed description of each of the above steps.

First, the S110 subscription platform resource is responsible for realizing the dynamic resource allocation setting of the storage space allocation and the computing resource Vcore (virtual core) and the memory size according to the resource subscription content proposed by the user. In the step S120 of the present invention, a transaction mechanism for selling and purchasing data sets is provided, which is responsible for transaction processing of data sets between users, and the composition thereof is referred to FIG. 2, and the main structure is divided into a S210 shelf-up program and an S220 purchase program according to user behavior, and the execution thereof is executed. Details include S211 dataset labeling and information registration, S212 dataset pricing, S221 query data dictionary and data purchase, S222 record dataset purchase information, S223 instant feedback popularity analysis model to data dictionary and S224 data permission setting a total of 6 items program.

When the user wants to collect the data set in the data market for sale, the S210 will be put on the shelf. The S211 data set labeling and information registration program will be used for labeling and information registration according to the nature of the data set. The login information includes the necessary information content such as the attribute label, the data category and the format of the data set, wherein the attribute and the category belong to a multi-value type, and the login operation registers the related information into the metadata file of the data market.

After the preparation, the data set uploaded through the information registration operation in the program S211 shall conform to the data set specification on the Hadoop platform, and be a common file format in the Hadoop platform, thereby achieving the consistency of the data set format to provide the purchase user. The convenience, ease of use, updateability, and readability of using data sets.

The on-board user continues to perform the step S212 data set pricing, which provides a setting interface for the user to price the data set they are selling.

In the present invention, the definition of the data set purchase is limited in the nature of the use of the license, and the non-buy-off property, the price set by the user is the price at which the data consumer authorized to use the data set in a specific time unit, and the specific time unit is called The data use authorization time is a minimum non-separable time interval. The purchase behavior of the data consumer user needs to be purchased in integer multiples of time units. The platform also authorizes the data in integer multiples of time units, so the purchased data set During the period of the purchase authorization, the data consumer users will be able to use it continuously. If the data set authorization period is exceeded, the data set must be re-purchased. After completing the above S210 shelf-up program, the data set can be sold on the data market.

The data consumer user obtains the required data set by purchasing the data dictionary from the S221 in the purchase program and purchasing the data. The data dictionary mentioned in the S221 program contains a purchase interface and a query interface as an entry point for the query and purchase of the data consumer user data set.

The query interface of the data dictionary provides systematic classification browsing according to the data labeling attribute, and also has the methods of keyword query, comprehensive keyword query and data labeling attribute for quick screening. The data dictionary internal query engine sorts the query results based on the data model generated by the popularity analysis, and provides a list of results that the queryer may be interested in, in order to help reduce the time spent on query exploration.

In order to make the query information more reference value, the information presented in the data dictionary contains the data field description, data format, update frequency, number of data, file size, sale price, and information provided by the collector in the data set. News.

The characteristic of the data dictionary is to provide users with a more efficient and referenced data search interface through data labeling and query, keyword and popularity query, to help users quickly find the applicable data set, so that users can query after evaluation. Purchase the required data set for use.

After the purchase behavior occurs, the program S222 records the data set purchase information, and the record program collects the data set information purchased by the user on the data market. The data set information includes at least the data set name, the data set label attribute, and the user. Purchase date, purchase count, purchase unit price, dataset space usage, and snapshot information of the current data set in the data dictionary.

In order to make the data dictionary achieve more accurate query results, the present invention uses an analysis framework for instantly updating the data model, and instantly updates the popularity analysis model to the data dictionary program via S223, and instantly updates the data model generated by the latest purchase data. The data dictionary query engine solves the phenomenon that the model is too old to reflect the current stage of data and is inaccurate.

The operation procedure is based on the information collection information recorded in the S222 program. The real-time streaming component record data of the Hadoop platform is transmitted to the S223 program for the popularity analysis operation. The analysis includes one or more data analysis algorithms, depending on the requirements. It can be adjusted, combined, and exchanged. It will be based on the latest sales information of the data set on the data market, and the user purchase history record popularity recommendation model. This popularity recommendation model will then use the real-time streaming component of the Hadoop platform to feed back the data. The dictionary's query engine performs model updates.

Then execute the S224 data permission setting. The purpose of this program is to solve the problem of the lack of the authority and control ability on the Hadoop platform, and the design limitation of the combination of the authorization time management and the multi-group user permission setting.

The program S224 is responsible for processing the data set access authority management, and generating a permission rule file for the rule setting operation, and setting the user's read permission for the purchased data set according to the user account and the purchased time unit, and After setting the authorization time, the data set will not be used after the date exceeds the authorized use period.

The rights management improvement is based on the fact that each agent in the Hadoop platform (multiple hosts in a hadoop platform) buryes agents (agents, each of which must be installed with an agent to control permissions), and the agent performs access level. Control, with each set of user authentication key, authorization data set item and data authorization effective date message of the permission rule file record, and the above permission rule file will be transmitted to the Agent embedded in each host on the Hadoop platform through the encryption mechanism. The Agent performs user authentication according to the permission rule file and grants access rights to the corresponding data set.

In order to prevent the risk of the privilege file being falsified, the design uses the multi-party privilege file SHA256 code confirmation mechanism. When the Agent receives the access permission request, it will first connect with other Agents to perform the privilege file comparison operation, and the permission through the SHA256 algorithm. The code obtained by the file operation is compared with the SHA256 code of other host permission files. If the same is the same, the integrity verification is performed, indicating that it has not been modified. Otherwise, if the comparison result of the SHA256 code of any host permission file does not match, the file is started. The majority synchronization mechanism synchronizes the rights files with most SHA256 codes to other hosts for permission repair.

Referring to FIG. 1, in the program of S130, the improved privilege control design and the characteristics of the Hadoop platform are used to construct a convenient environment for ready-to-use. After the user purchases the data set in the data market, the data set permission is transferred to the agent embedded in each host on the Hadoop platform through the permission rule file set by the S224 program, and the agent sets up a control. Layers in front of each Hadoop computing service, and automatically implements the permission setting of the corresponding data set, and controls the actual file of the data set and the data set in the data warehouse by setting the authority and matching the data usage time limit in the permission rule file. Read access.

After the permission is opened, the user can access and read the actual file of the data set and the data set in the warehouse, and the purchase user can seamlessly access the purchased data set without the additional data carrying, and each computing service The interface can also access the actual file of the data set or data set in the warehouse, allowing users to use the computing resources they rent and use the computing services on the Hadoop platform, such as Spark, Hive, etc., directly on the platform. The analysis, calculation and utilization of the data set do not require additional analysis and computing environments to achieve the purpose of data set and computing resource integration.

Each billing cycle platform (Hadoop platform) will carry out the S140 integrated billing program, which integrates the user's expenses for renting resources on the platform and the income and expenses of the data market, thereby generating the user's bill for the billing cycle. For the composition, please refer to Figure 3. The S140 integrated billing program includes the S410 platform resource billing program, the S420 data market billing program, and the S430 billing project output user income and expenditure sheet. The following details will be explained.

In the storage resource part, the storage resources used by the user will be processed by the S411: Hadoop Distributed File System (HDFS) application space collection program, which collects information about the HDFS subscription space of the user, and the collected data includes the application. The information about the size of the rent space, the space occupied by the original data, the space occupied by the data backup, the number of data backup copies, the remaining available space, etc., and the data is collected and written into the billing database of the platform.

In the computing resource part, when the user performs data analysis on the Hadoop platform, the resource usage information will be recorded in the log file by the resource management system in the platform, and the resource usage is calculated by executing S412: virtual core (Vcore) (Vcore). Statistics and S413: Memory computing resource usage statistics program analyzes the resource usage log files generated by each user to obtain the user's resource usage information and usage time data in the platform.

The quantization method used is quantized by multiplying the number of Vcores used in the calculation by the actual execution time, and Memory is quantized by the memory size allocated by the operation operation by the actual execution time, and the collected amount is used. The summary is written into the billing database of the platform.

In the implementation details of the S412 and S413 program operation resource usage statistics, in order to reduce the execution time cost of the huge resource usage log file in the analysis of the posting period, the present invention adopts a micro batch analysis strategy in analyzing the usage log operation. The billing cycle is cut into multiple data collection periods. Each usage data collection cycle will analyze and summarize the resource usage records of the log files belonging to the cycle time, and use the file reading indicators to record the current position after each analysis. Speed up the next read job.

The above design can effectively use and release the analysis resources used, and the time cost of dispersing the single resource usage analysis can more effectively calculate the amount of computing resources used by the Vcore and Memory used by the user on the platform.

The information collected above will be submitted to the S414 renting resource billing program to calculate the cost of the user's leased resources in the set billing cycle. S411: HDFS lease space collection, S412: Vcore computing resource usage statistics, 413Memory computing resources The usage information obtained by the usage statistics is calculated by using the unit price of the resource, and the three items are used as the platform resource subscription fee in the user's billing cycle.

The other part of the account processing for the data market is executed by the S420 data market billing program. The first step is to collect the data market transaction record program through S421 according to the user purchase record recorded in the S222 procedure mentioned above. The user calculates the total purchase cost of the user during the current billing cycle, and calculates the total sales information of the individual data sets in the current period for the data set, and passes the summarized transaction records to the S422 data market billing program. Perform account calculation processing.

Then, the S422 data market billing program is responsible for calculating the account information of the data market according to the billing cycle set by the platform, and the output account information is divided into two parts: one is the purchase price that the data consumer user should pay. According to the purchase unit of the data set purchased by the user, the sales unit price of the data set is multiplied; the second is the income of the user sales data set as the data, and the total sales amount of the data set is deducted according to the total sales amount of the data set. The ratio is used as the cost of the shelf, and the final income is the data to provide the user with the proceeds from the data market during the billing cycle.

Finally, the S430 integrated billing program is responsible for processing the billing total settlement of the platform users according to the billing cycle set by the platform, and the fee generated by the resource renting provided by the above S414 renting resource billing, and the S422 data market billing The account set purchase and sale account provided provides the bill for the user of the billing cycle.

Then, an application embodiment is used to specifically describe the actual operation of the data set transaction and computing resource integration method of the present invention.

The invention is based on the Hadoop platform, provides user renting resources and constructs a data flow pipeline between users, and the user can use the resources on the Hadoop platform to calculate the data set, and can choose to upload the generated data set to the platform. The information provided is sold on the market.

Hadoop platform commonly includes distributed file system (HDFS, Hadoop Distributed File System), resource management system (YARN), distributed data repository database (Hive), memory analysis operation (Spark) and other components, distributed file system use Decentralized storage of archives, resource management system YARN is responsible for resource management and scheduling of Hadoop clusters, decentralized data repository database Hive and memory analysis operations Spark can be used to perform data query, data analysis, data exploration, and distributed data storage The database application Hive and the memory analysis operation Spark perform the analysis application query by the resource management system YARN for resource allocation and scheduling management.

FIG. 1 is a flowchart of the present invention. The user will rent a resource through the program S110, including the HDFS space and the platform computing resource, and may store the prepared data or the analyzed data. The platform then uses the resource subscription content proposed by the user. The Hadoop platform performs dynamic resource allocation setting of the storage space allocation and the number of computing resources Vcore and Memory, and then performs data market trading in the program S120.

After the data set transaction is completed, the platform computing resource and the purchased data set can be integrated in the program S130. The present invention provides ready-to-use integration and convenience, and the user purchases the data set immediately after passing through the distributed Hadoop platform. The authorization mechanism is opened for authorization, and since the data set provided in the data market has been prepared and imported into the decentralized data storage database, a standard data source interface is provided, and the user purchases the data set and authorizes the data set. After that, you can use the Hadoop platform to provide analysis and calculation tools that can interface with the distributed data repository database interface or the distributed file system interface, such as Hive, Spark, etc., and the data analysis results can also be stored in In the HDFS space for rent, the time and cost of erecting the environment and loading data are eliminated.

The program S140 comprehensive billing will be based on the user's transaction results in the data marketplace and the user's space and computing resource usage on the Hadoop platform. The following will be explained in a practical use case.

In the following embodiments, the tenant users on the Hadoop platform are defined as "data providing users" and "data consumer users" according to the roles of the two ends.

The "data providing user" in the program S110 rents a distributed file system (HDFS) storage space to the Hadoop platform to store the data, and loads the data into the data storage, such as the distributed data storage database (Hive) table, said For the "data set", the "data providing user" can apply for the data set in the data storage in the data market.

2 is a flow chart of the data market transaction of the present invention, which is divided into a S210 racking program and an S220 purchasing program. The following is a data set listing procedure for the "data providing user" and a data set purchasing procedure for the "data consumer user". Description.

The data collection user program of the "data providing users" is as follows: The data providing user has a data set of the locations and attendances of the 2015 Taipei bus stations, and the data collection is carried out through the S211 data set labeling and information registration procedures. The function of attribute labeling and classification, labeling the data set with the labeling information of traffic, bus, Taipei city, person, location, etc., and registering the name of the data set, the original data format, the data set field definition, and the data sample through the interface. Basic information such as files.

In the S211 program, the data set on the shelf is converted into the data storage and storage format provided by the platform, and then the user interface of the S212 data set pricing program is used to set the unit price of the data use authorization time unit for the data set. The authorized time unit is set to 1 month, and the price marked 1000 yuan is the price charged for the data set within 1 month. The Hadoop platform will write the unit price of the data set set by the "data provider" into the database. In the middle store, the data set after the shelf is displayed in the data dictionary for sale.

The data collection process of "data consumer users" is as follows: "data consumer users" in the program S221 to query the data dictionary and carry out the data purchase process, the user can sort, query, browse and search through the query interface of the data dictionary to select the purchase According to the information set labeling information, the data set will be found in the traffic and bus categories. It can also be searched by keyword search or comprehensive keyword search. For example, searching for "bus" by keyword will be based on The popularity and relevance of the data returned by the S223 program show the ranking of the search results. The higher the display order of the most relevant and hottest data, the more convenient the data consumer can find the data.

The query results displayed in the data dictionary contain information such as each data set name, dataset original data format, dataset field definition, data sample file, dataset browsing times, dataset download times, data popularity analysis results, etc. Data consumer users" browse.

When the data consumer purchases the data set purchase behavior, it is necessary to determine the use period of the data set to be purchased. The time unit purchased must be an integral multiple of the data use authorization time unit. In this embodiment, the data consumer user purchases 2 Monthly data usage period.

After the purchase is completed, the program S222 will be started to record the data collection purchase information. The Hadoop platform will record the purchase behavior of the "data consumer user". The record content includes the data set name, the data set labeling attribute, the user purchase date, The number of purchases, the purchase price, the purchase period, the data transfer amount, and the information in the data dictionary.

When each purchase occurs, the program S223 immediately returns the popularity analysis model to the data dictionary, and uses the instant streaming component to transmit the purchase record to the analysis framework for the update operation of the popularity model, and each new purchase information will be included in the analysis. Basic data, analysis algorithms can use, for example, collaborative filtering analysis with associated rule analysis algorithms, but not limited to the above combination, instant calculation of the popularity recommendation model, and the latest popularity recommendation model is instantly fed back to the data through the streaming component In the dictionary to optimize the query, other "data consumer users" can be recommended and optimized based on the latest sales status when querying the data dictionary.

The Hadoop platform will set the file according to the authorized data set of the data set purchased by the user in the data permission setting of the program S224 according to the data set purchased by the “data consumer user”. The file contains the user identification code, the data set identification code, Dataset type, dataset path, and usage time. The permission rule file is transferred to the S130 program. The Agent (agent) embedded in each of the Hadoop platforms actually manages the rights, and controls the read permission and time limit of the data set and its actual files.

The data set purchased by the Hadoop platform for "data consumer users" will be authorized for use time limit by the time limit for use. The purchased data set will be authorized for use within the period of use purchased by the "data consumer user", in this embodiment. The period of time that the data consumer can purchase the data set is 2 months. Once the data set is used, the platform will automatically be deemed to be overdue and must be re-purchased before it can be used again.

Through the S130 integration platform computing resources and purchased data sets, users will immediately activate the authorization through the Hadoop platform authorization mechanism modified in this patent after purchasing the data set. After the permission is opened, the “data consumer users” can purchase 2 Within the life of the month, analyze and utilize the data of the data set, and automatically manage the permissions through the control layer constructed by the embedded agent on the Hadoop platform. The Hadoop platform tool is used. The data file and the actual file of the data set can be read in the warehouse. Therefore, the purchase user can use the analysis and calculation tool provided on the Hadoop platform to interface with the distributed data storage database interface, and directly perform analysis and calculation without any data copying and moving. The procedure to the user space.

3 is a comprehensive billing flowchart of the present invention. The billing item is mainly divided into two parts, namely, the S410 platform resource billing program and the S420 data market billing program, and the S410 program is mainly the HDFS space and operation on the Hadoop platform. The application fee for the resource, the S420 program is the related fee generated by the data exchange between users. The following describes the operation mode of the two programs.

In the S410 program, the S411: HDFS subscription space collection collects the HDFS space information that each user subscribes to. In this embodiment, the user subscribes to the HDFS space of 30T, and when the user performs data operation analysis (for example, Hive query, During the process of Spark Job...), the Vcore computing resources and Memory computing resources used by the S412: Vcore computing resource usage statistics program and the S413: Memory computing resource usage statistics program are collected, and the user operations in the resource management system are analyzed. The resource usage log file generated by the process is quantified by the number of Vcores used in the operation process multiplied by the actual execution time, the memory size configured by the operation process, and the actual execution time as the collection of usage statistics.

In this embodiment, the usage data collection period of the platform is defined as one day, and the data collection cycle programs 411, 412, and 413 are executed every other day to collect space and resource usage and continuously collect and store them in the database of the platform.

User 1 day Vcore usage = Σ (the number of Vcores used in the calculation process * the number of seconds used in the analysis operation) = 4500

Memory usage by user for 1 day = Σ (memory size (MB) configured in the calculation process * number of seconds used for analysis operation) = 3200000

The program S414 leases the resource billing by the one month billing cycle, and the relationship with the foregoing usage data collection period is: the billing period is composed of several usage data collection periods, and must be the usage data collection period. Integer multiple. The program S414 takes out the values written by the three programs S411, S412, S413 and the like from the database, and is the usage value of the resource used by the user on the platform in the current month, wherein the S412 and S413 programs will use the one-month billing cycle as the The statistical basis is to capture the usage within one month.

User Vcore usage within 1 month = Σ (user's 1 day Vcore usage) = 60000

User Memory usage within 1 month = Σ (user 1 day memory usage) = 5000000

The usage value information in the database includes the user's unique identification code, the number of accounts, the HDFS subscription space, the total Vcore usage, and the total amount of Memory. It is matched with the HDFS space unit price, the Vcore resource unit price, and the Memory resource unit price set on the Hadoop platform. Calculate the cost of the user's resources used on the Hadoop platform. The calculation formula is: user resource billing = (Vcore resource unit price * Vcore 1 month usage) + (Memory resource unit price * Memory within 1 month Usage) + (HDFS resource unit price * HDFS subscription space)

For example, if the system sets the Vcore resource unit price to be 0.005 yuan per core, the memory resource unit price is 0.0001 yuan per second, and the HDFS resource unit price is 80 yuan per TB, the user's platform in the current month. The resource billing is 3,200 yuan, and its calculation method is as follows: user resource billing = (60000 * 0.005) + (5000000 * 0.0001) + (30 * 80) = 3200

In the S420 program, the data market transaction record program is first collected through the S421 program, and the user purchase records recorded in the S222 record data set purchase and use information program mentioned in the previous billing cycle are summarized. The user purchases the data sets and data sets of the stations and the number of passengers in Taipei City in 2015. The unit price is 1000 yuan, the data usage authorization time unit is 2 months, and the collected transaction records are transmitted to the S422 data market billing program. Perform account calculation processing.

Then, the S422 data market billing program is responsible for calculating the account information of the data market according to the one-month billing cycle set by the platform. The account information is divided into two parts: one is the purchase of the "data consumer user". The price is based on the data usage authorization time unit of the data set purchased by the “data consumer user” multiplied by the unit price of the data set indicated in the data dictionary; the second is the “data provider user” sales data set. The proceeds are based on the total sales of the data set minus a certain percentage of the total sales of the platform. The handling fee in this embodiment is 10%. The final income is the data provided by the user in the data market during the billing cycle. The income, the calculation formula is as follows: data consumer users in the data market billing = data set unit price * purchase unit n = 1000 * 2 = 2000

Data to provide user income = total sales of the shelf data set * (1 - handling fee ratio) = 2000 * 90% = 1800

The fee generated by the user in using the platform resource is obtained through the program S414, and the consumption (data consumer user) or the profit (data providing user) in the data market is obtained through the program S422, and the "S430 comprehensive billing" will be produced. Based on this, the user's income and expenditure is sent to the user. The Hadoop platform operator will charge the user resource billing and data market transaction fees for each user. The calculation formula and examples are as follows: data consumption user comprehensive billing = resource billing + data market billing = 3200 + 2000 = 5200

Hadoop platform operator income = resource charge + (data market total sales * commission fee ratio) = 3200 + (2000 * 10%) = 3400

In summary, the data set transaction and computing resource integration method of the present invention allows the data requester to pay for the required data set and can immediately perform computational analysis on the platform, thereby greatly reducing the inconvenience of constructing the analysis environment, and The data provider brings benefits and meets the needs of both ends of the data supply and demand.

The above is intended to be illustrative only and not limiting. Any equivalent modifications or alterations to the spirit and scope of the invention are intended to be included in the scope of the appended claims.

Claims (9)

 A data set transaction and computing resource integration method includes: allocating storage space and computing resources on a Hadoop platform according to a resource claim request by a user; running a data set trading platform on the Hadoop platform to provide the resource for renting The user corresponding to the request performs a data set transaction; and an authorization management agent module is set on the Hadoop platform, and the permission integration is performed according to the data set transaction request submitted by the user, and the user is directly stored on the Hadoop platform. Take a data set corresponding to the transaction through the data set.    The data set transaction and computing resource integration method of claim 1, further comprising the steps of: calculating a fee according to the data transaction of the user and the resource subscription request; and transmitting the fee to the user.    The data set transaction and computing resource integration method according to claim 1, wherein the step of running the data set trading platform on the Hadoop platform to provide the resource subscription request to the user to perform the data set transaction further comprises the following steps: Providing a data dictionary as a query purchase interface; the query purchase interface further provides the user a recommended data set according to a category browsing, a keyword query and a popularity analysis; and setting the data according to the data set selected by the user A set of usage rights.    The data set transaction and computing resource integration method according to claim 3, wherein the step of the authorization management agent module performing permission integration according to the data set transaction request submitted by the user further comprises the following steps: performing permission integration according to the usage authority And analyzing and computing the data set on a platform.    The data set transaction and computing resource integration method of claim 2, further comprising the steps of: charging according to the result of the data set transaction and a resource subscription status on the Hadoop platform; wherein the resource is rented The situation includes multiple usage log files.    The data set transaction and computing resource integration method according to claim 3, further comprising the steps of: determining that the data set transaction is a data set shelf program or a purchase data set program; if the data set is on the shelf program, performing the following Step: Label the data set; log in to the data set; and price the data set.    The data set transaction and computing resource integration method according to claim 6, if the purchase data set program, the following steps are performed: querying the data dictionary; purchasing the data set; recording a data set purchase information; purchasing according to the data set The information provides the popularity analysis to the data dictionary; and sets the usage rights.    The data set transaction and computing resource integration method according to claim 5, wherein the charging according to the resource subscription status on the Hadoop platform further comprises the following steps: performing a platform resource charging procedure, which comprises the following steps: obtaining The user rents a system space of one of the distributed file systems; obtains a computing amount of a virtual core; obtains a resource usage amount of a memory; and calculates the fee according to the system space, the operation amount, and the resource usage.    The data set transaction and computing resource integration method according to claim 5, wherein the result of the data set transaction further comprises the following steps: performing a data set transaction charging procedure, comprising the following steps: obtaining the data set trading platform a transaction result; and calculating the fee based on the result of the transaction.