CN109445919B - Online computing resource transaction system based on cloud service - Google Patents
Online computing resource transaction system based on cloud service Download PDFInfo
- Publication number
- CN109445919B CN109445919B CN201811220112.2A CN201811220112A CN109445919B CN 109445919 B CN109445919 B CN 109445919B CN 201811220112 A CN201811220112 A CN 201811220112A CN 109445919 B CN109445919 B CN 109445919B
- Authority
- CN
- China
- Prior art keywords
- computing resource
- user
- computing
- job
- platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/06—Buying, selling or leasing transactions
- G06Q30/0601—Electronic shopping [e-shopping]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Abstract
The invention provides an online computing resource transaction system based on cloud service, which comprises a computing resource transaction platform, at least one super computer cluster accessed to the computing resource transaction platform and a private cloud storage system appointed by the computing resource transaction platform. The invention ensures that the computing resources of the supercomputer can be shared by pushing the computing resources to the Internet for on-line job submission.
Description
Technical Field
The invention relates to the technical field of computers, in particular to an online computing resource transaction system based on cloud service.
Background
At present, research units, companies, colleges and universities and social circles build more and more supercomputer clusters, and the clusters of the units basically meet respective computing requirements. When the computing task is insufficient, very large computing resource waste is often caused, including the consumption of a series of public resources such as maintenance, power, network and the like of the supercomputer. Therefore, the utilization of the computing resources of the super computer is not maximized, and the computing resources of the super computer cannot be shared.
In the process of implementing the invention, the inventor finds that at least the following technical problems exist in the prior art:
each unit or company with the super computer is basically isolated from the internet, online transaction of computing resources cannot be achieved, and a user cannot directly and remotely send computing operation to the super computer; in addition, each super computer cannot share the operation calculation software, software calculation resources of own industry are basically met, the use frequency of the calculation software is low, resource waste on a software layer is caused, and the network transmission bottleneck cannot solve the operation of calculating larger data analysis class.
Disclosure of Invention
In order to solve the above problems, the present invention provides an online computing resource transaction system based on cloud services, which enables the computing resources of a supercomputer to be shared by pushing the computing resources onto the internet to submit online jobs.
The invention provides an online computing resource transaction system based on cloud service, which comprises a computing resource transaction platform, at least one supercomputer cluster accessed to the computing resource transaction platform and a private cloud storage system appointed by the computing resource transaction platform;
the computing resource trading platform is used for selecting the type of a scheduling system for operation, required application software, a hardware environment for operation and a payment mode according to the computing requirements of a user to complete the configuration of a computing resource environment, generating an online computing resource trading order, preparing the computing resource environment configured by the user and informing the user to submit job data;
the private cloud storage system is used for receiving and storing the operation data file which is uploaded by a user and encrypted and compressed by using a file compression password specified by the computing resource trading platform, wherein the file compression password is notified to the user by the computing resource trading platform through the online computing resource trading order;
the computing resource transaction platform is used for extracting corresponding job data files from the private cloud storage system by using job metadata in the online computing resource transaction order for security analysis after the job data files are transmitted, notifying a user to confirm if the job data files have security problems, reading the corresponding job data files into a super computer computing queue with computing capacity to wait for a job scheduling system of a super computer cluster in charge of computing to schedule if the job security verification is passed, so that the job scheduling system can compute the predicted start time of job computation and the queuing time required to be waited, and notifying the user of job running conditions in real time through the computing resource transaction platform;
the super computer responsible for calculation is used for performing operation calculation, generating a compression key after the operation calculation is finished, carrying out encryption compression on operation calculation results by using the compression key, and uploading the operation calculation results subjected to encryption compression processing to the private cloud storage system; the system is also used for generating a file extraction password, a file decompression password and a file extraction path after the operation calculation result after the encryption and compression processing is uploaded, and sending the file extraction password, the file decompression password and the file extraction path to the computing resource transaction platform;
and the computing resource transaction platform is used for sending the file extraction password, the file decompression password, the file extraction path and the charging bill to the user according to the payment mode selected by the user so as to facilitate the user to pay the fee and extract the operation computing result of the transaction.
Compared with the prior art, firstly, the online computing resource transaction system based on the cloud service can enable users to use online transaction computing resources through a network, enable common consumers to purchase computing resources at low cost, avoid independently constructing the supercomputer cluster by themselves, enable high-cost computer resources to be shared, and improve the utilization rate of the supercomputer; and secondly, the remote shared storage is used for carrying out remote operation submission, so that the data security of the user can be guaranteed from a data plane, and meanwhile, the network security of the super computer is better protected as no service port is required to be arranged on the super computer.
Drawings
Fig. 1 is a schematic structural diagram of an online computing resource transaction system based on cloud services according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention provides an online computing resource transaction system based on cloud service, which comprises a computing resource transaction platform, at least one supercomputer cluster accessed to the computing resource transaction platform and a private cloud storage system appointed by the computing resource transaction platform, as shown in figure 1.
The computing resource trading platform is used for selecting the type of a scheduling system for operation, required application software, a hardware environment for operation and a payment mode according to the computing requirements of a user to complete the configuration of a computing resource environment, generating an online computing resource trading order, preparing the computing resource environment configured by the user and informing the user to submit job data.
The scheduling system type may be PBS, slurm, LSF.
The hardware environment includes the number of computing nodes, whether a storage system or a faster data transfer channel is required, whether a user's computing job requires video rendering computation, and the like.
The private cloud storage system is used for receiving and storing the operation data file which is uploaded by a user and encrypted and compressed by using the file compression password specified by the computing resource trading platform, wherein the file compression password is notified to the user by the computing resource trading platform through the online computing resource trading order.
The computing resource transaction platform is used for extracting corresponding job data files from the private cloud storage system by using job metadata in the online computing resource transaction order for security analysis after the job data files are transmitted, notifying a user to confirm if the job data files have security problems, reading the corresponding job data files into a super computer computing queue with computing capacity to wait for a job scheduling system of a super computer cluster in charge of computing to schedule if the job security verification is passed, so that the job scheduling system can compute the predicted start time of job computation and the queuing time required to wait, and notifying the user of job running conditions in real time through the computing resource transaction platform.
The super computer responsible for calculation is used for performing operation calculation, generating a compression key after the operation calculation is finished, carrying out encryption compression on operation calculation results by using the compression key, and uploading the operation calculation results subjected to encryption compression processing to the private cloud storage system; and the encryption and compression module is also used for generating a file extraction password, a file decompression password and a file extraction path after uploading the operation calculation result after the encryption and compression processing is finished, and sending the file extraction password, the file decompression password and the file extraction path to the computing resource transaction platform.
And the computing resource transaction platform is used for sending the file extraction password, the file decompression password, the file extraction path and the charging bill to the user according to the payment mode selected by the user so as to facilitate the user to pay the fee and extract the operation computing result of the transaction.
Compared with the prior art, firstly, the online computing resource transaction system based on the cloud service can enable users to use online transaction computing resources through a network, enable common consumers to purchase computing resources at low cost, avoid independently constructing the supercomputer cluster by themselves, enable high-cost computer resources to be shared, and improve the utilization rate of the supercomputer; secondly, the remote operation submission by using the remote shared storage can ensure the user data security from a data layer, and meanwhile, the network security of the super computer is better protected as no service port is required to be arranged at the super computer end; moreover, the separated transmission of the order and the data ensures the safety of the user operation data, and even if the user intercepts the data when transmitting the data, the user can not see the real data of the user.
The computing resource transaction platform is further used for completing registration processes of online computing resource consuming users and the super computer cluster on the computing resource transaction platform, wherein the super computer cluster submits information of software and hardware environments, computing capacity and scheduling system types of the super computer cluster when the computing resource transaction platform registers.
The computing resource trading platform is connected with each super computer through a special network, monitors the busy situation of computing tasks of each super computer in real time to determine whether cluster computing resources meeting online computing resource trading orders exist or not, selects a super computer cluster meeting the requirements of a user working environment according to the neighborhood of a city where a user is located when the cluster computing resources meeting the online computing resource trading orders are determined, and informs an online computing resource consuming user to carry out remote job submission.
The computing resource transaction platform provides two payment modes of prepayment and credit real-time charging.
The prepayment mode is that when the user estimates the computer, the computing resource transaction platform evaluates the user operation and the charging standard of each machine, and calculates and deducts the cost when the operation is calculated.
And stopping charging when the user operation calculation is completed, suspending the operation if the user balance is insufficient, and performing scheduling operation by the scheduling system according to the real-time environment evaluation again after the user continues to charge.
The credit real-time charging mode is that the computing resource transaction platform records the number of machine hours of operation, calculates the charge after the operation calculation is completed and provides a charging bill.
The private cloud storage system is further used for storing application software which is purchased in advance by the computing resource trading platform and used for computing, so that all super computer clusters accessing the computing resource trading platform can be used in an open mode.
When the private cloud storage system does not have application software required by the user specification, the computing resource trading platform prompts the user to upload application software provided by the user to the private cloud storage system or to purchase the application software specified by the online computing resource trading order.
If the authorization of the application software provided by the user can be repeatedly used, the computing resource transaction platform gives a certain proportion of preference to the user when computing the cost; if the application software provided by the user can not be authorized to be used repeatedly, the unloading operation is executed after the computing task is completed.
Compared with the prior art, the invention has the following remarkable advantages: the computing resource transaction platform can inform a user of the queuing time and the job running condition of the expected start time of job calculation to be waited in real time, so the user can log in the job scheduling system again to control job task calculation, for example, the priority of job calculation can be adjusted, specifically, the user can adjust the job and the priority after the job enters the calculation queue, the higher the priority is, the higher the charge is, if the job starts to be calculated, the user has a problem in sending job data or job scripts, the job can be stopped or hung midway, and the job charging and the computer are correspondingly stopped, thereby reducing the loss for the user to the maximum extent.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The above description is only for the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (6)
1. An online computing resource transaction system based on cloud service is characterized by comprising a computing resource transaction platform, at least one supercomputer cluster accessed to the computing resource transaction platform and a private cloud storage system designated by the computing resource transaction platform;
the computing resource trading platform is used for selecting the type of a scheduling system for operation, required application software, a hardware environment for operation and a payment mode according to the computing requirements of a user to complete the configuration of a computing resource environment, generating an online computing resource trading order, preparing the computing resource environment configured by the user and informing the user to submit job data;
the private cloud storage system is used for receiving and storing the operation data file which is uploaded by a user and encrypted and compressed by using a file compression password specified by the computing resource trading platform, wherein the file compression password is notified to the user by the computing resource trading platform through the online computing resource trading order;
the computing resource transaction platform is used for extracting corresponding job data files from the private cloud storage system by using job metadata in the online computing resource transaction order for security analysis after the job data files are transmitted, notifying a user to confirm if the job data files have security problems, reading the corresponding job data files into a super computer computing queue with computing capacity to wait for a job scheduling system of a super computer cluster in charge of computing to schedule if the job security verification is passed, so that the job scheduling system can compute the predicted start time of job computation and the queuing time required to be waited, and notifying the user of job running conditions in real time through the computing resource transaction platform;
the super computer cluster responsible for the calculation is used for performing operation calculation, generating a compression key after the operation calculation is completed, carrying out encryption compression on operation calculation results by using the compression key, and uploading the operation calculation results subjected to the encryption compression processing to the private cloud storage system; the system is also used for generating a file extraction password, a file decompression password and a file extraction path after the operation calculation result after the encryption and compression processing is uploaded, and sending the file extraction password, the file decompression password and the file extraction path to the computing resource transaction platform;
and the computing resource transaction platform is used for sending the file extraction password, the file decompression password, the file extraction path and the charging bill to the user according to the payment mode selected by the user so as to facilitate the user to pay the fee and extract the operation computing result of the transaction.
2. The system of claim 1, wherein the computing resource trading platform is further configured to perform a registration process between the online computing resource consuming user and the cluster of supercomputers on the computing resource trading platform, wherein the cluster of supercomputers submits information about its own software and hardware environment, computing power, and scheduling system type when the computing resource trading platform registers.
3. The system of claim 1, wherein the computing resource trading platform is connected with each supercomputer through a private network, monitors the busy situation of the computing task of each supercomputer in real time to determine whether cluster computing resources meeting the online computing resource trading order exist, selects a supercomputer cluster meeting the demands of the user working environment according to the proximity of the city where the user is located when determining that the cluster computing resources meeting the online computing resource trading order exist, and informs the online computing resource consuming user to carry out remote job submission.
4. The system of claim 1, wherein the computing resource trading platform provides both prepaid and credit real-time charging;
when the prepayment mode is that a user estimates a computer, the computing resource transaction platform evaluates user operation and a charging standard per machine time, and calculates and deducts cost when the operation is calculated;
the credit real-time charging mode is that the computing resource transaction platform records the number of machine hours of operation, calculates the charge after the operation calculation is completed and provides a charging bill.
5. The system of claim 1, wherein the private cloud storage system is further configured to store application software for all computing pre-purchased by the computing resource trading platform for open use to all supercomputer clusters accessing the computing resource trading platform.
6. The system of claim 1, wherein when the private cloud storage system does not have application software that the user specifies to be needed, the computing resource trading platform prompts the user to upload application software provided by the user to the private cloud storage system or to make a purchase according to application software specified by an online computing resource trading order.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811220112.2A CN109445919B (en) | 2018-10-19 | 2018-10-19 | Online computing resource transaction system based on cloud service |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811220112.2A CN109445919B (en) | 2018-10-19 | 2018-10-19 | Online computing resource transaction system based on cloud service |
Publications (2)
Publication Number | Publication Date |
---|---|
CN109445919A CN109445919A (en) | 2019-03-08 |
CN109445919B true CN109445919B (en) | 2021-07-23 |
Family
ID=65546757
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201811220112.2A Active CN109445919B (en) | 2018-10-19 | 2018-10-19 | Online computing resource transaction system based on cloud service |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN109445919B (en) |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110599148B (en) * | 2019-09-16 | 2022-05-31 | 广州虎牙科技有限公司 | Cluster data processing method and device, computer cluster and readable storage medium |
CN111198755B (en) * | 2019-12-23 | 2023-07-21 | 曙光信息产业(北京)有限公司 | Pre-charging device and method based on SLURM job scheduling system |
CN113703952B (en) * | 2020-05-20 | 2023-10-10 | 山东省计算中心(国家超级计算济南中心) | Resource allocation method for queue resource scheduling based on supercomputer |
CN114827259B (en) * | 2022-03-02 | 2023-09-22 | 杭州华橙软件技术有限公司 | Data processing method, computer readable storage medium and device |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102546722A (en) * | 2010-12-27 | 2012-07-04 | 北京爱国者信息技术有限公司 | Cloud service sharing and trading platform and system |
US20170123858A1 (en) * | 2014-10-02 | 2017-05-04 | International Business Machines Corporation | Resource substitution and reallocation in a virtual computing environment |
CN106959884A (en) * | 2017-03-23 | 2017-07-18 | 联想(北京)有限公司 | A kind of data processing method, electronic equipment and cluster |
CN107665456A (en) * | 2017-09-25 | 2018-02-06 | 广州地理研究所 | A kind of space big data transaction platform based on cloud GIS |
CN107705025A (en) * | 2017-10-16 | 2018-02-16 | 曙光信息产业(北京)有限公司 | Supercomputer and its operating method |
CN108427601A (en) * | 2017-02-13 | 2018-08-21 | 北京航空航天大学 | A kind of cluster transaction processing method of privately owned chain node |
-
2018
- 2018-10-19 CN CN201811220112.2A patent/CN109445919B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102546722A (en) * | 2010-12-27 | 2012-07-04 | 北京爱国者信息技术有限公司 | Cloud service sharing and trading platform and system |
US20170123858A1 (en) * | 2014-10-02 | 2017-05-04 | International Business Machines Corporation | Resource substitution and reallocation in a virtual computing environment |
CN108427601A (en) * | 2017-02-13 | 2018-08-21 | 北京航空航天大学 | A kind of cluster transaction processing method of privately owned chain node |
CN106959884A (en) * | 2017-03-23 | 2017-07-18 | 联想(北京)有限公司 | A kind of data processing method, electronic equipment and cluster |
CN107665456A (en) * | 2017-09-25 | 2018-02-06 | 广州地理研究所 | A kind of space big data transaction platform based on cloud GIS |
CN107705025A (en) * | 2017-10-16 | 2018-02-16 | 曙光信息产业(北京)有限公司 | Supercomputer and its operating method |
Non-Patent Citations (1)
Title |
---|
《一种云计算资源的定价与分配方法》;勾建磊;《中国优秀硕士学位论文全文数据库·信息科技辑》;20170228;全文 * |
Also Published As
Publication number | Publication date |
---|---|
CN109445919A (en) | 2019-03-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109445919B (en) | Online computing resource transaction system based on cloud service | |
US11711268B2 (en) | Methods and apparatus to execute a workload in an edge environment | |
US11368517B2 (en) | Secure automated resource-exchange system | |
US9229603B2 (en) | Methods, systems, apparatuses, and computer-readable mediums for provisioning petrotechnical workflows in a cloud computing environment | |
Goudarzi et al. | Energy-efficient virtual machine replication and placement in a cloud computing system | |
US9742652B2 (en) | Proactive identification of hotspots in a cloud computing environment | |
US8996647B2 (en) | Optimizing storage between mobile devices and cloud storage providers | |
CN103761147B (en) | The management method and system of calculated examples in a kind of cloud platform | |
CN106506670B (en) | A kind of cloud platform virtual resource high speed dispatching method and system | |
CN103544064A (en) | Cloud computing method, cloud management platform and client | |
US9253048B2 (en) | Releasing computing infrastructure components in a networked computing environment | |
CN103092671A (en) | Method and system for determining virtual machine image pattern distributions in a networked computing environment | |
US11113244B1 (en) | Integrated data pipeline | |
US20120323821A1 (en) | Methods for billing for data storage in a tiered data storage system | |
Singh et al. | Multi-disciplinary research issues in cloud computing | |
Aggarwal et al. | Solving user priority in cloud computing using enhanced optimization algorithm in workflow scheduling | |
CN102117468A (en) | Method for leasing instance based on cloud computing | |
Rosa et al. | Bionimbuz: A federated cloud platform for bioinformatics applications | |
CN105357239B (en) | Method and device for providing service and method and device for obtaining service | |
CN113032947A (en) | Industrial design simulation system based on virtual application technology | |
Yang | Energy-efficient cloud data center with fair service level agreement for green computing | |
Muruganandam et al. | Blockchain based adaptive resource allocation in cloud computing | |
CN111861611A (en) | Resource processing method and device based on block chain, electronic equipment and storage medium | |
CN110515725A (en) | Service request processing method, computer storage medium, server and system | |
KR20200100470A (en) | Blockchain Network Node for Processing Transaction |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
TR01 | Transfer of patent right |
Effective date of registration: 20220727 Address after: 100089 building 36, courtyard 8, Dongbeiwang West Road, Haidian District, Beijing Patentee after: Dawning Information Industry (Beijing) Co.,Ltd. Patentee after: DAWNING INFORMATION INDUSTRY Co.,Ltd. Address before: 100193 No. 36 Building, No. 8 Hospital, Wangxi Road, Haidian District, Beijing Patentee before: Dawning Information Industry (Beijing) Co.,Ltd. |
|
TR01 | Transfer of patent right |