CN115883657A - Cloud disk service accelerated scheduling method and system - Google Patents
Cloud disk service accelerated scheduling method and system Download PDFInfo
- Publication number
- CN115883657A CN115883657A CN202211497026.2A CN202211497026A CN115883657A CN 115883657 A CN115883657 A CN 115883657A CN 202211497026 A CN202211497026 A CN 202211497026A CN 115883657 A CN115883657 A CN 115883657A
- Authority
- CN
- China
- Prior art keywords
- distribution network
- content distribution
- user
- scheduling
- content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 22
- 230000008569 process Effects 0.000 claims abstract description 4
- 230000006399 behavior Effects 0.000 claims description 27
- 238000004458 analytical method Methods 0.000 claims description 23
- 230000005540 biological transmission Effects 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 9
- 238000001514 detection method Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 2
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000001133 acceleration Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 230000014759 maintenance of location Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Abstract
The invention relates to a cloud disk service accelerated scheduling method, which is characterized in that the scheduling method comprises the following steps: when a user requests to access the cloud disk, the user side service scheduling collects user behavior data, stores the user behavior data into a user side database, and sends the user behavior data and the user request to a content distribution network; the content distribution network judges whether the content distribution network node server stores the user request content according to the user behavior data, and returns a result when the content distribution network node server stores the user request content, otherwise, the content distribution network sends the user request to the object storage server; and the object storage server processes the user request content and stores the user request access content to a content distribution network for the accelerated scheduling of the cloud disk service. The invention solves the technical problems that the access, uploading and downloading speeds of the cloud disk server are low, and the storage layout of the cloud disk cannot be optimized according to the preference of a client.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method and a system for accelerating scheduling of cloud disk services.
Background
In the prior art, in order to improve the service performance of a product, a network service provider continuously increases network servers in various areas, and also performs content delivery through a CDN technology, and redirects a user's access to a server to a streaming media server closest to the user by using a global load balancing technology, which not only can improve the user experience, but also improves the availability of a website. With the continuous development of cloud disk services, the uploading and downloading of cloud disk servers and the massive HTTP access of cloud disk servers (including PC terminals, mobile terminals, and applet terminals) all bring huge access flow pressure to the cloud disk servers and the object storage node servers. For servers in different regions, if the access pressure borne by a server at a certain node is too high, the operation efficiency of the node is too low, and even the node stops service in severe cases; for a cloud disk system, the uploading and downloading rate directly affects the use experience of a user, and how to reasonably and effectively schedule a cloud disk server and an object storage server through the system enables the cloud disk server and the object storage server to meet various increasing access requirements of the user and ensures the uploading and downloading rate when the user uses a cloud disk is a key problem faced by the current cloud disk system.
Therefore, it is desirable to have a method and a system for cloud disk service accelerated scheduling, which can solve the problems in the prior art.
Disclosure of Invention
In view of the above disadvantages and shortcomings of the prior art, the present invention provides a method and a system for scheduling cloud disk services in an accelerated manner, which solve the technical problems that a cloud disk server is slow in access and upload/download speed and cannot optimize the storage layout of a cloud disk according to the preference of a client.
In order to achieve the above object, the present invention provides a method for accelerating scheduling of cloud disk services, where the scheduling method includes:
when a user requests to access the cloud disk, the user side service scheduling collects user behavior data, stores the user behavior data into a user side database, and sends the user behavior data and the user request to a content distribution network;
the content distribution network judges whether the content distribution network node server stores the user request content according to the user behavior data, when the content distribution network node server stores the user request content, a result is returned, otherwise, the content distribution network sends the user request to the object storage server;
and the object storage server processes the user request content and stores the user request access content to a content distribution network for accelerated scheduling and use of the cloud disk service.
A system for cloud disk service expedited scheduling, the system comprising:
the user service scheduling module is used for storing the user behavior analysis data and sending the user behavior analysis data and the user request to the content distribution network service scheduling module;
the content distribution network service scheduling module determines a content distribution network server to return a result according to the user behavior analysis data, or sends a user request to the object storage server;
and the object storage server scheduling module executes the user request according to the load balancing strategy and stores the access content of the user request to the content distribution network scheduling module.
Optionally, the user behavior analysis data includes data of IP attribution, mobile phone number attribution, resident location, whether the user is a rights user, and user end platform analysis user behavior characteristics when the user accesses the mobile phone.
Optionally, the content distribution network service scheduling module includes: and detecting and selecting nodes by the content distribution network link and processing the failure of the content distribution network.
Optionally, the content delivery network link probing selecting node:
sending a detection message to a content distribution network node of a specified object to acquire link transmission time;
storing all responded content distribution network nodes into a content distribution network service scheduling module according to link transmission time, selecting a content distribution network node with the shortest link transmission time from a user side to a content distribution network edge node, and sending user request content to the content distribution network edge node along the selected content distribution network node;
and sending a message to the content distribution network node at regular time, acquiring the transmission time of the current link, and directly returning to the content distribution network node according to the cache content in the content distribution network service scheduling module when receiving the request of the user side again.
Optionally, the content delivery network failure handling: when the content distribution network node fails and the content distribution network scheduling module cannot acquire the transmission time of the corresponding link, the current content distribution network node is set as a failure node, and the content distribution network scheduling module detects the connection condition of the relevant link at regular time.
Optionally, the load balancing policy includes: a proximity principle, a residual storage space principle, a current connection number principle and a rights and interests user principle.
An electronic device, comprising: a memory and a server; the service is used for executing the computer program stored in the memory to realize the system for executing the cloud disk service accelerated scheduling.
The invention has the beneficial effects that:
the invention provides a method and a system for accelerating scheduling of cloud disk services, which are special for an intelligent scheduling system of the cloud disk services, and provide an accelerating flow for the services of a cloud disk server by applying basic load balancing and content distribution network services.
Drawings
Fig. 1 is a schematic diagram of a system for accelerated scheduling based on cloud disk services according to the present invention.
Fig. 2 is a flowchart of a cloud disk service-based accelerated scheduling method according to the present invention.
Fig. 3 is a flow chart of content delivery network service scheduling.
Detailed Description
For the purpose of better explaining the present invention and to facilitate understanding, the present invention will be described in detail by way of specific embodiments with reference to the accompanying drawings.
In order to better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
A Content Delivery Network (CDN) refers to a computer Network system connected to each other via the internet, and a server closest to each user is used to send music, pictures, videos, application programs, and other files to the user more quickly and reliably. The content distribution network is capable of providing high performance, scalable and low cost network content to users. The main scheduling modules of the scheduling system at the content distribution network side are as follows: detecting a content distribution network link, selecting a content distribution network node, and processing a content distribution network fault.
The content distribution network avoids bottlenecks and links on the internet which can affect the data transmission speed and stability, so that the content transmission is faster and more stable. The content distribution network system can redirect the user's request to the service node closest to the user in real time according to the network traffic and the comprehensive information such as the connection of each node, the load condition, the distance to the user, the response time, and the like. The method aims to enable a user to access required content nearby, solve the congestion problem of the Internet network and improve the response speed of the user for accessing the site.
The content distribution network adopts more cache servers (content distribution network edge nodes) and is arranged in a region or network with relatively concentrated user access. When a user accesses a website, the access of the user is pointed to the cache server closest to the user by using a global load technology, and the cache server responds to the user request.
The use of content distribution network services alone has its disadvantages: for specific server requirements such as a cloud disk server, a general content distribution network acceleration solution cannot completely meet daily scheduling requirements, and cannot collect user data for subsequent analysis to provide more intelligent services; secondly, for a server with a certain expansion requirement, a simple content distribution network service cannot completely meet the requirement.
Load Balance (Load Balance) is to share a plurality of operation units to execute, such as WEB servers, FTP servers, enterprise key application servers and other key task servers, so as to complete work tasks together. Load balancing builds on existing network architectures and provides an inexpensive, efficient, transparent way to extend the bandwidth of network devices and servers, increase throughput, enhance network data processing capabilities, and increase network flexibility and availability.
If a user wants to access the content of a website, the specific CDN acceleration principle steps are:
(1) when a user clicks the content on the APP, the APP searches for IP address resolution by a local DNS (domain name resolution system) according to the URL address; (2) the local DNS system will transfer the domain name resolution right to the DNS server special for the content distribution network; (3) the DNS server special for the content distribution network returns the IP address of the global load balancing equipment of the content distribution network to the user; (4) a user initiates a content URL access request to load balancing equipment of a content distribution network; (5) the CDN load balancing equipment selects a cache server of a region to which the user belongs according to the user IP address and the content URL requested by the user; (6) the load balancing equipment informs the IP address of the cache server of the user, and the user sends a request to the selected cache server; (7) a user sends a request to a cache server, and the cache server responds to the user request and transmits the content required by the user to a user terminal; (8) if the cache server does not have the content desired by the user, the cache server requests the content from the source server of the website; (9) and the source server returns the content to the cache server, the cache server sends the content to the user, and the content is judged not to be cached on the cache server according to a cache strategy defined by the user.
The content distribution network adopts more cache servers (content distribution network edge nodes) and is arranged in a region or network with relatively concentrated user access. When a user accesses a website, the access of the user is pointed to the cache server closest to the user by using a global load technology, and the cache server responds to the user request.
Example one
As shown in fig. 1 and 2, a system for cloud disk service accelerated scheduling, the system comprising:
the user side service scheduling module 201 is used for storing the user behavior analysis data and sending the user behavior analysis data and the user request to the content distribution network service scheduling module;
the content delivery network side service scheduling module 202 determines that the content delivery network server returns a result according to the user behavior analysis data, or sends a user request to the object storage server;
and the server side scheduling module 203 executes the user request according to the load balancing strategy and stores the content accessed by the user request to the content distribution network scheduling module.
The user-side service scheduling module 201 includes a user behavior analysis system, which is used for providing insight into secrets hidden in user behaviors, and the platform provides a plurality of analysis modes and scenarios such as event analysis, retention analysis, transformation analysis, user grouping, user retention and the like. Some access data can be recorded in a database when a user accesses, a data base can be laid for generating a user portrait, the storage layout of a cloud disk file of the user can be optimized, and the IP attribution/mobile phone number attribution/resident position/whether the IP attribution/mobile phone number attribution/resident position/the VIP (interest user) is sent to a CDN service for subsequent scheduling, so that the exclusive CDN service of the cloud disk server is required to be deployed on the user side, static website resources and files are required to be intelligently cached for HTTP requests of the cloud disk server, the access speed of the cloud disk website server is increased on the user side, the files of the cloud disk are intelligently cached according to local bandwidth cost and idle conditions, the downloading speed of the cloud disk files is increased, and specific requests are intelligently scheduled to a specific cloud disk server according to a self-defined routing rule.
The user behavior analysis system mainly relates to data acquisition and data analysis, and information transmission:
data acquisition and data analysis: no matter whether the scheduling system is a WEB end, a PC end, an applet end or an APP end, the scheduling system can be accessed to a data acquisition function, data acquisition information mainly comprises information which can identify a user such as a mobile phone number of the user, an IP (Internet protocol) of the user using the cloud disk service currently and the location, resident place, the type of the current operation of the user, the size of related files and the like, the information is stored in a database, the scheduling system of a subsequent cloud disk can be accessed to AI (Internet access) capacity, the common operation of the user is analyzed by adopting a deep learning method, and reasonable scheduling is carried out aiming at different users.
Information transmission: the information is mainly used for subsequent CDN and object storage server scheduling, information such as a mobile phone number, an IP address, a place of residence and the like in a request message of a user is transmitted to a scheduling system, and scheduling and transmission optimization are performed according to different strategies.
User side routing: performing hash operation on a user side according to a custom routing rule, for example, according to an IP address or mobile phone home location information, so as to ensure that a user can normally acquire file information uploaded on a server at any time, and when a resident position changes, redirecting the resident position to a server at the resident position according to database information recorded by a user behavior analysis system; for a cloud disk system, in order to provide better service for a VIP (rightful user), a user behavior analysis system can simultaneously acquire whether a current request user is the rightful user, a server of the rightful user can be faster and more reliable, the customized routing rule further comprises CDN link RTT calculation and room link RTT calculation, the current optimal CDN link and the room link after scheduling are intelligently detected, and faster and more reliable cloud disk service is provided. And finally, the user side scheduling selects an optimal CDN according to a self-defined routing rule, sends the request message to the server, and then performs the intelligent scheduling system CDN side service scheduling module.
The content delivery network side service scheduling module 202: detecting a CDN link, selecting a CDN node and processing a CDN fault.
The flow of the content delivery network side service scheduling module 202 is as follows:
link probing and node selection rules at CDN: (1) Sending a detection message to a specified object CDN node, acquiring link transmission time (RTT), and selecting a proper CDN node according to the length of a link; (2) Storing all the CDN nodes which obtain the response into a database according to the RTT, returning a CDN node server with the shortest current RTT, and forwarding the request to the CDN server; (3) And sending a message to the CDN node at regular time to obtain the latest connection condition of the current link, and directly returning to the CDN node according to the cache content when receiving the request of the client again.
CDN fault processing: when the CDN node fails and the scheduling module cannot obtain the corresponding RTT, the current CDN node is set as an unreachable node (failed node), and the scheduling system still detects the connection condition of the relevant link at regular time, and can continue to forward the request to the node when the CDN node recovers.
Under special conditions, for example, corresponding resources or files requested by the user are not cached in the CDN node, the CDN node needs to directly forward the request packet to the corresponding object storage server to perform server-side service scheduling, and the CDN node obtains the resources requested by the current user, so that the user can enjoy the accelerated service on the CDN side when requesting the same resources next time. For the situation that the CDN edge node does not have the resource, the CDN needs to send the request to the server-side service scheduling module continuously to perform server-side scheduling.
The server-side scheduling module 203: object storage server performance detection, object server load balancing
Object storage server performance probing:
although the CDN node can provide a large amount of acceleration services, for a file or a resource that is not stored on the CDN node, the CDN needs to send a request for the resource to the target storage server, so a server-side scheduling module in the scheduling system is also indispensable, and the main workflow thereof is as follows:
(1) After receiving the request message, the CDN node indicates that the resource is stored in the object server if the current file or resource is not stored in its own internal server, and the CDN cannot provide acceleration service at this time and needs to send the request to the object server.
(2) Searching the link length from the current node to all the cloud disk OSS node servers, and the number of the current processing requests (the current task number) of the target node server, if the request is related to the cloud disk file, acquiring the physical storage residual capacity of the target node server and the supported storage mode
(3) The scheduling system performs performance detection on the object storage server, and returns to the object storage server with the shortest RTT, for example, a request sent by a user in south china is generally sent to an object storage server cluster in south china.
Detecting the performance of the server: the current read-write frequency of the target node server; and acquiring the amount of free data storage space of the target node server for the object storage server.
Load Balance means that the Load Balance is shared by a plurality of operation units to be executed, such as a WEB server, an FTP server, an enterprise key application server and other key task servers, so as to jointly complete work tasks. Load balancing builds on existing network architectures and provides an inexpensive, efficient, transparent way to extend the bandwidth of network devices and servers, increase throughput, enhance network data processing capabilities, and increase network flexibility and availability.
And (3) load balancing strategy:
(1) The principle of the approach is as follows: the distance from the IP request to the server
(2) Residual capacity principle: disk residual capacity size of OSS node server
(3) Current connection number principle: number of current connection requests of server
(4) The equity user principle: the user's equity level
In order to face an emergency, the scheduling system needs to further forward a request message of the user according to a load balancing policy, so as to ensure that the user can obtain a normal server response.
Example two
As shown in fig. 2, the flow of the content distribution network service scheduling module includes: the method comprises the steps of obtaining a message requested by a user terminal, analyzing the name of a requested resource in the current message, judging whether a current content distribution network edge node has a cache of the resource, if so, directly returning a result from the current content distribution network edge node, and if not, forwarding the requested content of the user terminal to an object storage server for load balancing strategy scheduling, and returning the result.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, the claims should be construed to include preferred embodiments and all such variations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include such modifications and variations.
Claims (10)
1. A cloud disk service accelerated scheduling method is characterized in that the scheduling method comprises the following steps:
when a user requests to access the cloud disk, the user side service scheduling acquires user behavior data, stores the user behavior data into a user side database, and sends the user behavior data and the user request to a content distribution network;
the content distribution network judges whether the content distribution network node server stores the user request content according to the user behavior data, and returns a result when the content distribution network node server stores the user request content, otherwise, the content distribution network sends the user request to the object storage server;
and the object storage server processes the user request content and stores the user request access content to a content distribution network for the accelerated scheduling of the cloud disk service.
2. The method for cloud disk service accelerated scheduling of claim 1, wherein: and the content distribution network selects a content distribution network node server according to a preset routing rule.
3. The method for cloud disk service accelerated scheduling of claim 1, wherein: and the object storage server selects a proper object storage server to process the content requested by the user according to the load balancing scheduling strategy.
4. A cloud disk service accelerated scheduling system is characterized in that: the system comprises:
the user service scheduling module is used for storing the user behavior analysis data and sending the user behavior analysis data and the user request to the content distribution network service scheduling module;
the content distribution network service scheduling module determines a return result of the content distribution network server according to the user behavior analysis data, or sends a user request to the object storage server;
and the object storage server scheduling module executes the user request according to the load balancing strategy and stores the access content of the user request to the content distribution network scheduling module.
5. The system for cloud disk service expedited scheduling of claim 4, wherein: the user behavior analysis data comprises IP attribution, mobile phone number attribution and resident position when the user accesses, and data of whether the user behavior analysis data is a rights and interests user and user side platform analysis user behavior characteristics.
6. The system for cloud disk service expedited scheduling of claim 4, wherein: the content distribution network service scheduling module comprises: and detecting and selecting nodes by the content distribution network link and processing the fault of the content distribution network.
7. The system for cloud disk service expedited scheduling of claim 6, wherein: the content delivery network link probing selection node:
sending a detection message to a content distribution network node of a specified object to acquire link transmission time;
storing all responded content distribution network nodes into a content distribution network service scheduling module according to link transmission time, selecting a content distribution network node with the shortest link transmission time from a user side to a content distribution network edge node, and sending user request content to the content distribution network edge node along the selected content distribution network node;
and sending a message to the content distribution network node at regular time, acquiring the transmission time of the current link, and directly returning to the content distribution network node according to the cache content in the content distribution network service scheduling module when receiving the request of the user side again.
8. The system for cloud disk service accelerated scheduling of claim 6, wherein: the content distribution network failure processing: when the content distribution network node fails and the content distribution network scheduling module cannot acquire the transmission time of the corresponding link, the current content distribution network node is set as a failure node, and the content distribution network scheduling module detects the connection condition of the relevant link at regular time.
9. The system for cloud disk service expedited scheduling of claim 4, wherein: the load balancing strategy comprises the following steps: a proximity principle, a residual storage space principle, a current connection number principle and a rights and interests user principle.
10. An electronic device, comprising: a memory and a server; the service is configured to execute the computer program stored in the memory to implement the system for performing cloud disk service accelerated scheduling of any of claims 4 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211497026.2A CN115883657A (en) | 2022-11-25 | 2022-11-25 | Cloud disk service accelerated scheduling method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211497026.2A CN115883657A (en) | 2022-11-25 | 2022-11-25 | Cloud disk service accelerated scheduling method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115883657A true CN115883657A (en) | 2023-03-31 |
Family
ID=85764197
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211497026.2A Pending CN115883657A (en) | 2022-11-25 | 2022-11-25 | Cloud disk service accelerated scheduling method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115883657A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117194054A (en) * | 2023-11-06 | 2023-12-08 | 西安一二三云计算有限公司 | Cloud disk resource scheduling method based on user image |
-
2022
- 2022-11-25 CN CN202211497026.2A patent/CN115883657A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117194054A (en) * | 2023-11-06 | 2023-12-08 | 西安一二三云计算有限公司 | Cloud disk resource scheduling method based on user image |
| CN117194054B (en) * | 2023-11-06 | 2024-02-23 | 西安一二三云计算有限公司 | Cloud disk resource scheduling method based on user image |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10506029B2 (en) | Content distribution network | |
| RU2549135C2 (en) | System and method for providing faster and more efficient data transmission | |
| US8903950B2 (en) | Personalized content delivery using peer-to-peer precaching | |
| CN106031130B (en) | Content distribution network framework with edge proxies | |
| US7035907B1 (en) | Manipulating content objects to control their display | |
| US7555532B2 (en) | Advanced content and data distribution techniques | |
| US8402137B2 (en) | Content management | |
| CN107094176B (en) | Method and system for caching data traffic on a computer network | |
| US9588854B2 (en) | Systems and methods for a secondary website with mirrored content for automatic failover | |
| US20100115613A1 (en) | Cacheable Mesh Browsers | |
| US8370420B1 (en) | Web-integrated display of locally stored content objects | |
| US11159642B2 (en) | Site and page specific resource prioritization | |
| EP3066577B1 (en) | Content node selection using network performance profiles | |
| US11281730B1 (en) | Direct leg access for proxy web scraping | |
| CN115883657A (en) | Cloud disk service accelerated scheduling method and system | |
| CN107682281A (en) | A kind of application management method of SDN switch and SDN switch | |
| EP4227829A1 (en) | Web scraping through use of proxies, and applications thereof | |
| US8832281B2 (en) | Utilizing resources of a peer-to-peer computer environment | |
| CN114268631A (en) | Low-delay network system, communication connection method thereof and readable storage medium | |
| KR20150011087A (en) | Distributed caching management method for contents delivery network service and apparatus therefor | |
| WO2023280593A1 (en) | Web scraping through use of proxies, and applications thereof | |
| EP3347825A1 (en) | Execution of per-user functions at cache nodes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |