CN114979271A - CDN cache layered scheduling method based on edge cloud computing - Google Patents
CDN cache layered scheduling method based on edge cloud computing Download PDFInfo
- Publication number
- CN114979271A CN114979271A CN202210506958.2A CN202210506958A CN114979271A CN 114979271 A CN114979271 A CN 114979271A CN 202210506958 A CN202210506958 A CN 202210506958A CN 114979271 A CN114979271 A CN 114979271A
- Authority
- CN
- China
- Prior art keywords
- data
- cdn
- cache
- fragmentation
- cloud computing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000013467 fragmentation Methods 0.000 claims abstract description 36
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 36
- 238000004364 calculation method Methods 0.000 claims description 21
- 239000012634 fragment Substances 0.000 claims description 9
- 238000004458 analytical method Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 5
- 238000005457 optimization Methods 0.000 claims 1
- 238000001914 filtration Methods 0.000 description 8
- 230000000694 effects Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a CDN cache hierarchical scheduling method based on edge cloud computing, which relates to the technical field of distributed caches and is characterized in that data to be cached are classified and identified, a corresponding routing algorithm is selected according to a strategy for selecting CDN edge nodes to perform logic computation and data fragmentation on the data after the classification and identification, and the data are routed to the designated CDN edge nodes according to the computation results and the fragmentation number of the edge nodes.
Description
Technical Field
The invention discloses a method, relates to the technical field of distributed caches, and particularly relates to a CDN cache hierarchical scheduling method based on edge cloud computing.
Background
With the increasing demand of the edge cloud CDN in a large-scale public cloud environment, it is difficult to satisfy cache management of a plurality of edge nodes and reasonable distribution of content. At present, a technical scheme of additionally arranging a hierarchical cache device is adopted, and a second-layer cache service is provided to request data from a source station to be pulled to a middle-layer cache device so as to reduce the concurrent request pressure of the source station. There still remains the problem of redundant backup of cached data and the accumulation of stored bearer pressure.
Meanwhile, a second layer of cache service is erected in a link between the approach source station and the edge cloud CDN node, and the complexity of operation and maintenance is increased. The problems of deviation of flow requirements of CDN edge nodes at different moments, deviation of flow bandwidth capacity of intermediate layer cache devices and the like also cause a situation that extremely complex logic judgment is added on a management mechanism for ensuring that cache contents keep customized requirement strategies of a user side synchronous, and business flows of different users are possibly processed by the same intermediate layer cache devices, so that the problem of centralized pressure bearing of the intermediate devices is caused to a certain extent.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a CDN cache hierarchical scheduling method based on edge cloud computing, which is a distributed cache technology for performing data fragmentation on cache contents based on classification identification and collaborative filtering of cache data, can realize index establishment and content distribution, and ensures effective improvement of the comprehensive utilization rate of resources of edge cloud CDN nodes.
The specific scheme provided by the invention is as follows:
the invention provides a CDN cache hierarchical scheduling method based on edge cloud computing, which is characterized in that data to be cached is subjected to classification identification, a corresponding routing algorithm is selected according to a strategy for selecting CDN edge nodes to perform logic computation and data fragmentation on the data subjected to the classification identification, and the data is routed to an appointed CDN edge node according to a computation result and the fragmentation number of the edge nodes.
Further, in the CDN cache hierarchical scheduling method based on edge cloud computing, selecting a corresponding routing algorithm according to a policy for selecting a CDN edge node to perform logic computation and data fragmentation on data after the classification identification includes:
selecting a corresponding routing algorithm according to the search efficiency optimal strategy to perform logic calculation and data fragmentation on the data after the classification identification;
or selecting a corresponding routing algorithm according to the maximum utilization node resource strategy to perform logic calculation and data fragmentation on the classified and identified data.
Further, in the CDN cache hierarchical scheduling method based on edge cloud computing, a corresponding routing algorithm is selected according to a search efficiency optimal policy to perform logic computation and data fragmentation on data after the classification identification, and the data is routed to an assigned CDN edge node, with the process:
obtaining a request for caching data, carrying out classification identification on the data,
performing primary routing calculation and fragmentation processing on the classified and identified data according to the corresponding routing algorithm, wherein the formula of the routing algorithm is as follows:
shard＝hash(routing)％number_of_primary_shards，
the data on the fragment is subjected to the reverse index construction of a b-tree,
and tracing and finding the calculated CDN edge node according to the route, and writing the data needing to be cached into the CDN edge node.
Further, the CDN cache hierarchical scheduling method based on edge cloud computing judges whether the CDN edge node already stores cache data, judges whether the cache data is live data if the cache data exists, and writes the latest data to be cached into the CDN edge node if the cache data does not exist.
Further, in the edge cloud computing-based CDN cache hierarchical scheduling method, a corresponding routing algorithm is selected according to a maximum utilization node resource policy to perform logic computation and data fragmentation on the data after the classification identification, and the data is routed to an assigned CDN edge node, with the process:
obtaining a cache data request, carrying out classification identification on the data,
performing primary routing calculation and fragmentation processing on the classified and identified data according to the corresponding routing algorithm, wherein the formula of the routing algorithm is as follows:
shard＝hash(routing)％number_of_primary_shards+λ(load_values)*weight
acquiring load _ values load factor data of the CDN edge node, and calculating a weight factor of the CDN edge node;
the data on the fragment is subjected to the reverse index construction of a b-tree,
and tracing and finding the calculated CDN edge node according to the route, and writing the data needing to be cached into the CDN edge node.
Further, the CDN cache hierarchical scheduling method based on edge cloud computing judges whether the CDN edge node already stores cache data, judges whether the cache data is live data if the cache data exists, and writes the latest data to be cached into the CDN edge node if the cache data does not exist.
Further, in the CDN cache hierarchical scheduling method based on edge cloud computing, an index tree, a back-to-source policy, and a TTL policy of cache data are recorded at the CDN edge node.
The invention also provides a CDN cache hierarchical scheduling device based on edge cloud computing, which comprises a classification identification module, an analysis computing module and a routing module,
the classification identification module carries out classification identification on data needing to be cached, the analysis calculation module selects a corresponding routing algorithm according to a strategy for selecting CDN edge nodes to carry out logic calculation and data fragmentation on the data after the classification identification, and the routing module routes the data to the designated CDN edge nodes according to the calculation result and the fragmentation number of the edge nodes.
The invention has the advantages that:
the invention provides a CDN cache layered scheduling method based on edge cloud computing, which realizes data fragmentation distributed caching of cache contents based on classification identification and collaborative filtering of cache data. Through the real-time load of CDN edge nodes and the algorithm of routing fragmentation, the required cache data is reasonably distributed and used, the complex cache layering framework is simplified, the data can be reasonably read and written, and the effect of maximizing the utilization of data distribution resources is achieved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic flow chart of the method of the present invention using a routing algorithm 1.
Fig. 2 is a schematic flow chart of the method of the present invention using the routing algorithm 2.
Detailed Description
The present invention is further described below in conjunction with the following figures and specific examples so that those skilled in the art may better understand the present invention and practice it, but the examples are not intended to limit the present invention.
The invention provides a CDN cache hierarchical scheduling method based on edge cloud computing, which is characterized in that data needing to be cached is subjected to classification identification, a corresponding routing algorithm is selected according to a strategy of selecting a CDN edge node, the data after the classification identification is subjected to logic computation and data fragmentation, and the data is routed to an appointed CDN edge node according to a computation result and the number of fragments of the edge node.
The method disclosed by the invention is used for carrying out data fragmentation and distributed caching on the cached content based on the classification identification and collaborative filtering of the cached data, so that the effective improvement of the comprehensive resource utilization rate of the edge cloud CDN node is ensured.
In a specific application, in some embodiments of the method of the present invention, when performing CDN cache hierarchical scheduling based on edge cloud computing, data to be cached and accelerated is classified and identified, the identified classified data is subjected to logic computation and data fragmentation, and a route is determined to an assigned edge CDN node in combination with the number of fragments of the edge node.
Further, selecting a corresponding routing algorithm according to the optimal search efficiency strategy to perform logic calculation and data fragmentation on the classified and identified data, wherein the formula of the routing algorithm 1 is as follows:
shard＝hash(routing)％number_of_primary_shards
primary _ shards refers to the number of core fragments and determines which shard the cached data is on, routing value parameters are calculated through a hash function, and the hash values obtained by the same routing value can be consistent. And performing reverse index construction of a b-tree on the classified and identified data to reduce the number of times of seeking a disk, and storing the constructed index tree into the memory for one copy, so that the next query of the board can be directly read from the memory. With reference to the attached drawing 1, the method comprises the following specific steps:
step 1: acquiring a data request, and carrying out classification identification and filtering on data;
step 2: carrying out primary route calculation and fragmentation processing on the data subjected to classification identification and filtration by using a routing algorithm 1;
and step 3: searching the computed CDN edge node according to the route tracing, and writing the cache data into the node;
and 4, step 4: backing up the constructed index tree at the CDN edge node;
and 5: recording cache setting methods such as a back-source strategy and TTL of cache data at an edge CDN node;
step 6: if the cached data already exists in the CDN edge node hit in the step 3, judging whether the cached data is fresh data or not, if so, directly returning, otherwise, searching a source station according to a source returning strategy in the step 5 to read the latest data;
and 7: and returning the cache data in the step 6 to the user, and judging whether to reset according to a refreshing strategy in the cache rule.
In addition, the logic computation mentioned in the above steps may also be added to the data report of the real-time load condition of the CDN edge node, or used in other data services that need to accelerate different types of caches.
Further, or selecting a corresponding routing algorithm according to the maximum utilization node resource strategy to perform logic calculation and data fragmentation on the classified and identified data, wherein the formula of the routing algorithm 2 is as follows:
shard＝hash(routing)％number_of_primary_shards+λ(load_values)*weight
load real-time data added into CDN edge nodes through the routing algorithm 2 can effectively utilize node resources to the maximum extent, a small weight is set for a full-load node, and a large weight is set for an idle node, so that the resource utilization of the whole edge CDN node is maximized. Load _ values refer to load factors of the CDN edge nodes, and weight refers to weight factors of the CDN edge nodes. With reference to fig. 2, the following steps are performed:
step 1: acquiring a data request, and carrying out classification identification and filtering on data;
and 2, step: acquiring load _ values data of the edge CDN node, and calculating a weight;
and step 3: carrying out primary routing calculation and fragmentation processing on the data subjected to classification identification and filtration according to a routing algorithm 2;
and 4, step 4: searching the computed CDN edge node according to the route tracing, and writing the cache data into the node;
and 5: backing up the constructed index tree at the CDN edge node;
step 6: recording cache setting methods such as a back-to-source strategy and TTL of cache data at an edge CDN node;
and 7: if the cached data exists in the CDN node hit in the step 3, judging whether the cached data is fresh data or not, if so, directly returning, otherwise, searching a source station according to a back source strategy in the step 5 to read the latest data;
and 8: and returning the cache data in the step 7 to the user, and judging whether to reset according to a refreshing strategy in the cache rule.
The method solves the difficult problems of complex logic, close coupling of network topology and complex deployment and construction of the traditional CDN multi-level hierarchical architecture, and can distribute the cache data as required.
The invention also provides a CDN cache hierarchical scheduling device based on edge cloud computing, which comprises a classification identification module, an analysis computing module and a routing module,
the classification identification module carries out classification identification on data needing to be cached, the analysis calculation module selects a corresponding routing algorithm according to a strategy for selecting CDN edge nodes to carry out logic calculation and data fragmentation on the data after the classification identification, and the routing module routes the data to the designated CDN edge nodes according to the calculation result and the fragmentation number of the edge nodes.
Because the content of information interaction, execution process, and the like among the modules in the device is based on the same concept as the method embodiment of the present invention, specific content can be referred to the description in the method embodiment of the present invention, and is not described herein again.
Similarly, the device realizes the distributed caching of the data fragments of the cached content based on the classification identification and the collaborative filtering of the cached data. Through the real-time load of CDN edge nodes and the algorithm of routing fragmentation, the required cache data is reasonably distributed and used, the complex cache layering framework is simplified, the data can be reasonably read and written, and the effect of maximizing the utilization of data distribution resources is achieved.
It should be noted that not all steps and modules in the above flows and system structure diagrams are necessary, and some steps or modules may be omitted according to actual needs. The execution sequence of the steps is not fixed and can be adjusted according to the needs. The system structure described in the above embodiments may be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by a plurality of physical entities, or some components in a plurality of independent devices may be implemented together.
The above-mentioned embodiments are merely preferred embodiments for fully illustrating the present invention, and the scope of the present invention is not limited thereto. The equivalent substitutions or changes made by the person skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the invention is subject to the claims.
Claims (8)
1. A CDN cache hierarchical scheduling method based on edge cloud computing is characterized in that data needing to be cached are subjected to classification identification, a corresponding routing algorithm is selected according to a strategy of selecting CDN edge nodes, logic computation and data fragmentation are carried out on the data subjected to the classification identification, and the data are routed to designated CDN edge nodes according to computation results and the number of fragments of the edge nodes.
2. The CDN cache hierarchical scheduling method based on edge cloud computing according to claim 1 wherein said selecting a corresponding routing algorithm according to a policy for selecting CDN edge nodes to perform logic computation and data fragmentation on data after classification identification comprises:
selecting a corresponding routing algorithm according to a searching efficiency optimal strategy to perform logic calculation and data fragmentation on the classified and identified data;
or selecting a corresponding routing algorithm according to the maximum utilization node resource strategy to perform logic calculation and data fragmentation on the classified and identified data.
3. The CDN cache hierarchical scheduling method based on edge cloud computing as claimed in claim 2 wherein a corresponding routing algorithm is selected according to a lookup efficiency optimization strategy to perform logic computation and data fragmentation on the classified identified data and route the data to an assigned CDN edge node, the process is:
obtaining a request for caching data, carrying out classification identification on the data,
performing primary routing calculation and fragmentation processing on the classified and identified data according to the corresponding routing algorithm, wherein the formula of the routing algorithm is as follows:
shard＝hash(routing)％number_of_primary_shards，
the data on the fragment is subjected to the reverse index construction of a b-tree,
and tracing and finding the calculated CDN edge node according to the route, and writing the data needing to be cached into the CDN edge node.
4. The CDN cache hierarchical scheduling method based on edge cloud computing as claimed in claim 3 wherein said CDN edge node is determined whether cache data already exists, if so, it is determined whether the cache data is fresh data, otherwise, the latest data to be cached is written into said CDN edge node.
5. The CDN cache hierarchical scheduling method based on edge cloud computing of claim 2 wherein the corresponding routing algorithm is selected according to the maximum utilization node resource policy to perform logic computation and data fragmentation on the classified identified data and route the data to the designated CDN edge node, the process is:
obtaining a cache data request, carrying out classification identification on the data,
performing primary routing calculation and fragmentation processing on the classified and identified data according to the corresponding routing algorithm, wherein the formula of the routing algorithm is as follows:
shard＝hash(routing)％number_of_primary_shards+λ(load_values)*weight
acquiring load _ values load factor data of the CDN edge node, and calculating a weight factor of the CDN edge node;
the data on the fragment is subjected to the reverse index construction of a b-tree,
and tracing and finding the calculated CDN edge node according to the route, and writing the data needing to be cached into the CDN edge node.
6. The CDN cache hierarchical scheduling method based on edge cloud computing as claimed in claim 5 wherein determining if said CDN edge node already has cache data, if so, determining if the cache data is live data, otherwise, writing the latest data to be cached into said CDN edge node.
7. The CDN cache hierarchical scheduling method based on edge cloud computing as claimed in claim 3 or 5 wherein index tree, back source policy and TTL policy of cached data are recorded at the CDN edge node.
8. A CDN cache layered scheduling device based on edge cloud computing is characterized by comprising a classification identification module, an analysis computing module and a routing module,
the classification identification module carries out classification identification on data needing to be cached, the analysis calculation module selects a corresponding routing algorithm according to a strategy for selecting CDN edge nodes to carry out logic calculation and data fragmentation on the data after the classification identification, and the routing module routes the data to the designated CDN edge nodes according to the calculation result and the fragmentation number of the edge nodes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210506958.2A CN114979271A (en) | 2022-05-11 | 2022-05-11 | CDN cache layered scheduling method based on edge cloud computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210506958.2A CN114979271A (en) | 2022-05-11 | 2022-05-11 | CDN cache layered scheduling method based on edge cloud computing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114979271A true CN114979271A (en) | 2022-08-30 |
Family
ID=82982002
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210506958.2A Pending CN114979271A (en) | 2022-05-11 | 2022-05-11 | CDN cache layered scheduling method based on edge cloud computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114979271A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117295199A (en) * | 2023-11-14 | 2023-12-26 | 深圳市铭灏天智能照明设备有限公司 | Internet of things garage illumination system using edge side calculation |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106161569A (en) * | 2015-04-24 | 2016-11-23 | 华为软件技术有限公司 | The recommendation of Web content, buffer replacing method and equipment |
| CN107181695A (en) * | 2017-07-28 | 2017-09-19 | 成都谛听科技股份有限公司 | Distributed caching method and device |
| CN107277561A (en) * | 2016-04-08 | 2017-10-20 | 北京优朋普乐科技有限公司 | Content distributing network |
| CN109582758A (en) * | 2018-12-06 | 2019-04-05 | 重庆邮电大学 | A kind of Elasticsearch index fragment optimization method |
-
2022
- 2022-05-11 CN CN202210506958.2A patent/CN114979271A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106161569A (en) * | 2015-04-24 | 2016-11-23 | 华为软件技术有限公司 | The recommendation of Web content, buffer replacing method and equipment |
| CN107277561A (en) * | 2016-04-08 | 2017-10-20 | 北京优朋普乐科技有限公司 | Content distributing network |
| CN107181695A (en) * | 2017-07-28 | 2017-09-19 | 成都谛听科技股份有限公司 | Distributed caching method and device |
| CN109582758A (en) * | 2018-12-06 | 2019-04-05 | 重庆邮电大学 | A kind of Elasticsearch index fragment optimization method |
Non-Patent Citations (2)
| Title |
|---|
| SYSU_LLUOZH: "【网络协议趣谈】CDN静态和动态数据缓存", pages 2 - 9, Retrieved from the Internet <URL:http://t.csdnimg.cn/jA8HY> * |
| 信徒FAVOR: "【Elasticsearch】吐血整理", pages 1 - 2, Retrieved from the Internet <URL:http://t.csdnimg.cn/QsDfh> * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117295199A (en) * | 2023-11-14 | 2023-12-26 | 深圳市铭灏天智能照明设备有限公司 | Internet of things garage illumination system using edge side calculation |
| CN117295199B (en) * | 2023-11-14 | 2024-06-25 | 深圳市铭灏天智能照明设备有限公司 | Internet of things garage illumination system using edge side calculation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110191148B (en) | Statistical function distributed execution method and system for edge calculation | |
| CN103425756B (en) | The replication strategy of data block in a kind of HDFS | |
| CN107590182B (en) | Distributed log collection method | |
| CN102831120B (en) | A kind of data processing method and system | |
| CN101674233B (en) | Peterson graph-based storage network structure and data read-write method thereof | |
| CN106161610A (en) | A kind of method and system of distributed storage | |
| CN107800808A (en) | A kind of data-storage system based on Hadoop framework | |
| Wang et al. | Hybrid pulling/pushing for i/o-efficient distributed and iterative graph computing | |
| CN104679594A (en) | Middleware distributed calculating method | |
| CN106933868A (en) | A kind of method and data server for adjusting data fragmentation distribution | |
| CN110087250A (en) | A kind of network slice layout scheme and its method based on multiple target combined optimization model | |
| CN111597168A (en) | Block chain capacity recovery scheme based on integrity value | |
| CN112162789A (en) | Edge calculation random unloading decision method and system based on software definition | |
| CN114979271A (en) | CDN cache layered scheduling method based on edge cloud computing | |
| CN115481110A (en) | Resource sharing exchange design method based on metadata | |
| CN102404406A (en) | Massive real-time data synchronization system based on private cloud storage | |
| CN104580422A (en) | Cluster rendering node data access method based on shared cache | |
| CN103399943A (en) | Communication method and communication device for parallel query of clustered databases | |
| CN106790705A (en) | A kind of Distributed Application local cache realizes system and implementation method | |
| CN112463881B (en) | Efficient block chain system and processing method for cloud edge aggregation computing environment | |
| Guo et al. | Blockchain-assisted caching optimization and data storage methods in edge environment | |
| CN112559459A (en) | Self-adaptive storage layering system and method based on cloud computing | |
| CN104683480A (en) | Distribution type calculation method based on applications | |
| Aswini et al. | Artificial Intelligence Based Smart Routing in Software Defined Networks. | |
| US20230156074A1 (en) | Multi-cloud edge system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |