CN111669629A - Video CDN node instant capacity expansion method, scheduler and CND storage system - Google Patents
Video CDN node instant capacity expansion method, scheduler and CND storage system Download PDFInfo
- Publication number
- CN111669629A CN111669629A CN202010424901.9A CN202010424901A CN111669629A CN 111669629 A CN111669629 A CN 111669629A CN 202010424901 A CN202010424901 A CN 202010424901A CN 111669629 A CN111669629 A CN 111669629A
- Authority
- CN
- China
- Prior art keywords
- node
- file
- machines
- machine
- time interval
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/262—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists
- H04N21/26208—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists the scheduling operation being performed under constraints
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/262—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists
- H04N21/26208—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists the scheduling operation being performed under constraints
- H04N21/26241—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists the scheduling operation being performed under constraints involving the time of distribution, e.g. the best time of the day for inserting an advertisement or airing a children program
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/858—Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
- H04N21/8586—Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot by using a URL
Abstract
The invention discloses a video CDN node instant capacity expansion method, a scheduler and a CND storage system.A plurality of groups of node machines are divided into all node machines, and the plurality of groups of node machines are used for storing files distributed in a plurality of time intervals; selecting a node machine storing the key information from the corresponding group of node machines to provide services according to the key information of the file requested by the user; all the node machines comprise the existing node machine and the newly-added node machine. When the node is expanded, the video does not need to be migrated, the newly added node machine is immediately available, and the node server and the request video URL are provided with time attributes, so that the migration is not needed during the expansion, the time required by the expansion migration is saved, the operation and maintenance cost and the total resource cost are greatly reduced, and the expansion efficiency is improved.
Description
Technical Field
The invention relates to the technical field of CDN resource storage, in particular to a video CDN node instant capacity expansion method, a scheduler and a CND storage system.
Background
And the mainstream video CDN node adopts a Hash algorithm to position files, and the positioning comprises file distribution and file scheduling.
1. File distribution
Before the file is on line for a user to access, the file is distributed to each computer room through a network protocol (HTTP, P2P and the like), and a distribution program selects one node machine to store the file through a hash algorithm by taking a characteristic value (MD 5 value of the file) of a video as a keyword.
2. File scheduling
When a user opens a webpage to play a video, a file scheduling service is requested, and the scheduling service selects a node machine to provide service for the user through a hash algorithm by taking a characteristic value (an MD5 value of a file) of the video as a keyword.
Since the distribution and scheduling algorithms, the eigenvalues, and the hash sets are consistent, they can be addressed correctly (the same node machine is selected for distribution and scheduling).
When the disk space of the node computer is full, new files cannot be distributed, and new node computers need to be added in an expansion mode. And as the hash set is changed, the result of the scheduling hash algorithm is inconsistent with the actual storage position of the file. Therefore, files need to be migrated, services are provided after the result of the hash algorithm to be scheduled is consistent with the file storage position, and the operation and maintenance cost and the overall resource cost are high.
Disclosure of Invention
The invention aims to solve the technical problem that in order to overcome the defects of the prior art, the invention provides a method for instantly expanding the capacity of a video CDN node, a scheduler and a CND storage system.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows: a video CDN node instant capacity expansion method comprises the following steps: dividing all node machines into a plurality of groups, wherein the plurality of groups of node machines are used for storing files distributed in a plurality of time intervals; according to key information (for example, time meta information contained in a URL) of a file requested by a user, selecting a node machine storing the file with the key information from a corresponding group of node machines to provide a service; all the node machines comprise the existing node machine and the newly-added node machine.
In the invention, the newly added node machine and the existing node machine are respectively used for storing the files distributed in different time intervals, namely the newly added node machine and the existing node machine are respectively arranged in different groups, thereby ensuring the effectiveness of capacity expansion.
For a certain file, the distribution time of the file is taken as a time point, a time interval corresponding to the file is determined, and a node machine is selected to store the file in the time interval.
The key information of the user request file comprises a characteristic value and a distribution time point of the video, for example, time attributes are carried on a node server and a request video URL, so that migration is not needed during capacity expansion (migration is needed in the prior art), and time needed by capacity expansion migration is saved.
Specifically, the specific process of selecting the node machine according to the user request file comprises the following steps: and determining a time interval corresponding to the user request file according to the key information of the user request file, and selecting the node machine storing the file with the key information to provide service in the time interval.
Accordingly, the present invention also provides a scheduler comprising:
the distribution program is used for determining a time interval corresponding to the file to be distributed by taking the distribution starting time as a time point, and selecting one node machine in the node machine group corresponding to the time interval to store the file to be distributed in the time interval;
and the scheduling program is used for determining a time interval corresponding to the user request file according to the key information of the user request file, and selecting the node machine storing the file with the key information to provide service in the time interval.
The invention also provides a CDN system adopting the scheduler, which comprises a node machine room; the node machine room comprises a plurality of groups of node machines, and the plurality of groups of node machines are used for storing files distributed in a plurality of time intervals; the multiple groups of node machines comprise an existing node machine set and a newly-added node machine set.
Compared with the prior art, the invention has the beneficial effects that: when the node is expanded, the video does not need to be migrated, the newly added node machine is immediately available, and the node server and the request video URL are provided with time attributes, so that the migration is not needed during the expansion, the time required by the expansion migration is saved, the operation and maintenance cost and the total resource cost are greatly reduced, and the expansion efficiency is improved.
Detailed Description
According to the invention, CDN nodes are abstracted into a three-level structure of a machine room-a time bucket-a node machine. The time bucket is a time interval, and the node machines in the bucket only store the files distributed in the time interval. For example: some machine room has 5 machines (numbers 1,2,3,4, 5), and needs to expand 2 machines (numbers 6, 7). The current time is 20181225, the expansion time point is 20190101, and the barrel is divided into { "-20190101": 1,2,3,4,5], "20190101-": 6,7 }.
In the file distribution process, the distribution program determines which time bucket the file should be distributed to by taking the distribution starting time as a time point, and then selects a node machine to store the file in the time bucket through a hash algorithm. The file of the invention needs to contain time information, such as/C2/2018/01/31 _0/00013D94AAE045C3766D8077619D5764_ v02_1_1_1510.mp4, the time stamp of the file is 20180131, the file is distributed into a barrel of "-20190101", and the file is stored in a machine selected by Hash algorithm in [1,2,3,4,5 ].
In the process of file scheduling, a user requests a file to carry key information such as a characteristic value and a distribution time point of a video, and similar to the distribution process, a scheduler determines a time bucket to which the file should be distributed, and then selects a node machine in which the file is located to serve the user through a hash algorithm in the time bucket.
The main process steps of this embodiment are as follows:
if 5 node machines exist in a certain machine room, the node machines are numbered 1,2,3,4 and 5 respectively, 2 node machines are required to be added, the node machines are numbered 6 and 7, and operation and maintenance personnel hope to take effect when the node machines are added at 2018-08-2300: 00: 00. (Current time before 2018-08-2300: 00: 00)
And 1, configuring a node time bucket, and keeping the node machines 1,2,3,4 and 5 from the computer room to provide services all the time, stopping distributing new files at 2018-08-2300: 00:00 due to full disks, and distributing the files to the node machines 6 and 7. The machine room time bucket configuration may be recorded as { "-20180823": 1,2,3,4,5"," +20180823": 6,7" }, which takes effect immediately;
2, when the request URL is http:// disp.com/2017/08/22/BDA16493E80217E61B91EA2E6AB664D.mp4 scheduler, determining that the file is issued at 2018-08-22 and before 2018-08-2300: 00:00, selecting nodes at node machines 1,2,3,4 and 5;
3, when the request URL is http:// disp.com/2018/08/24/BDA16493E80217E61B91EA2E6AB664D.mp4 scheduler determines that the file is issued at 2018-08-24, after 2018-08-2300: 00:00, the node machine 6,7 is selected as the node.
Correspondingly, the invention also provides an embodiment of the structure of the scheduler, wherein the scheduler comprises a distribution program and a scheduler. The distribution program is used for determining a time interval corresponding to the file to be distributed by taking the distribution starting time as a time point, and selecting one node machine in the node machine group corresponding to the time interval to store the file to be distributed in the time interval; the scheduling program is used for determining a time interval corresponding to a user request file according to key information of the user request file, and selecting a node machine storing the file with the key information to provide service in the time interval.
On the basis of the scheduler, the invention also provides an embodiment of the CDN system, which comprises a node machine room, wherein the node machine room is communicated with the scheduler; the node machine room comprises a plurality of groups of node machines, and the plurality of groups of node machines are used for storing files distributed in a plurality of time intervals; the multiple groups of node machines comprise an existing node machine set and a newly-added node machine set.
Therefore, the time attribute is carried on the node server and the request video URL, so that migration is not needed during capacity expansion (migration is needed in the prior art), and the time needed by capacity expansion migration is saved.
Claims (8)
1. A video CDN node instant capacity expansion method is characterized by comprising the following steps: dividing all node machines into a plurality of groups, wherein the plurality of groups of node machines are used for storing files distributed in a plurality of time intervals; selecting a node machine storing the key information from the corresponding group of node machines to provide services according to the key information of the file requested by the user; all the node machines comprise the existing node machine and the newly-added node machine.
2. The video CDN node instant volume expansion method of claim 1, wherein the newly added node machine and the existing node machine are respectively configured to store files distributed in different time intervals.
3. The video CDN node immediate capacity expansion method of claim 1, wherein for a file, a time interval corresponding to the file is determined using a distribution time of the file as a time point, and a node machine is selected to store the file within the time interval.
4. The video CDN node immediate capacity expansion method of claim 1, wherein the key information of the user request file includes a feature value and a delivery time point of a video.
5. The video CDN node instant capacity expansion method of claim 1, wherein the specific process of selecting a node machine according to the user request file includes: and determining a time interval corresponding to the user request file according to the key information of the user request file, and selecting the node machine storing the file with the key information to provide service in the time interval.
6. A scheduler, comprising:
the distribution program is used for determining a time interval corresponding to the file to be distributed by taking the distribution starting time as a time point, and selecting one node machine in the node machine group corresponding to the time interval to store the file to be distributed in the time interval;
and the scheduling program is used for determining a time interval corresponding to the user request file according to the key information of the user request file, and selecting the node machine storing the file with the key information to provide service in the time interval.
7. A CDN storage system employing the scheduler of claim 6, comprising a node room; the node machine room comprises a plurality of groups of node machines, and the plurality of groups of node machines are used for storing files distributed in a plurality of time intervals.
8. The CDN storage system of claim 7 wherein the plurality of groups of node machines comprise existing groups of node machines and newly added groups of node machines.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010424901.9A CN111669629A (en) | 2020-05-19 | 2020-05-19 | Video CDN node instant capacity expansion method, scheduler and CND storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010424901.9A CN111669629A (en) | 2020-05-19 | 2020-05-19 | Video CDN node instant capacity expansion method, scheduler and CND storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111669629A true CN111669629A (en) | 2020-09-15 |
Family
ID=72383923
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010424901.9A Pending CN111669629A (en) | 2020-05-19 | 2020-05-19 | Video CDN node instant capacity expansion method, scheduler and CND storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111669629A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113596119A (en) * | 2021-07-16 | 2021-11-02 | 中移（杭州）信息技术有限公司 | Edge capability distribution method, system, device and computer readable storage medium |
| CN115174696A (en) * | 2022-09-08 | 2022-10-11 | 北京达佳互联信息技术有限公司 | Node scheduling method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140344413A1 (en) * | 2012-12-13 | 2014-11-20 | Level 3 Communications, Llc | Collector mechanisms in a content delivery network |
| CN104469392A (en) * | 2014-12-19 | 2015-03-25 | 北京奇艺世纪科技有限公司 | Video file storage method and device |
| CN104580393A (en) * | 2014-12-18 | 2015-04-29 | 北京蓝汛通信技术有限责任公司 | Method and device for expanding server cluster system and server cluster system |
| CN108540546A (en) * | 2018-03-28 | 2018-09-14 | 深圳市网心科技有限公司 | Network node enters network control method, electronic equipment, network system and storage medium |
| CN109005056A (en) * | 2018-07-16 | 2018-12-14 | 网宿科技股份有限公司 | Storage capacity evaluation method and apparatus based on CDN application |
-
2020
- 2020-05-19 CN CN202010424901.9A patent/CN111669629A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140344413A1 (en) * | 2012-12-13 | 2014-11-20 | Level 3 Communications, Llc | Collector mechanisms in a content delivery network |
| CN104580393A (en) * | 2014-12-18 | 2015-04-29 | 北京蓝汛通信技术有限责任公司 | Method and device for expanding server cluster system and server cluster system |
| CN104469392A (en) * | 2014-12-19 | 2015-03-25 | 北京奇艺世纪科技有限公司 | Video file storage method and device |
| CN108540546A (en) * | 2018-03-28 | 2018-09-14 | 深圳市网心科技有限公司 | Network node enters network control method, electronic equipment, network system and storage medium |
| CN109005056A (en) * | 2018-07-16 | 2018-12-14 | 网宿科技股份有限公司 | Storage capacity evaluation method and apparatus based on CDN application |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113596119A (en) * | 2021-07-16 | 2021-11-02 | 中移（杭州）信息技术有限公司 | Edge capability distribution method, system, device and computer readable storage medium |
| CN115174696A (en) * | 2022-09-08 | 2022-10-11 | 北京达佳互联信息技术有限公司 | Node scheduling method and device, electronic equipment and storage medium |
| CN115174696B (en) * | 2022-09-08 | 2023-01-20 | 北京达佳互联信息技术有限公司 | Node scheduling method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108287660A (en) | Date storage method and equipment | |
| CN104050250A (en) | Distributed key-value query method and query engine system | |
| CN102891869B (en) | CDN (Content Delivery Network) and content distribution and scheduling method | |
| CN101799797B (en) | Dynamic allocation method of user disk quota in distributed storage system | |
| CN103095806B (en) | A kind of load balancing management system of the real-time dataBase system towards bulk power grid | |
| BR112014000597B1 (en) | METHOD AND SYSTEM FOR DOWNLOADING A MULTIMEDIA FILE | |
| CN103929500A (en) | Method for data fragmentation of distributed storage system | |
| CN102591970A (en) | Distributed key-value query method and query engine system | |
| CN104317742A (en) | Automatic thin-provisioning method for optimizing space management | |
| CN101472166A (en) | Method for caching and enquiring content as well as point-to-point medium transmission system | |
| US20130031221A1 (en) | Distributed data storage system and method | |
| CN103186554A (en) | Distributed data mirroring method and data storage node | |
| US9110820B1 (en) | Hybrid data storage system in an HPC exascale environment | |
| CN109218100A (en) | Distributed objects storage cluster and its request responding method, system and storage medium | |
| CN104967873A (en) | Streaming live scheduling method, system and scheduling server | |
| CN103167026A (en) | Processing method, system and device for cloud storage environmental data | |
| CN104469392A (en) | Video file storage method and device | |
| CN111669629A (en) | Video CDN node instant capacity expansion method, scheduler and CND storage system | |
| CN112947860B (en) | Hierarchical storage and scheduling method for distributed data copies | |
| WO2015031378A1 (en) | System and method for supporting partition level journaling for synchronizing data in a distributed data grid | |
| CN102982182A (en) | Data storage planning method and device | |
| CN109756573A (en) | A kind of file system based on block chain | |
| CN109871365A (en) | A kind of distributed file system | |
| CN107181773A (en) | Data storage and data managing method, the equipment of distributed memory system | |
| KR20130038517A (en) | System and method for managing data using distributed containers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200915 |