CN112543354B - Service-aware distributed video cluster efficient telescoping method and system - Google Patents
Service-aware distributed video cluster efficient telescoping method and system Download PDFInfo
- Publication number
- CN112543354B CN112543354B CN202011368368.5A CN202011368368A CN112543354B CN 112543354 B CN112543354 B CN 112543354B CN 202011368368 A CN202011368368 A CN 202011368368A CN 112543354 B CN112543354 B CN 112543354B
- Authority
- CN
- China
- Prior art keywords
- streaming media
- media server
- cluster
- distributed
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
- H04N21/2405—Monitoring of the internal components or processes of the server, e.g. server load
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0896—Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
- H04N21/2402—Monitoring of the downstream path of the transmission network, e.g. bandwidth available
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/101—Server selection for load balancing based on network conditions
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a service-aware distributed video cluster efficient telescoping method and a system, wherein the method comprises the following steps: generating a program input source plug flow address according to the attribute of the program to be distributed and the running state information of a streaming media server in the distributed cluster system; acquiring a real-time video stream of a program to be distributed, and pushing the real-time video stream to a streaming media server corresponding to the service type; monitoring the resource utilization rate of the streaming media server and judging whether the capacity expansion condition is met or not; if the capacity expansion condition is met, a new streaming media server instance is created according to the dynamic expansion strategy, and the new streaming media server instance is scheduled to the working node. The invention solves the problems that the distributed video cluster service is not perceived and dynamic expansion cannot be realized, realizes the perception of the distributed video cluster service, and simultaneously supports various application services and the dynamic efficient elastic expansion function of the cluster.
Description
Technical Field
The application relates to the technical field of distributed video clusters, in particular to a service-aware distributed video cluster efficient telescoping method and system.
Background
With the development of video industry and the increase of video application scenes, multimedia traffic presents an explosive growth trend, and multimedia content becomes a main component of current network traffic. In the video Internet era, on one hand, online video extremely consumes network bandwidth resources; on the other hand, users have increasingly high quality of video service, and the transmission of multimedia content has faced significant technical challenges. The traditional distributed video cluster can not meet the requirements of differentiated service scenes because of the problems of inflexible dynamic elastic expansion, service imperceptibility and the like.
The inventor of the application finds that the prior art distributed video cluster has the following technical problems:
firstly, the service is not perceived, the service types cannot be distinguished, the same processing mode is adopted for all the services, the bandwidth resources of the cluster system cannot be reasonably utilized, the cluster bandwidth utilization rate is possibly low, the requirements cannot be met when the service with some high code rate is handled, and the playing end cannot watch normally;
secondly, service distribution and scheduling are completed by a specific server in the cluster, the coupling degree is high, the problem of single-point failure exists at the same time, the distribution and scheduling server does not realize high availability, and if the server is down, the distributed cluster cannot work normally;
Thirdly, dynamic elastic expansion and contraction of the distributed clusters are not supported, the distributed clusters cannot realize dynamic automatic expansion or contraction according to actual service demands, and the service rapid growth demands cannot be met.
Therefore, it is important to provide a distributed video clustering method and system supporting multiple business applications, and dynamic efficient elastic scaling.
Disclosure of Invention
The invention mainly aims to provide a service-aware distributed video cluster efficient telescoping method and system, which aim to solve the problem that the service of a distributed video cluster is not aware and dynamic telescoping cannot be realized.
In order to achieve the above purpose, the present application provides a service-aware distributed video cluster efficient scaling method, which includes the following steps:
generating a program input source plug flow address according to the attribute of the program to be distributed and the running state information of a streaming media server in the distributed cluster system; wherein, the attribute information comprises program information, input source protocol and service type;
acquiring a real-time video stream of a program to be distributed, and pushing the real-time video stream to a streaming media server corresponding to the service type;
monitoring the resource utilization rate of the streaming media server and judging whether the capacity expansion condition is met or not;
If the capacity expansion condition is met, a new streaming media server instance is created according to the dynamic expansion strategy, and the new streaming media server instance is scheduled to the working node.
In an embodiment, the service aware distributed video cluster efficient scaling method further includes:
in response to a program creation request, configuring a program to be distributed;
and storing the programs to be distributed in a database cluster according to the priority.
In an embodiment, the service types include at least two of the following: on demand service, live service.
In an embodiment, the obtaining the real-time video stream of the program to be distributed, pushing the real-time video stream to the streaming media server corresponding to the service type, includes:
acquiring an operating streaming media server list according to streaming media server cluster information;
configuring a real-time encoder to acquire a real-time video stream of a program to be distributed, pushing the real-time video stream to a streaming media server corresponding to a program input source pushing address of the service type so as to update the program state in real time;
wherein each service type runs a corresponding streaming server instance and each streaming server instance runs in a separate container.
In an embodiment, the streaming media server cluster comprises a source station cluster and an edge cluster; the pushing the real-time video stream to a streaming media server corresponding to a push address of a program input source of the service type to update a program state in real time includes:
pushing a real-time video stream of a program to be distributed to a streaming media server cluster nearest to a push network, and storing the real-time video stream in a distributed file system;
if the push network is closest to the source station cluster, pushing the real-time video stream to the source station cluster;
and if the push network is closest to the edge cluster, pushing the real-time video stream to the edge cluster, and automatically acquiring the real-time video stream in the edge cluster through the source station cluster.
In an embodiment, the resources include at least: CPU, memory and network bandwidth; the monitoring the resource utilization rate of the streaming media server, and judging whether the current streaming media server meets the capacity expansion condition, comprises the following steps:
respectively defining expansion thresholds of CPU, memory and network bandwidth;
the resource utilization rate of each streaming media server is monitored and obtained in real time;
and if the resource utilization rate of any one of the CPU, the memory and the network bandwidth is larger than the capacity expansion threshold, the capacity expansion condition is met.
In an embodiment, the service aware distributed video cluster efficient scaling method further includes:
respectively defining capacity shrinkage thresholds of a CPU, a memory and a network bandwidth;
when the streaming media server no longer receives the real-time video stream of the program to be distributed, acquiring the resource utilization rate of the corresponding streaming media server;
and if the utilization rate of all resources in the CPU, the memory and the network bandwidth is simultaneously smaller than the capacity reduction threshold value, the capacity reduction condition is satisfied.
In an embodiment, the creating a new streaming media server instance according to the dynamic scaling strategy, and scheduling the new streaming media server instance to the working node includes:
calculating the expected cost of the streaming media server according to a dynamic expansion strategy, wherein the expected cost of the streaming media server is calculated according to the relation between the resource utilization rate and the capacity expansion threshold;
creating a corresponding number of streaming media server instances according to the expected number of copies of the streaming media server;
and adjusting the streaming media server instance to a proper working node according to the running state of the current streaming media server cluster and the streaming media server instance.
The embodiment of the application also provides a service-aware distributed video cluster efficient telescopic system, which comprises a control unit, a main unit and a working unit;
The control unit includes:
the controller is used for receiving the program information to be distributed and generating an input source plug flow address according to the program information and the information of the current streaming media server cluster;
the monitor is connected with the controller and used for monitoring the running state and the resource utilization rate of each streaming media server in real time;
the main unit includes:
the expansion controller is connected with the monitor and used for acquiring the resource utilization rate to calculate the expansion copy number; the copy controller and the telescopic controller create a corresponding number of telescopic copies to dynamically telescopic;
the scheduler is used for scheduling the created copy number to a proper node in the current distributed cluster to work;
the communication module is used for acquiring cluster information of the streaming media server in the current distributed cluster system;
the working unit includes: and the streaming media server cluster is connected with the communication module and used for receiving and distributing video streams.
In an embodiment, the system further comprises:
a distributed file system cluster for storing all video stream data in the system;
and the database cluster is used for storing the program information to be distributed.
The technical scheme of the service-aware distributed video cluster efficient telescoping method and system provided by the embodiment of the application has at least the following technical effects:
1. The technical scheme that the program input source plug flow address is generated according to the program information to be distributed, the input source protocol and the operation state information of the streaming media server in the distributed cluster system so as to connect the real-time encoder and the streaming media server is adopted, so that the problem that the service is not perceived in the prior art is solved, the effect of establishing the input source plug flow address and updating the program state is realized.
2. Because the real-time encoder is configured to obtain the real-time video stream of the program to be distributed, each service type is correspondingly operated on one streaming media server, different streaming media server clusters are constructed according to different service conditions, and the real-time video stream of different service types is forwarded to the corresponding streaming media server clusters.
3. The method comprises the steps of calculating the number of streaming media server instances to be created according to a set dynamic expansion strategy by adopting a CPU, a memory and network bandwidth resource for calculating the streaming media server instances, scheduling a copy controller to create the streaming media server instances with the corresponding number, and scheduling the streaming media server instances to be created to the most suitable nodes in the cluster according to the current state of the distributed cluster to create and run the streaming media server instances.
Drawings
Fig. 1 is a schematic structural diagram of a service-aware distributed video cluster efficient telescopic system according to an embodiment of the present application;
fig. 2 is a schematic flow chart of a first embodiment of a service aware distributed video cluster efficient telescoping method according to the present application;
fig. 3 is a schematic flow chart of a second embodiment of a service aware distributed video cluster efficient telescoping method according to the present application;
fig. 4 is a specific flowchart of step S240 of the second embodiment of the service aware distributed video cluster efficient scaling method of the present application;
fig. 5 is a specific flowchart of step S242 of the second embodiment of the service aware distributed video cluster efficient scaling method of the present application;
fig. 6 is a specific flowchart of step S250 of the second embodiment of the service aware distributed video cluster efficient scaling method of the present application;
fig. 7 is a schematic flowchart of a step S260 of a second embodiment of the service aware distributed video cluster efficient scaling method of the present application;
the achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
In order to solve the problem that the distributed video cluster service is not perceived and dynamic expansion cannot be realized, the program input source plug-flow address is generated according to the attribute of the program to be distributed and the running state information of a streaming media server in a distributed cluster system; acquiring a real-time video stream of a program to be distributed, and pushing the real-time video stream to a streaming media server corresponding to the service type; monitoring the resource utilization rate of the streaming media server and judging whether the capacity expansion condition is met or not; if the capacity expansion condition is met, a new streaming media server instance is created according to a dynamic expansion strategy, and the new streaming media server instance is scheduled to a technical scheme of a working node, so that distributed video cluster service perception is realized, and meanwhile, multiple application services and a cluster dynamic efficient elastic expansion function are supported.
In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
In order to better understand the above technical solutions, the following detailed description will refer to the accompanying drawings and specific embodiments.
As shown in fig. 2, fig. 2 is a flow chart of a first embodiment of a service aware distributed video cluster efficient scaling method according to the present application, including the following steps:
step S110, generating a program input source push address according to the attribute of the program to be distributed and the running state information of the streaming media server in the distributed cluster system; wherein the attribute information includes program information, input source protocol, and service type.
In this embodiment, the controller obtains an input source protocol and a service type in user-created program information from the database cluster, obtains the cluster information of the streaming media servers in the current distributed cluster system through the communication module, obtains a list of the streaming media servers running in the current system, and generates an input source push address for programs of different service types according to the service type and the input source protocol of the user-created program.
Step S120, obtaining a real-time video stream of a program to be distributed, and pushing the real-time video stream to a streaming media server corresponding to the service type.
In this embodiment, a real-time encoder is configured to obtain a real-time video stream of a program to be distributed, and the real-time video stream is pushed to a corresponding streaming media server in real time according to the push addresses of input sources of programs with different service types; for example, acquiring a real-time video stream of a live broadcast service to be distributed, and pushing the real-time video stream to a live broadcast streaming media server corresponding to the live broadcast service; or, acquiring a real-time video stream of the on-demand service to be distributed, and pushing the real-time video stream to an on-demand streaming media server corresponding to the on-demand service; the specific process is as described in step S241 to step S243, and will not be described here again.
And step S130, monitoring the resource utilization rate of the streaming media server and judging whether the capacity expansion condition is met.
In this embodiment, the resources include at least: CPU, memory and network bandwidth, monitor CPU, memory, network bandwidth utilization rate of all flow media server of business in real time, and report these information through the communication module, define the flow media server and expand the threshold value or shrink the threshold value, judge whether to meet the condition of expanding or shrinking through judging the relation of the resource utilization rate of the current flow media server and expansion threshold value or shrink the threshold value, CPU, memory and network bandwidth utilization rate of all flow media server container are sent to the flexible controller in order to expand and contract and control; the specific determination process is as described in step S251 to step S253, and will not be described here.
Step S140, if the capacity expansion condition is met, a new streaming media server instance is created according to the dynamic expansion strategy, and the new streaming media server instance is scheduled to the working node.
In this embodiment, if the capacity expansion condition is met, the number of expected copies of the streaming media server is calculated according to the relationship between the resource utilization rate and the preset threshold value of the resource, and meanwhile, a corresponding number of streaming media server instances are created, and the new streaming media server instances are scheduled to the working node, and the specific process is as described in step S261 to step S263, which is not repeated here.
Generating a program input source push address to connect a real-time encoder and a streaming media server according to program information to be distributed, an input source protocol and operation state information of the streaming media server in the distributed cluster system; the method comprises the steps of adopting a configuration real-time encoder to obtain real-time video streams of programs to be distributed, enabling each service type to correspondingly run on one streaming media server, constructing different streaming media server clusters according to different service conditions, and forwarding the real-time video streams of different service types to the corresponding streaming media server clusters; the CPU, the memory and the network bandwidth resource utilization rate of the streaming media server instance are calculated, the number of the streaming media server instances to be created is calculated according to a set dynamic expansion strategy, a copy controller is scheduled to create the streaming media server instances with corresponding number, the streaming media server instances to be created are scheduled to the most suitable nodes in the cluster to create and run according to the state of the current distributed cluster, the problems that the service is not perceived and the dynamic expansion control cannot be realized in the prior art are solved, the service perception is realized, the dynamic expansion efficiency is improved through the dynamic expansion control, and the system resource is saved when the user quantity is less.
As shown in fig. 3, fig. 3 is a flow chart of a second embodiment of a service aware distributed video cluster efficient scaling method according to the present application, including the following steps:
step S210, in response to the program creation request, a program to be distributed is configured.
In this embodiment, the response to the request for creating the program refers to that the user submits a request for creating the program, where the program includes a live broadcast service or an on-demand service, and the user may create a service type to be published by logging in a device with a visual interface, and configure different attribute information for different programs; for example, a user may create a live broadcast service or an on-demand service on a web service website page by logging in to the web service website, and configure different attribute information for the live broadcast service or the on-demand service; or the user can also configure different attribute information on the application program by logging in the application program, and the user can also log in through a mobile phone, a computer or other intelligent equipment to create the attribute information.
Step S220, storing the programs to be distributed in a database cluster according to the priority.
In this embodiment, the database cluster means that at least two or more database servers are utilized to form a virtual single database logical image, and transparent data service is provided to the user side like a single database system; in this embodiment, the attribute information of the different programs configured in step S210 is stored in the database cluster, and the attribute information is stored in the database cluster in the form of a priority message queue.
Step S230, generating a program input source plug address according to attribute information of a program to be distributed and running state information of a streaming media server in a distributed cluster system; wherein, the attribute information comprises program information, input source protocol and service type;
in this embodiment, the attribute information of the program to be distributed includes an input source protocol, a service type, a program name, and a program description; the input source protocol comprises RTMP, RTSP and HTTP; the streaming media server has the main functions of collecting, caching, scheduling, transmitting and playing the audio, video and media forms of media files. In this embodiment, the controller obtains an input source protocol and a program type in attribute information of a program created by a user from the database cluster, and obtains cluster information of a streaming media server in a current distributed cluster system through the communication module at the same time, so as to obtain a streaming media server list running in the current system, and generates a push address of an input source for the program according to a service type and the input source protocol in the attribute information of the program to be distributed created by the user, and the video data is pushed to a corresponding streaming media server in real time through the push address of the input source, so that real-time update of the video data is realized.
Step S240, obtaining a real-time video stream of a program to be distributed, and pushing the real-time video stream to a streaming media server corresponding to the service type.
As shown in fig. 4, fig. 4 is a specific flowchart of step S240 of the second embodiment of the service aware distributed video cluster efficient scaling method of the present application, which includes the following steps:
step S241, according to the cluster information of the streaming media server, the running streaming media server list is obtained.
In this embodiment, the streaming media server cluster includes at least two streaming media servers processing two different service types, and an operating streaming media server list is obtained according to streaming media server cluster information.
Step S242, configuring a real-time encoder to obtain a real-time video stream of a program to be distributed, and pushing the real-time video stream to a streaming media server corresponding to a program input source push address of the service type to update a program state in real time.
In this embodiment, the real-time encoder is configured to push the video stream to the push address of the input source generated in step S230; the push stream address of the input source is generated by the controller, so that the video stream pushed by the real-time encoder can be automatically forwarded to a streaming media server of a service type corresponding to the released program, and the controller monitors the input source and updates the program state in real time; for example, according to the push address of the input source of the live broadcast service, the live broadcast video pushed by the real-time encoder is automatically forwarded to the streaming media server corresponding to the live broadcast service, and the controller detects the pushed live broadcast service information and updates the program state.
As shown in fig. 5, fig. 5 is a specific flowchart of step S242 of the second embodiment of the service aware distributed video cluster efficient scaling method of the present application, which includes the following steps:
in step S2421, the real-time video stream of the program to be distributed is pushed to the streaming server cluster closest to the push network, and the real-time video stream is stored in the distributed file system.
In this embodiment, the distributed file system stores all video files, including files uploaded by a user, live broadcast service recording files, and on-demand service slicing files, the user can upload video files through a visual interface as source files of release programs, the uploaded video files are stored in the distributed file system of the system, the user can select to record live broadcast services for subsequent playback when in live broadcast service, and for programs to be recorded, the live broadcast streaming media server can store the files in the distributed file system.
Step S2422, wherein if the push network is closest to the source station cluster, the real-time video stream is pushed to the source station cluster.
In this embodiment, the streaming media server cluster is responsible for receiving and distributing the real-time video stream, and constructs a source station cluster and an edge cluster for the live broadcast service. When a management user creates a live broadcast service and pushes a real-time video stream to a distributed video cluster, the video stream is pushed to a streaming media server nearest to a push network, and if the push network is nearest to a source station cluster, the real-time video stream is directly pushed to the source station cluster.
Step S2423, if the push network is closest to the edge cluster, pushing the real-time video stream to the edge cluster, and automatically obtaining the real-time video stream in the edge cluster through the source cluster.
In this embodiment, if the push network is closest to a certain edge cluster, the real-time video stream is pushed to the edge cluster closest to the edge cluster, and the source cluster automatically acquires the video stream from the edge cluster for distribution, and the user can pull stream from the edge cluster through the CDN for distribution; for example: when a user watches the released live broadcast service through a player or a webpage, after the user sends a play request, an edge server nearest to the user network acquires the program from a source station cluster and sends the program to a common user for watching; after the user creates the on-demand service, the on-demand service can be watched through the player or the webpage, and the on-demand file is acquired from the distributed file system through the on-demand cluster server closest to the push network and is sent to the user for watching.
In this embodiment, when a user needs to record a live broadcast service, the recording work is completed by a source station cluster server, and an edge cluster server is only responsible for stream distribution; the edge cluster and the source station cluster can be automatically and dynamically telescopic and are mutually independent.
The real-time video stream of the program to be distributed is obtained by configuring the real-time encoder, if the push network is closest to the source station cluster, the real-time video stream is pushed to the source station cluster, and if the push network is closest to the edge cluster, the real-time video stream is pushed to the edge cluster, and the technical scheme of the real-time video stream in the edge cluster is automatically obtained through the source station cluster. Therefore, the problem of unstable system in the process of pushing the real-time data stream is effectively solved, the real-time video stream of the program to be distributed is pushed to the streaming media server cluster nearest to the push network, and the high stability of the system is improved.
Step S243, wherein each service type runs a corresponding streaming server instance, and each streaming server instance runs in a separate container.
In this embodiment, after the distributed video cluster is started normally, for different service types, a streaming media server and a telescopic controller are operated in the cluster, and for each service type, a corresponding streaming media server instance is operated; for example, the streaming media servers corresponding to the on-demand service and the live service are independent of each other, and each streaming media server instance runs in a separate container, and the streaming media server containers of all services are managed by using the same telescopic controller. The telescoping controller specifies the policies and conditions of telescoping at creation time.
The method comprises the steps of acquiring a running streaming media server list according to streaming media server cluster information, configuring a real-time encoder to acquire a real-time video stream of a program to be distributed, pushing the real-time video stream to a streaming media server corresponding to the service type according to a pushing address of an input source, running a corresponding streaming media server instance in each service type, and running the technical scheme that each streaming media server instance is in a separate container. Therefore, the problem that the service in the prior art is not perceived is effectively solved, and the effects that different services use different streaming media server clusters and a set of streaming media server systems integrate multiple services are realized.
Step S250, monitoring the resource utilization rate of the streaming media server and judging whether the capacity expansion condition is met;
as shown in fig. 6, fig. 6 is a specific flowchart of step S250 of the second embodiment of the service aware distributed video cluster efficient scaling method of the present application, which includes the following steps:
step S251, define the expansion threshold of CPU, memory and network bandwidth separately.
In this embodiment, the capacity expansion threshold refers to a condition that the streaming media server satisfies capacity expansion, and the capacity expansion threshold of the CPU, the memory and the network bandwidth is defined in the expansion controller.
Step S252, the resource utilization rate of each streaming media server is monitored and obtained in real time.
In this embodiment, the monitoring module monitors the CPU, the memory and the network bandwidth usage of each streaming media server in real time, and reports these information to the expansion controller to determine whether the capacity expansion condition is reached.
Step S253, if the resource utilization rate of any one of the CPU, the memory and the network bandwidth is greater than the capacity expansion threshold, the capacity expansion condition is satisfied.
In this embodiment, for example, the resource usage rate of each streaming media server is monitored and obtained in real time, and if it is detected that the resource usage rate of any one of the CPU, the memory and the network bandwidth in the streaming media server is greater than the capacity expansion threshold, capacity expansion is performed; if any two resource utilization rates in the streaming media server are detected to be larger than the set capacity expansion threshold value, carrying out capacity expansion; and if detecting that the utilization rate of all resources in the streaming media server is greater than the set capacity expansion threshold, carrying out capacity expansion.
In other embodiments, the capacity reduction threshold values of the CPU, the memory and the network bandwidth are respectively defined, when the streaming media server no longer receives the real-time video stream of the program to be distributed, the resource utilization rate of the corresponding streaming media server is obtained, and if all the resource utilization rates in the CPU, the memory and the network bandwidth are simultaneously smaller than the capacity reduction threshold values, the capacity reduction condition is met; for example: when the management user stops pushing the input video stream, the corresponding streaming media server does not receive the program video stream any more, the resource utilization rate of each streaming media server is monitored and obtained in real time, and when the utilization rate of all resources in the CPU, the memory and the network bandwidth is simultaneously smaller than the capacity reduction threshold value, capacity reduction is carried out.
Because the capacity expansion threshold and the capacity contraction threshold which respectively define the CPU, the memory and the network bandwidth are adopted, the resource utilization rate of each streaming media server is monitored and obtained in real time, and if the resource utilization rate of any one of the CPU, the memory and the network bandwidth is larger than the capacity expansion threshold, the capacity expansion condition is met, and the capacity expansion is carried out; if the utilization rate of all resources in the CPU, the memory and the network bandwidth is simultaneously smaller than the capacity reduction threshold value, the capacity reduction condition is met, and the capacity reduction technical scheme is carried out. Therefore, the problem that dynamic elastic expansion and contraction of the distributed clusters are not supported is effectively solved, dynamic automatic expansion or contraction is realized according to actual service demands, and the service rapid growth demands are met.
Step S260, if the capacity expansion condition is satisfied, a new streaming media server instance is created according to the dynamic expansion strategy, and the new streaming media server instance is scheduled to the working node.
As shown in fig. 7, fig. 7 is a specific flowchart of step S260 of the second embodiment of the service aware distributed video cluster efficient scaling method of the present application, which includes the following steps:
step S261, the expected cost number of the streaming media server is calculated according to the dynamic expansion strategy, wherein the expected cost number of the streaming media server is calculated according to the relation between the resource utilization rate and the expansion threshold.
In this embodiment, the dynamic scaling strategy is used to calculate the expected number of copies of the streaming media server, where at least one expected number of copies exists; under the capacity expansion condition, the dynamic expansion strategy refers to: expected cost = current utilization/expansion threshold of resources, where resources refer to CPU, memory and network bandwidth, if the calculated expected copy number has decimal places, the value after the decimal place is greater than 0.1, then enter one place, for example, the calculated expected cost is 4.2, then the expected cost is 5; if the calculated expected cost is less than 1, the expected cost is 1; for example, setting the capacity expansion threshold of a CPU of a streaming media server in a telescopic controller to be 50%, setting the CPU utilization rate of a certain streaming media server instance in the current system to be 60%, and calculating the expected cost number of the streaming media server to be 0.6/0.5=1.2 according to the dynamic telescopic strategy;
the embodiment can also calculate the expected number of copies of the corresponding number of shrinkage according to a dynamic expansion strategy, and under the condition of shrinkage, the dynamic expansion strategy refers to: the expected number of copies=the current utilization rate of the resource/the capacity reduction threshold, and the calculation process is the same as that of the capacity expansion expected number of copies, and will not be described in detail here.
Step S262, according to the expected number of copies of the streaming media server, a corresponding number of streaming media server instances are created.
In this embodiment, according to the number of expected copies of the streaming media server, where the number of expected copies of the streaming media server refers to the number of streaming media server instances that need to be expanded, the copy controller is scheduled to create the streaming media server instances that need to be expanded.
Step S263, according to the current running state of the streaming media server cluster and the streaming media server instance, the streaming media server instance is adjusted to a suitable working node.
In this embodiment, the copy controller stores templates of streaming media server instances of each service type, and the same copy controller is responsible for creating and deleting all types of streaming media server instances; the copy controller creates corresponding quantity of streaming media server copy information according to the corresponding streaming media server templates, the dispatcher monitors the streaming media server instance information created by the copy controller, and the dispatcher adjusts the streaming media server instance to a proper working node according to the running state of the current streaming media server cluster and the streaming media server instance according to various dispatching strategies arranged in the dispatcher.
The method comprises the steps of calculating the number of streaming media server instances to be created according to a set dynamic expansion strategy by adopting a CPU, a memory and network bandwidth resource for calculating the streaming media server instances, scheduling a copy controller to create the streaming media server instances with the corresponding number, and scheduling the streaming media server instances to be created to the most suitable nodes in the cluster according to the current state of the distributed cluster to create and run the streaming media server instances.
Based on the same inventive concept, as shown in fig. 1, fig. 1 is a schematic structural diagram of a service-aware distributed video cluster efficient telescopic system according to an embodiment of the present application, where the system includes a control unit 10, a main unit 30, and a working unit 40.
In the present embodiment, the control unit 10 includes: the controller 13 is configured to receive program information to be distributed created by the Web server 11, connect with the communication module 33 in the main unit 30, obtain information of a streaming media server cluster in the current distributed cluster system, and generate an input source push address according to the program information and the information of the current streaming media server cluster, so that the real-time encoder 50 can push a video stream to a corresponding streaming media server according to the input source push address; and the monitor 12 is connected with the controller 13 and is used for monitoring the running state and the resource utilization rate of each streaming media server in real time.
In the present embodiment, the main unit 30 includes: the expansion controller 31 is connected with the monitor 12, acquires the CPU, the memory and the network bandwidth of the streaming media server of the monitor 12 for monitoring all services in real time through the communication module 33, and acquires the resource utilization rate to calculate the expansion expected cost number; the copy controller 32 and the expansion controller 31 report the information to the expansion controller 31 through the communication module 33, and inform the copy controller 32 to create a corresponding number of expansion copies for dynamic expansion; the scheduler 34 is configured to schedule the number of created copies to a suitable node in the current distributed cluster for operation, and when the capacity expansion condition is satisfied, the scheduler 34 is responsible for scheduling the container copies created by the expansion controller 31 and the copy controller 32 to the most suitable node in the current distributed cluster for operation; the communication module 33 may be further configured to obtain cluster information of the streaming media server in the current distributed cluster system.
In the present embodiment, the working unit 40 includes: and the streaming media server cluster is connected with the communication module 33 and is used for receiving and distributing video streams, wherein the streaming media server cluster at least comprises a live edge cluster 41 and an on-demand cluster 42.
In this embodiment, the system further includes: the distributed file system cluster is used for storing all video stream data in the system, and mainly comprises files uploaded by a user, live program recording files and on-demand program slicing files, wherein the user can upload the video files through a Web server 11 to serve as source files of release programs, the uploaded video files are stored in a distributed file system of the system, the user can select to record live programs for subsequent playback when live programs are broadcast, and for programs needing to be recorded, a live streaming media server can store the files in the distributed file system; the database cluster 20 is used for storing the program information to be distributed, including user information, program information submitted by the user, running state and information of the distributed program in the system, state and information of each function module of the system, and the like, which are created by the Web server 11.
The system receives program information to be distributed, acquires information of a streaming media server cluster in a current distributed cluster system through a communication module, generates an input source plug flow address according to the program information and the information of the current streaming media server cluster, sends a real-time video stream to streaming media servers of different services corresponding to the input source plug flow address by using a real-time encoder, monitors the running state and resource utilization rate of each streaming media server in real time through a monitor, acquires the resource utilization rate through the telescopic controller to calculate telescopic copy numbers, creates corresponding telescopic copy numbers according to a copy controller, and informs a scheduler to schedule the created copy numbers to a proper node working technical scheme in the current distributed cluster. Therefore, the problems that the service is not perceived and the service cannot be automatically stretched are effectively solved, and an operation environment is provided for the control method of high-efficiency stretching of the service perceived video screen cluster.
It will be appreciated by those skilled in the art that the schematic structural diagram of the system for efficient scaling of a service aware distributed video cluster shown in fig. 1 is not limiting of the system, and that the system for efficient scaling of a service aware distributed video cluster may include more or fewer components than shown, or may combine certain components, or may be a different arrangement of components.
Because the service-aware distributed video cluster efficient telescopic system provided by the embodiment of the present application is a service-aware distributed video cluster efficient telescopic system adopted for implementing the method of the embodiment of the present application, based on the method introduced by the embodiment of the present application, a person skilled in the art can understand the specific structure and deformation of the service-aware distributed video cluster efficient telescopic system, and therefore, the detailed description thereof is omitted herein. The service-aware distributed video cluster efficient telescopic system adopted by the method of the embodiment of the application belongs to the scope of protection required by the application.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flowchart and/or block of the flowchart illustrations and/or block diagrams, and combinations of flowcharts and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (8)
1. A service-aware distributed video cluster efficient telescoping method is characterized in that,
generating a program input source plug address according to attribute information of a program to be distributed and running state information of a streaming media server in a distributed cluster system; wherein, the attribute information comprises program information, input source protocol and service type;
acquiring a real-time video stream of a program to be distributed, and pushing the real-time video stream to a streaming media server corresponding to the service type;
monitoring the resource utilization rate of the streaming media server and judging whether the capacity expansion condition is met or not;
If the capacity expansion condition is met, creating a new streaming media server instance according to a dynamic expansion strategy, and scheduling the new streaming media server instance to a working node, wherein the dynamic expansion strategy comprises an expected copy number = current utilization rate of resources/capacity expansion threshold value;
the step of obtaining the real-time video stream of the program to be distributed and pushing the real-time video stream to the streaming media server corresponding to the service type comprises the following steps:
acquiring an operating streaming media server list according to streaming media server cluster information;
configuring a real-time encoder to acquire a real-time video stream of a program to be distributed, pushing the real-time video stream to a streaming media server corresponding to a program input source pushing address of the service type so as to update the program state in real time;
wherein each service type runs a corresponding streaming media server instance, and each streaming media server instance runs in a separate container;
the service-aware distributed video cluster efficient telescoping method further comprises the following steps:
in response to a program creation request, configuring a program to be distributed;
storing the programs to be distributed in a database cluster according to the priority, wherein the database cluster is a virtual single database logic image formed by at least two or more database servers, and transparent data service is provided for a user side.
2. The method for efficient scaling of service aware distributed video clusters of claim 1, wherein said service types include at least two of: on demand service, live service.
3. The efficient telescoping method of traffic aware distributed video clusters of claim 2, wherein said streaming media server clusters comprise source station clusters and edge clusters; the pushing the real-time video stream to a streaming media server corresponding to a push address of a program input source of the service type to update a program state in real time includes:
pushing a real-time video stream of a program to be distributed to a streaming media server cluster nearest to a push network, and storing the real-time video stream in a distributed file system;
if the push network is closest to the source station cluster, pushing the real-time video stream to the source station cluster;
and if the push network is closest to the edge cluster, pushing the real-time video stream to the edge cluster, and automatically acquiring the real-time video stream in the edge cluster through the source station cluster.
4. The traffic-aware distributed video cluster efficient telescoping method as recited in claim 3, wherein said resources comprise at least: CPU, memory and network bandwidth; the monitoring the resource utilization rate of the streaming media server, and judging whether the capacity expansion condition is met, comprises the following steps:
Respectively defining expansion thresholds of CPU, memory and network bandwidth;
the resource utilization rate of each streaming media server is monitored and obtained in real time;
and if the resource utilization rate of any one of the CPU, the memory and the network bandwidth is larger than the capacity expansion threshold, the capacity expansion condition is met.
5. The traffic-aware distributed video cluster efficient telescoping method of claim 3, further comprising:
respectively defining capacity shrinkage thresholds of a CPU, a memory and a network bandwidth;
when the streaming media server no longer receives the real-time video stream of the program to be distributed, acquiring the resource utilization rate of the corresponding streaming media server;
and if the utilization rate of all resources in the CPU, the memory and the network bandwidth is simultaneously smaller than the capacity reduction threshold value, the capacity reduction condition is satisfied.
6. The method for efficient scaling of traffic aware distributed video clusters of claim 4, wherein creating a new streaming server instance based on a dynamic scaling policy and scheduling the new streaming server instance to a working node comprises:
calculating the expected cost of the streaming media server according to a dynamic expansion strategy, wherein the expected cost of the streaming media server is calculated according to the relation between the resource utilization rate and the capacity expansion threshold;
Creating a corresponding number of streaming media server instances according to the expected number of copies of the streaming media server;
and adjusting the streaming media server instance to a proper working node according to the running state of the current streaming media server cluster and the streaming media server instance.
7. The system is characterized by comprising a control unit, a main unit and a working unit;
the control unit includes:
the controller is used for receiving the program information to be distributed and generating an input source plug flow address according to the program information and the information of the current streaming media server cluster;
the monitor is connected with the controller and used for monitoring the running state and the resource utilization rate of each streaming media server in real time;
the main unit includes:
the expansion controller is connected with the monitor and used for acquiring the resource utilization rate to calculate the expansion copy number; the copy controller and the telescopic controller create a corresponding number of telescopic copies to dynamically telescopic;
the scheduler is used for scheduling the created copy number to a proper node in the current distributed cluster to work; the communication module is used for acquiring cluster information of the streaming media server in the current distributed cluster system;
The working unit includes: the streaming media server cluster is connected with the communication module and used for receiving and distributing real-time video streams of programs to be distributed;
wherein the system further comprises configuring a program to be distributed in response to the request for creating the program;
storing the programs to be distributed in a database cluster according to the priority, wherein the database cluster is a virtual single database logic image formed by at least two or more database servers, and transparent data service is provided for a user side.
8. The system for efficient scaling of traffic-aware distributed video clusters of claim 7, further comprising:
the distributed file system cluster is used for storing real-time video stream data of all programs to be distributed in the system;
and the database cluster is used for storing the program information to be distributed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011368368.5A CN112543354B (en) | 2020-11-27 | 2020-11-27 | Service-aware distributed video cluster efficient telescoping method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011368368.5A CN112543354B (en) | 2020-11-27 | 2020-11-27 | Service-aware distributed video cluster efficient telescoping method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112543354A CN112543354A (en) | 2021-03-23 |
| CN112543354B true CN112543354B (en) | 2023-05-09 |
Family
ID=75016415
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011368368.5A Active CN112543354B (en) | 2020-11-27 | 2020-11-27 | Service-aware distributed video cluster efficient telescoping method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112543354B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113630438B (en) * | 2021-06-25 | 2024-01-12 | 微梦创科网络科技（中国）有限公司 | Stream processing task scheduling method and distributed stream processing system |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103561285A (en) * | 2013-10-11 | 2014-02-05 | 深圳市同洲电子股份有限公司 | Program broadcasting method and network plug-flow server |
| CN108668178B (en) * | 2017-03-31 | 2020-12-04 | 华为技术有限公司 | Multicast implementation method and related network equipment |
| CN110233984A (en) * | 2018-03-06 | 2019-09-13 | 北京视联动力国际信息技术有限公司 | A kind of monitoring system and method based on view networking |
| CN109873718A (en) * | 2019-01-23 | 2019-06-11 | 平安科技（深圳）有限公司 | A kind of container self-adapting stretching method, server and storage medium |
| CN210405337U (en) * | 2019-05-30 | 2020-04-24 | 联想（上海）信息技术有限公司 | Dynamic expansion and contraction capacity device of AR server |
| CN110418154B (en) * | 2019-07-30 | 2021-10-19 | 腾讯科技（深圳）有限公司 | Multimedia data pushing method, device and system |
| CN110706824A (en) * | 2019-09-27 | 2020-01-17 | 京东方科技集团股份有限公司 | Interactive method, apparatus, server, system and medium for remote consultation |
| CN111444015A (en) * | 2020-03-24 | 2020-07-24 | 北京奇艺世纪科技有限公司 | Server control method and device, computer equipment and storage medium |
-
2020
- 2020-11-27 CN CN202011368368.5A patent/CN112543354B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112543354A (en) | 2021-03-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110769278B (en) | Distributed video transcoding method and system | |
| US20140165119A1 (en) | Offline download method, multimedia file download method and system thereof | |
| RU2343536C2 (en) | Mechanism of peer broadcasting of information content | |
| CN102882829A (en) | Transcoding method and system | |
| US20150237113A1 (en) | Method and system for file transmission | |
| CN104378665A (en) | Distributed transcoding system and method based on digital television | |
| CN109348264B (en) | Video resource sharing method and device, storage medium and electronic equipment | |
| CN114064211B (en) | Video stream analysis system and method based on end-side-cloud computing architecture | |
| US11290777B2 (en) | High-speed parallel engine for processing file-based high-resolution images | |
| CN104735044A (en) | Streaming media live broadcast method and system | |
| CN103813181A (en) | System and method for optimizing video | |
| CN104349177A (en) | Method for turning to play multimedia file under desktop cloud, virtual machine and system | |
| CN112543354B (en) | Service-aware distributed video cluster efficient telescoping method and system | |
| CN113630438B (en) | Stream processing task scheduling method and distributed stream processing system | |
| WO2021073405A1 (en) | Data distribution method, electronic device, and storage medium | |
| CN110996114A (en) | Live broadcast scheduling method and device, electronic equipment and storage medium | |
| CN105338371A (en) | Multimedia transcoding scheduling method and apparatus | |
| AU2020226900B2 (en) | Adaptive retrieval of objects from remote storage | |
| WO2014117508A1 (en) | Content delivery network and content management method in content delivery network | |
| CN112953944A (en) | Audio-video transcoding method based on MapReduce | |
| CN112689166A (en) | Method and system for flexibly increasing and decreasing CDN hot content in real time | |
| WO2024037161A1 (en) | Data transmission method, apparatus and system, device, and storage medium | |
| CN112995241A (en) | Service scheduling method and device | |
| Sahoo et al. | Resource allocation for video transcoding in the multimedia cloud | |
| WO2017071524A1 (en) | Multimedia resource publishing method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |