CN115514996A - Method and device for determining state of live transcoding machine - Google Patents
Method and device for determining state of live transcoding machine Download PDFInfo
- Publication number
- CN115514996A CN115514996A CN202110691790.2A CN202110691790A CN115514996A CN 115514996 A CN115514996 A CN 115514996A CN 202110691790 A CN202110691790 A CN 202110691790A CN 115514996 A CN115514996 A CN 115514996A
- Authority
- CN
- China
- Prior art keywords
- transcoding
- load
- task
- target machine
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/262—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234336—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
Abstract
The invention discloses a method and a device for determining the state of a live transcoding machine, wherein the method is used for acquiring the transcoding historical load of a transcoding task and the transcoding real-time load at the current moment by aiming at each transcoding task in a target machine; and obtaining the estimated load of the transcoding task according to the transcoding real-time load and the transcoding historical load. Then, determining the observation state of the target machine according to the estimated loads of all transcoding tasks in the target machine; wherein, the observation state is used for representing the resource consumption of the target machine; finally, according to the conditional probability of each hidden state of the target machine when the target machine is in the observation state, obtaining the current state prediction result of the target machine; wherein the hidden state indicates whether the target machine is available. The invention can avoid the idleness or overload of the transcoding machine during task scheduling.
Description
Technical Field
The invention relates to the technical field of networks, in particular to a method and a device for determining the state of a live transcoding machine.
Background
In order to ensure the stability of live broadcast transcoding, the live broadcast of the video usually adopts fixed-rate coding, the transcoding task load is relatively stable, and the load can be accurately predicted by using the code rate, the resolution and the frame rate of the source stream. However, on the premise of ensuring the image quality, in order to further reduce the code rate and save the bandwidth, a related technology of dynamic code rate coding is introduced. Compared with a fixed bitrate, the video transcoding task load using a dynamic bitrate can fluctuate greatly along with the change of a video picture, and the load estimation by using a traditional method can often cause two task scheduling extremes, so that a transcoding machine is idle or overloaded.
Therefore, the state of the transcoding machine in the prior art is not accurately predicted, and the transcoding machine is easy to idle or overload.
Disclosure of Invention
In view of the above problems, the present invention provides a method and an apparatus for determining a status of a live transcoding machine, which can avoid the occurrence of idleness or overload of the transcoding machine during task scheduling. .
In a first aspect, the present application provides the following technical solutions through an embodiment:
a method for determining the status of a live transcoding machine, comprising:
aiming at each transcoding task in a target machine, acquiring a transcoding historical load of the transcoding task and a transcoding real-time load at the current moment; obtaining an estimated load of the transcoding task according to the transcoding real-time load and the transcoding historical load; determining the observation state of the target machine according to the estimated loads of all transcoding tasks in the target machine; wherein the observation state is used to characterize a resource consumption size of the target machine; obtaining a current state prediction result of the target machine according to the conditional probability of each hidden state of the target machine in the observation state; wherein the hidden state represents whether the target machine is available.
Optionally, the obtaining of the transcoding real-time load includes:
acquiring a feature vector of the transcoding task; determining a task load of the transcoding task according to the feature vector; and acquiring the transcoding real-time load according to the task load.
Optionally, the obtaining a feature vector of the transcoding task includes:
acquiring task parameters of the transcoding task; the task parameters include one or more of: video code rate, video time complexity, video space complexity, coding parameters, processor occupation and memory occupation; processing the task parameters according to the height, the width and the frame rate of the video frame of the transcoding task to obtain the characteristic parameters of the unit pixel; and obtaining the feature vector according to the feature parameters.
Optionally, the obtaining the transcoding real-time load according to the task load includes:
acquiring the transcoding real-time load according to the task load and a preset smooth function; wherein the smoothing function is to smooth fluctuations of the task load.
Optionally, the obtaining an estimated load of the transcoding task according to the transcoding real-time load and the transcoding historical load includes:
distributing a first weight to the transcoding real-time load, and distributing a second weight to the transcoding historical load; wherein the sum of the first weight and the second weight is 1, and the value range of the first weight and the second weight is [0,1]; and determining the sum of the transcoding real-time load after the first weight is distributed and the transcoding historical load after the second weight is distributed as the estimated load of the transcoding task.
Optionally, the determining the observation state of the target machine according to the estimated loads of all transcoding tasks in the target machine includes:
acquiring the basic consumption load of a target machine; the basic consumption load is the load necessary for the target machine to run; and determining the sum of the estimated load of all transcoding tasks in the target machine and the basic consumption load as the observation state of the target machine.
Optionally, the obtaining a current state prediction result of the target machine according to the conditional probability of each hidden state in which the target machine is in the observation state includes:
when the hidden state is available or high-load, obtaining a first conditional probability according to the historical probability that the target machine is available or high-load in the observation state; when the hidden state is unavailable, obtaining a second conditional probability according to the machine resource state of the target machine in the observation state; wherein the machine resource state represents a state of whether a target machine can execute the transcoding task; constructing a transition probability matrix according to the first conditional probability and the second conditional probability; and obtaining the current state prediction result of the target machine based on the transition probability matrix.
Optionally, the machine resource status includes: hardware consumption state, task number and historical maximum load sum; the obtaining a second conditional probability according to the machine resource state of the target machine in the observation state includes:
the target machine, while in the observation state: if the hardware consumption state is larger than a preset first threshold value, determining that the second conditional probability is 1; if the task number is larger than a preset second threshold value, determining that the second conditional probability is 1; if the historical maximum load sum is larger than a preset third threshold value, determining that the second conditional probability is 1; the historical maximum load is the sum of the maximum values of the transcoding historical loads corresponding to all transcoding tasks in the target machine; otherwise, determining that the second conditional probability is 0.
In a second aspect, based on the same inventive concept, the present application provides the following technical solutions through an embodiment:
a state determination device of a live transcoding machine, comprising:
the system comprises an acquisition module, a transcoding module and a transcoding module, wherein the acquisition module is used for acquiring the transcoding historical load of a transcoding task and the transcoding real-time load at the current moment aiming at each transcoding task in a target machine; the estimation module is used for acquiring the estimated load of the transcoding task according to the transcoding real-time load and the transcoding historical load; the determining module is used for determining the observation state of the target machine according to the estimated loads of all transcoding tasks in the target machine; wherein the observation state is used to characterize a resource consumption size of the target machine; the prediction module is used for obtaining the current state prediction result of the target machine according to the conditional probability of each hidden state of the target machine in the observation state; wherein the hidden state represents whether the target machine is available.
In a third aspect, based on the same inventive concept, the present application provides the following technical solutions through an embodiment:
a computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of any of the first aspects.
The method and the device for determining the state of the live transcoding machine provided by the embodiment of the invention are characterized in that the transcoding historical load of a transcoding task and the transcoding real-time load at the current moment are obtained by aiming at each transcoding task in a target machine; and obtaining the estimated load of the transcoding task according to the transcoding real-time load and the transcoding historical load. Then, determining the observation state of the target machine according to the estimated loads of all transcoding tasks in the target machine; wherein, the observation state is used for representing the resource consumption of the target machine; finally, according to the conditional probability of each hidden state of the target machine when the target machine is in the observation state, obtaining the current state prediction result of the target machine; wherein the hidden state indicates whether the target machine is available. In the embodiment of the invention, the estimated load of the transcoding task is determined through two dimensions, namely transcoding real-time load and transcoding historical load, so that abnormal fluctuation data in the estimated load is weakened, and the accuracy of the estimated load is improved; furthermore, the stability and the accuracy of the observation state of the target machine are improved; and finally, predicting the state change of the target machine by observing the conditional probabilities of different hidden states in the state, thereby determining the state change trend of the target machine, improving the state judgment accuracy, and avoiding the phenomenon that the transcoding machine is idle or overloaded due to extreme scheduling caused by dynamic fluctuation of the load during task scheduling.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts. In the drawings:
fig. 1 shows a flowchart of a method for determining a state of a live transcoding machine according to a first embodiment of the present invention;
fig. 2 shows a functional module structural diagram of a state determining apparatus of a live transcoding machine according to a second embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
When a dynamic code rate coding technology is used, task load of a transcoding task fluctuates in a large range along with different live broadcast contents and pictures, and the fluctuation of the load directly causes the state of a transcoding machine to fluctuate, so that task scheduling of the transcoding task is more difficult and complicated. Therefore, in order to improve the accuracy of state judgment of the transcoding machine, ensure the high efficiency and reliability of transcoding task scheduling and avoid the extreme scheduling situation, the invention provides a method and a device for determining the state of the live transcoding machine. The technical solution of the present invention is explained in detail by the following specific examples.
First embodiment
Referring to fig. 1, a method for determining a status of a live transcoding machine according to a first embodiment of the present invention is shown, where the method includes:
step S10: and aiming at each transcoding task in the target machine, acquiring the historical transcoding load of the transcoding task and the real-time transcoding load at the current moment.
In step S10, the target machine is the transcoding machine that currently needs to determine the state. The transcoding machine is a machine for transcoding a live stream of a live video, and the transcoding machine may be a specific computer, a server, or other devices with data processing capability, and the like, without limitation. A transcoding task means transcoding a live source stream, and each anchor correspondingly generates a transcoding task after starting the live stream. One or more transcoding tasks may exist on a target machine.
Aiming at any transcoding task, the transcoding real-time load of the transcoding task is available; generally, there is also a transcoding history load corresponding to the transcoding task.
1. The transcoding real-time load obtaining process of the transcoding task is as follows:
first, a feature vector of a transcoding task may be obtained.
The feature vector may be determined by task parameters of the transcoding task. Specifically, the task parameters include one or more of the following: video bitrate, video temporal complexity, video spatial complexity, coding parameters, processor occupancy and memory occupancy. And then, processing the task parameters according to the height, the width and the frame rate of the video frame of the transcoding task to obtain the characteristic parameters of the unit pixel. The specific treatment process is as follows:
wherein, x' i Is a characteristic parameter of a unit pixel, x i As the parameters of the task, it is,width is the video frame width, height is the video frame height, and fps is the video frame rate. Normalization is realized through the characteristic parameters of the unit pixels, the load evaluation consistency of different transcoding tasks is guaranteed, and the load condition of the unit pixels can be accurately measured. Then, according to the characteristic parameters, a characteristic vector is obtained, and elements in the characteristic vector are formed by the characteristic parameters.
And then, determining the task load of the transcoding task according to the feature vector.
Specifically, in the actual transcoding process, different characteristic parameters have different influences on the transcoding load; in this embodiment, the weight distribution can be performed on the characteristic parameters according to the actual situation, so that the reliability of the task load is improved, and the actual load situation of the transcoding task can be more accurately reflected. The weight corresponding to each feature parameter can be obtained by a linear regression method based on historical data, so that a better correlation between each feature parameter and the task load is ensured, and is not described in detail in this embodiment. Thus, the task load may be specifically expressed as follows:
L o (v,t)＝ω·x
wherein L is o (v, t) represents the task load of the transcoding task v at the time t, ω represents the weight, and x represents the feature vector composed of the feature parameters.
And finally, acquiring a transcoding real-time load according to the task load.
Generally, the task load can be regarded as a transcoding real-time load, but the load generally has certain fluctuation in the transcoding process. In order to avoid adverse effects such as distortion and jump caused by the volatility on the coded real-time load, in this embodiment, the volatility of the task load is smoothed by a preset smoothing function g (t), so that the fluctuation condition of the task load is reduced. I.e. the smoothing function is used to smooth the fluctuations of the task load, g (t) may specifically be
Wherein c is 1 And c 2 Is a positive number, g (t) is a value range [0,1]The transcoding real-time load is specifically as follows:
wherein the content of the first and second substances,
for transcoding task v transcoding real-time load at time t, L o (v, τ) represents the task load at a distance τ time from time t, and g (t- τ) represents the smoothing parameter at time (t- τ). Therefore, the task loads near the current t moment are all used for smoothing the task loads at the t moment through g (t), the influence caused by instantaneous fluctuation of the loads can be effectively reduced, and the accuracy and the reliability of transcoding real-time loads are improved. Further, s may take a value of not more than 30s, for example, 5s, 10s, 15s, and the like. If the value is too large, the calculated amount is rapidly increased, the real task load at the time t is weakened, and the accuracy of transcoding real-time load is reduced; if the value is too small, the task load is not well smoothed, the transcoding real-time load is easy to generate instantaneous jump, and the accuracy is reduced.
In the embodiment, the task load is obtained through the method, and then the transcoding real-time load is obtained, so that the actual influence of each characteristic parameter on the transcoding real-time load can be refined, and the objectivity and the accuracy of the transcoding real-time load can be effectively improved.
2. The transcoding historical load obtaining process of the transcoding task is as follows:
first, it should be noted that, in some cases, a live broadcast platform may have a relatively novel live broadcast content, and the new live broadcast content cannot find a corresponding reference in a historical live broadcast, and at this time, it may be considered that a transcoding task corresponding to the new live broadcast has no transcoding historical load. In addition, if the historical data corresponding to the transcoding task cannot be acquired, the transcoding task corresponding to the new live broadcast can be considered to have no transcoding historical load.
In practical application, for each live stream, the played contents of the live stream have similarity, and the estimated load can be corrected through the statistical characteristics of the transcoding historical load. That is, the transcoding historical load may be determined by statistical characteristics. In this embodiment, the statistical features may be divided into two main dimensions: 1. transcoding load quantiles; 2. continuously stabilizing the load duration. Whether the transcoding task is similar to the historical task can be judged through the habit characteristic, the partition characteristic, the content characteristic and the like of the anchor. For example, the habit features can be a code rate, a frame rate, a picture scale and the like selected in the live broadcast; the partition characteristics may be a partition type in which the live broadcast is located, and the partition types include: online game competition, entertainment heaven and earth, science and technology culture, voice interaction and the like; the content features may be live specific content, including: cars, music, dancing, games, etc. For example, when the habit characteristics, the partition characteristics, and the content characteristics of a historical task are all the same, the transcoding real-time load of the historical task during transcoding can be considered as the transcoding historical load corresponding to the current transcoding task.
Specifically, a historical task similar to the current transcoding task may be determined first; and then, carrying out quantile statistics and continuous stable load duration statistics on the corresponding transcoding real-time loads of all historical tasks in the transcoding process of the historical tasks to obtain corresponding quantile statistical data and stable time statistical data.
Then, when determining the transcoding historical load of the transcoding task, the following manner may be used:
1. and determining the transcoding historical load of the transcoding task based on the quantile statistical data.
Specifically, a target quantile can be preset; and then, acquiring the transcoding real-time load of the historical task corresponding to the target quantile as the transcoding historical load of the current transcoding task. The target quantile may be a number greater than 70%. Specifically, the transcoding load may be 75% to 95%, for example, 80%, 85%, 90%, and so on, and when the current transcoding real-time load is modified by using the transcoding historical load, it is guaranteed that the transcoding machine has sufficient load when the transcoding task is distributed.
2. And determining the transcoding historical load of the transcoding task based on the stability time statistical data.
The stability time statistic data includes the load condition of the historical task in a certain stability period. The stable time period is the time period within which the transcoding real-time load corresponding to the historical task does not change or changes very little, for example, the change degree is within 2%. The length of the stable period may be determined based on experience, for example, the length of the stable period should be greater than 15min, and may be specifically 20min, 30min, and so on. The value range of the stable time period can reflect the stable value of the dynamic change of the load in the live broadcast process. Further, if a plurality of stable time periods exist, the transcoding real-time load corresponding to each stable time period historical task can be obtained. And finally, selecting the largest transcoding real-time load corresponding to the historical task in a stable period as the transcoding historical load, so that when the load dynamically changes to the largest load in the live broadcasting process, the transcoding machine has enough load. In addition, the mode can also avoid the situation that the load forecast is too high or too low, namely, the adverse effect caused by the transient fluctuation of the load is avoided.
3. And determining the transcoding historical load of the transcoding task based on the quantile statistical data and the stable time statistical data. In this implementation, the larger value of the determined transcoding historical loads in 1 and 2 above may be used as the final transcoding historical load. Therefore, after final prediction is further ensured, the transcoding machine has enough load, and the reliability of the prediction result is improved.
Step S20: and obtaining the estimated load of the transcoding task according to the transcoding real-time load and the transcoding historical load.
In step S20, a first weight may be first assigned to the real-time load of the code, and a second weight may be assigned to the historical load of the code; wherein the sum of the first weight and the second weight is 1, and the value range of the first weight and the second weight is [0,1]; and then, determining the sum of the transcoding real-time load after the first weight is distributed and the transcoding historical load after the second weight is distributed as the estimated load of the transcoding task. Specific reference is made to the following:
wherein, the first and the second end of the pipe are connected with each other,
for the estimated load of the transcoding task v at time t,
for transcoding task v, transcoding real-time load at time t, c is a first weight, L h (v, t) is the transcoding historical load of transcoding task v at time t. In this embodiment, when there is no transcoding history load, c =1; when the transcoding task is just started, the transcoding real-time load has a climbing stage, and generally can reach a stable state within about 5-15 s, so that c =0 can be determined when the transcoding task is just started; in other cases, c is determined by the live broadcast room, the partition type, the live broadcast content and the like, and the value range is (0, 1). For example, when live broadcast with high load such as a large game is carried out, the value of c is close to 1, and the weight of real-time load is larger; the value of the low-load live broadcast c such as the color live broadcast, the singing live broadcast and the like is closer to one end of 0. The specific value of the weight can be obtained through actual test and debugging, and the functional relationship between the live broadcast type and the weight value can be set in a self-defined way without limitation.
The weight control strategy can ensure that the real fluctuation condition of the load is larger when the high-load live broadcast is carried out, and the first weight of the distribution of the transcoding real-time load is larger, so that the accuracy of the state judgment of the transcoding machine can be ensured; when the low-load live broadcast is carried out, the load fluctuation is often abnormal fluctuation of unreal consumed load, the distribution of transcoding real-time load is smaller first weight, the load fluctuation condition of a smooth transcoding task is facilitated, and the reliability of transcoding machine state judgment is improved.
In addition, in step S20, the transcoding real-time load is corrected by introducing the transcoding historical load, so that the state prediction of the transcoding machine can be more stable, and the actual load situation can be matched. Extreme situations of prediction results caused by large load fluctuation are avoided.
Step S30: determining the observation state of the target machine according to the estimated loads of all transcoding tasks in the target machine; wherein the observation state is used to characterize a resource consumption size of the target machine.
In step S30, the target machine state may be predicted based on the estimated load prediction, and specifically, may be modeled by a hidden markov model. Specifically, the real-time status of the target machine is also related to the load necessary for the target machine itself to operate. Firstly, acquiring the basic consumption load of a target machine; the basic consumption load is the load necessary for the operation of the target machine; and then, determining the sum of the estimated load and the basic consumption load of all transcoding tasks in the target machine as the observation state of the target machine. The method comprises the following specific steps:
where O (t) is the observed state of the target machine at time t, v n For the transcoding task at time t in the target machine,
the base consumption load of the target machine at time t. The basic consumption load is all loads except the transcoding task and can be directly obtained from the target machine.
In the step, the observation load is obtained by combining the basic consumption load of the target machine, so that the accuracy of judging the state of the target machine can be further improved.
Step S40: obtaining a current state prediction result of the target machine according to the conditional probability of each hidden state of the target machine in the observation state; wherein the hidden state represents whether the target machine is available.
In step S40, there is a corresponding hidden state in the observation state of the target device, and the hidden state indicates whether the target device is available. Specifically, in this embodiment, if the hidden state is M, M = { unavailable, high load, available }, and the high load state is used as a critical state between available state and unavailable state. Further, the hidden state may correspond to an initial state π, which may be expressed as follows:
where O (0) is the initial observed state and threshold is the threshold at which the target machine is unavailable.
Further, when the hidden state is available or high-load, the first conditional probability can be obtained according to the historical probability that the target machine is available or high-load when in the observation state; when the hidden state is unavailable, obtaining a second conditional probability according to the machine resource state of the target machine in the observation state; and the machine resource state represents the state of whether the target machine can execute the transcoding task. Specifically, it can be expressed as follows:
conditional probability of being in state i in observation state O:
wherein, P e (i) To observe the conditional probability of being in state i under state O, p (i | O) is the first conditional probability, p c Is the second conditional probability. For the first conditional probability, it can be obtained by counting the probability of being in state i when observing state O. However, the second conditional probability is relatively few in occurrence of the unusable state, and thus the statistical data is easily lost, and if the second conditional probability is obtained based on statistics, a large error occurs, and the reliability of the final determination result is lowered. Therefore, the second conditional probability is determined in the following manner in the present embodiment:
when the target machine is in a certain observation state:
1. if the hardware consumption state is larger than a preset first threshold value, determining that the second conditional probability is 1; wherein the hardware comprises any one or more of: hard disk, network card, memory, graphics processor, etc. The hardware consumption state represents the usage load state of these hardware. If the hardware consumption state is a plurality of hardware, the hardware consumption state is a plurality of hardware consumption states, and the number of the first threshold values corresponds to the number of the hardware consumption states one by one. For example, if the hardware consumption status of any hardware is greater than the first threshold, it indicates that the hardware has reached the upper performance limit, and the target machine cannot increase the transcoding task any more, and the second conditional probability is determined to be 1.
2. If the task number is larger than a preset second threshold value, determining that the second conditional probability is 1; the size of the second threshold may be determined by counting the number of historical tasks, or may be fixed based on the parallel processing performance of the transcoding machine. When the number of the tasks is too large, a large load may be caused, and the transcoding efficiency is reduced, so that when the number of the tasks is greater than a preset second threshold, the second conditional probability is determined to be 1, and the target machine is prevented from further increasing the transcoding tasks.
3. If the historical maximum load sum is larger than a preset third threshold value, determining that the second conditional probability is 1; and the historical maximum load is the sum of the maximum values of the transcoding historical loads corresponding to all transcoding tasks in the target machine. The transcoding task running in the target machine does not necessarily reach the maximum load dynamically at the current moment, but in fact at some future moment the load of the transcoding task in the current target machine may rise resulting in the target machine reaching a high load state. Therefore, to avoid continuing to assign transcoding tasks to the target machine at the current time, the sum of the estimated loads in the target machine at some future time exceeds the maximum design load of the target machine. In this embodiment, the maximum values of the transcoding historical loads corresponding to all transcoding tasks are summed and compared with a preset third threshold, and the third threshold may be valued to be or slightly smaller than an observation state corresponding to a high load state. When the historical maximum load sum is greater than a preset third threshold, the situation that the target machine runs at full load with a certain probability due to all transcoding tasks in the target machine is shown. At this time, the second conditional probability is determined to be 1, and the target machine can be guaranteed to normally operate at the future time.
4. In addition to the above 3 cases, it can be determined that the second conditional probability is 0. I.e. the first conditional probability.
After the determination of the conditional probability is completed, a transition probability matrix can be constructed according to the first conditional probability and the second conditional probability, which is specifically as follows:
wherein, P t (i, j) represents the probability that the target machine transitions from the previous state i to state j in observation state O, and states 1,2,3 represent unavailable, high load, and available, respectively.
And finally, obtaining the current state prediction result of the target machine based on the transition probability matrix. When the state prediction result is obtained, a viterbi (viterbi) algorithm can be adopted for solving, so that the hidden state of the current target machine can be obtained, namely the hidden state is used as the current state prediction result of the target machine.
The state prediction result obtained by the transition probability matrix mode is a result obtained by integrating the unavailability, high load and availability probability in an observation state, and is more accurate and reliable.
After obtaining the state prediction result, the following application scenarios are also used in this embodiment to explain the effect of the method in this embodiment:
the transcoding system generally comprises a plurality of transcoding rooms, each transcoding room is provided with a plurality of transcoding machines, and the target machine can be any one of the transcoding machines, so that the current state of any transcoding machine can be judged, and a reliable basis is provided for the scheduling of transcoding tasks. And the transcoding machines in the computer room are divided into a plurality of transcoding clusters. When the task is scheduled, the load of the transcoding task and the state of the transcoding machine are combined to select an optimal machine.
Transcoding the state of the machine room: on the basis of the transcoding machine state, the number of machines available in the machine room exceeds a threshold value, and the machine room is considered to be available when no network fault exists in the machine room.
Transcoding the cluster: the transcoding machines in the same machine room are divided into a plurality of logic transcoding clusters; in this embodiment, the method is mainly divided into two categories: the system comprises a high-load transcoding task cluster and a low-load transcoding task cluster.
The transcoding task with high load is predicted, and only the transcoding task machine with high load can be dispatched; and (4) scheduling the low-load transcoding task machines to the low-load transcoding task machines in priority, wherein the low-load transcoding task machines are expected to be low-load transcoding tasks. When the low-load transcoding machines are in a high-load state, the low-load transcoding tasks can be dispatched to the high-load transcoding task machines. Therefore, the rationality of the distribution of transcoding tasks with different loads is ensured, and the scheduling efficiency is improved.
When live streaming and new transcoding tasks are issued, the scheduling strategy is as follows:
1. before live streaming, a transcoding scheduling system is inquired to obtain the transcoding historical load L of the same live room h (v) (ii) a That is, the transcoding history load L of the current transcoding task h (v)。
2. Selecting a transcoding machine room which meets the transcoding historical load and has enough available state transcoding machines according to a near access principle; the transcoding machine satisfying the transcoding historical load indicates that the transcoding machine is still available after increasing the transcoding historical load.
3. If one live broadcast room needs to issue a plurality of different transcoding tasks, the tasks are distributed to different transcoding machines as much as possible;
4. when the transcoding task is distributed, one of all the machines which are available in the state M and meet the historical transcoding load is randomly selected.
When a problem occurs in the transcoding machine, the problem situation is as follows:
1. when the transcoding cluster has enough transcoding load, the machines are in an available state, but no single machine meets the transcoding load requirement of a new transcoding task;
2. the transcoding load suddenly rises to cause the load of a single transcoding machine to be insufficient;
3. and if the transcoding machine fails or is abnormal, other transcoding machines are required to continuously complete the transcoding task.
Aiming at the problem situation, the ongoing transcoding task migration strategy is as follows:
in order to ensure smooth migration of the transcoding task and avoid live broadcast interruption or problems caused by task interruption, the original transcoding task needs to be recovered after a new repeated transcoding task is started through transcoding task scheduling. When the above problem situations 2 and 3 occur, the selection strategy of the transcoding machine is the same as the scheduling strategy when a new transcoding task is issued. When the problem 1 occurs, the scheduling steps are as follows:
step 1: and selecting a target transcoding machine, and arranging the transcoding tasks in the target transcoding machine from large to small according to the estimated load to serve as alternative transcoding tasks needing to be migrated. When the target transcoding machine is selected, the transcoding machine with the largest available load can be selected, and the transcoding machine with the smallest observation state can also be selected, so that the scheduling efficiency is improved, and the selection calculation times are reduced; in addition, a transcoding machine can be randomly determined without limitation.
Step 2: transferring transcoding tasks in the target transcoding machine to other transcoding machines in sequence, and transferring the transcoding tasks in the target transcoding machine according to a scheduling strategy for issuing a new transcoding task during transferring;
and step 3: judging whether a transcoding machine meets the load requirement of a new transcoding task or not; if no transcoding machine meets the load requirement of the new transcoding task, returning to the step 1 to continue executing; and if the transcoding machines meet the load requirement of the new transcoding task, migrating the transcoding task in the target transcoding machine.
The state prediction result obtained based on the implementation method can be used for remarkably improving the reliability when the existing transcoding task and the new transcoding task are scheduled, and the transcoding machine can effectively deal with different load conditions. And the state judgment of the transcoding machine does not fall into an extreme condition, the transcoding task can be flexibly scheduled in the execution process, the utilization rate of the transcoding machine is improved, and the overload of the machine caused by large-amplitude dynamic fluctuation of the load can be effectively coped with.
In summary, according to the method for determining the status of the live transcoding machine provided in this embodiment, a transcoding history load of a transcoding task and a transcoding real-time load at a current moment are obtained for each transcoding task in a target machine; and obtaining the estimated load of the transcoding task according to the transcoding real-time load and the transcoding historical load. Then, determining the observation state of the target machine according to the estimated loads of all transcoding tasks in the target machine; wherein, the observation state is used for representing the resource consumption of the target machine; finally, according to the conditional probability of each hidden state of the target machine when the target machine is in the observation state, obtaining the current state prediction result of the target machine; wherein the hidden state indicates whether the target machine is available. In the embodiment, the estimated load of the transcoding task is determined through two dimensions, namely the transcoding real-time load and the transcoding historical load, so that abnormal fluctuation data in the estimated load is weakened, and the accuracy of the estimated load is improved; furthermore, the stability and the accuracy of the observation state of the target machine are improved; and finally, predicting the state change of the target machine by observing the conditional probabilities of different hidden states in the state, thereby determining the state change trend of the target machine, improving the state judgment accuracy, and avoiding the phenomenon that the transcoding machine is idle or overloaded due to extreme scheduling caused by dynamic fluctuation of the load during task scheduling.
Second embodiment
Referring to fig. 2, based on the same inventive concept, a second embodiment of the present invention provides an apparatus 300 for determining a status of a live transcoding machine, including:
the obtaining module 301 is configured to obtain, for each transcoding task in the target machine, a transcoding history load of the transcoding task and a transcoding real-time load at the current moment; the estimation module 302 is configured to obtain an estimated load of the transcoding task according to the transcoding real-time load and the transcoding historical load; a determining module 303, configured to determine an observation state of the target machine according to estimated loads of all transcoding tasks in the target machine; wherein the observation state is used to characterize a resource consumption size of the target machine; a prediction module 304, configured to obtain a current state prediction result of the target machine according to conditional probabilities of hidden states of the target machine in the observation state; wherein the hidden state represents whether the target machine is available.
As an optional implementation manner, the obtaining module 301 is specifically configured to:
acquiring a feature vector of the transcoding task; determining task load of the transcoding task according to the feature vector; and acquiring the transcoding real-time load according to the task load.
As an optional implementation manner, the obtaining module 301 is further specifically configured to:
acquiring task parameters of the transcoding task; the task parameters include one or more of: video code rate, video time complexity, video space complexity, coding parameters, processor occupation and memory occupation; processing the task parameters according to the height, the width and the frame rate of the video frame of the transcoding task to obtain characteristic parameters of unit pixels; and obtaining the feature vector according to the feature parameters.
As an optional implementation manner, the obtaining module 301 is further specifically configured to:
acquiring the transcoding real-time load according to the task load and a preset smooth function; wherein the smoothing function is to smooth fluctuations of the task load.
As an optional implementation manner, the estimation module 302 is specifically configured to:
distributing a first weight to the transcoding real-time load, and distributing a second weight to the transcoding historical load; wherein the sum of the first weight and the second weight is 1, and the value range of the first weight and the second weight is [0,1]; and determining the sum of the transcoding real-time load after the first weight is distributed and the transcoding historical load after the second weight is distributed as the estimated load of the transcoding task.
As an optional implementation manner, the determining module 303 is specifically configured to:
acquiring the basic consumption load of a target machine; the basic consumption load is the load necessary for the target machine to run; and determining the sum of the estimated load of all transcoding tasks in the target machine and the basic consumption load as the observation state of the target machine.
As an optional implementation manner, the prediction module 304 is specifically configured to:
when the hidden state is available or high-load, obtaining a first conditional probability according to the historical probability that the target machine is available or high-load in the observation state; when the hidden state is unavailable, obtaining a second conditional probability according to the machine resource state of the target machine in the observation state; wherein the machine resource state represents a state of whether a target machine can execute the transcoding task; constructing a transition probability matrix according to the first conditional probability and the second conditional probability; and obtaining the current state prediction result of the target machine based on the transition probability matrix.
As an optional implementation, the machine resource status includes: hardware consumption state, task number and historical maximum load sum; the prediction module 304 is specifically configured to:
the target machine, while in the observation state: if the hardware consumption state is larger than a preset first threshold value, determining that the second conditional probability is 1; if the task number is larger than a preset second threshold, determining that the second conditional probability is 1; if the historical maximum load sum is larger than a preset third threshold value, determining that the second conditional probability is 1; the maximum historical load is the sum of the maximum values of the transcoding historical loads corresponding to all transcoding tasks in the target machine; otherwise, determining that the second conditional probability is 0.
It should be noted that, the implementation and technical effects of the apparatus 300 for determining the status of a live transcoding machine provided in the embodiment of the present invention are the same as those of the foregoing method embodiment, and for a brief description, reference may be made to corresponding contents in the foregoing method embodiment where no part of the apparatus embodiment is mentioned.
Third embodiment
Based on the same inventive concept, the third embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the method according to any one of the first aspect.
It should be noted that, in the computer-readable storage medium provided by the embodiment of the present invention, the specific implementation and the resulting technical effect of each step are the same as those of the foregoing method embodiment, and for a brief description, corresponding contents in the foregoing method embodiment may be referred to for the non-mentioned points of the embodiment.
The term "and/or" appearing herein is merely one type of association that describes an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship; the word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A method for determining the state of a live transcoding machine is characterized by comprising the following steps:
aiming at each transcoding task in a target machine, acquiring a transcoding historical load of the transcoding task and a transcoding real-time load at the current moment;
obtaining an estimated load of the transcoding task according to the transcoding real-time load and the transcoding historical load;
determining the observation state of the target machine according to the estimated loads of all transcoding tasks in the target machine; wherein the observation state is used to characterize a resource consumption size of the target machine;
obtaining a current state prediction result of the target machine according to the conditional probability of each hidden state of the target machine in the observation state; wherein the hidden state represents whether the target machine is available.
2. The method of claim 1, wherein the obtaining of the transcoding real-time payload comprises:
acquiring a feature vector of the transcoding task;
determining task load of the transcoding task according to the feature vector;
and acquiring the transcoding real-time load according to the task load.
3. The method of claim 2, wherein the obtaining the feature vector of the transcoding task comprises:
acquiring task parameters of the transcoding task; the task parameters include one or more of: video code rate, video time complexity, video space complexity, coding parameters, processor occupation and memory occupation;
processing the task parameters according to the height, the width and the frame rate of the video frame of the transcoding task to obtain characteristic parameters of unit pixels;
and obtaining the feature vector according to the feature parameters.
4. The method of claim 2, wherein obtaining the transcoding real-time load according to the task load comprises:
acquiring the transcoding real-time load according to the task load and a preset smooth function; wherein the smoothing function is to smooth fluctuations of the task load.
5. The method of claim 1, wherein obtaining an estimated load of the transcoding task based on the transcoding real-time load and the transcoding historical load comprises:
distributing a first weight to the transcoding real-time load, and distributing a second weight to the transcoding historical load; wherein the sum of the first weight and the second weight is 1, and the value range of the first weight and the second weight is [0,1];
and determining the sum of the transcoding real-time load after the first weight is distributed and the transcoding historical load after the second weight is distributed as the estimated load of the transcoding task.
6. The method of claim 1, wherein determining the observed state of the target machine based on the estimated load of all transcoding tasks in the target machine comprises:
acquiring the basic consumption load of a target machine; the basic consumption load is the load necessary for the target machine to run;
and determining the sum of the estimated load of all transcoding tasks in the target machine and the basic consumption load as the observation state of the target machine.
7. The method of claim 1, wherein obtaining the current state prediction result of the target machine according to the conditional probabilities of the hidden states of the target machine in the observation state comprises:
when the hidden state is available or high-load, obtaining a first conditional probability according to the historical probability that the target machine is available or high-load in the observation state;
when the hidden state is unavailable, obtaining a second conditional probability according to the machine resource state of the target machine in the observation state; wherein the machine resource state represents a state of whether a target machine can execute the transcoding task;
constructing a transition probability matrix according to the first conditional probability and the second conditional probability;
and obtaining the current state prediction result of the target machine based on the transition probability matrix.
8. The method of claim 7, wherein the machine resource state comprises: hardware consumption state, task number and historical maximum load sum; the obtaining a second conditional probability according to the machine resource state of the target machine in the observation state includes:
the target machine, while in the observation state:
if the hardware consumption state is larger than a preset first threshold value, determining that the second conditional probability is 1;
if the task number is larger than a preset second threshold value, determining that the second conditional probability is 1;
if the historical maximum load sum is larger than a preset third threshold value, determining that the second conditional probability is 1; the maximum historical load is the sum of the maximum values of the transcoding historical loads corresponding to all transcoding tasks in the target machine;
otherwise, determining that the second conditional probability is 0.
9. A state determination apparatus of a live transcoding machine, comprising:
the acquisition module is used for acquiring the transcoding historical load of the transcoding task and the transcoding real-time load at the current moment aiming at each transcoding task in the target machine;
the estimation module is used for acquiring the estimated load of the transcoding task according to the transcoding real-time load and the transcoding historical load;
the determining module is used for determining the observation state of the target machine according to the estimated loads of all transcoding tasks in the target machine; wherein the observation state is used to characterize a resource consumption size of the target machine;
the prediction module is used for obtaining the current state prediction result of the target machine according to the conditional probability of each hidden state of the target machine in the observation state; wherein the hidden state represents whether the target machine is available.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110691790.2A CN115514996A (en) | 2021-06-22 | 2021-06-22 | Method and device for determining state of live transcoding machine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110691790.2A CN115514996A (en) | 2021-06-22 | 2021-06-22 | Method and device for determining state of live transcoding machine |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115514996A true CN115514996A (en) | 2022-12-23 |
Family
ID=84499725
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110691790.2A Pending CN115514996A (en) | 2021-06-22 | 2021-06-22 | Method and device for determining state of live transcoding machine |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115514996A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101523372A (en) * | 2006-10-05 | 2009-09-02 | 澳大利亚国家Ict有限公司 | Decentralised multi-user online environment |
| CN108307206A (en) * | 2017-12-25 | 2018-07-20 | 北京奇艺世纪科技有限公司 | A kind of distribution method and device of live streaming encoding tasks |
| CN110417610A (en) * | 2018-04-30 | 2019-11-05 | 慧与发展有限责任合伙企业 | Storage system postpones Outlier Detection |
| US20200082316A1 (en) * | 2018-09-12 | 2020-03-12 | International Business Machines Corporation | Cognitive handling of workload requests |
| CN111581174A (en) * | 2019-02-18 | 2020-08-25 | 北京搜狗科技发展有限公司 | Resource management method and device based on distributed cluster system |
| CN112860403A (en) * | 2021-02-22 | 2021-05-28 | 中国联合网络通信集团有限公司 | Cluster load resource scheduling method, device, equipment, medium and product |
-
2021
- 2021-06-22 CN CN202110691790.2A patent/CN115514996A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101523372A (en) * | 2006-10-05 | 2009-09-02 | 澳大利亚国家Ict有限公司 | Decentralised multi-user online environment |
| CN108307206A (en) * | 2017-12-25 | 2018-07-20 | 北京奇艺世纪科技有限公司 | A kind of distribution method and device of live streaming encoding tasks |
| CN110417610A (en) * | 2018-04-30 | 2019-11-05 | 慧与发展有限责任合伙企业 | Storage system postpones Outlier Detection |
| US20200082316A1 (en) * | 2018-09-12 | 2020-03-12 | International Business Machines Corporation | Cognitive handling of workload requests |
| CN111581174A (en) * | 2019-02-18 | 2020-08-25 | 北京搜狗科技发展有限公司 | Resource management method and device based on distributed cluster system |
| CN112860403A (en) * | 2021-02-22 | 2021-05-28 | 中国联合网络通信集团有限公司 | Cluster load resource scheduling method, device, equipment, medium and product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7117499B2 (en) | Virtual computer systems and computer virtualization programs | |
| US9755935B2 (en) | Method, system and computer storage medium for bandwidth optimization of network application | |
| CN115134368B (en) | Load balancing method, device, equipment and storage medium | |
| CN112181613B (en) | Heterogeneous resource distributed computing platform batch task scheduling method and storage medium | |
| CN113434294A (en) | Data processing method, computer equipment and readable storage medium | |
| CN111506434B (en) | Task processing method and device and computer readable storage medium | |
| CN113608871A (en) | Service processing method and device | |
| CN112422977A (en) | Distribution method and distribution device for audio and video transcoding tasks | |
| CN114968521A (en) | Distributed rendering method and device | |
| CN111752710B (en) | Data center PUE dynamic optimization method, system and equipment and readable storage medium | |
| CN115033340A (en) | Host selection method and related device | |
| CN113515382A (en) | Cloud resource allocation method and device, electronic equipment and program product | |
| CN115543626A (en) | Power defect image simulation method adopting heterogeneous computing resource load balancing scheduling | |
| US20130339935A1 (en) | Adjusting Programs Online and On-Premise Execution | |
| CN104679444A (en) | Dynamic adjustment method and device for virtualized storage resources | |
| CN112416568A (en) | Duration estimation method and duration estimation device for audio and video transcoding task | |
| CN115514996A (en) | Method and device for determining state of live transcoding machine | |
| Désiré et al. | Fractional rider deep long short term memory network for workload prediction-based distributed resource allocation using spark in cloud gaming | |
| CN117009042A (en) | Information calculation load scheduling method, device, equipment and medium in Internet of things mode | |
| CN114880079A (en) | Kubernetes cluster scale adjustment method, system and equipment based on reinforcement learning | |
| CN110941489A (en) | Method and device for scaling stream processing engine | |
| CN116089021B (en) | Deep learning-oriented large-scale load mixed part scheduling method, device and medium | |
| CN114997401B (en) | Adaptive inference acceleration method, apparatus, computer device, and storage medium | |
| WO2023226312A1 (en) | Service cluster scaling method and related device | |
| CN114201282A (en) | Dynamic task scheduling method for real-time rendering calculation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |