CN115002513A - Audio and video scheduling method and device, electronic equipment and computer readable storage medium - Google Patents

Audio and video scheduling method and device, electronic equipment and computer readable storage medium Download PDF

Info

Publication number
CN115002513A
CN115002513A CN202210580614.6A CN202210580614A CN115002513A CN 115002513 A CN115002513 A CN 115002513A CN 202210580614 A CN202210580614 A CN 202210580614A CN 115002513 A CN115002513 A CN 115002513A
Authority
CN
China
Prior art keywords
link
user experience
audio
video
index value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210580614.6A
Other languages
Chinese (zh)
Other versions
CN115002513B (en
Inventor
饶明佺
王林鑫
李琳
顾明
刘倍余
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Migu Cultural Technology Co Ltd
China Mobile Communications Group Co Ltd
Original Assignee
Migu Cultural Technology Co Ltd
China Mobile Communications Group Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Migu Cultural Technology Co Ltd, China Mobile Communications Group Co Ltd filed Critical Migu Cultural Technology Co Ltd
Priority to CN202210580614.6A priority Critical patent/CN115002513B/en
Publication of CN115002513A publication Critical patent/CN115002513A/en
Application granted granted Critical
Publication of CN115002513B publication Critical patent/CN115002513B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/2662Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234381Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/24Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/24Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
    • H04N21/2402Monitoring of the downstream path of the transmission network, e.g. bandwidth available
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440281Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the temporal resolution, e.g. by frame skipping
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/44209Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

The invention provides an audio and video scheduling method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of audio and video. The method comprises the following steps: acquiring a first user experience index value corresponding to a target moment; under the condition that the first user experience index value is lower than a first threshold value, acquiring a first link portrait corresponding to a first preset time period before a target moment; inputting the first link portrait into a user experience model to obtain a target user experience index value corresponding to a second time period; and adjusting the link and/or code rate of the audio and video playing corresponding to the second time period according to the target user experience index value and the user experience model. The scheme of the embodiment of the invention can dynamically schedule the link and/or code rate of audio and video playing according to the user experience index value of audio playing, avoids the influence of audio and video playing caused by the weak network environment of the public network, and ensures the high-efficiency operation of the network link.

Description

Audio and video scheduling method and device, electronic equipment and computer readable storage medium
Technical Field
The present invention relates to the field of audio and video technologies, and in particular, to an audio and video scheduling method and apparatus, an electronic device, and a computer-readable storage medium.
Background
With the gradual popularity of the concept of zero-contact and remote office covering the world, how to provide high-availability, high-quality and low-delay real-time audio and video service for customers in the public network environment is more and more important to ensure the efficient operation of a network link. At present, in order to ensure low delay and high quality of real-time audio and video in a public network environment, when a network is detected to be in a weak network scene, the audio and video communication quality is ensured by means of key frame retransmission, code rate control and retry or bandwidth reduction of Forward Error Correction (FEC) of the audio and video, and the influence of the weak network scene on audio and video playing is reduced. However, if the network is always in a weak network scenario, retransmission storms or too low code rates can be caused by no lower limit of retransmission or bandwidth reduction, thereby affecting user experience.
Disclosure of Invention
The invention aims to provide an audio and video scheduling method, an audio and video scheduling device, electronic equipment and a computer readable storage medium, which are used for solving the problem that retransmission storms or too low code rates can be caused by no lower limit retransmission or bandwidth reduction if a network is always in a weak network scene in the prior art.
In a first aspect, an embodiment of the present invention provides an audio and video scheduling method, including:
acquiring a first user experience index value corresponding to the target moment;
under the condition that the first user experience index value is lower than a first threshold value, acquiring a first link portrait corresponding to a first preset time period before a target moment; the link image is used for indicating related information of the audio and video playing link;
inputting the first link portrait into a user experience model to obtain a target user experience index value corresponding to a second time period; the second time period comprises a second preset time period after the target time; the user experience model is used for representing the corresponding relation between a user video playing experience index value and a link portrait under a target audio and video playing code rate;
and adjusting the link and/or code rate of the audio and video playing corresponding to the second time period according to the target user experience index value and the user experience model.
Optionally, the user experience metrics include at least one of: video resolution, video frame rate, video stuck rate, audio stuck rate, second on rate, and packet loss rate.
Optionally, the obtaining a first user experience index value corresponding to a target time includes:
acquiring a first link portrait corresponding to a target moment, and determining a link portrait characteristic value according to the first link portrait;
and determining the user experience index value according to the link portrait characteristic value.
Optionally, the obtaining the corresponding first link representation at the target time includes:
receiving audio and video link data reported by a user at the target moment;
and analyzing and processing the audio and video link data to generate the first link portrait.
Optionally, the audio-video link data comprises at least one of: the method comprises the following steps of timestamp, edge server, terminal machine type, operating system, network type, packet loss rate, video resolution, frame rate, sending code rate and receiving code rate;
the first link representation includes at least one of: time attribute, terminal attribute, network attribute, region attribute, edge server attribute and experience attribute;
the link portrayal feature values include at least one of: time elements, terminal elements, network elements, region elements, edge server elements, experience elements.
Optionally, the user experience model is a model obtained by performing model training on a first model:
the first model is:
Figure BDA0003662225290000021
wherein f (V | θ) represents a user experience index value; v is a link portrait characteristic value; mi is the ith link portrait experience index value corresponding to the link portrait characteristic value, δ i is the weight of the ith user experience index, and θ is the audio and video playing code rate; n is an integer greater than or equal to 1;
the link portrayal experience metric value comprises at least one of: video resolution value, video frame rate value, video cadence rate reciprocal value, audio cadence rate reciprocal value, second open rate value.
Optionally, performing model training on the first model to obtain the user experience model, including:
acquiring a historical link image set; the historical link portraits comprise link portraits before the current moment;
performing model training on the first model through a logistic stewart regression algorithm and a gradient descent method according to the historical link image set to obtain the weight of each user experience index;
determining the experience model according to the first model and the weight of each user experience index.
Optionally, the determining the user experience index value according to the link portrait characteristic value includes:
and calculating the user experience index value through the user experience model according to the link portrait characteristic value.
Optionally, the adjusting, according to the target user experience index value and the user experience model, a link and/or a code rate of audio and video playing corresponding to the second time period includes:
under the condition that the target user experience index value is smaller than a second threshold value, adjusting a link and/or a code rate corresponding to a user experience model until the target user experience index value corresponding to the user experience model is larger than or equal to the second threshold value;
and when the target user experience index value is larger than or equal to a second threshold value, taking the link and the code rate corresponding to the user experience model as the link and the code rate of the audio and video playing corresponding to the second time period after adjustment.
In a second aspect, an embodiment of the present invention provides an audio and video scheduling apparatus, including:
the first acquisition module is used for acquiring a first user experience index value corresponding to the target moment;
the second acquisition module is used for acquiring a first link portrait corresponding to a first preset time period before a target time under the condition that the first user experience index value is lower than a first threshold value; the link image is used for indicating relevant information of the audio and video playing link;
the processing module is used for inputting the first link portrait into a user experience model to obtain a target user experience index value corresponding to a second time period; the second time period comprises a second preset time period after the target time; the user experience model is used for representing the corresponding relation between the user video playing experience index value and the link portrait under the target audio and video playing code rate;
and the adjusting module is used for adjusting the link and/or code rate of the audio and video playing corresponding to the second time period according to the target user experience index value and the user experience model.
In a third aspect, an embodiment of the present invention provides an electronic device, including: a transceiver, a processor, a memory, and a program or instructions stored on the memory and executable on the processor; the processor implements the audio and video scheduling method as described above when executing the program or instructions.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a program or instructions are stored, which when executed by a processor implement the steps in the audio-video scheduling method as described above.
According to the audio and video scheduling method, the user experience of audio and video playing can be monitored by acquiring the first user experience index value corresponding to the target moment, and the target user experience index value is predicted according to the first link portrait and through the user experience model under the condition that the first user experience index value is detected to be lower than the first threshold value; and adjusting the link and/or code rate of the audio and video playing corresponding to the second time period according to the predicted target user experience index value. According to the scheme, the link and/or the code rate of audio and video playing can be dynamically scheduled according to the user experience index value of the audio playing, the influence of the audio and video playing caused by the fact that the public network is in a weak network environment is avoided, high-efficiency operation of the network link is guaranteed, and the problem that in the prior art, if the network is always in a weak network scene, the code rate is too low due to the fact that retransmission storms are caused infinitely or bandwidth is reduced infinitely is solved.
Drawings
Fig. 1 is a schematic flow diagram of an audio and video scheduling method according to an embodiment of the present invention;
FIG. 2 is a diagram of a first link image element according to an embodiment of the invention;
fig. 3 is a schematic block diagram of an audio/video scheduling apparatus according to an embodiment of the present invention;
fig. 4 is a schematic diagram of an entity device of an audio/video scheduling device according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantages of the present invention more apparent, the following detailed description is given with reference to the accompanying drawings and specific embodiments. In the following description, specific details such as specific configurations and components are provided only to help the full understanding of the embodiments of the present invention. It will therefore be apparent to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. In addition, the description of the suppression function and configuration is omitted for the sake of clarity and conciseness.
It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
In various embodiments of the present invention, it should be understood that the sequence numbers of the following processes do not mean the execution sequence, and the execution sequence of each process should be determined by the function and the inherent logic of the process, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
In addition, the terms "system" and "network" are often used interchangeably herein.
In the embodiments provided herein, it should be understood that "B corresponding to a" means that B is associated with a from which B can be determined. It should also be understood that determining B from a does not mean determining B from a alone, but may be determined from a and/or other information.
As shown in fig. 1, an audio/video scheduling method according to an embodiment of the present invention includes the following steps:
step 101, obtaining a first user experience index value corresponding to a target time.
According to the audio and video scheduling method, the user experience index value of the audio playing can be monitored in real time by acquiring the first user experience index value corresponding to the target moment.
102, acquiring a first link portrait corresponding to a first preset time period before a target time under the condition that the first user experience index value is lower than a first threshold value; the link image is used for indicating the related information of the audio and video playing link.
It should be noted that, when the first user experience index value is lower than the first threshold value, that is, the public network is in a weak network environment, the environment of the public network is determined according to the first user experience index value.
In an embodiment of the present invention, the link image may include: time attribute, terminal attribute, network attribute, region attribute, server attribute, experience attribute. The link portrait represents the relevant information of the audio and video playing link, so that the magnitude of link data can be greatly reduced, and the speed of calculating the target user experience index value is improved.
Step 103, inputting the first link portrait into a user experience model to obtain a target user experience index value corresponding to a second time period; the second time period comprises a second preset time period after the target time; the user experience model is used for representing the corresponding relation between the user video playing experience index value and the link portrait under the target audio and video playing code rate.
It should be noted that the user experience model is a model after learning and training, and a target user experience index value corresponding to the second time period is predicted through the user experience model according to the first link representation.
And step 104, adjusting the link and/or code rate of the audio and video playing corresponding to the second time period according to the target user experience index value and the user experience model.
In an embodiment of the present invention, the link adjustment mainly refers to adjusting an edge server, so that the audio and video experience of the user is optimal in the second time period.
According to the scheme of the invention, the dynamic adjustment of the link and/or the code rate can be triggered when the user experience index value of the audio and video playing is detected to be lower than the first threshold value. The influence of audio and video playing due to the fact that the public network is in a weak network environment is avoided, efficient operation of a network link is guaranteed, and the problem that in the prior art, if the network is always in a weak network scene, infinite retransmission causes retransmission storms or infinite bandwidth reduction causes too low code rate is solved.
Optionally, the user experience metrics include at least one of: video resolution, video frame rate, video stuck rate, audio stuck rate, second on rate, and packet loss rate.
In an embodiment of the present invention, the first threshold corresponding to the video resolution may be 800 × 480 or 1280 × 720; the first threshold corresponding to the video frame rate may be 30 fps; the first threshold corresponding to the video morton rate may be 6%; the first threshold value corresponding to the audio morton rate may be 2%.
According to the embodiment of the invention, whether the user public network is in the weak network state is judged according to the user experience index value.
Optionally, the obtaining a first user experience index value corresponding to a target time includes:
acquiring a first link portrait corresponding to a target moment, and determining a link portrait characteristic value according to the first link portrait;
and determining the user experience index value according to the link portrait characteristic value.
In an embodiment of the present invention, the determining a characteristic value of a link portrait according to the first link portrait includes:
and processing the first link portrait through a Hash function to obtain the characteristic value of the link portrait.
Through the data format of the link portrait, the magnitude of link data can be effectively reduced, and therefore the calculation amount for determining the user experience index value is reduced.
Optionally, the obtaining the corresponding first link representation at the target time includes:
receiving audio and video link data reported by a user at the target moment;
and analyzing and processing the audio and video link data to generate the first link portrait.
The link representation is an important factor affecting the quality of the link communication. In one embodiment of the invention, a user reports audio and video link data every 5 seconds, and stores the received audio and video link data to obtain an audio and video link data set; and analyzing the audio and video link data set to generate a link portrait.
Optionally, the audiovisual link data comprises at least one of: the method comprises the following steps of timestamp, edge server, terminal machine type, operating system, network type, packet loss rate, video resolution, frame rate, sending code rate and receiving code rate;
as shown in FIG. 2, the first link representation includes at least one of: time attribute, terminal attribute, network attribute, region attribute, edge server attribute and experience attribute;
the link portrayal feature values include at least one of: time elements, terminal elements, network elements, region elements, edge server elements, experience elements.
According to the scheme of the embodiment of the invention, the link portrayal is established by establishing the terminal machine type, the audio and video service quality index, the server element and the like, so that the link can be more comprehensively described and measured. The method avoids the condition that the adjustment of the communication quality of the public network link is only based on the service end, and does not consider the quality factors such as time period, terminal (model), service end load, code rate, packet loss rate, resolution ratio and the like.
Optionally, the user experience model is a model obtained by performing model training on a first model:
the first model is:
Figure BDA0003662225290000071
wherein f (V | θ) represents a user experience index value; v is a link portrait characteristic value; mi is the ith link portrait experience index value corresponding to the link portrait characteristic value, δ i is the weight of the ith user experience index, and θ is the audio and video playing code rate; n is an integer greater than or equal to 1;
the link portrayal experience metric value comprises at least one of: video resolution value, video frame rate value, video cadence rate reciprocal value, audio cadence rate reciprocal value, second open rate value.
In an embodiment of the present invention, the user experience index value is calculated by the first model under the condition that the link portrait characteristic value V is at the code rate θ.
Optionally, performing model training on the first model to obtain the user experience model, including:
acquiring a historical link image set; the historical link portraits comprise link portraits before the current moment;
performing model training on the first model through a logistic stewart regression algorithm and a gradient descent method according to the historical link image set to obtain the weight of each user experience index;
determining the experience model according to the first model and the weight of each user experience index.
And the historical link image set is determined after analyzing, processing and storing the audio and video link data received at the historical moment.
The first model is subjected to learning training through the historical link portrait set, the weight of each user experience index is determined, and then the user experience model is obtained, so that the user experience model meets the expectation that the model performance is good, the network state is good, the load of an edge server is low, and the user experience score is higher under the condition of a certain code rate theta.
According to the scheme of the embodiment of the invention, model training is carried out on the first model according to the historical link portrait set, and the first model is learned and trained through the historical link portrait set stored with a large amount of historical data, so that the accuracy of the user experience model obtained after training can be ensured; and the target user experience index value is calculated by only adopting the first link portrait corresponding to the first preset time period, so that the occupation of calculation resources is saved, and the calculation efficiency is improved.
Optionally, the determining the user experience index value according to the link portrait characteristic value includes:
and calculating the user experience index value through the user experience model according to the link portrait characteristic value.
According to the practical scheme of the invention, the network environment of the public network used by the user is judged according to the user experience index value, and under the condition that the user experience index value is lower than the first threshold value, the network operation environment of the public network is considered to be bad, so that the experience of the audio and video playing of the user is influenced, and a trigger mechanism is provided for the dynamic adjustment of the link and/or the code rate of the audio and video playing.
Optionally, the adjusting, according to the target user experience index value and the user experience model, a link and/or a code rate of audio and video playing corresponding to the second time period includes:
under the condition that the target user experience index value is smaller than a second threshold value, adjusting a link and/or a code rate corresponding to a user experience model until the target user experience index value corresponding to the user experience model is larger than or equal to the second threshold value;
and when the target user experience index value is larger than or equal to a second threshold value, taking the link and the code rate corresponding to the user experience model as the link and the code rate of the audio and video playing corresponding to the second time period after adjustment.
In an embodiment of the present invention, when calculating the target user experience index value, according to the user experience model, the first link representation is input, and the target user experience index value is output; when the link and/or code rate of the audio and video playing corresponding to the second time period is adjusted, the target user experience index value is input, for example, the second threshold value is input, and according to the user experience model, the link and/or code rate of the audio and video playing corresponding to the second time period is output.
According to the practical scheme of the invention, the link and the code rate of the audio and video playing corresponding to the second time period are adjusted through the second threshold value according to the user experience model, so that the audio and video playing keeps good link and code rate, and better user audio and video experience is obtained.
As shown in fig. 3, an embodiment of the present invention provides an audio/video scheduling apparatus 300, including:
a first obtaining module 301, configured to obtain a first user experience index value corresponding to a target time;
a second obtaining module 302, configured to obtain a first link portrait corresponding to a first preset time period before a target time when the first user experience index value is lower than a first threshold; the link image is used for indicating related information of the audio and video playing link;
a processing module 303, configured to input the first link representation into a user experience model, so as to obtain a target user experience index value corresponding to a second time period; the second time period comprises a second preset time period after the target time; the user experience model is used for representing the corresponding relation between the user video playing experience index value and the link portrait under the target audio and video playing code rate;
and an adjusting module 304, configured to adjust a link and/or a code rate of the audio/video playing corresponding to the second time period according to the target user experience index value and the user experience model.
The audio and video scheduling device can monitor the user experience of audio and video playing by acquiring a first user experience index value corresponding to a target moment, and under the condition that the first user experience index value is detected to be lower than a first threshold value, the target user experience index value is predicted according to the first link portrait and through a user experience model; and adjusting the link and/or code rate of the audio and video playing corresponding to the second time period according to the predicted target user experience index value. The scheme of the invention can dynamically schedule the link and/or the code rate of the audio and video playing according to the user experience index value of the audio playing, avoids the influence of the audio and video playing because the public network is in a weak network environment, ensures the high-efficiency operation of the network link, and solves the problem of the prior art that the code rate is too low because the network is always in a weak network scene and the bandwidth is infinitely reduced or the retransmission is caused.
Optionally, the first obtaining module further includes:
the acquisition subunit is used for acquiring a first link portrait corresponding to a target moment and determining a link portrait characteristic value according to the first link portrait;
and the determining subunit is used for determining the user experience index value according to the link portrait characteristic value.
Optionally, the obtaining subunit is further configured to:
receiving audio and video link data reported by a user at the target moment;
and analyzing the audio and video link data to generate the first link portrait.
Optionally, the determining subunit is further configured to:
and calculating the user experience index value through the user experience model according to the link portrait characteristic value.
Optionally, the adjusting module is further configured to:
under the condition that the target user experience index value is smaller than a second threshold value, adjusting a link and/or a code rate corresponding to a user experience model until the target user experience index value corresponding to the user experience model is larger than or equal to the second threshold value;
and when the target user experience index value is larger than or equal to a second threshold value, taking the link and the code rate corresponding to the user experience model as the link and the code rate of the audio and video playing corresponding to the second time period after adjustment.
As shown in fig. 4, in an embodiment of the present invention, the entity device of the audio/video scheduling device may correspond to a data acquisition module, a quality data module, a link portrait module, a machine learning module, a calculation module, an instruction module, and a dynamic scheduling module.
Optionally, the data acquisition module is configured to:
acquiring a first user experience index value corresponding to a target moment; and (c) a second step of,
and under the condition that the first user experience index value is lower than a first threshold value, acquiring a first link portrait corresponding to a first preset time period before a target time.
Optionally, the computing module is configured to:
and inputting the first link portrait into a user experience model to obtain a target user experience index value corresponding to a second time period.
Optionally, the instruction module is configured to generate a scheduling instruction according to the target user experience index value.
Optionally, the dynamic scheduling module is configured to:
and adjusting the link and/or code rate of the audio and video playing corresponding to the second time period according to the target user experience index value and the user experience model.
Optionally, the link representation module is to:
the method comprises the steps of obtaining a first link portrait corresponding to a target moment, and determining a link portrait characteristic value according to the first link portrait.
Optionally, the computing module is further configured to:
and determining the user experience index value according to the link portrait characteristic value.
Optionally, the data acquisition module is further configured to:
receiving audio and video link data reported by a user at the target moment;
and analyzing and processing the audio and video link data to generate the first link portrait.
Optionally, the data acquisition module is further configured to:
acquiring a historical link image set; the historical set of link portraits comprises link portraits prior to a current time instant.
Optionally, the machine learning module is to:
according to the historical link image set, performing model training on the first model through a logistic regression algorithm and a gradient descent method to obtain the weight of each user experience index;
determining the experience model according to the first model and the weight of each user experience index.
Optionally, the computing module is configured to:
and calculating the user experience index value through the user experience model according to the link portrait characteristic value.
Optionally, the dynamic scheduling module is further configured to:
under the condition that the target user experience index value is smaller than a second threshold value, adjusting a link and/or a code rate corresponding to a user experience model until the target user experience index value corresponding to the user experience model is larger than or equal to the second threshold value;
and when the target user experience index value is larger than or equal to a second threshold value, taking the link and the code rate corresponding to the user experience model as the link and the code rate of the audio and video playing corresponding to the second time period after adjustment.
As shown in fig. 5, an embodiment of the present invention provides an electronic device, including a transceiver 510, a processor 500, a memory 520, and a program or instructions stored on the memory 520 and executable on the processor 500; the processor 500, when executing the program or the instructions, implements the audio/video scheduling method applied as described above.
The transceiver 510 is used for receiving and transmitting data under the control of the processor 500.
Wherein in fig. 5, the bus architecture may include any number of interconnected buses and bridges, with one or more processors, represented by processor 500, and various circuits, represented by memory 520, being linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface. The transceiver 510 may be a number of elements, including a transmitter and a receiver, providing a means for communicating with various other apparatus over a transmission medium. For different user devices, the user interface 530 may also be an interface capable of interfacing with a desired device externally, including but not limited to a keypad, display, speaker, microphone, joystick, etc.
The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 may store data used by the processor 500 in performing operations.
Optionally, the transceiver 510 is configured to obtain a first user experience metric value corresponding to a target time;
under the condition that the first user experience index value is lower than a first threshold value, acquiring a first link portrait corresponding to a first preset time period before a target moment; the link image is used for indicating relevant information of the audio and video playing link.
The processor 500 is configured to input the first link representation into a user experience model to obtain a target user experience index value corresponding to a second time period; the second time period comprises a second preset time period after the target time; the user experience model is used for representing the corresponding relation between a user video playing experience index value and a link portrait under a target audio and video playing code rate;
and adjusting the link and/or code rate of the audio and video playing corresponding to the second time period according to the target user experience index value and the user experience model.
Optionally, the transceiver 510 is further configured to:
acquiring a first link portrait corresponding to a target moment, and determining a link portrait characteristic value according to the first link portrait;
and determining the user experience index value according to the link portrait characteristic value.
Optionally, the transceiver 510 is also used for
Receiving audio and video link data reported by a user at the target moment;
and analyzing the audio and video link data to generate the first link portrait.
Optionally, the transceiver 510 is further configured to:
acquiring a historical link image set; the historical link portrait set comprises link portraits before the current moment;
optionally, the processor 500 is further configured to:
performing model training on the first model through a logistic stewart regression algorithm and a gradient descent method according to the historical link image set to obtain the weight of each user experience index;
determining the experience model according to the first model and the weight of each user experience index.
Wherein the first model is:
Figure BDA0003662225290000131
wherein f (V | θ) represents a user experience index value; v is a link portrait characteristic value; mi is the ith link portrait experience index value corresponding to the link portrait characteristic value, δ i is the weight of the ith user experience index, and θ is the audio and video playing code rate; n is an integer greater than or equal to 1;
the link portrayal experience metric value comprises at least one of: video resolution value, video frame rate value, video cadence rate reciprocal value, audio cadence rate reciprocal value, second open rate value.
Optionally, the processor 500 is further configured to:
and calculating the user experience index value through the user experience model according to the link portrait characteristic value.
Optionally, the processor 500 is further configured to:
under the condition that the target user experience index value is smaller than a second threshold value, adjusting a link and/or a code rate corresponding to a user experience model until the target user experience index value corresponding to the user experience model is larger than or equal to the second threshold value;
and when the target user experience index value is larger than or equal to a second threshold value, taking the link and the code rate corresponding to the user experience model as the link and the code rate of the audio and video playing corresponding to the second time period after adjustment.
Embodiments of the present invention provide a computer readable storage medium on which a program or instructions are stored, which when executed by a processor implement the steps in the audio-video scheduling method as described above.
The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
It is further noted that the terminals described in this specification include, but are not limited to, smart phones, tablets, etc., and that many of the functional components described are referred to as modules in order to more particularly emphasize their implementation independence.
In embodiments of the present invention, modules may be implemented in software for execution by various types of processors. An identified module of executable code may, for instance, comprise one or more physical or logical blocks of computer instructions which may, for instance, be constructed as an object, procedure, or function. Nevertheless, the executables of an identified module need not be physically located together, but may comprise disparate instructions stored in different bits which, when joined logically together, comprise the module and achieve the stated purpose for the module.
Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified within modules, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices, and may exist, at least partially, merely as electronic signals on a system or network.
When a module can be implemented by software, considering the level of existing hardware technology, a module implemented by software may build a corresponding hardware circuit to implement a corresponding function, without considering cost, and the hardware circuit may include a conventional Very Large Scale Integration (VLSI) circuit or a gate array and an existing semiconductor such as a logic chip, a transistor, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices or the like.
The exemplary embodiments described above are described with reference to the drawings, and many different forms and embodiments of the invention may be made without departing from the spirit and teaching of the invention, therefore, the invention is not to be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. In the drawings, the size and relative sizes of components may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Unless otherwise indicated, a range of values, when stated, includes the upper and lower limits of the range, and any subranges therebetween.
While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims (12)

1. An audio and video scheduling method, comprising:
acquiring a first user experience index value corresponding to the target moment;
under the condition that the first user experience index value is lower than a first threshold value, acquiring a first link portrait corresponding to a first preset time period before a target moment; the link image is used for indicating relevant information of the audio and video playing link;
inputting the first link portrait into a user experience model to obtain a target user experience index value corresponding to a second time period; the second time period comprises a second preset time period after the target time; the user experience model is used for representing the corresponding relation between the user video playing experience index value and the link portrait under the target audio and video playing code rate;
and adjusting the link and/or code rate of the audio and video playing corresponding to the second time period according to the target user experience index value and the user experience model.
2. The audio-video scheduling method of claim 1, wherein the user experience indicator comprises at least one of: video resolution, video frame rate, video stuck rate, audio stuck rate, second on rate, and packet loss rate.
3. The audio and video scheduling method according to claim 1, wherein the obtaining a first user experience index value corresponding to a target time includes:
acquiring a first link portrait corresponding to a target moment, and determining a link portrait characteristic value according to the first link portrait;
and determining the user experience index value according to the link portrait characteristic value.
4. The audio-video scheduling method of claim 3, wherein the obtaining the corresponding first link representation at the target time comprises:
receiving audio and video link data reported by a user at the target moment;
and analyzing the audio and video link data to generate the first link portrait.
5. The audio-video scheduling method of claim 4 wherein the audio-video link data comprises at least one of: the method comprises the following steps of timestamp, edge server, terminal machine type, operating system, network type, packet loss rate, video resolution, frame rate, sending code rate and receiving code rate;
the first link representation includes at least one of: time attribute, terminal attribute, network attribute, region attribute, edge server attribute, experience attribute;
the link portrayal feature values include at least one of: time elements, terminal elements, network elements, region elements, edge server elements, experience elements.
6. The audio-video scheduling method according to claim 1, wherein the user experience model is a model obtained by model training a first model:
the first model is:
Figure FDA0003662225280000021
wherein f (V | θ) represents a user experience index value; v is a link portrait characteristic value; mi is the ith link portrait experience index value corresponding to the link portrait characteristic value, δ i is the weight of the ith user experience index, and θ is the audio and video playing code rate; n is an integer greater than or equal to 1;
the link portrayal experience metric value comprises at least one of: video resolution value, video frame rate value, video cadence rate reciprocal value, audio cadence rate reciprocal value, second open rate value.
7. The audio-video scheduling method according to claim 6, wherein performing model training on the first model to obtain the user experience model comprises:
acquiring a historical link image set; the historical link portrait set comprises link portraits before the current moment;
performing model training on the first model through a logistic stewart regression algorithm and a gradient descent method according to the historical link image set to obtain the weight of each user experience index;
determining the experience model according to the first model and the weight of each user experience index.
8. The audio-video scheduling method according to claim 7, wherein the determining the user experience index value according to the link portrait characteristic value includes:
and calculating the user experience index value through the user experience model according to the link portrait characteristic value.
9. The audio/video scheduling method according to claim 1, wherein the adjusting the link and/or the code rate of the audio/video playing corresponding to the second time period according to the target user experience index value and the user experience model comprises:
under the condition that the target user experience index value is smaller than a second threshold value, adjusting a link and/or a code rate corresponding to a user experience model until the target user experience index value corresponding to the user experience model is larger than or equal to the second threshold value;
and when the target user experience index value is larger than or equal to a second threshold value, taking the link and the code rate corresponding to the user experience model as the link and the code rate of the audio and video playing corresponding to the second time period after adjustment.
10. An audio/video scheduling apparatus, comprising:
the first acquisition module is used for acquiring a first user experience index value corresponding to the target moment;
the second acquisition module is used for acquiring a first link portrait corresponding to a first preset time period before a target time under the condition that the first user experience index value is lower than a first threshold value; the link image is used for indicating related information of the audio and video playing link;
the processing module is used for inputting the first link portrait into a user experience model to obtain a target user experience index value corresponding to a second time period; the second time period comprises a second preset time period after the target time; the user experience model is used for representing the corresponding relation between the user video playing experience index value and the link portrait under the target audio and video playing code rate;
and the adjusting module is used for adjusting the link and/or code rate of the audio and video playing corresponding to the second time period according to the target user experience index value and the user experience model.
11. An electronic device, comprising: a transceiver, a processor, a memory, and a program or instructions stored on the memory and executable on the processor; characterized in that the processor, when executing the program or instructions, implements the audio-video scheduling method according to any of claims 1 to 9.
12. A computer readable storage medium having a program or instructions stored thereon, which when executed by a processor implement the steps in the audio-video scheduling method according to any one of claims 1-9.
CN202210580614.6A 2022-05-25 2022-05-25 Audio and video scheduling method and device, electronic equipment and computer readable storage medium Active CN115002513B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210580614.6A CN115002513B (en) 2022-05-25 2022-05-25 Audio and video scheduling method and device, electronic equipment and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210580614.6A CN115002513B (en) 2022-05-25 2022-05-25 Audio and video scheduling method and device, electronic equipment and computer readable storage medium

Publications (2)

Publication Number Publication Date
CN115002513A true CN115002513A (en) 2022-09-02
CN115002513B CN115002513B (en) 2023-10-20

Family

ID=83028232

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210580614.6A Active CN115002513B (en) 2022-05-25 2022-05-25 Audio and video scheduling method and device, electronic equipment and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN115002513B (en)

Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102802089A (en) * 2012-09-13 2012-11-28 浙江大学 Shifting video code rate regulation method based on experience qualitative forecast
WO2014078744A2 (en) * 2012-11-16 2014-05-22 Vid Scale, Inc. Systems and methods for implementing model-based qoe scheduling
WO2015188678A1 (en) * 2014-06-13 2015-12-17 珠海全志科技股份有限公司 Transmission control method and system for video stream of mobile device
US20160088322A1 (en) * 2014-09-22 2016-03-24 Arris Enterprises, Inc. Video Quality of Experience Based on Video Quality Estimation
CN105872594A (en) * 2016-03-30 2016-08-17 乐视控股(北京)有限公司 Real time bit rate adjusting method and device, and server device
WO2019056499A1 (en) * 2017-09-20 2019-03-28 平安科技(深圳)有限公司 Prediction model training method, data monitoring method, apparatuses, device and medium
CN109618229A (en) * 2018-12-21 2019-04-12 广州酷狗计算机科技有限公司 Association playback method, device, server and the storage medium of audio-video
US20190121417A1 (en) * 2018-12-20 2019-04-25 Intel Corporation Data rate adaptive link speed
CN110708578A (en) * 2019-10-22 2020-01-17 北方工业大学 Video code rate self-adaptive adjusting method
WO2020034082A1 (en) * 2018-08-14 2020-02-20 海能达通信股份有限公司 Slicing-based rtp stream transmission method, device, terminal and server
CN110996125A (en) * 2019-11-18 2020-04-10 腾讯科技(深圳)有限公司 Video stream generation method and device, electronic equipment and storage medium
WO2021026509A1 (en) * 2019-08-08 2021-02-11 Intel Corporation Link performance prediction and media streaming technologies
CN112543366A (en) * 2020-12-02 2021-03-23 北京五八信息技术有限公司 Video playing method and device
CN112788373A (en) * 2019-11-07 2021-05-11 中国电信股份有限公司 Scheduling decision method, device and computer readable storage medium for content distribution network
WO2022028304A1 (en) * 2020-08-07 2022-02-10 腾讯科技(深圳)有限公司 Multimedia data processing method and apparatus, device and readable storage medium
WO2022033452A1 (en) * 2020-08-10 2022-02-17 北京字节跳动网络技术有限公司 Volume recommendation method and apparatus, device and storage medium
CN114222169A (en) * 2021-12-14 2022-03-22 咪咕互动娱乐有限公司 Video stream transmission method and device and communication equipment
CN114449282A (en) * 2019-12-26 2022-05-06 腾讯科技(深圳)有限公司 Video transmission control method and device, equipment and storage medium

Patent Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102802089A (en) * 2012-09-13 2012-11-28 浙江大学 Shifting video code rate regulation method based on experience qualitative forecast
WO2014078744A2 (en) * 2012-11-16 2014-05-22 Vid Scale, Inc. Systems and methods for implementing model-based qoe scheduling
WO2015188678A1 (en) * 2014-06-13 2015-12-17 珠海全志科技股份有限公司 Transmission control method and system for video stream of mobile device
US20160088322A1 (en) * 2014-09-22 2016-03-24 Arris Enterprises, Inc. Video Quality of Experience Based on Video Quality Estimation
CN105872594A (en) * 2016-03-30 2016-08-17 乐视控股(北京)有限公司 Real time bit rate adjusting method and device, and server device
WO2019056499A1 (en) * 2017-09-20 2019-03-28 平安科技(深圳)有限公司 Prediction model training method, data monitoring method, apparatuses, device and medium
WO2020034082A1 (en) * 2018-08-14 2020-02-20 海能达通信股份有限公司 Slicing-based rtp stream transmission method, device, terminal and server
US20190121417A1 (en) * 2018-12-20 2019-04-25 Intel Corporation Data rate adaptive link speed
CN109618229A (en) * 2018-12-21 2019-04-12 广州酷狗计算机科技有限公司 Association playback method, device, server and the storage medium of audio-video
WO2021026509A1 (en) * 2019-08-08 2021-02-11 Intel Corporation Link performance prediction and media streaming technologies
CN110708578A (en) * 2019-10-22 2020-01-17 北方工业大学 Video code rate self-adaptive adjusting method
CN112788373A (en) * 2019-11-07 2021-05-11 中国电信股份有限公司 Scheduling decision method, device and computer readable storage medium for content distribution network
CN110996125A (en) * 2019-11-18 2020-04-10 腾讯科技(深圳)有限公司 Video stream generation method and device, electronic equipment and storage medium
CN114449282A (en) * 2019-12-26 2022-05-06 腾讯科技(深圳)有限公司 Video transmission control method and device, equipment and storage medium
WO2022028304A1 (en) * 2020-08-07 2022-02-10 腾讯科技(深圳)有限公司 Multimedia data processing method and apparatus, device and readable storage medium
WO2022033452A1 (en) * 2020-08-10 2022-02-17 北京字节跳动网络技术有限公司 Volume recommendation method and apparatus, device and storage medium
CN112543366A (en) * 2020-12-02 2021-03-23 北京五八信息技术有限公司 Video playing method and device
CN114222169A (en) * 2021-12-14 2022-03-22 咪咕互动娱乐有限公司 Video stream transmission method and device and communication equipment

Also Published As

Publication number Publication date
CN115002513B (en) 2023-10-20

Similar Documents

Publication Publication Date Title
US11070794B2 (en) Video quality assessment method and apparatus
US11228794B2 (en) Method, computer device, and storage medium for media code rate self-adaptation
CN111628847B (en) Data transmission method and device
CN104394486B (en) A kind of video stream processing method and device based on cloud desktop
US20090234919A1 (en) Method of Transmitting Data in a Communication System
CN105430532A (en) Control method and system for adaptive adjustment of video data transmission
CN111225209B (en) Video data plug flow method, device, terminal and storage medium
US11924255B2 (en) Data transmission method and apparatus, server, storage medium, and program product
CN111629210A (en) Data processing method and device and electronic equipment
CN109040830B (en) Live broadcast pause prediction method, switching method and device
CN115037416B (en) Data forward error correction processing method, device, electronic equipment and storage medium
CN112929712A (en) Video code rate adjusting method and device
CN111669617A (en) Live video stream transmission method based on intelligent edge
US10044833B2 (en) Data processing method and apparatus used for terminal application
JP2022546108A (en) Information processing method, device, equipment and computer-readable storage medium
CN115002513A (en) Audio and video scheduling method and device, electronic equipment and computer readable storage medium
CN114629797B (en) Bandwidth prediction method, model generation method and device
US20230011483A1 (en) Method and device for adjusting bit rate during live streaming
CN114390006B (en) Data packet transmission method, device, equipment and storage medium
US11936698B2 (en) Systems and methods for adaptive video conferencing
CN112866767B (en) Screen projection control method and device, electronic equipment and storage medium
CN110809127B (en) Video call method and device based on deep simulation learning
CN115190359A (en) Automatic conversion method and system for video playing format
US8791981B2 (en) Bit rate control apparatus and method thereof
Singh et al. Modeling and Optimization of Video Transmission in Data Compression & Source Coding

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant