CN116939318A - Audio and video transmission system based on perceptual quality optimization - Google Patents
Audio and video transmission system based on perceptual quality optimization Download PDFInfo
- Publication number
- CN116939318A CN116939318A CN202311199771.3A CN202311199771A CN116939318A CN 116939318 A CN116939318 A CN 116939318A CN 202311199771 A CN202311199771 A CN 202311199771A CN 116939318 A CN116939318 A CN 116939318A
- Authority
- CN
- China
- Prior art keywords
- data
- transmission
- audio
- steps
- mode
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 289
- 238000005457 optimization Methods 0.000 title claims abstract description 18
- 238000011156 evaluation Methods 0.000 claims abstract description 120
- 238000000034 method Methods 0.000 claims description 106
- 239000000872 buffer Substances 0.000 claims description 55
- 238000012545 processing Methods 0.000 claims description 40
- 230000003044 adaptive effect Effects 0.000 claims description 24
- 238000011084 recovery Methods 0.000 claims description 24
- 239000011159 matrix material Substances 0.000 claims description 18
- 238000013441 quality evaluation Methods 0.000 claims description 18
- 238000012937 correction Methods 0.000 claims description 16
- 230000009123 feedback regulation Effects 0.000 claims description 12
- 239000013598 vector Substances 0.000 claims description 12
- 238000005311 autocorrelation function Methods 0.000 claims description 7
- 238000001303 quality assessment method Methods 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 3
- 230000003362 replicative effect Effects 0.000 claims 1
- 230000009471 action Effects 0.000 description 16
- 230000000694 effects Effects 0.000 description 8
- 230000007246 mechanism Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 4
- 230000000977 initiatory effect Effects 0.000 description 3
- 230000005236 sound signal Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 230000008713 feedback mechanism Effects 0.000 description 2
- 238000009472 formulation Methods 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 238000012913 prioritisation Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000010420 art technique Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention aims to provide an audio and video transmission system based on perceptual quality optimization, which relates to the technical field of data transmission, and comprises the following components: an analyzer, a policy selector, a transmitter, a receiver, an evaluator, and a feedback adjuster; the analyzer is configured to analyze the data to be transmitted to judge the data type of the data to be transmitted, so as to obtain pre-transmitted data; the strategy selector is configured to formulate a data transmission initial strategy according to the data type of the data to be transmitted; the transmitter transmits the pre-transmitted data and the data transmission initial strategy to the receiver; the receiver is configured to receive pre-sent data and a data transmission initial strategy; the evaluator is configured to evaluate the quality of the received pre-sent data to obtain an evaluation result. The invention formulates feedback adjustment transmission strategy through pre-sent data, realizes dynamic self-adaptive transmission, and improves transmission quality and user experience.
Description
Technical Field
The invention relates to the technical field of data transmission, in particular to an audio and video transmission system based on perceptual quality optimization.
Background
In the present digital age, audio and video transmission technology plays a vital role in various fields, such as video conferencing, live broadcasting in real time, network games, multimedia applications and the like. With the popularization of high-definition, 4K and even 8K videos and the diversification of audios, the requirements on the performance and the quality of an audio and video transmission system are also increasing. However, in practical applications, a series of challenges such as bandwidth limitation, packet loss, delay, quality degradation, etc. are faced in audio/video transmission.
The traditional audio and video transmission system generally adopts a fixed strategy, and cannot be adaptively adjusted according to specific scenes and data types, so that the transmission effect is different under different network environments and data quality, and the stability and good performance of user experience cannot be ensured. Some presently disclosed techniques attempt to cope with different types of data by using a single transmission policy or to transmit using a preset policy. However, these methods have limitations, and cannot meet the requirements of different data types and network environments, resulting in still limited transmission efficiency and quality.
For example, some prior art techniques treat audio and video data as a unified data type, with the same transmission strategy being employed for transmission. This approach does not fully exploit the characteristics of the audio and video data, resulting in possible problems in both audio communication and video transmission. In addition, the transmission policy in the prior art is often fixed, and cannot be flexibly adjusted according to the network condition and the data characteristic. When network bandwidth is limited or data packets are lost, transmission quality and user experience will be severely affected.
In addition, there is no effective evaluation mechanism in some prior art to monitor the quality of the transmitted data. In the transmission process, a real-time quality evaluation and feedback mechanism is lacked, so that the problem in the transmission cannot be found and solved in time. The lack of accurate quality assessment cannot determine proper transmission strategies and adjustment methods, so that the performance and quality of an audio and video transmission system cannot be effectively improved.
In addition, some prior art may not adequately consider the diversity and specificity of multimedia data. For example, in video data, the importance of different frames varies, I frames contain all information, P/B frames depend on the previous I/P frames, and the frequency of sound fluctuations in audio data also varies from content to content. However, some techniques may ignore these differences, resulting in that appropriate processing measures are not taken for the different data types during transmission, and thus stability and quality of transmission cannot be guaranteed to the maximum extent.
Disclosure of Invention
In view of the above, the main purpose of the present invention is to provide an intelligent audio/video transmission system based on perceptual quality optimization, and the present invention makes feedback adjustment transmission policy by pre-sending data, so as to realize dynamic adaptive transmission and improve transmission quality and user experience.
In order to achieve the above purpose, the technical scheme of the invention is realized as follows: an audio-video transmission system based on perceptual quality optimization, the system comprising: an analyzer, a policy selector, a transmitter, a receiver, an evaluator, and a feedback adjuster; the analyzer is configured to obtain data to be transmitted, analyze the data to be transmitted, determine a data type of the data to be transmitted, and copy the data to be transmittedFrame data to obtain pre-sent data; the data types include: audio data and video data; the strategy selector is configured to formulate a data transmission initial strategy according to the data type of the data to be transmitted and a preset initial strategy; the transmitter is configured to transmit pre-transmitted data and a data transmission initial strategy to the receiver according to the formulated data transmission initial strategy; the receiver is configured to receive pre-sent data and a data transmission initial strategy; the evaluator is configured to evaluate the quality of the received pre-sent data to obtain an evaluation result; the feedback adjuster is configured to perform feedback adjustment based on the initial data transmission policy according to the evaluation result, formulate a feedback adjustment transmission policy, and send the feedback adjustment transmission policy to the policy selector to replace the initial data transmission policy in the policy selector And sending a control command to the analyzer to prompt the analyzer to start sending data to be sent; the analyzer transmits the data to be transmitted to the transmitter through the strategy selector; and the transmitter transmits the data to be transmitted to the receiver according to the formulated feedback adjustment transmission strategy.
Further, the data transmission policy and the feedback adjustment transmission policy both include: transmission protocol, transmission rate, packet loss processing mode, buffer processing mode, error recovery mode, bandwidth allocation mode, multicast mode and frame priority scheduling mode; the transmission protocol includes: TCP or UDP; the transmission rate includes: fixed rate or adaptive rate; the packet loss processing mode comprises the following steps: retransmission or omission; the buffer processing mode comprises the following steps: short buffer or long buffer; the error recovery method comprises the following steps: forward error correction or adaptive retransmission; the bandwidth allocation method comprises the following steps: average or on demand; the multicast mode comprises the following steps: unicast or multicast; the frame priority scheduling method comprises the following steps: i-frame priority or P/B-frame priority.
Further, when the data type of the data to be transmitted is video data, the transmitter firstly separates an audio part and a video part in the pre-transmitted data, and establishes different data transmission initial strategies for the audio part and the video part; the data transmission initial strategy of the audio part comprises the following steps: the transmission protocol is TCP; the transmission rate is a fixed rate; the packet loss processing mode is neglected; the buffer treatment mode is long buffer; the error recovery mode is forward error correction; the bandwidth allocation mode is allocation according to the need; the multicast mode is multicast; the frame priority scheduling mode is P/B frame priority; the data transmission initial strategy of the video part comprises the following steps: the transmission protocol is UDP; the transmission rate is an adaptive rate; the packet loss processing mode is retransmission; the buffer treatment mode is long buffer; the error recovery mode is self-adaptive retransmission; the bandwidth allocation mode is allocation according to the need; the multicast mode is unicast; the frame priority scheduling mode is I frame priority; when the data type of the data to be transmitted is audio data, the data transmission initial strategy of the pre-transmitted data comprises the following steps: the transmission protocol is TCP; the transmission rate is a fixed rate; the packet loss processing mode is neglected; the buffer processing mode is short buffer; the error recovery mode is forward error correction; the bandwidth allocation mode is average allocation; the multicast mode is multicast; the frame priority scheduling mode is P/B frame priority.
Further, in copying data to be transmittedWhen frame data, ->The following constraints are satisfied:；
and, in addition, the method comprises the steps of,；
wherein, is the total number of frames of data to be transmitted.
Further, the evaluator evaluates the quality of the received pre-sent data, and the method for obtaining the evaluation result comprises the following steps: when the data to be transmitted obtain the data type as audio data, the evaluator evaluates the quality of the pre-transmitted data to obtain an evaluation result; when the data to be transmitted is video data, the evaluator firstly carries out quality evaluation on the audio data separated from the pre-transmitted data to obtain an audio evaluation result, then carries out quality evaluation on the video data separated from the pre-transmitted data to obtain a video evaluation result, and calculates to obtain an evaluation result based on the audio evaluation result and the video evaluation result.
Further, the evaluator performs quality evaluation on the pre-sent data to obtain an evaluation result or the evaluator first performs quality evaluation on the audio data separated from the pre-sent data, and the method for obtaining the audio evaluation result includes: dividing the audio data into 8 equal-length segments, and calculating the number of local extreme points for each segmentLength of wave form curve />And mean fluctuation intensity>The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Taking the subscript as an integer, and taking the value range of 1 to 8; setting three first judgment thresholds as respectivelyThe first evaluation judgment was performed using the following rule, obtaining 8 first evaluation values:
if it isAnd->Then->；
If it isOr->Then->；
Wherein, is the first evaluation value; based on the first evaluation value, an evaluation result or an audio evaluation result is obtained as follows:。
further, the number of local extremum points is calculated using the following formula:
wherein the function isAt->Returning to 1 when the local extremum is found, otherwise returning to 0; />For dividing audio data into 8 equal-length segments, the +.>First part of the individual section>A sample number; />For the length of each segment; the length of the waveform curve;
the waveform curve length is calculated using the following formula:
the average fluctuation intensity is calculated using the following formula:
；
wherein, is->First part of the individual section>Samples.
Further, the pre-sent dataThe method for obtaining the video evaluation result by performing quality evaluation on the separated video data comprises the following steps: dividing video data into 8 parts, randomly selecting a frame of image for each part, and respectively calculating image contrast Image clarity->And color consistency->The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Taking the subscript as an integer, and taking the value range of 1 to 8; construction of a feature vector +.>Then calculate its time sequence autocorrelation function +.>The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>The value is an integer, and the value range is 1 to 8; setting two second judging thresholds which are respectively +.>And->The method comprises the steps of carrying out a first treatment on the surface of the Performing second evaluation judgment by using the following formula to obtain 8 second evaluation values:
；
wherein, is a second evaluation value; based on the second evaluation value, obtaining a video evaluation result as follows: />。
Further, the image contrastCalculated using the following formula:
；
wherein the method comprises the steps ofIs->Gray intensity values of pixels of an image of a frame selected partly at random,/for each pixel of the frame>Is a small constant to prevent denominator from being zero;
the image definitionCalculated using the following formula:
；
wherein, the abscissa variable of the pixels of the image for a randomly selected frame; />A vertical variable for a pixel of an image of a randomly selected frame;
the color consistency is calculated using the following formula:
；
wherein the method comprises the steps ofIs->Color channel of an image of a frame selected partly at random +.>Average value of (2); />Is->Color channel of an image of a frame selected partly at random +. >Average value of (2).
Calculating a time series autocorrelation function using the formula：
；
Wherein, is the average of the feature vectors.
Further, the method for recalculating the evaluation result based on the audio evaluation result and the video evaluation result includes: the evaluation result was calculated using the following formula:
；
wherein, & is AND operation;
the feedback regulator carries out feedback regulation on the basis of the initial strategy of data transmission according to the evaluation result, and the method for formulating the transmission strategy of feedback regulation comprises the following steps: setting a matrix of initial strategies of data transmission as follows:
the method comprises the steps of carrying out a first treatment on the surface of the Wherein, when the transport protocol is TCP, </i >>The method comprises the steps of carrying out a first treatment on the surface of the When the transport protocol is UDP, the +.>The method comprises the steps of carrying out a first treatment on the surface of the When the transmission rate is a fixed rate +.>The method comprises the steps of carrying out a first treatment on the surface of the When the transmission rate is an adaptive rate,the method comprises the steps of carrying out a first treatment on the surface of the When the packet loss processing mode is retransmission, the method comprises the steps of +.>When the packet loss processing mode is neglect, the method is carried out by +.>The method comprises the steps of carrying out a first treatment on the surface of the When the buffer processing mode is short buffer, the +.>When the buffer processing mode is long buffer, the +.>The method comprises the steps of carrying out a first treatment on the surface of the When the error recovery mode is forward error correction, < >>When the error recovery mode is adaptive retransmission, < +.>The method comprises the steps of carrying out a first treatment on the surface of the When the bandwidth allocation mode is average allocation, +.>When the bandwidth allocation is on-demand allocation, the method comprises the steps of>The method comprises the steps of carrying out a first treatment on the surface of the When the multicast mode is unicast, the +. >When the multicast mode is multicast, the +.>The method comprises the steps of carrying out a first treatment on the surface of the When the frame priority scheduling mode is I-frame priority, the method comprises the step of +.>When the frame priority scheduling mode is the P/B frame priority, the frame priority scheduling mode is +.>The method comprises the steps of carrying out a first treatment on the surface of the And performing matrix OR operation on the matrix of the initial data transmission strategy and the evaluation result to obtain a matrix of the feedback adjustment transmission strategy, and further obtaining the feedback adjustment transmission strategy.
By adopting the technical scheme, the invention has the following beneficial effects: firstly, the invention adopts an analyzer to classify and separate the data to be transmitted, and respectively formulates different data transmission initial strategies according to different characteristics of audio and video data. The classification mechanism enables the audio data and the video data to respectively adopt the optimal transmission strategy according to the characteristics of the audio data and the video data, and the respective advantages are fully exerted. By formulating different transmission strategies for the audio part and the video part, the transmission delay can be effectively reduced, the transmission quality can be improved, and the conflict of audio and video data caused by sharing the same transmission strategy can be avoided, so that the overall efficiency and quality of audio and video transmission can be improved.
The invention introduces an evaluator for quality evaluation of the received pre-sent data, including evaluation of the audio data and the video data. By evaluating the quality of the received data in real time, possible problems in the transmission process, such as packet loss, delay, quality degradation, etc., are found out in time. The existence of the evaluator enables the system to adjust the transmission strategy in time according to the actual transmission condition, and the user requirement is met with the optimal transmission quality. The evaluation mechanism effectively ensures the real-time performance and stability of audio and video transmission, and improves the transmission effect and user experience.
The feedback regulator in the invention dynamically adjusts feedback based on the initial strategy of data transmission according to the real-time evaluation result of the evaluator. The transmission quality is monitored in real time, and the transmission strategy is adjusted in time, so that the method can be better adapted to the change of network environment and the difference of data types, and the transmission efficiency and quality are improved. The feedback regulator enables the system to dynamically optimize the transmission strategy according to the actual situation, so that delay and data loss in transmission are reduced to the greatest extent, and the transmission stability and reliability are improved.
The invention adopts a multi-index evaluation mechanism, which comprises indexes such as local extreme point number, waveform curve length, average fluctuation intensity, image contrast, image definition, color consistency and the like. The indexes fully consider the diversity and the specificity of the audio and video data, and can evaluate the quality and the performance of the data more accurately. The multi-index evaluation mechanism enables the system to comprehensively consider factors in different aspects, so that the transmission effect is evaluated more comprehensively, and the accuracy and the effectiveness of the evaluation are improved.
The adaptive transmission framework in the invention can realize real-time data transmission strategy adjustment instead of adopting a fixed transmission strategy. The self-adaptive mechanism enables the system to flexibly cope with transmission requirements under different network environments and data quality, dynamically optimizes transmission and improves transmission efficiency and quality. The self-adaptive transmission framework enables the audio and video transmission system to automatically select the optimal transmission strategy under different conditions, thereby meeting the requirements of users on transmission quality and performance to the greatest extent.
In summary, the feedback adjustment transmission strategy is formulated by pre-sending data, so that dynamic self-adaptive transmission is realized, the audio and video transmission quality is optimized, and the user experience is improved. The pre-sent data is used as a part of the data to be sent, and is sent before transmission, so that a basis is provided for evaluation and adjustment in the transmission process. Through real-time evaluation and feedback mechanisms, the system can dynamically optimize transmission strategies, adapt to different network environments and data characteristics, and realize optimal audio and video transmission effects.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the accompanying drawings in which:
fig. 1 is a schematic system structure diagram of an intelligent audio/video transmission system based on perceptual quality optimization according to an embodiment of the present invention.
Detailed Description
The method of the present invention will be described in further detail with reference to the accompanying drawings.
Example 1: an audio-video transmission system based on perceptual quality optimization, the system comprising: an analyzer, a policy selector, a transmitter, a receiver, an evaluator, and a feedback adjuster; the analyzer is configured to obtain data to be transmitted, analyze the data to be transmitted, determine a data type of the data to be transmitted, and copy the data to be transmitted Frame data to obtain pre-sent data; the data types include: audio data and video data; the strategy selector is configured to formulate a data transmission initial strategy according to the data type of the data to be transmitted and a preset initial strategy; the transmitter is configured to transmit pre-transmitted data and a data transmission initial strategy to the receiver according to the formulated data transmission initial strategy; the receiver is configured to receive pre-sent data and a data transmission initial strategy; the evaluator is configured to evaluate the quality of the received pre-sent data to obtain an evaluation result; the feedback regulator is configured to perform feedback regulation based on the initial data transmission strategy according to the evaluation result, formulate a feedback regulation transmission strategy, send the feedback regulation transmission strategy to the strategy selector to replace the initial data transmission strategy in the strategy selector, and send a control command to the analyzer to prompt the analyzer to start sending data to be sent; the analyzer transmits the data to be transmitted to the transmitter through the strategy selector; the transmitter adjusts the data to be transmitted according to the formulated feedback The transmission policy is sent to the receiver.
Specifically, the system aims at optimizing audio and video quality perceived by a user, and does not simply pursue the bandwidth of transmission or the data transmission speed. By performing quality evaluation on the received pre-sent data and performing feedback adjustment on a transmission strategy according to an evaluation result, the system can preferentially ensure the quality of audio and video data under a limited bandwidth and provide better user experience.
Example 2: on the basis of the above embodiment, the data transmission policy and the feedback adjustment transmission policy each include: transmission protocol, transmission rate, packet loss processing mode, buffer processing mode, error recovery mode, bandwidth allocation mode, multicast mode and frame priority scheduling mode; the transmission protocol includes: TCP or UDP; the transmission rate includes: fixed rate or adaptive rate; the packet loss processing mode comprises the following steps: retransmission or omission; the buffer processing mode comprises the following steps: short buffer or long buffer; the error recovery method comprises the following steps: forward error correction or adaptive retransmission; the bandwidth allocation method comprises the following steps: average or on demand; the multicast mode comprises the following steps: unicast or multicast; the frame priority scheduling method comprises the following steps: i-frame priority or P/B-frame priority.
Specifically, the transmission protocol: for defining the transmission rules and communication modes of data in the network. Common transport protocols include TCP (transmission control protocol) and UDP (user datagram protocol). TCP: TCP provides reliable data transmission that ensures reliability of data through acknowledgement and retransmission mechanisms. However, TCP introduces a large transmission delay and additional data overhead, and is suitable for a scenario with high requirements for data reliability. UDP: UDP provides connectionless transport services, and has no acknowledgement and retransmission mechanisms, so that the transport efficiency is high. However, UDP does not guarantee the reliability of data, and is suitable for audio and video transmission scenarios with high requirements for real-time performance.
Transmission rate: refers to the rate at which data is transmitted in the network and may be a fixed rate or an adaptive rate. Fixed rate: the data is transmitted at a fixed rate. The method is suitable for the condition that the network bandwidth is stable and predictable. Adaptive rate: the data transmission rate is dynamically adjusted according to the network conditions to adapt to the fluctuation of the network bandwidth. The adaptive rate can better cope with network jitter and congestion, and provides a more stable transmission effect.
The packet loss processing mode comprises the following steps: handling the loss situation that may occur during the transmission of the data packets. Retransmission: and for the lost data packet, the transmitting end can retransmit so as to ensure that the receiving end receives complete data. This way, the reliability of the data can be guaranteed, but the transmission delay can be increased. Neglecting: for lost packets, no retransmission is directly ignored. The method is suitable for audio and video transmission with high real-time requirements, but can cause loss of partial data and reduce transmission quality.
The buffer treatment mode comprises the following steps: the data is buffered at the receiving end to balance the transmission rate difference between the transmitting end and the receiving end. Short buffer: with smaller buffers, data can be displayed faster with lower transmission delays, but under-buffering conditions may occur, resulting in data loss or stuck. Long buffer: the use of a larger buffer area can better handle the situation of larger transmission delay, ensure the integrity and stability of data, but increase the buffer time of the receiving end.
Error recovery mode: and errors possibly occurring in data transmission are processed, and the integrity and the accuracy of the data are ensured. Forward error correction: the transmitting end performs redundancy coding on the data, and the receiving end can correct a small amount of errors through the redundancy information, so that the reliability of the data is improved. Adaptive retransmission: when an error occurs, the transmitting end retransmits according to the feedback information of the receiving end, so that the correct transmission of the data is ensured.
Bandwidth allocation mode: and determining a bandwidth allocation mode of the data stream aiming at the shared bandwidth of the plurality of data streams in the network. Average distribution: the available bandwidth is distributed equally to multiple data streams, ensuring fairness, but may result in some data streams failing to meet real-time transmission requirements. And (5) distributing according to the requirement: and dynamically distributing bandwidth according to the real-time transmission requirement and importance of the data stream, and preferentially guaranteeing the transmission requirement of the important data stream.
Multicast mode: the manner in which data is transmitted in the network. Unicast: the data is transmitted in a point-to-point mode, and the method is suitable for one-to-one data transmission scenes. Multicast: the data is transmitted in a one-to-many mode, can be simultaneously transmitted to a plurality of receiving ends, and is suitable for one-to-many data distribution scenes.
Frame priority scheduling mode: and carrying out priority scheduling on frames of different types in the audio and video data so as to ensure the transmission priority of key frames (I frames). I-frame priority: the key frames have higher transmission priority, so that the receiving end can decode in time, and the continuity of video transmission is ensured. P/B frame priority: the non-key frames (P/B frames) have higher transmission priority, are suitable for the condition of limited network bandwidth, and can reduce transmission overhead.
Through the combination and adjustment of the strategies, the audio and video transmission system can optimize the transmission effect according to different network environments and user requirements, and better audio and video transmission experience is provided. The selection between different combinations and adjustment modes can be determined according to specific scenes and requirements, so that the system has flexibility and adaptability in different application scenes
Example 3: on the basis of the above embodiment, when the data type of the data to be transmitted is video data, the transmitter first separates an audio part and a video part in the pre-transmitted data, and makes different data transmission initial strategies for the audio part and the video part; the data transmission initial strategy of the audio part comprises the following steps: the transmission protocol is TCP; the transmission rate is a fixed rate; the packet loss processing mode is neglected; the buffer treatment mode is long buffer; the error recovery mode is forward error correction; the bandwidth allocation mode is allocation according to the need; the multicast mode is multicast; the frame priority scheduling mode is P/B frame priority; the data transmission initial strategy of the video part comprises the following steps: the transmission protocol is UDP; the transmission rate is an adaptive rate; the packet loss processing mode is retransmission; the buffer treatment mode is long buffer; the error recovery mode is self-adaptive retransmission; the bandwidth allocation mode is allocation according to the need; the multicast mode is unicast; the frame priority scheduling mode is I frame priority; when the data type of the data to be transmitted is audio data, the data transmission initial strategy of the pre-transmitted data comprises the following steps: the transmission protocol is TCP; the transmission rate is a fixed rate; the packet loss processing mode is neglected; the buffer processing mode is short buffer; the error recovery mode is forward error correction; the bandwidth allocation mode is average allocation; the multicast mode is multicast; the frame priority scheduling mode is P/B frame priority.
Specifically, the audio data characteristics and transmission requirements: the real-time requirement is higher: audio data is generally used in scenes such as voice communication, music playing, and the like, and real-time transmission is required to ensure that a receiving end can play audio content in real time, so that the problem of delay or asynchronism during communication is avoided. The amount of data is relatively small: the amount of data of the audio data is small compared to the video data, and the load of transmission is relatively light. The loss tolerance is higher: small amounts of audio data loss may not be apparent in the human ear, so for audio data, real-time tends to be more important than reliability, tolerating partial data loss is acceptable.
Video data characteristics and transmission requirements: the real-time requirement is higher: video data is generally used in video conference, real-time monitoring and other scenes, and real-time transmission is required to ensure that a receiving end can display video content in real time, so that the problems of picture delay or asynchronism are avoided. The data size is large: the data volume of video data is relatively large, contains a large amount of image information, and the load of transmission is heavy. High data integrity requirements: each frame of video data has a dependency relationship, and a key frame (I frame) is the basis of other frames (P/B frames), so that for video data, the integrity of the data is very important, and it is required to ensure that each frame can be correctly transmitted.
Because audio data and video data differ significantly in application scenarios and transmission requirements. Because the real-time performance of the audio data is required to be high and the data volume is relatively small, strategies such as packet loss omission, small buffer use, forward error correction and the like can be adopted to ensure the real-time performance preferentially during transmission, and the listening feeling is not influenced obviously for a small amount of lost audio data.
In contrast, video data is also important in real-time, but the data integrity requirement is higher, and frame loss cannot be tolerated because of the dependency between frames. In addition, the amount of video data is large, so that a more efficient transmission strategy is needed, for example, UDP protocol is used to reduce transmission delay, adaptive rate is adopted to dynamically adjust transmission rate, and retransmission is used to ensure complete transmission of data.
Data transmission initiation strategy for audio part:
transmission protocol: TCP (Transmission control protocol)
Principle and action: by using TCP as a transmission protocol, reliable transmission of audio data can be ensured, loss of audio data packets is avoided, and better audio quality and stability are provided.
Transmission rate: fixed rate
Principle and action: the fixed rate transmission is adopted, so that the audio data can be ensured to be sent according to the fixed rate, the stable transmission speed is maintained, and the method is suitable for the transmission of real-time audio data.
The packet loss processing mode comprises the following steps: neglecting
Principle and action: for audio data, real-time performance is important, and the influence of a small amount of data loss on audio quality is relatively small, so that a neglect mode is adopted to avoid delay caused by retransmission.
The buffer treatment mode comprises the following steps: long buffer
Principle and action: the long buffer is adopted to balance the transmission rate difference between the sending end and the receiving end, ensure that the receiving end can stably receive the audio data, and reduce the audio jamming phenomenon.
Error recovery mode: forward error correction
Principle and action: the forward error correction can correct partial errors through redundant information, so that the reliability of audio data transmission is improved, and the errors and losses of audio data are reduced.
Bandwidth allocation mode: on demand dispensing
Principle and action: bandwidth is allocated according to the requirement, bandwidth resources can be dynamically allocated according to the transmission requirement and importance of the real-time audio data, and preferential transmission of the audio data is ensured.
Multicast mode: multicast
Principle and action: the multicast mode is adopted to transmit the audio data to a plurality of receiving ends at one time, thereby reducing network transmission overhead and improving transmission efficiency.
Frame priority scheduling mode: P/B frame prioritization
Principle and action: for audio data, the concept of key frames is not applicable, so that the transmission overhead can be reduced by preferentially transmitting the P/B frames, and the transmission efficiency of the audio data can be improved.
Data transmission initiation strategy for video part:
transmission protocol: UDP (user Datagram protocol)
Principle and action: the UDP is used as a transmission protocol, so that the transmission delay can be reduced, and the method is suitable for video data transmission with higher real-time requirements.
Transmission rate: adaptive rate
Principle and action: the self-adaptive rate can dynamically adjust the transmission rate according to the network condition, and ensures continuous and stable transmission of video data.
The packet loss processing mode comprises the following steps: retransmission
Principle and action: for video data, it is very important to ensure the integrity of the data, so that a retransmission mode is adopted to ensure the accurate transmission of video data packets.
The buffer treatment mode comprises the following steps: long buffer
Principle and action: the long buffer can process the condition of larger transmission delay, ensure that the receiving end can stably receive video data, and improve the fluency of video playing.
Error recovery mode: adaptive retransmission
Principle and action: the self-adaptive retransmission can dynamically adjust the retransmission strategy according to the error condition and the network condition, and the reliability and the efficiency of video data transmission are improved.
Bandwidth allocation mode: on demand dispensing
Principle and action: bandwidth is allocated according to the requirement, bandwidth resources are dynamically allocated according to the real-time transmission requirement and importance of video data, and priority transmission of the video data is ensured.
Multicast mode: unicast of
Principle and action: video data typically needs to be transmitted separately for each receiving end, and unicast is used to ensure that each receiving end can obtain complete video data.
Frame priority scheduling mode: i-frame prioritization
Principle and action: for video data, an I-frame is a key frame, and the decoding of other frames (P/B frames) depends on the I-frame. Therefore, the priority transmission of the I frames can ensure the continuity and stability of video transmission.
Example 4: on the basis of the above embodiment, in copying data to be transmittedWhen frame data, ->The following constraints are satisfied:
；
and, in addition, the method comprises the steps of,；
wherein, is the total number of frames of data to be transmitted.
Specifically, this constraint is to limit the number of frames of the pre-sent data copy. The constraint requires that the number of frames of the pre-sent data copy be between 10% and 50% of the total number of frames of the data to be sent. The constraint condition can ensure that the pre-sent data contains enough information so as to decode and restore the audio and video data at the receiving end, and meanwhile, excessive transmission overhead caused by copying excessive data is avoided.
Duplicating excessive frame data can result in an increased amount of data being transmitted, increasing overhead of network transmissions. The percentage of the number of the copying frames is limited in a reasonable range through the constraint condition, so that the transmission overhead can be reduced, and the transmission efficiency is improved.
Example 5: based on the above embodiment, the method for evaluating the quality of the received pre-sent data by the evaluator to obtain the evaluation result includes: when the data to be transmitted obtain the data type as audio data, the evaluator evaluates the quality of the pre-transmitted data to obtain an evaluation result; when the data to be transmitted is video data, the evaluator firstly carries out quality evaluation on the audio data separated from the pre-transmitted data to obtain an audio evaluation result, then carries out quality evaluation on the video data separated from the pre-transmitted data to obtain a video evaluation result, and calculates to obtain an evaluation result based on the audio evaluation result and the video evaluation result.
In particular, the audio data and the video data are separated and subjected to separate quality evaluations, because the audio data and the video data have different characteristics and requirements in the transmission process, and different quality evaluation methods are required to ensure the transmission quality of the audio data and the video data. Audio data: the audio data is usually a continuous sound signal, which has high real-time requirements, and the continuity and stability of the audio data need to be maintained so as to avoid the problems of delay or asynchronization during conversation. Quality assessment of audio data typically involves distortion, noise, sound clarity, etc. of the audio signal.
Video data: the video data contains continuous image frames, and has high real-time requirement, and the continuity and fluency of the video data need to be maintained so as to avoid the phenomenon of picture blocking or discontinuity. Quality assessment of video data generally involves aspects of sharpness, distortion, fluency of motion, etc. of the image.
Example 6: on the basis of the above embodiment, the evaluator performs quality evaluation on the pre-sent data to obtain an evaluation result or the evaluator first performs quality evaluation on the audio data separated from the pre-sent data, and the method for obtaining the audio evaluation result includes: dividing the audio data into 8 equal-length segments, and calculating the number of local extreme points for each segmentWaveform curve length->And mean fluctuation intensity>The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Taking the subscript as an integer, and taking the value range of 1 to 8; setting three first judgment thresholds which are respectively +.>The first evaluation judgment was performed using the following rule, obtaining 8 first evaluation values:
if it isAnd->Then->；
If it isOr->Then->；
Wherein, is the first evaluation value; based on the first evaluation value, an evaluation result or an audio evaluation result is obtained as follows:。
specifically, the number of local extreme points : in audio data, a waveform typically contains a series of extreme points, i.e., local maxima or minima. The number of local extrema reflects the fluctuations and variations of the audio waveform. Application: when the local pole of the audio waveformWhen the number of value points is large, it indicates that the audio data has more fluctuation in this sub-segment, and more audio changes may exist, corresponding to a complex audio scene or richer sound content. Waveform curve length->: the waveform curve length is the sum of the lengths of the continuous fold line segments that connect over the audio waveform. The waveform curve length reflects the smoothness and stability of the audio waveform. Application: when the curve length of the audio waveform is longer, indicating that the audio data is more intense in this sub-segment, noise or other disturbances may be present, corresponding to a less stable audio signal. Mean fluctuation intensity->: the average fluctuation intensity measures the amplitude variation of the audio waveform in a subsection, i.e. the amplitude fluctuation degree of the waveform. Application: when the average fluctuation intensity of the audio waveform is large, it indicates that the audio data is large in fluctuation in this sub-segment, the amplitude is severely changed, and noise, distortion or other quality problems may exist. Judgment threshold- >: three judging thresholds are set for judging the number of local extreme points, the length of the waveform curve and the average fluctuation intensity of the audio data. The setting of the threshold value can be adjusted according to the actual scene and the data characteristics so as to adapt to different audio quality requirements. The setting of the judgment threshold is critical to the accuracy of the evaluation result, and the reasonable threshold can accurately judge the quality of the audio data and provide effective feedback information. The evaluator can obtain the first evaluation value of each sub-segment by dividing the audio data into sub-segments and calculating local features, and judging according to the set judgment threshold>Thereby forming an overall evaluation result->. The evaluation result may provide information about the quality of the audio data to optimize the transmission strategy of the audio data, providing a better audio transmission experience. The evaluation method fully utilizes the local characteristics of the audio waveform, comprehensively evaluates the quality of the audio data and provides an important reference basis for an audio and video transmission system. />
Example 7: on the basis of the above embodiment, the number of local extreme points is calculated using the following formula:
；
wherein the function isAt- >Returning to 1 when the local extremum is found, otherwise returning to 0; />For dividing audio data into 8 equal-length segments, the +.>First part of the individual section>A sample number; />For the length of each segment; in this formula, the function +.>To determine whether each sample is a local extremum point. Function->Return 1 indicates sample +.>Is a local extreme point, otherwise, returns to 0. Specific judgmentThe rules are as follows: if->Greater than two samples adjacent thereto +.>Andthen->Is a local maximum and returns to 1. If->Less than two samples adjacent thereto +.>Andthen->Is a local minimum and returns to 1. Otherwise->Not local extreme points, return 0. Finally, the number of all local extreme points in the segment is added to obtain +.>Represents->Number of local extremum points within a segment.
The waveform curve length is calculated using the following formula:
；
this formula is obtained by calculating the absolute value of the difference between each sample and the previous sample and then adding all the absolute values of the differencesTo the point of. This corresponds to sampling a waveform profile of audio data, and measuring the amplitude of variation between adjacent samples, thereby obtaining the length of the waveform profile.
The average fluctuation intensity is calculated using the following formula:
；
The formula first calculates the absolute value of the difference between all samples in the segment and the average value, then adds all the absolute values of the differences, and finally divides the absolute value by the number of samples in the segment to obtain the average fluctuation intensity. This is equivalent to measuring the average amplitude fluctuation of the waveform of the audio data in the segment, reflecting the stability and amplitude variation of the audio data.
Wherein, is->First part of the individual section>Samples.
Example 8: based on the above embodiment, the method for performing quality evaluation on the video data separated from the pre-sent data to obtain a video evaluation result includes: dividing video data into 8 parts, randomly selecting a frame of image for each part, and respectively calculating image contrastImage clarity->And color consistency->The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Taking the subscript as an integer, and taking the value range of 1 to 8; construction of a feature vector +.>Then calculate its time sequence autocorrelation functionThe method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>The value is an integer, and the value range is 1 to 8; setting two second judging thresholds which are respectively +.>And->The method comprises the steps of carrying out a first treatment on the surface of the Performing second evaluation judgment by using the following formula to obtain 8 second evaluation values:
；
wherein, is a second evaluation value; based on the second evaluation value, obtaining a video evaluation result as follows: / >。
Specifically, the video data quality evaluation method adopts analysis of image contrast, image definition and color consistency of video frame images. By calculating feature vectorsIn (c) and time series autocorrelation function->The quality of the video frame images may be evaluated from a temporal perspective.Two judgment thresholds are set>And->And judging the quality of the video frame image by comparing the image definition and the image contrast with the threshold value.
Overall evaluation resultsIs a vector of 8 binary values, each value representing the result of the evaluation of the corresponding part. The evaluation result can be used for measuring the quality of video data, and is further used for integrally evaluating and feeding back and adjusting a transmission strategy so as to optimize the transmission effect of the video data and provide better video transmission experience. The evaluation method fully utilizes the characteristics of the video frame images and the time sequence correlation, comprehensively evaluates the quality of video data and provides important reference for an audio and video transmission system.
Example 9: on the basis of the previous embodiment, the image contrastCalculated using the following formula:
；
wherein the method comprises the steps ofIs->Gray intensity values of pixels of an image of a frame selected partly at random,/for each pixel of the frame >Is a small constant to prevent denominator from being zero; />Representation ofFirst->The gray scale intensity values of pixels of a frame of image selected randomly by the portion. The numerator in the formula represents the difference between the maximum value and the minimum value of the pixel intensity values of the frame image, and the denominator represents the sum of the maximum value and the minimum value plus a small constant +.>To prevent the denominator from being zero. By such calculation, the contrast of the frame image can be obtained.
The image definitionCalculated using the following formula:
；
wherein, the abscissa variable of the pixels of the image for a randomly selected frame; />A vertical variable for a pixel of an image of a randomly selected frame;
gradient information of the image is used here to represent the image sharpness. For a randomly selected frame of image, the gradients (namely, the absolute values of partial derivatives) of the image in the horizontal direction and the vertical direction are calculated respectively, and then the sum of the absolute values of the gradients in the two directions is taken as the definition of the image.
The color consistency is calculated using the following formula:
；
wherein the method comprises the steps ofIs->Color channel of an image of a frame selected partly at random +.>Average value of (2); />Is->Color channel of an image of a frame selected partly at random +. >Average value of (2).
Here, each color channel is calculated for a randomly selected frame of imageMean value of>. Then calculate the color channel of the part +.>Mean value of>Color channel with all parts->Average value->The sum of squares of the differences between them gives the color consistency +.>。
Calculating a time series autocorrelation function using the formula：；
Wherein, is the average of the feature vectors. For feature vector->First, calculate the average value of the sum and the whole feature vector +.>Is a difference in (c). Then for each distance->Feature vector +.>And->The product is performed and all the product results are added and divided by 8 to obtain the time-series autocorrelation function +.>。
Example 10: based on the above embodiment, the method for recalculating the evaluation result based on the audio evaluation result and the video evaluation result includes: the evaluation result was calculated using the following formula:
；
wherein, & is AND operation;
the feedback regulator carries out feedback regulation on the basis of the initial strategy of data transmission according to the evaluation result, and the method for formulating the transmission strategy of feedback regulation comprises the following steps: setting a matrix of initial strategies of data transmission as follows:
the method comprises the steps of carrying out a first treatment on the surface of the Wherein, when the transport protocol is TCP, </i >>The method comprises the steps of carrying out a first treatment on the surface of the When the transport protocol is UDP, the +. >The method comprises the steps of carrying out a first treatment on the surface of the When the transmission rate is a fixed rate +.>The method comprises the steps of carrying out a first treatment on the surface of the When the transmission rate is an adaptive rate,the method comprises the steps of carrying out a first treatment on the surface of the When the packet loss processing mode is retransmission, the method comprises the steps of +.>When the packet loss processing mode is neglect, the method is carried out by +.>The method comprises the steps of carrying out a first treatment on the surface of the When the buffer processing mode is short buffer, the +.>When the buffer processing mode is long buffer, the +.>The method comprises the steps of carrying out a first treatment on the surface of the When the error recovery mode is forward error correction, < >>When the error recovery mode is adaptive retransmission, < +.>The method comprises the steps of carrying out a first treatment on the surface of the When the bandwidth allocation mode is average allocation, +.>When the bandwidth allocation is on-demand allocation, the method comprises the steps of>The method comprises the steps of carrying out a first treatment on the surface of the When the multicast mode is unicast, the +.>When the multicast mode is multicast, the +.>The method comprises the steps of carrying out a first treatment on the surface of the When the frame priority scheduling mode is I-frame priority, the method comprises the step of +.>When the frame priority scheduling mode is the P/B frame priority, the frame priority scheduling mode is +.>The method comprises the steps of carrying out a first treatment on the surface of the And performing matrix OR operation on the matrix of the initial data transmission strategy and the evaluation result to obtain a matrix of the feedback adjustment transmission strategy, and further obtaining the feedback adjustment transmission strategy.
Specifically, the evaluation resultThe calculation of (1) involves the evaluation of two parts: audio evaluation result->And video evaluation result->. The two evaluation results are respectively vectors containing 8 binary values, each value representing the evaluation result of the corresponding part. For example, a->Possibly +.>，/>Possibly +. >。
To obtain the final evaluation resultAdopts and is connected withOperation (or operation). The corresponding steps are as follows:
will beAnd->And performing OR operation bit by bit. For example, for the first bit,wherein->Representing an or operation. Performing OR operation on each position in turn to obtain final evaluation result +.>. For example, a->. Thus, the evaluation results of the audio and video are combined together by the AND operation to form the final evaluation result +.>. The evaluation result will affect the formulation of the subsequent feedback adjustment transmission strategy.
Formulation of feedback-adjusted transmission strategies involves matrix of data transmission initiation strategiesAnd evaluation result->。Is a vector comprising 8 elements, each element +>The state representing the corresponding policy may be 0 or 1. For example, a->。
In order to obtain a matrix of feedback-adjusted transmission strategies, and operation (or operation) is employed. The corresponding steps are as follows: matrix for initial policy of data transmissionAnd evaluation result->And performing OR operation bit by bit. For example, for the first bit, the +.>. And performing OR operation on each position in sequence to obtain a matrix of the feedback adjustment transmission strategy. For example, a->. Thus, by AND operation, according to the evaluation result +. >And adjusting the initial strategy of data transmission to obtain a final matrix of feedback adjustment transmission strategies. The policy matrix will contain a series of values of 0 and 1 representing the states of the different transmission policies. The adjustment can automatically optimize the data transmission process according to the actual transmission condition, and better audio and video transmission quality and experience are provided. Through cooperation of the evaluator and the feedback adjuster, the system can dynamically adjust the transmission strategy according to the type and quality of the transmission data so as to provide more optimized audio/video transmission effect.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. A plurality of units or means recited in the apparatus claims can also be implemented by means of one unit or means in software or hardware. The terms first, second, etc. are used to denote a name, but not any particular order.
Claims (10)
1. An audio-video transmission system based on perceptual quality optimization, the system comprising: an analyzer, a policy selector, a transmitter, a receiver, an evaluator, and a feedback adjuster; the analyzer is configured to obtain data to be transmitted, analyze the data to be transmitted, determine a data type of the data to be transmitted, and copy the data to be transmittedFrame data to obtain pre-sent data; the data types include: audio data and video data; the strategy selector is configured to formulate a data transmission initial strategy according to the data type of the data to be transmitted and a preset initial strategy; the transmitter is configured to transmit pre-transmitted data and a data transmission initial strategy to the receiver according to the formulated data transmission initial strategy; the receiver is configured to receive pre-sent data and a data transmission initial strategy; the evaluator is configured to evaluate the quality of the received pre-sent data to obtain an evaluation result; the feedback regulator is configured to perform feedback regulation based on the initial data transmission strategy according to the evaluation result, formulate a feedback regulation transmission strategy, send the feedback regulation transmission strategy to the strategy selector to replace the initial data transmission strategy in the strategy selector, and send a control command to the analyzer to prompt the analyzer to start sending data to be sent; the analyzer transmits the data to be transmitted to the transmitter through the strategy selector; and the transmitter transmits the data to be transmitted to the receiver according to the formulated feedback adjustment transmission strategy.
2. The perceptual quality optimization-based audio video transmission system of claim 1, wherein the data transmission strategy and the feedback adjustment transmission strategy each comprise: transmission protocol, transmission rate, packet loss processing mode, buffer processing mode, error recovery mode, bandwidth allocation mode, multicast mode and frame priority scheduling mode; the transmission protocol includes: TCP or UDP; the transmission rate includes: fixed rate or adaptive rate; the packet loss processing mode comprises the following steps: retransmission or omission; the buffer processing mode comprises the following steps: short buffer or long buffer; the error recovery method comprises the following steps: forward error correction or adaptive retransmission; the bandwidth allocation method comprises the following steps: average or on demand; the multicast mode comprises the following steps: unicast or multicast; the frame priority scheduling method comprises the following steps: i-frame priority or P/B-frame priority.
3. The audio/video transmission system based on perceptual quality optimization of claim 2, wherein when the data type of the data to be transmitted is video data, the transmitter first separates an audio part and a video part in pre-transmitted data, and establishes different data transmission initial strategies for the audio part and the video part; the data transmission initial strategy of the audio part comprises the following steps: the transmission protocol is TCP; the transmission rate is a fixed rate; the packet loss processing mode is neglected; the buffer treatment mode is long buffer; the error recovery mode is forward error correction; the bandwidth allocation mode is allocation according to the need; the multicast mode is multicast; the frame priority scheduling mode is P/B frame priority; the data transmission initial strategy of the video part comprises the following steps: the transmission protocol is UDP; the transmission rate is an adaptive rate; the packet loss processing mode is retransmission; the buffer treatment mode is long buffer; the error recovery mode is self-adaptive retransmission; the bandwidth allocation mode is allocation according to the need; the multicast mode is unicast; the frame priority scheduling mode is I frame priority; when the data type of the data to be transmitted is audio data, the data transmission initial strategy of the pre-transmitted data comprises the following steps: the transmission protocol is TCP; the transmission rate is a fixed rate; the packet loss processing mode is neglected; the buffer processing mode is short buffer; the error recovery mode is forward error correction; the bandwidth allocation mode is average allocation; the multicast mode is multicast; the frame priority scheduling mode is P/B frame priority.
4. An audio-visual transmission system based on perceived quality optimization as recited in claim 3, wherein in replicating the data to be transmittedWhen frame data, ->The following constraints are satisfied:
；
and, in addition, the method comprises the steps of,；
wherein, is the total number of frames of data to be transmitted.
5. The audio-video transmission system based on perceptual quality optimization of claim 4, wherein the method of evaluating the quality of the received pre-sent data by the evaluator to obtain the evaluation result comprises: when the data to be transmitted obtain the data type as audio data, the evaluator evaluates the quality of the pre-transmitted data to obtain an evaluation result; when the data to be transmitted is video data, the evaluator firstly carries out quality evaluation on the audio data separated from the pre-transmitted data to obtain an audio evaluation result, then carries out quality evaluation on the video data separated from the pre-transmitted data to obtain a video evaluation result, and calculates to obtain an evaluation result based on the audio evaluation result and the video evaluation result.
6. Such asThe audio/video transmission system based on perceptual quality optimization of claim 5, wherein the method for evaluating the quality of the pre-transmitted data to obtain an evaluation result or for evaluating the quality of the audio data separated from the pre-transmitted data to obtain an audio evaluation result comprises: dividing the audio data into 8 equal-length segments, and calculating the number of local extreme points for each segment Waveform curve length->And mean fluctuation intensity>The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Taking the subscript as an integer, and taking the value range of 1 to 8; setting three first judgment thresholds which are respectively +.>The first evaluation judgment was performed using the following rule, obtaining 8 first evaluation values:
if it isAnd->Then->；
If it isOr->Then->；
Wherein, is the first evaluation value; based on the first evaluation value, an evaluation result or an audio evaluation result is obtained as follows: />。
7. The perceptual quality optimization-based audio video transmission system of claim 6, wherein the number of local extremal points is calculated using the formula:
；
wherein the function isAt->Returning to 1 when the local extremum is found, otherwise returning to 0; />For dividing audio data into 8 equal-length segments, the +.>First part of the individual section>A sample number; />For the length of each segment; the waveform curve length is calculated using the following formula:
；
the average fluctuation intensity is calculated using the following formula:
；
wherein, is->First part of the individual section>Samples.
8. The audio-video transmission system based on perceptual quality optimization of claim 7, wherein the method for performing quality assessment on the video data separated from the pre-transmission data to obtain the video assessment result comprises: dividing video data into 8 parts, randomly selecting a frame of image for each part, and respectively calculating image contrast Image clarity->And color consistency->The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Taking the subscript as an integer, and taking the value range of 1 to 8; construction of a feature vector +.>Then calculate its time sequence autocorrelation function +.>The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>The value is an integer, and the value range is 1 to 8; setting two second judging thresholds which are respectively +.>And->The method comprises the steps of carrying out a first treatment on the surface of the Performing second evaluation judgment by using the following formula to obtain 8 second evaluation values:
；
wherein, is a second evaluation value; based on the second evaluation value, obtaining a video evaluation result as follows: />。
9. The perceptual quality optimization-based audio video transmission system of claim 8, wherein the image contrast ratioCalculated using the following formula:
；
wherein the method comprises the steps ofIs->Gray intensity values of pixels of an image of a frame selected partly at random,/for each pixel of the frame>Is a small constant to prevent denominator from being zero;
the image definitionCalculated using the following formula: />；
Wherein, the abscissa variable of the pixels of the image for a randomly selected frame; />A vertical variable for a pixel of an image of a randomly selected frame;
the color consistency is calculated using the following formula:the method comprises the steps of carrying out a first treatment on the surface of the Wherein->Is->Color channel of an image of a frame selected partly at random +. >Average value of (2); />Is->Color channel of an image of a frame selected partly at random +.>Average value of (2);
calculating a time series autocorrelation function using the formula：
The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Is the average of the feature vectors.
10. The audio-visual transmission system based on perceptual quality optimization of claim 8, wherein the method of recalculating the evaluation result based on the audio evaluation result and the video evaluation result comprises: the evaluation result was calculated using the following formula:the method comprises the steps of carrying out a first treatment on the surface of the Wherein, &is AND operation; the feedback regulator carries out feedback regulation on the basis of the initial strategy of data transmission according to the evaluation result, and the method for formulating the transmission strategy of feedback regulation comprises the following steps: setting a matrix of initial strategies of data transmission as follows: />The method comprises the steps of carrying out a first treatment on the surface of the Wherein, when the transport protocol is TCP, </i >>The method comprises the steps of carrying out a first treatment on the surface of the When the transport protocol is UDP, the +.>The method comprises the steps of carrying out a first treatment on the surface of the When the transmission rate is a fixed rate +.>The method comprises the steps of carrying out a first treatment on the surface of the When the transmission rate is an adaptive rate,the method comprises the steps of carrying out a first treatment on the surface of the When the packet loss processing mode is retransmission, the method comprises the steps of +.>When the packet loss processing mode is neglect, the method is carried out by +.>The method comprises the steps of carrying out a first treatment on the surface of the When the buffer processing mode is short buffer, the +.>When the buffer processing mode is long buffer, the +.>The method comprises the steps of carrying out a first treatment on the surface of the When the error recovery mode is forward error correction, < > >When the error recovery mode is adaptive retransmission, < +.>The method comprises the steps of carrying out a first treatment on the surface of the When the bandwidth allocation mode is average allocation, +.>When the bandwidth allocation is on-demand allocation, the method comprises the steps of>The method comprises the steps of carrying out a first treatment on the surface of the When the multicast mode is unicast, the +.>When the multicast mode is multicast, the +.>The method comprises the steps of carrying out a first treatment on the surface of the When the frame priority scheduling mode is I-frame priority, the method comprises the step of +.>When the frame priority scheduling mode is the P/B frame priority, the frame priority scheduling mode is +.>The method comprises the steps of carrying out a first treatment on the surface of the And performing matrix OR operation on the matrix of the initial data transmission strategy and the evaluation result to obtain a matrix of the feedback adjustment transmission strategy, and further obtaining the feedback adjustment transmission strategy.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311199771.3A CN116939318B (en) | 2023-09-18 | 2023-09-18 | Audio and video transmission system based on perceptual quality optimization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311199771.3A CN116939318B (en) | 2023-09-18 | 2023-09-18 | Audio and video transmission system based on perceptual quality optimization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116939318A true CN116939318A (en) | 2023-10-24 |
| CN116939318B CN116939318B (en) | 2023-12-19 |
Family
ID=88382950
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311199771.3A Active CN116939318B (en) | 2023-09-18 | 2023-09-18 | Audio and video transmission system based on perceptual quality optimization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116939318B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117376613A (en) * | 2023-12-07 | 2024-01-09 | 方图智能(深圳)科技集团股份有限公司 | Distributed audio and video transmission control method, system and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200112472A1 (en) * | 2018-10-05 | 2020-04-09 | Samsung Eletrônica da Amazônia Ltda. | Method for digital video transmission adopting packaging forwarding strategies with path and content monitoring in heterogeneous networks using mmt protocol, method for reception and communication system |
| CN116489473A (en) * | 2023-05-05 | 2023-07-25 | 深圳市云屋科技有限公司 | Dynamic compensation system and method for audio and video transmission optimization |
| CN116599904A (en) * | 2023-06-15 | 2023-08-15 | 北京交通大学 | Parallel transmission load balancing device and method |
-
2023
- 2023-09-18 CN CN202311199771.3A patent/CN116939318B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200112472A1 (en) * | 2018-10-05 | 2020-04-09 | Samsung Eletrônica da Amazônia Ltda. | Method for digital video transmission adopting packaging forwarding strategies with path and content monitoring in heterogeneous networks using mmt protocol, method for reception and communication system |
| CN116489473A (en) * | 2023-05-05 | 2023-07-25 | 深圳市云屋科技有限公司 | Dynamic compensation system and method for audio and video transmission optimization |
| CN116599904A (en) * | 2023-06-15 | 2023-08-15 | 北京交通大学 | Parallel transmission load balancing device and method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117376613A (en) * | 2023-12-07 | 2024-01-09 | 方图智能(深圳)科技集团股份有限公司 | Distributed audio and video transmission control method, system and storage medium |
| CN117376613B (en) * | 2023-12-07 | 2024-03-15 | 方图智能(深圳)科技集团股份有限公司 | Distributed audio and video transmission control method, system and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116939318B (en) | 2023-12-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7782779B2 (en) | Apparatus and method for transmitting a multimedia data stream | |
| US10355998B2 (en) | Adaptive video over multicast | |
| RU2487407C2 (en) | System and method of compressing streaming interactive video | |
| KR100787314B1 (en) | Method and apparatus for adaptive media playout for intra-media synchronization | |
| CN116939318B (en) | Audio and video transmission system based on perceptual quality optimization | |
| US8036125B2 (en) | Apparatus and method for transmitting multimedia stream using virtual machines based on a number of transmissions at a data rate | |
| Ray et al. | Vantage: optimizing video upload for time-shifted viewing of social live streams | |
| US20120317300A1 (en) | Multipath rate adaptation | |
| US20050249240A1 (en) | Multimedia server with simple adaptation to dynamic network loss conditions | |
| EP3022884A1 (en) | Quality optimization with buffer and horizon constraints in adaptive streaming | |
| RU2598805C2 (en) | Method for dynamic adaptation of repetition frequency of bits when receiving and appropriate receiver | |
| CN101917613A (en) | Acquiring and coding service system of streaming media | |
| US7526565B2 (en) | Multiple description hinting and switching for adaptive media services | |
| Muntean et al. | Region of interest-based adaptive multimedia streaming scheme | |
| KR20050091054A (en) | Device and process for adjusting the bit rate of a stream of contents and associated products | |
| US20200213640A1 (en) | Method and apparatus for automatic hls bitrate adaptation | |
| US10349105B2 (en) | Channel change processing using stored content | |
| US7720067B2 (en) | Data transfer apparatus and transfer control method | |
| US11178205B2 (en) | System and method for providing live streaming of video data in a low-bandwidth network | |
| CN112055165B (en) | Video communication method, device, system, server, client and storage medium | |
| KR100861594B1 (en) | Apparatus and method for controlling multimedia data rate | |
| Dhamodaran et al. | Adaptive bitstream prioritization for dual TCP/UDP streaming of HD video | |
| Deshpande | High quality video streaming using content-awareadaptive frame scheduling with explicit deadlineadjustment | |
| Vijaykumar et al. | A cross-layer frame work for adaptive video streaming over wireless networks | |
| Kinoshita et al. | Low Latency Live Streaming System with Congestion Control |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |