JP5099697B2 - Audio / video output method, audio / video output method realization program, and audio / video output device - Google Patents

Description

  The present invention relates to an audio / video output method, an audio / video output method realization program, and an audio / video output device in a receiving terminal of an audio / video stream transmitted in packets.

  As access lines become faster, applications that handle audio and video on IP (Internet Protocol) networks have become widespread. However, an IP network is basically a best-effort type, resulting in packet loss and transmission delay.

  Packet loss and transmission delay disturb the audio temporal structure and video temporal / spatial structure, and lower the quality of service (QoS) of audio / video transmission. If the QoS of audio / video transmission decreases, the user experience quality (Quality of Experience: QoE, see Non-Patent Document 1) will decrease.

  Therefore, in order to suppress a drop in QoE due to packet loss or transmission delay, either or both of an error compensation method and a frame skip method are often implemented in video output control (see Non-Patent Document 2). .

  Video error compensation is to deal with the loss of information on the network and the loss of information due to loss of synchronization, and to interpolate the lost information from other information. Since video frames can be output by interpolating the gaps, the temporal quality of the video is improved. However, since the interpolation does not necessarily restore the original image information, there is a possibility that the spatial quality is degraded by error compensation. In addition, when a frame with degraded spatial quality is used as a reference frame for the next motion prediction, there is a problem that the degraded spatial quality is propagated in GOP (Group of Pictures) units.

  On the other hand, in the frame skip method, if there is a video frame disturbed due to packet loss or the like, the frame output is temporarily interrupted, and the interrupt is continued until the next normal I frame appears. In other words, in this method, video frames whose spatial structure is disturbed due to missing or the like are not output, and only frames with high spatial quality can be output. However, the freeze of the stream causes the user to feel a decrease in the time quality of the video.

  As can be seen from the above, error compensation preserves the temporal quality of video and frame skip preserves spatial quality.

  Conventionally, in audio / video packet transmission, degradation of video spatial quality caused by error compensation and temporal quality caused by frame skip have been taken up as separate problems. And most studies have considered only one of video spatial quality or temporal quality, and audio quality has not been taken into account.

  On the other hand, in the broadcasting world, several inventions related to methods for properly using error compensation and frame skip have been made. However, these do not take into account degradation of time quality due to packet arrival delay peculiar to packet transmission.

  In Patent Document 1, a recording / reproducing apparatus for digital broadcasting is invented. In this apparatus, the image quality deterioration of the reproduced video is suppressed by deleting the video decoding data unit or the audio decoding data unit in a portion where the number of decoding error signals of the transport stream data is equal to or greater than a predetermined threshold. Here, the video decoding data unit is GOP. However, this apparatus assumes recording and reproduction, and does not target a streaming type that reproduces while receiving. For this reason, time quality such as the effect of packet arrival delay on the reproduced video is not considered. Also, the interaction between audio and video is not considered.

In Patent Document 2, a digital broadcast receiving apparatus is invented that includes video data processing means that performs arithmetic processing in a plurality of processing modes corresponding to the degree of received video degradation. In this method, video output is performed by thinning out inter-frame predictive encoded images in the MPEG compression method according to the mode of image degradation. The image deterioration mode is detected by a plurality of types of parameters, and the threshold value for determining the processing mode is characterized by changing in accordance with the genre. In addition, the degree of image degradation during display output input by the operator can be set as a threshold value. Furthermore, the volume of sound is reduced according to the degree of deterioration. However, it is not obvious that simply thinning out inter-frame predictive encoded images according to video degradation has led to an improvement in the quality of the user's experience. In addition, although it has a mechanism for reflecting the user's quality of experience in the threshold setting, it requires input of information from the user, which is troublesome for the user. Further, in this patent document, time quality deterioration due to arrival delay of information is not taken into consideration.
JP 2007-124445 A JP 2004-112654 A ITU-T Rec. G.100 / P.10 Amendment 1, `` Amendment 1: new appendix I definition of Quality of Experience (QoE), '' Jan. 2007. N. Feamster and H. Balakrishnan, `` Packet loss recovery for streaming video, '' Proc.IEEE 12th International Packetvideo Workshop (PV2002), Apr. 2002. ITU-T Rec. P.800.1, `` Mean Opinion Score (MOS) terminology, '' Mar. 2003. Yoshihisa Tanaka, `` Psychological Measurement Second Edition '', The University of Tokyo Press, 1997. JP Guilford, Psychometric methods, McGraw-Hill, NY, 1954. Mindcraft Inc, `` WebStone benchmark information, '' http://www.mindcraft.com/webstone/. `` H.264 / MPEG-4 AVC reference software JM11.0, '' http://iphome.hhi.de/suehring/tml/index.htm The Video Quality Experts Group, `` The video quality experts group web site, '' http://www.its.bldrdoc.gov/vqeg/. ITU-T Rec. P.911, `` Subjective audiovisual quality assessment methods for multimedia applications, '' Dec. 1998. S. Tasaka and Y. Ito, `` Psychometric analysis of the mutually compensatory property of multimedia QoS, '' Conf. Rec.IEEE ICC 2003, pp. 1880-1886, May 2003. F. Mosteller, `` Remarks on the method of paired comparisons: III. a test of significance for paired comparisons when equal standard deviations and equal correlations are assumed, '' Psychometrika, vol.16, no.2, pp.207-218, 1951.

  In previous research on audio and video packet transmission, the spatial quality degradation caused by error compensation and the temporal quality degradation caused by frame skip were taken up as separate problems. And most studies have considered only one of video spatial quality or temporal quality, and audio quality has not been taken into account. Therefore, the best user experience quality is not always obtained in audio / video output using either the conventional error compensation method or the frame skip method.

  Therefore, the present invention uses the trade-off relationship between the temporal quality and spatial quality of video to maximize the quality of voice (QoE) of the user who views the packet-transmitted audio / video. It is possible to output video.

The audio / video output system according to the first aspect of the invention is a buffer unit that absorbs packet arrival delay fluctuations, a decoding unit that decodes audio / video, and a decoding unit that receives and outputs packet-transmitted audio and video. A determination unit for determining whether or not a video frame can be output, and an output unit for outputting decoded audio / video. The buffering time in the buffer unit and the error compensation rate of the video frame decoded by the decoding unit in response, an audio and video output format toggle its or skip or the frame to display the frame subjected to error compensation in the determination unit, sound at audio and video output is obtained at the output and the video decoding unit Estimating section for estimating user experience quality in real time from time quality information, video time quality information and video space quality information The threshold for switching in the determination unit, and wherein the benzalkonium be set using the estimated value of the estimator. The audio / video output method realization program of the second invention comprises a buffer unit that absorbs packet arrival delay fluctuations when receiving and outputting packet-transmitted audio and video, a decoding unit that decodes audio / video, A determination unit that determines whether or not a decoded video frame can be output, and an output unit that outputs decoded audio and video. The buffering time in the buffer unit and the error compensation of the video frame decoded by the decoding unit depending on the rate, a or a toggle its audio-video system implementing the program skips whether the frame to display the frame subjected to error compensation in the determination unit, when the audio-video output, the output unit and the video decoding unit User in real time from audio time quality information, video time quality information and video space quality information An estimation unit that estimates a sensitive quality is provided, the threshold for switching in the determination unit, characterized and Turkey set by using an estimate of the estimator. The audio / video output device according to the third aspect of the invention is a buffer unit that absorbs packet arrival delay fluctuations, a decoding unit that decodes audio / video, and a decoding unit that receives and outputs packet-transmitted audio and video. A determination unit for determining whether or not a video frame can be output, and an output unit for outputting decoded audio / video. The buffering time in the buffer unit and the error compensation rate of the video frame decoded by the decoding unit in response, a determination unit toggle its audio-video output device or skip whether the frame frame displays the performing error compensation in a voice at the time of audio and video output is obtained at the output and the video decoding unit Estimating section for estimating user experience quality in real time from time quality information, video time quality information and video space quality information The threshold for switching in the determination unit, characterized and Turkey set by using an estimate of the estimator.

  The present invention relates to an output method provided in a receiving side apparatus in audio / video packet transmission. In the present invention, no special function is required for the transmission side device. For this reason, the conventional transmission side apparatus can be used.

  As shown in the block diagram of FIG. 1, the present invention determines a buffer unit that absorbs packet arrival delay fluctuations, a decoding unit that decodes audio and video, and whether or not a decoded video frame can be output. A decision unit and an output unit for outputting decoded audio / video, and the error compensation in the decision unit according to the buffering time in the buffer unit and the degree of error compensation of the video frame decoded in the decoding unit It is possible to realize high user experience quality using the trade-off between video temporal quality and spatial quality by switching whether to display a frame that has been performed or to skip the frame.

  In the system of the present invention, the received audio / video packet is buffered in the buffer unit. This absorbs delay fluctuations during transmission over the network. An arbitrary upper limit is set for the buffering time indicating the buffered time. That is, a packet that arrives with a large delay is determined to be meaningless information and is discarded. Thus, this buffering time affects both temporal quality and spatial quality.

  Therefore, considering the effect of improving the time and space quality due to buffering and the tradeoff between the effect of space compensation and the effect on time quality due to error compensation, it is important to perform appropriate control to improve the user's quality of experience. It becomes important for improvement.

  Further, the system of the present invention is characterized in that high user experience quality can be realized by utilizing the interaction between modalities in audio / video transmission.

  In audio / video transmission, two different media, audio and video, are transmitted. Audio is recognized by hearing, and video is recognized by vision. There is a nature of interaction between these different modalities. For example, when the video output quality is poor and the sound quality is good, the video information can be complemented.

  In the present invention, this interaction between modalities is used. For this reason, as a measure of user experience quality, instead of MOS (Mean Opinion Score, see Non-Patent Document 3) used in many of the ITU-T standards, it is a series that is one of the psychometric measurement methods. A distance scale obtained by the category method (see Non-Patent Documents 4 and 5) is used. In MOS, only the order between the scale values is guaranteed, and the equidistantness of the values is not guaranteed. MOS is suitable for quality evaluation of single media transmission. However, during multimedia transmission, the scale value interval is likely not to be constant for the user due to the interaction between modalities. For this reason, MOS is not suitable for evaluating the quality of multimedia transmission.

  By using the distance measure obtained by the sequence category method, it is possible to appropriately evaluate and estimate the user experience quality including the interaction between modalities. Then, by performing control in view of the user experience quality evaluated by such a method, it is possible to realize high user experience quality.

  Further, in the system of the present invention, as shown in the block diagram of FIG. 2, at the time of audio / video output, audio time quality information, video time quality information and video space quality information obtained at the output unit and video decoding unit Provides an estimation unit that estimates the user experience quality in real time, and uses this estimate to set a threshold for judging the switching from the trade-off between video temporal quality and spatial quality. It is characterized by enabling the realization of quality.

  By using the distance scale as an evaluation scale of the user experience quality, it is possible to estimate the user experience quality with high accuracy from the time quality information and the spatial quality information obtained at the output unit, that is, the application level QoS. Then, it is possible to obtain a high user experience quality by estimating the user experience quality with respect to several output forms and adopting an output form with the highest estimation result.

  In this way, it is possible to perform control closer to the user's sense by dynamically deciding whether to prioritize temporal quality or spatial quality based on the user experience quality estimated in real time. Thus, the user experience quality can be improved.

  With the configuration of the present invention, it is possible to provide a high user experience quality to a user who views a voice / video transmitted in packets.

  Embodiments 1 and 2 embodying the present invention will be described below with reference to the drawings.

  As a first embodiment of the present invention, a method is conceivable in which error compensation and frame skip are switched and the spatial quality and temporal quality of video are controlled according to the ratio of slices in which error compensation is performed in one video frame. The specific operation will be described with reference to the flowchart of FIG. As shown in FIG. 3, first, frame information to be output is acquired from a buffer for storing received frames and decoded. Then, error compensation is performed on a slice that is missing in the decoded frame. At this time, the error compensation method is not limited to a specific one, and any error compensation method can be used as long as the error compensation ratio can be calculated. The rate of error compensation is called the error compensation rate (unit:%) and is defined by the following equation.

Error compensation rate =
Number of slices with error compensation / number of slices in one frame x 100 [%]
When the obtained error compensation rate is larger than a preset threshold value, the output is canceled and a non-output frame is set. Even when the error compensation rate is less than or equal to the threshold, if the motion prediction reference frame of the target frame is a non-output frame, the output is similarly canceled and a non-output frame is set. A frame is output when the error compensation rate is equal to or less than the threshold and the motion prediction reference frame of the target frame is an output frame. This operation is performed for each frame to switch between the error compensation method and the frame skip method.

  FIG. 4 shows an operation example when the threshold value is 100%, 40%, 20%, and 0%. The solid parallelogram in the figure represents the output frame, and the light solid parallelogram represents the non-output frame. Error-compensated slices are shown as small gray parallelograms. The numerical value in parentheses in the figure represents the error compensation rate of the frame. Note that when the threshold value is 100%, it corresponds to a pure error compensation method, and the threshold value 0% corresponds to a pure frame skip method.

  A method of experiments conducted to evaluate the effectiveness of this method is shown. In the experiment, QoE is evaluated by changing the threshold of the switching method described above. Then, based on the obtained QoE evaluation results, the effect of changes in threshold on QoE is discussed. The experimental system and QoE evaluation experiment method are shown below.

  The network configuration used in this experiment is shown in FIG. It consists of 4 PCs and 2 routers. Each of the four PCs functions as an audio / video transmitting terminal, audio / video receiving terminal, Web server, and Web client. All lines are 100BASE-TX. The router is RS3000 manufactured by Alcatel Lucent (formerly Riverstone Networks).

  The audio / video is transmitted from the audio / video transmitting terminal to the audio / video receiving terminal. Audio and video are transmitted in separate transport streams. RTP / UDP is used as the transport protocol. At that time, the receiving terminal performs playout buffering control for 1 second in order to absorb fluctuations in transmission delay.

  Table 1 shows the audio and video specifications. The MU (Media Unit) in the table is a unit of media synchronization processing, which corresponds to 20 ms of sampling data for audio and 1 frame for video. The voice MU is packed into one UDP datagram. The video MU is divided into 15 UDP datagrams in units of slices and transmitted. Three types of picture patterns are considered in which n-1 P frames (n = 1, 5, 15) follow an I frame.

  In this network, Web traffic is transmitted from the Web server to the Web client. For the Web client, WebStone (refer to Non-Patent Document 6) which is a Web server performance evaluation tool is used. The number of clients set in Webstone is 6, 30, 40, 50, 75, and 100.

  As the video error compensation method, the one implemented in JM11.0 (see Non-Patent Document 7) is used. In error compensation in I frames, missing information is interpolated from surrounding information. Two types of error compensation methods can be used for error compensation in the P frame. In this embodiment, a method (Frame Copy) for copying a missing portion from the previous output frame is selected.

  Switching control between the error compensation method and the frame skip method is performed according to the procedure described above. The threshold values are 100%, 40%, 20%, and 0%.

  The content selection was made with reference to the VQEG test plan (see Non-Patent Document 8). In this study, as outlined in Table 2, we selected three types of content, prepared two contents for each, and used a total of six contents. Table 3 shows the average bit rate. For reference, ITU-TP is used for all contents. The TI (Temporal Information) value proposed in 911 (see Non-Patent Document 9) is shown. This represents the degree of movement, and the larger the value, the greater the movement. The TI values in Table 3 are calculated excluding scene changes.

  Audio / video output from the media receiving terminal is recorded, and using this as a stimulus, QoE evaluation using the sequence category method is performed by the method of Non-Patent Document 10. As a category in the rating scale method, the five-step disturbance category rating (DCR) shown in Table 4 is used. The evaluation criteria at that time is the audio / video output when Web traffic is not generated.

  The test subjects were 32 men in their 20s. The number of evaluation targets (stimulus) is 432 (= 6 × 3 × 4 × 6), which is 6 contents, 3 picture patterns, 4 types of switching thresholds, and 6 types of Web clients. The presentation order of each content is random for each subject. Also, when evaluating individual content, the presentation order of the combination of the switching threshold and the number of Web clients is also random. The regeneration time of one stimulus was 10 seconds, and the evaluation time per person was about 4.5 hours including breaks.

  Mosteller's fitness test (see Non-Patent Document 11) was performed on the scale values obtained by the series category method. As a result, at the significance level of 5%, the scale value obtained by the series category method did not match the measurement results. Therefore, the stimuli were removed one by one in descending order of the error between the estimated value and the actual measurement value among the 432 stimuli. As a result, by removing 40 stimuli, the obtained scale value matched the measurement result at 5% significance level. In this embodiment, these scale values are used as QoE parameters.

  Since an arbitrary value can be set as the origin in the distance scale, in this embodiment, the scale origin is determined so that the minimum value of the QoE parameter is zero. As a result, the lower limit of each category was 4.649 (Category 5), 3.532 (Category 4), 2.371 (Category 3), and 1.010 (Category 2).

  FIG. 6 shows psychological scale values for the number of Web clients when the content is sport 2 and the picture pattern is only an I frame. In this figure, evaluation values deleted as a result of Mosteller's fitness test are not shown. From this figure, it can be seen that, regardless of the number of Web clients, high QoE can be obtained with the frame skip method with a threshold of 0%. When the picture pattern is only I frames, it was effective to obtain a high QoE for all contents by using a threshold of 0%, that is, only the frame skip method.

  FIG. 7 shows psychological scale values for the number of Web clients when the content is Sports 2 and the picture pattern is IPPPP. From this figure, it can be seen that QoE when the threshold is set to a value larger than 0% is better than when the threshold is set to 0% in a situation where the number of Web clients is large and the network is congested. Therefore, when the P pattern is included in the picture pattern, it can be said that setting the threshold value to a value larger than 0% is effective for obtaining a high QoE in the content where the video influences dominantly.

  FIG. 8 shows psychological scale values for the number of Web clients when the content is animation 2 and the picture pattern is IPPPP, and FIG. 9 shows psychological scale values when the picture pattern is IPPPPPPPPPPPPPP. From these figures, it can be seen that it is more effective to set the threshold value to a larger value as the number of P frames included in the picture pattern increases.

  7, 8, and 9, in animation, the effect of setting a threshold value larger than that of video-dominated content begins to appear. You can see that it takes a lot. The same tendency as the animation was obtained for the audio-dominated content.

  In addition, it was found that the highest QoE can be obtained only for error compensation systems with a threshold of 100% for content with a lot of video motion and content that contains many P frames in picture patterns.

  From these results, there is a trade-off relationship between video temporal quality and spatial quality, which depends on the type of content, the degree of video motion, and the picture pattern. It turns out to be effective.

  Based on the QoE evaluation result of Example 1, the method of Example 2 can be considered. In this method, the error compensation method and the frame skip method are switched according to the user experience quality estimation result during a certain learning period. A specific operation of this method will be described with reference to the flowchart of FIG.

  In the system of the second embodiment, first, the system to be used is switched according to the picture pattern. If the picture pattern is only an I frame, the threshold is set to 0% and only the frame skip method is used. Otherwise, the output start method is determined according to the content type. For video-oriented content, set the threshold to 100% and start output using the error compensation method. For audio-oriented content, the threshold is set to 0% and output is started using the frame skip method.

  A certain learning period is set after the start of output. Meanwhile, when audio / video is output according to the above settings, the switching method is applied with multiple thresholds (eg 0%, 20%, 40%, 100%, etc.) using decoded video. Deriving QoE estimates. At the end of the learning period, these multiple QoE estimates are compared. The threshold at which the highest QoE estimate is obtained is used for subsequent video output. Further, such threshold selection is repeated at regular intervals.

  Although the embodiment of the present invention has been described above, instead of using the estimated QoE value, it is also possible to use a threshold value uniquely determined from attribute information of contents exchanged when an audio / video transmission session is established.

  The present invention can be widely used for an audio / video streaming service via a packet transmission network.

Block configuration diagram of the system shown in claims 1 and 2 Block diagram of the system shown in claim 3 Flow chart of the first embodiment Video stream output example Experimental system Psychological scale value when content is sport 2 and picture pattern is I Psychological scale value when the content is sport 2 and the picture pattern is IPPPP Psychological scale value when the content is animation 2 and the picture pattern is IPPPP Psychological scale value when the content is animation 2 and the picture pattern is IPPPPPPPPPPPPPP Flow chart of embodiment 2

Claims (3)

When receiving and outputting packet-transmitted audio and video, a buffer unit that absorbs packet arrival delay fluctuations, a decoding unit that decodes audio and video, and a determination unit that determines whether a decoded video frame can be output If, and an output unit for outputting the decoded audio and video, in accordance with the error compensation rate of video frames decoded by the buffering time and the decoder in the buffer unit, the error compensation in the evaluation unit a toggle its audio and video output format or skip or the frame to display the frame was, at the time of the audio-video output, audio time quality information obtained at the output unit and the video decoding unit, a video time An estimator for estimating a user experience quality in real time from the quality information and the video space quality information; The threshold for performing Oite switching, the estimating portion of the audio-video system and setting using the estimated value. When receiving and outputting packet-transmitted audio and video, a buffer unit that absorbs packet arrival delay fluctuations, a decoding unit that decodes audio and video, and a determination unit that determines whether a decoded video frame can be output If, and an output unit for outputting the decoded audio and video, in accordance with the error compensation rate of video frames decoded by the buffering time and the decoder in the buffer unit, the error compensation in the evaluation unit a switching or skip or the frame to display the frame Ru audio-video system implementing the program went, when audio-video output, audio time quality information obtained at the output unit and the video decoding unit, Establishing an estimation unit that estimates user experience quality in real time from video temporal quality information and video spatial quality information The threshold for switching in the determination unit, the audio-video system implemented programs and setting using the estimated value of the estimator. When receiving and outputting packet-transmitted audio and video, a buffer unit that absorbs packet arrival delay fluctuations, a decoding unit that decodes audio and video, and a determination unit that determines whether a decoded video frame can be output If, and an output unit for outputting the decoded audio and video, in accordance with the error compensation rate of video frames decoded by the buffering time and the decoder in the buffer unit, the error compensation in the evaluation unit a toggle its audio-video output device or skip whether the frame frame Show subjected to, when the audio-video output, audio time quality information obtained at the output unit and the video decoding unit, a video time An estimator for estimating a user experience quality in real time from the quality information and the video space quality information; The threshold for performing Oite switching, the estimating portion estimates the audio-video output device and setting utilized.