CN114339197A - Video playback test method, device and equipment - Google Patents

Abstract

The application discloses a video playing testing method, device and equipment, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring video playing information corresponding to a target video; determining corresponding playing quality characteristic information of the target video on at least two information modalities according to the visual information, the auditory information and the data stream information; and generating an audio-visual quality test result corresponding to the target video played by the target player based on the corresponding playing quality characteristic information on the at least two information modes. According to the technical scheme, the playing quality characteristics of the video on at least two information modes are determined by obtaining the multi-dimensional video playing information, so that the test result capable of representing the audio-visual information propagation quality is generated, the problem of inaccurate test result caused by single dimension is avoided, the accuracy of audio-video quality test is improved, and the labor cost required by the audio-video quality test is reduced.

Description

Video playback test method, device and equipment

technical field

The present application relates to the technical field of artificial intelligence, and in particular, to a testing method, device and equipment for video playback.

Background technique

In recent years, with the rapid growth of Internet audio and video data, the research field of audio and video quality evaluation has been relatively active.

In related technologies, the audio and video quality evaluation often relies heavily on the subjective evaluation of the human eye, that is, the audio and video quality is scored by means of human visual observation; for the audio and video quality evaluation scheme from an objective point of view, the video is often imaged. Dimensional quality evaluation, scoring audio and video quality through image quality.

In the related art, the audio and video quality evaluation has a single dimension, low accuracy, and high labor cost.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a video playback testing method, device, and equipment, which can use media content of multiple information modalities to test and evaluate audio and video quality, improve the accuracy of audio and video quality testing, and reduce testing labor costs.

According to an aspect of the embodiments of the present application, a method for testing video playback is provided, the method comprising:

Obtain video playback information corresponding to the target video, where the video playback information includes visual information, auditory information corresponding to the target video, and data stream information corresponding to the target video being played by the target player, and the data stream information is used to represent the video The processing quality of the data stream during playback;

Determine, according to the visual information, the auditory information and the data stream information, playback quality feature information corresponding to the target video in at least two information modalities;

Based on the corresponding playback quality feature information on the at least two information modalities, an audio-visual quality test result corresponding to the target video being played by the target player is generated, and the audio-visual quality test result is used to indicate that the target video is in the target video. The corresponding audiovisual information dissemination quality on the target player.

According to an aspect of the embodiments of the present application, a testing device for video playback is provided, the device comprising:

A playback information acquisition module, configured to acquire video playback information corresponding to the target video, where the video playback information includes visual information, auditory information corresponding to the target video, and data stream information corresponding to the target video being played by the target player. The data stream information is used to characterize the processing quality of the data stream during video playback;

a quality feature determination module, configured to determine playback quality feature information corresponding to the target video in at least two information modalities according to the visual information, the auditory information and the data stream information;

A test result generation module, configured to generate an audiovisual quality test result corresponding to the target video being played by the target player based on the corresponding playback quality feature information on the at least two information modalities, and the audiovisual quality test result is used for Characterizing the corresponding audiovisual information dissemination quality of the target video on the target player.

According to an aspect of the embodiments of the present application, a computer device is provided, the computer device includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, the at least one The instruction, the at least one piece of program, the code set or the instruction set are loaded and executed by the processor to implement the above-mentioned video playback test method.

According to an aspect of the embodiments of the present application, a computer-readable storage medium is provided, where at least one instruction, at least one segment of program, code set or instruction set is stored in the storage medium, the at least one instruction, the at least one segment of The program, the code set or the instruction set are loaded and executed by the processor to implement the above-mentioned testing method for video playback.

According to one aspect of the embodiments of the present application, a computer program product is provided, the computer program product includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the above-mentioned testing method for video playback.

The technical solutions provided in the embodiments of the present application can bring the following beneficial effects:

By acquiring the visual information, auditory information and data stream information that the player can perceive when the player plays the video, and the data stream information reflecting the data processing quality of the target player, the playback quality characteristics of the video in at least two information modalities can be determined. The playback quality features on the two information modalities are used to generate audio-visual quality test results that can characterize the quality of audio-visual information dissemination. The single problem that causes inaccurate test results improves the accuracy of audio and video quality testing, and can determine video playback quality without manual observation, reducing labor costs for audio and video quality testing.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

1 is a schematic diagram of an application running environment provided by an embodiment of the present application;

Fig. 2 is a flowchart of a method for testing video playback provided by an embodiment of the present application;

Fig. 3 is a flow chart 2 of a test method for video playback provided by an embodiment of the present application;

FIG. 4 exemplarily shows a schematic diagram of determining video playback quality indicator data;

FIG. 5 exemplarily shows a schematic diagram of determining audio playback quality indicator data;

FIG. 6 exemplarily shows a schematic diagram of determining image quality index data;

FIG. 7 exemplarily shows a schematic diagram of determining text quality indicator data;

FIG. 8 exemplarily shows a schematic diagram of a multimodal quality testing network;

Fig. 9 exemplarily shows a kind of schematic diagram of audio and video quality testing process in the process of terminal playing video;

10 is a block diagram of a test device for video playback provided by an embodiment of the present application;

FIG. 11 is a structural block diagram of a computer device provided by an embodiment of the present application.

Detailed ways

Before introducing the method embodiments of the present application, application scenarios, related terms or nouns that may be involved in the method embodiments of the present application are first introduced, so as to facilitate the understanding of those skilled in the art of the present application.

Human experience of the world is multimodal, such as seeing objects, hearing sounds, feeling textures, smelling smells, tasting tastes, etc. Modality refers to the way in which something happens or experiences, and a research question is characterized as multimodal when it contains multiple modalities. For AI to make progress in understanding the world around humans, it needs to be able to interpret these multimodal signals simultaneously.

In recent years, with the rapid growth of Internet audio and video data, except for information in plain text, richer data such as voice, image, and video have not been fully utilized and learned. In the field of audio and video, information learning and application in the field of images are often limited. Other information is not fully utilized. When general users watch videos, the evaluation of video quality is comprehensive in many aspects. Multimodal neural networks can learn the concepts between different modalities, and combine multimodal content such as text, voice, images, and videos. learning, so as to realize the integration of different modal information.

In the scenario where the terminal performs video playback, the terminal needs to finally present the video through decoding, rendering, post-processing and other processes, and considering the hardware performance of different devices, etc., the quality of the video viewed by the user on the terminal is the same as that of the video itself. The source is different. To evaluate and test the final video quality on the terminal, there is a lack of effective video playback quality evaluation tools. On the one hand, for newly launched videos, different video formats, video terminal post-processing algorithms, and the content quality evaluation of the video itself, it is often very dependent on the subjective evaluation of the human eye, which requires a lot of manpower for subjective evaluation, which is a waste of manpower and has poor timeliness. , the coverage rate is also very low. Even if there are evaluation tools, they are often single-dimensional evaluations such as images, and cannot be fully applied to video scenes. On the other hand, there is a lack of monitoring mechanisms for various quality problems such as clarity, fluency, and blurry screens that appear in video playback.

Therefore, in the embodiments of the present application, the multi-modal information mining and evaluation of the video is carried out, and the multi-modal quality test model is introduced to effectively quantify the video playback quality, which is helpful for a more comprehensive and convincing evaluation of the video quality. Playing scenes have a relatively large application value. First of all, the evaluation of video viewing quality is helpful for evaluating the effect of player algorithms and strategies; second, the evaluation of video content helps to screen out better videos for key pushes, and for short videos and live broadcasts and other timeliness requirements Higher scenarios are especially valuable.

The video playback test method provided in the embodiment of the present application involves artificial intelligence technology, which is briefly described below to facilitate understanding by those skilled in the art.

Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning, autonomous driving, and smart transportation.

Computer Vision Technology (Computer Vision, CV) Computer vision is a science that studies how to make machines "see". Further, it refers to the use of cameras and computers instead of human eyes to identify, track and measure targets. Machine vision, And further do graphics processing, so that computer processing becomes more suitable for human eye observation or transmission to the instrument detection image. As a scientific discipline, computer vision studies related theories and technologies, trying to build artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping Construction, autonomous driving, smart transportation and other technologies, as well as common biometric identification technologies such as face recognition and fingerprint recognition.

Machine Learning (ML) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or realize human learning behaviors to acquire new knowledge or skills, and to reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications are in all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, teaching learning and other technologies.

With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones , robots, intelligent medical care, intelligent customer service, Internet of Vehicles, autonomous driving, intelligent transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important value. In the embodiment of the method of the present application, the video playback quality can be tested and evaluated based on the media content of multiple information modalities through artificial intelligence technology, and multi-modal audio and video quality evaluation can be realized with the help of artificial intelligence technology.

In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

Please refer to FIG. 1 , which shows a schematic diagram of an application running environment provided by an embodiment of the present application. The application running environment may include: a terminal 10 and a server 20 .

The terminal 10 includes but is not limited to electronic devices such as mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, game consoles, e-book readers, multimedia playback devices, and wearable devices. A client in the terminal 10 that can install applications.

In this embodiment of the present application, the above-mentioned application program may be any application program that can provide a video playback service. Typically, the application is a video-type application. Of course, in addition to video applications, other types of applications can also provide video playback services. For example, news applications, social applications, interactive entertainment applications, browser applications, shopping applications, content sharing applications, virtual reality (Virtual Reality, VR) applications, augmented reality (Augmented Reality) applications , AR) applications, etc., which are not limited in the embodiments of the present application. In addition, for different applications, the video playback services involved will also be different, and the corresponding functions will also be different, which can be pre-configured according to actual needs, which is not limited in this embodiment of the present application . Optionally, the client terminal of the above application program runs in the terminal 10 .

The server 20 is used to provide background services for the client of the application program in the terminal 10 . For example, the server 20 may be the background server of the above application. The server 20 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, Cloud servers for basic cloud computing services such as middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the server 20 provides background services for applications in multiple terminals 10 at the same time.

Optionally, the terminal 10 and the server 20 can communicate with each other through the network 30 . The terminal 10 and the server 20 may be directly or indirectly connected through wired or wireless communication, which is not limited in this application.

Please refer to FIG. 2 , which shows a flowchart 1 of a method for testing video playback provided by an embodiment of the present application. The method can be applied to computer equipment, which refers to electronic equipment with data computing and processing capabilities. For example, the execution subject of each step may be the terminal 10 in the application running environment shown in FIG. 1 . The method may include the following steps (210-230).

Step 210: Obtain video playback information corresponding to the target video.

The above-mentioned target videos include but are not limited to offline videos, online videos, live videos, short videos, and other types of videos, which are not limited in this embodiment of the present application.

The video playback information includes visual information, auditory information corresponding to the target video, and data stream information corresponding to the target video being played by the target player. The data stream information is used to represent the processing quality of the data stream during the video playback process.

The video playback data refers to data associated with the playback quality of the target video. The video generation process mainly includes audio and video collection, audio and video encoding and compression, and audio and video encapsulation, so as to obtain a video file in a certain video format. The embodiments of the present application also do not limit the video format.

In an exemplary embodiment, the video playback process mainly includes audio and video decapsulation, audio and video decoding, audio and video synchronization and rendering. Among them, video playback can be performed through a video player, and the video player will perform operations such as audio and video decapsulation, decoding, audio and video synchronization, and rendering, and then generate the above-mentioned video playback information. Correspondingly, as shown in FIG. 3 , the implementation process of the above step 210 includes the following steps ( 211 to 213 ). FIG. 3 shows a second flowchart of a video playback testing method provided by an embodiment of the present application.

Step 211, in response to the video playback instruction, obtain at least two data streams corresponding to the target video, where the at least two data streams include an original video frame data stream and an original audio data stream.

For the video player, it is mainly to process the three data streams of video, audio and subtitles corresponding to the target video. For subtitles, some videos have subtitles embedded in the image, and no additional external subtitles are required, so the above target video corresponds to at least two data streams, namely the above-mentioned original video frame data stream and original audio data stream, the original video The frame data stream is used to render and display the sequence of video frames, and the raw audio data stream is used to render the playback audio signal.

Step 212: Perform image display processing on the original video frame data stream to obtain the video frame sequence displayed on the target page and the data stream information corresponding to the video frame sequence.

The above image display processing includes operations such as decapsulation, decoding, audio and video synchronization, and rendering of the original video frame data stream, so that the image information in the original video frame data stream is displayed on the target page.

A sequence of video frames is used to characterize visual information over image modalities. The video frame sequence is an image sequence composed of multiple consecutive video frames obtained by the player decoding the original video frame data of the target video. The above-mentioned video frame is the video image displayed on the target page after decoding, and is the visual information of the image mode that the user can see intuitively.

The data stream information corresponding to the video frame sequence includes at least one video playback quality indicator data that can reflect the playback quality of the target video in the video mode. The video playback quality indicator data in the above data stream information is the indicator data associated with the data processing performance of the player, including but not limited to the bitrate, frame rate, frame rate, frame rate, Fluency (fluency), decoding accuracy (decode accuracy), decoding time (decodetime) and so on. The calculation methods of each indicator and its role in measuring the playback quality of the video mode are as follows:

Code rate: code rate (unit kbps) = file size (KB)*8/time (s).

Frame rate (fps): Count the number of frames displayed per second.

Frame loss rate: The terminal receives the original video data package and assumes that the total number of frames is f. After decoding and rendering, the actual number of video frames on the screen is r. The frame loss rate is: (r-f)/f. Low, which proves that the player has fewer errors in decoding rendering.

Smoothness: The smoothness of video playback can be determined based on the above frame rate and frame loss rate. The higher the actual number of frames per second displayed during the playback of the target video, the higher the smoothness.

Decoding time: The time required to decode one video frame. For example, for different slice sources, HEVC slice source decoding takes longer than ordinary slice sources. Generally, hardware decoding takes less time than software decoding. Statistical decoding takes longer. Decoders are deployed on different device types of the terminal. Strategies help.

In one example, as shown in FIG. 4 , it exemplarily shows a schematic diagram of determining data of video playback quality indicators. Figure 4 shows the video cover of the movie whose file name is H264_864_486.mp4, whose size is 18.5MB (Mbyte, megabyte) and whose type is MPEG-4 (Moving Pictures Experts Group, Moving Picture Experts Group-4). picture. When playing the movie, the bit rate, frame rate, frame loss rate, and decoding time-consuming of the movie actually played by the client can be determined.

Step 213: Perform audio playback processing on the original audio data stream to generate an audio signal and data stream information corresponding to the audio signal.

The above audio playback processing includes operations such as decapsulation, decoding, audio and video synchronization, and rendering of the original audio data stream, so that the audio information in the original audio data stream is propagated to the user.

Audio signals are used to represent auditory information. The audio signal is a signal that is played after decoding and rendering, and the audio signal is auditory information of an audio mode that can be heard intuitively by a user.

The data stream information corresponding to the audio signal includes at least one audio playback quality indicator data corresponding to the audio mode. The audio playback quality indicator data in the above data stream information is indicator data related to the data processing performance of the player, including but not limited to the signal-to-noise ratio, fluency, short-term objectively understandable indicators, volume, etc. corresponding to the audio data stream. The calculation methods of each indicator are as follows:

Signal-to-Noise Ratio (SNR) has always been a conventional method to measure speech enhancement algorithms for wideband noise distortion. Therefore, SNR is mainly used in the simulation of algorithms for which both clean speech signals and noisy signals are known. The signal-to-noise ratio calculates the ratio of the average power of the speech signal to the noise signal over the entire time axis.

Fluency: Similar to the frame loss rate in video quality, assuming that the total duration of audio playback is t, and the duration of frame loss during the process is b, the fluency of a piece of audio is measured by (t-b)/t.

Short-Time Objective Intelligibility (STOI): The value ranges from 0 to 1. The larger the value, the higher the intelligibility.

In one example, as shown in FIG. 5 , it exemplarily shows a schematic diagram of determining the data of the audio playback quality indicator. For the audio signal in the target video, the signal-to-noise ratio, fluency, short-term objective intelligibility index, and volume corresponding to the audio signal can be determined.

In an exemplary embodiment, the above-mentioned at least two data streams further include original text data streams. Correspondingly, in order to process the above-mentioned original text data stream, as shown in FIG. 3 , the implementation process of the above-mentioned step 210 further includes the following step 214 .

Step 214: Perform text display processing on the original text data stream to obtain text content displayed on the target page and data stream information corresponding to the text content.

The text content is used to represent the visual information corresponding to the text modality. The text content in the video includes, but is not limited to, the text in the video frame image and the subtitles of the video. Subtitles refer to the display of video content in text form, including dialogue between video characters and descriptive language for pictures.

When the subtitles of the video are external subtitles or there are video comment texts, the above at least two data streams, in addition to the above-mentioned original video frame data stream and original audio data stream, will also include an original text data stream for displaying the target video. text information.

The data stream information corresponding to the text content includes at least one type of text playback quality indicator data corresponding to the text mode, and the text playback quality indicator data in the above-mentioned data stream information may be data associated with the data processing performance of the player, including but not limited to text Indicator data such as bit rate, fluency, and decoding time corresponding to the data stream.

It can be seen from the above content that the embodiments of the present application make full use of the audio-visual information of the target video and the audio-visual information in multiple modes and the data stream processing information in the player, so as to ensure the accuracy of the audio-video audio-visual quality test results. sex.

In addition, by starting a separate process to determine the video playback data during the user's video viewing process, the video playback quality evaluation can be performed during the user's viewing process. Information has the most authentic quality evaluation, which is of great help for developers to make decisions about the quality of algorithms and to discover and solve existing problems.

Step 220: Determine, according to the visual information, the auditory information, and the data stream information, playback quality feature information corresponding to the target video in at least two information modalities.

The above at least two information modalities include but are not limited to video modalities, audio modalities, image modalities, text modalities, and graphic modalities corresponding to the image modalities and the text modalities. Among them, the graphic mode refers to the fusion information mode corresponding to the image mode and the text mode.

When a user actually watches a video, the video quality is often judged from multiple dimensions. For example, if a video is played smoothly and the picture is clear, but there is no sound and subtitles at all, then the video is basically a video with zero viewing ability, but the current video quality evaluation angle is still very single, more from the perspective of images For the implementation, there is no effective quality evaluation system. Therefore, by acquiring the respective playback quality indicators corresponding to the above-mentioned various information modalities, the purpose of determining the video quality from multiple dimensions can be achieved.

For any one of the above two information modalities, the quality index data corresponding to the information modality can be obtained from the video playback data, and then the quality feature data of the target video on the information modality can be determined.

For video players, it is mainly for the processing of three data streams of video, audio and subtitles. Several information modalities associated with playback quality include: video, image, audio, and text. Therefore, visual information and auditory information can be and data flow information, perform feature extraction processing for each modal, and determine the quality feature data corresponding to these information modalities. This process can be simply understood as the process of multi-modal feature extraction.

In an exemplary embodiment, the visual information includes a video frame sequence corresponding to the target video, the auditory information includes an audio signal corresponding to the target video, and the data stream information includes data stream information corresponding to the video frame sequence and data stream information corresponding to the audio signal. Correspondingly, as shown in FIG. 3 , the implementation process of the above step 220 includes the following steps ( 221 - 223 ), which shows a second flowchart of the video playback testing method provided by an embodiment of the present application.

Step 221 , based on the data stream information corresponding to the video frame sequence, determine the playback quality feature information corresponding to the target video in the video mode.

The data stream information includes video playback quality indicator data associated with the player's processing performance, so the video playback quality feature vector can be determined based on at least one video playback quality indicator data in the data stream information.

In a possible implementation manner, the above-mentioned at least one video playback quality indicator data may be spliced to obtain the above-mentioned video playback quality feature vector.

In another possible implementation manner, feature extraction processing may be performed on the above at least one video playback quality indicator to obtain the above video playback quality feature vector.

The above-mentioned video playback quality feature vector may be used as playback quality feature information corresponding to the target video in the video mode.

The playback quality feature information corresponding to the target video in the video mode can also be determined according to the data stream information corresponding to the video frame sequence, audio signal, and text content. The above three kinds of data stream information can all represent the data of the player. Processing performance, cross-modal feature fusion of the above three data stream information can be used to represent the quality feature information of the video modality.

Optionally, the above-mentioned video playback quality feature vector is obtained by splicing the playback quality indicator data in the data stream information corresponding to the video frame sequence, audio signal, and text content. Optionally, feature extraction is performed on the playback quality indicator data in the data stream information corresponding to the video frame sequence, the audio signal, and the text content, to obtain the above-mentioned video playback quality feature vector.

Step 222 , based on the video frame sequence, determine the playback quality feature information corresponding to the target video in the image mode.

According to the image data of each video frame in the video frame sequence, at least one image quality indicator data can be determined, which is used to characterize the propagation quality of image modality information in the video content.

There are different dimensions for the measurement of image quality. For artworks, it is more inclined to aesthetics, rather than resolution, noise, etc.; for medical images, it is more concerned with the amount of image information, whether it reveals the crux of the patient; The video cover image is more concerned with whether it reflects the amount of video information to the greatest extent, and more attention is paid to important characters or plot scenes.

For the video playback process, the image content in the video frame sequence is diverse, so the image quality index data that can mainly reflect the image quality include but not limited to sharpness, resolution, color, texture blur, picture integrity, aesthetics relatively objective indicators such as degree. The above image quality indicator data can be obtained by performing corresponding analysis and processing on the image data of each video frame in the video frame sequence.

In one example, as shown in FIG. 6 , it exemplarily shows a schematic diagram of determining image quality index data. FIG. 6 shows 4 images with the same picture content but different tones, and these 4 images may be 4 video frames in the video. When the terminal plays the above four video frames, image quality index data such as definition, resolution, color, and picture integrity can be determined.

Based on at least one image quality indicator data, an image quality feature vector may be determined. In a possible implementation manner, the above-mentioned at least one image quality indicator data may be spliced to obtain the above-mentioned image quality feature vector. In another possible implementation manner, feature extraction processing may be performed on the above-mentioned at least one image quality index data to obtain the above-mentioned image quality feature vector.

The above image quality feature vector is used to represent the corresponding playback quality feature information of the target video on the image modality.

Step 223: Based on the audio signal and the data stream information corresponding to the audio signal, determine the playback quality feature information corresponding to the target video in the audio mode.

For video mode, the audio signal corresponding to each frame of image in the video is uncertain. The above-mentioned playback quality feature information of the video mode can be used as an independent evaluation dimension. Correspondingly, the playback quality feature information corresponding to the audio mode It can also be used as an independent evaluation dimension.

At least one audio quality indicator data associated with the audio content may be determined based on the content of the audio signal. The audio quality indicator data determined based on the content of the audio signal is the quality indicator data that has a weak correlation with the performance of the player, and pays more attention to the content itself, such as the type of audio content, the soundness of the audio content, etc. The signal is analyzed and processed.

The data stream information corresponding to the audio signal includes audio quality indicator data associated with the performance of the player, such as indicator data such as signal-to-noise ratio and fluency.

Based on the above audio quality indicator data, it may include either quality indicator data determined according to the content of the audio signal or audio quality indicator data included in the data stream information corresponding to the audio signal, and an audio playback quality feature vector may be determined.

In a possible implementation manner, the above-mentioned at least one audio content quality indicator data and the audio quality indicator data included in the data stream information corresponding to the audio signal may be spliced to obtain the above-mentioned audio playback quality feature vector.

In another possible implementation, feature extraction may be performed on the at least one audio content quality indicator data and the audio quality indicator data included in the data stream information corresponding to the audio signal to obtain the audio playback quality feature vector.

The above-mentioned audio playback quality feature vector may be used as playback quality feature information of the target video in the audio mode.

In an exemplary embodiment, the visual information further includes text content corresponding to the target video, and the data flow information further includes data flow information corresponding to the text content. Correspondingly, as shown in FIG. 3 , the implementation process of the above step 220 further includes the following steps (224-225).

Step 224 , based on the text content and the data stream information corresponding to the text content, determine the playback quality feature information corresponding to the target video in the text mode.

Based on the text content and the data flow information corresponding to the text content, at least one kind of text quality indicator data can be determined, which is used to characterize the dissemination quality of the text modal information. The above-mentioned text quality index data includes quality index data for measuring the text content and index data associated with the data processing performance of the player in the data stream information corresponding to the text content.

For text in video, such as subtitles, indicators that can measure the quality of subtitles include completeness, clarity, and text quality. The completeness is used to characterize whether the video contains subtitles, and further for foreign language episodes, the completeness is used to characterize whether the video has multilingual subtitles. Clarity includes subtitle clarity, which can represent the rendering effect of embedded and external subtitles. For example, if a 1080P video renders a 270P subtitle, the picture effect will be more inconsistent. The text quality can be determined according to the actual text content or the text generation method. For example, the quality of subtitles provided by a subtitle group and machine translation subtitles is quite different.

In an example, as shown in FIG. 7 , it exemplarily shows a schematic diagram of determining text quality index data. Figure 7 shows that the video frame image has subtitle content "Mother, do you remember?", the terminal can detect the subtitle content in the video image to determine the completeness, definition and text quality of the subtitle.

According to the above at least one text quality indicator data, a text quality feature vector can be determined. In a possible implementation manner, the above-mentioned at least one text quality indicator data may be spliced to obtain the above-mentioned text quality feature vector. In another possible implementation manner, feature extraction processing may be performed on the above-mentioned at least one image quality index data to obtain the above-mentioned text quality feature vector.

Step 225: Perform feature information fusion processing on the playback quality feature information corresponding to the image mode and the playback quality feature information corresponding to the text mode to obtain playback quality feature information corresponding to the target video in the graphic mode.

Graphical and textual modalities refer to the fusion information modalities corresponding to the image modalities and the text modalities.

Feature fusion processing is performed on the image quality feature vector and the text quality feature vector to obtain the image and text playback quality feature vector. The above-mentioned graphic and text playback quality feature vector is used to represent the playback quality feature information of the target video in the graphic and text mode.

For a frame of image in the video, it often has associated text information, such as subtitles or bullet screens, so the features corresponding to the image mode and the text mode are strongly correlated features, so the currently embedded subtitle features and image features can be combined. Stitching is performed as input to the subsequent playback quality test model. The feature fusion processing of the image quality feature vector and the text quality feature vector can use the bilinear pooling method, that is, by calculating the outer product of the two, and by linearizing the matrix generated by the outer product into a vector representation, the above graphic playback can be obtained. The quality feature vector is the fusion feature of the two.

Through the above process, multi-modal playback quality feature information can be extracted, so that the quality feature information extracted from various modalities can be used in the evaluation of audio-visual information test results.

Step 230 , based on the corresponding playback quality feature information on the at least two information modalities, generate an audiovisual quality test result corresponding to the target video being played by the target player.

The above audiovisual quality test results are used to characterize the audiovisual information dissemination quality corresponding to the target video on the target player.

In an exemplary embodiment, the audiovisual quality test result includes a corresponding playback quality attribution score of the target video on each of the at least two information modalities, and the playback quality attribution score is used to characterize single-modality information in the target video transmission quality.

Correspondingly, as shown in FIG. 3 , the implementation process of the above step 230 includes the following step 231 .

Step 231 , based on the playback quality feature information corresponding to each of the at least two information modalities, determine the corresponding playback quality attribution score of the target video on each of the information modalities.

After determining the playback quality feature information on the above-mentioned various information modalities, the pre-training process of multi-modal playback quality testing such as video, audio, text, image, and graphic text can be performed to obtain the quality evaluation of each information modal. Model.

In an exemplary embodiment, for the video modality in each information modality, the playback quality feature information of the target video on the video modality, that is, the video playback quality feature vector, is input to the video modality quality evaluation model, and the video modality is output. state quality score. The above-mentioned video modal quality is divided into the attribution score of the playback quality corresponding to the target video on the video modal, which is used to characterize the dynamic information dissemination quality in the target video and the video modal.

Among them, the above-mentioned video modal quality evaluation model takes the playback quality feature information of the sample video on the video modality as the training feature, that is, the above-mentioned video playback quality feature vector, and uses the label score as the label information. It is used to test the playback quality of the video mode information and output the quantization test result corresponding to the video mode, that is, the above-mentioned video mode quality score.

In an exemplary embodiment, for the audio modality in each information modality, the playback quality feature information of the target video on the audio modality, that is, the audio playback quality feature vector, is input into the audio modality quality evaluation model, and the audio modality is output. state quality score. The above-mentioned audio modality quality is divided into attribution scores of playback quality corresponding to the audio modality of the target video, which are used to characterize the propagation quality of audio modality information in the target video.

Among them, the above-mentioned audio modality quality evaluation model is a machine learning model that uses the playback quality feature information of the sample video on the audio modality as the training feature, that is, the audio playback quality feature vector, and uses the label score as the label information for training. The playback quality test of the audio modality information is performed, and the quantized test result corresponding to the audio modality is output, that is, the above-mentioned audio modality quality score.

In an exemplary embodiment, for the image modality in each information modality, the playback quality feature information of the target video on the image modality, that is, the image quality feature vector, is input into the image modality quality evaluation model, and the image modality is output. quality score. The above-mentioned image modality quality is divided into the attribution score of the playback quality corresponding to the image modality of the target video, which is used to characterize the dissemination quality of the image modality information in the target video.

Among them, the above-mentioned image modal quality evaluation model is a machine learning model that uses the playback quality feature information of the sample video on the image modality as the training feature, that is, the image quality feature vector, and uses the label score as the label information for training. Playing quality test of image modality information and outputting a quantified test result corresponding to the image modality, that is, the above-mentioned image modality quality score.

In an exemplary embodiment, for the text modality in each information modality, the playback quality feature information of the target video on the text modality, that is, the text quality feature vector, is input into the text modality quality evaluation model, and the text modality is output. quality score. The above-mentioned text modal quality is divided into the attribution score of the playback quality corresponding to the target video on the text modal, which is used to characterize the dissemination quality of the text modal information in the target video.

Among them, the above text modal quality evaluation model is a machine learning model that uses the playback quality feature information of the sample video on the text modality as the training feature, that is, the text quality feature vector, and uses the label score as the label information for training. Playing quality test of text modal information and outputting the quantitative test result corresponding to the text modal, that is, the above-mentioned text modal quality score.

In an exemplary embodiment, for the graphic mode in each information modality, the playback quality feature information of the target video on the graphic mode, that is, the graphic playback quality feature vector, is input into the graphic mode quality evaluation model , and output the modal quality score of the image and text. The above-mentioned image-text modal quality is divided into the attribution score of the playback quality corresponding to the target video on the image-text modal, which is used to characterize the dissemination quality of the joint information in the target video and the image-text modal.

Among them, the above-mentioned image-text modal quality evaluation model is a machine learning model that uses the playback quality feature information of the sample video on the graphic-text modality as the training feature, that is, the image-text playback quality feature vector, and uses the label score as the label information for training. , which is used to test the playback quality of the joint information of the graphic and text modes and output the quantized test results corresponding to the graphic and text modes, that is, the above-mentioned graphic and text mode quality scores.

In an exemplary embodiment, the audiovisual quality test result further includes an overall audiovisual quality score corresponding to the target video, and the overall audiovisual quality score is used to overall characterize the audiovisual information dissemination quality corresponding to the target video on the target player.

Correspondingly, as shown in FIG. 3 , after the above step 231 , the above method further includes the following step 232 .

Step 232: Perform fusion processing on the corresponding playback quality attribution scores on each information modality to obtain an overall audio-visual quality score.

In an exemplary embodiment, the above-mentioned image and text modal quality scores, audio modal quality scores, and video modal quality scores are weighted and averaged to obtain an overall audio-visual quality score. Optionally, the weight coefficients of the above-mentioned quality attribution points can be adjusted according to the actual situation. For example, the weight coefficients corresponding to the three playback quality attribution points are each 1/3. By calculating the above overall audiovisual quality score, and at the same time retaining the corresponding playback quality attribution score of each mode, it is convenient to provide video developers for reference and optimize in the dimension of poor quality.

In an exemplary embodiment, in the case where the image modality and the text modality are not fused, the above-mentioned audio modality quality score, video modality quality score, image modality quality score, and text modality quality score may be weighted and averaged , get the overall score of audio-visual quality.

In one example, as shown in FIG. 8 , it exemplarily shows a schematic diagram of a multimodal quality testing network. Input the playback data corresponding to the four modes of voice, image, subtitle and video in the video playback process to the voice network, image network, text network and video network respectively, and extract the media content of each mode from the corresponding network. The playback quality features are the voice features, image features, text features and video features shown in the figure, and then multi-modal feature weighted fusion based on the above voice features, image features, text features and video features can be achieved. To quantify the video playback quality. Based on the above-mentioned multi-modal quality testing network, the multi-dimensional features such as video, picture, audio, and text can be integrated when testing the video playback quality of the terminal, and different modalities can be mapped to a unified semantic space through their corresponding encoder feature networks. integrated into stable multimodal representations. By deploying the above multimodal quality test network, in addition to the quality evaluation of different information modalities such as video, audio, picture, and subtitles, a comprehensive quality score can also be given, which can be closer to the subjective quality evaluation of users when watching videos.

In another embodiment, cross-modal feature fusion processing may be performed directly on the playback quality feature information on at least two information modalities to obtain modality fusion feature information.

The above-mentioned cross-modal feature fusion processing refers to the processing of merging and mapping playback quality feature information corresponding to different modalities to the same feature space.

Under the premise that there is a cross-modal semantic gap, the playback quality features on various information modalities can be processed by tag sharing. For example, the features xa and xs corresponding to the audio modality and the text modality belong to different categories, and the subtitle feature xs can be forced to share the same label k as the audio xa. Combining embeddings through the learning of cross-modal features to narrow the possible semantic gaps between information modalities and capture task-relevant semantics to facilitate more knowledge transfer. Optionally, the above tag sharing rule includes tag sharing according to the video playback time information corresponding to the data of each information modality. The above tag may be a marked score, such as a user's playback quality score for the video playback content at a certain moment.

In a possible implementation manner, the above-mentioned video playback quality feature vector, audio playback feature vector, image quality feature vector, and text quality feature vector are multi-modally fused to obtain modal fusion feature data. The above vector fusion methods include, but are not limited to, superposition, splicing, combined embedding, and the like.

Specifically, the feature vector of video playback quality, the feature vector of audio playback, the feature vector of image quality, and the feature vector of text quality are spliced to obtain a modal fusion feature vector. Alternatively, a feature vector of modal fusion is obtained by superimposing the feature vector of video playback quality, the feature vector of audio playback, the feature vector of image quality, and the feature vector of text quality. Alternatively, the video playback quality feature vector, the audio playback feature vector, the image quality feature vector, and the text quality feature vector are combined and embedded to obtain a modal fusion feature vector. The above combined embedding process can be implemented by a multimodal knowledge transfer learning model.

Based on the modal fusion feature information, the overall audio-visual quality score corresponding to the target video is determined.

In a possible implementation, the above-mentioned modal fusion feature information, that is, the modal fusion feature vector, is input into the multimodal quality test model, and the overall audiovisual quality score corresponding to the target video is output. Optionally, the above-mentioned multi-modal quality test model is a machine learning model obtained by performing distillation training on the pre-training model according to the sample features and shared labels of each modal quality test model. Optionally, the above-mentioned shared label refers to the above-mentioned forced shared label.

By performing semantic fusion of different modalities through the extracted multi-dimensional playback quality features, the knowledge in the modal fusion features learned by each modal can be distilled into a general system, which can then be obtained according to the fusion feature knowledge. Comprehensive quality evaluation results of video playback.

In an example, as shown in FIG. 9 , it exemplarily shows a schematic diagram of an audio and video quality testing process in a process of a terminal playing a video. The user uses the player software installed in the terminal to play the video; after the terminal plays the video, the multi-modal quality test process runs in the background. The multi-modal quality test system can perform multi-dimensional evaluation of video playback, including different types of film sources, or newly launched video terminal processing algorithms, playback quality, and video content quality. In the process of multi-modal quality testing, for the information of each modality of video, image, audio and subtitle, the respective quality feature knowledge can be extracted from the corresponding network, and then synthesized and embedded into the multi-modal network to obtain the representation. The video score of audiovisual knowledge, that is, the above-mentioned overall score of playback quality. The above video score can be understood as the quantitative quality test result corresponding to the player strategy. Optionally, the above-mentioned multi-modal network is a neural network model obtained by performing machine learning training based on the multi-modal pre-training model. For the video playback quality test, the multi-modal pre-training model is used to evaluate the quality of multi-dimensional information content such as video, image, text, and voice contained in the video, and the audio-visual knowledge corresponding to the multi-modal information can be evaluated. Fusion makes the test score closer to the user's subjective quality evaluation when watching videos.

In an exemplary embodiment, as shown in FIG. 3 , after the above step 230 , the method further includes the following step 240 .

Step 240 , based on the audio-visual quality test result, generate policy adjustment information for the data processing policy information configured by the target player.

The data processing policy information includes at least one data processing policy for the data stream, and the policy adjustment information is used to adjust the data processing policy.

The player is configured with a data processing strategy library for processing various data streams. The data processing strategy library includes data strategies for various modal information.

For video modes, decoders for different encoding formats are configured in the player, such as decoders corresponding to H265 and H264 encoding modes. Data processing strategies for different decoding methods can also be configured in the player, such as hard decoding processing strategies MediaCodec, VideoToolbox, and soft decoding processing strategies FFmpeg. Among them, MediaCodec is the hardware encoder of the Android (Android operating system) platform; VideoToolbox is the video processing framework of the iOS platform; FFmpeg is a set of open source computers that can be used to record, convert digital audio and video, and convert them into streams. program.

For image modes, different image processing strategies are configured in the player, such as super resolution, super frame rate, color blindness, HDR (High-Dynamic Range), VR and other image processing strategies.

For audio mode, different audio processing strategies are configured in the player, such as denoising, echo cancellation, sound effect processing, power amplifier/enhancement, sound mixing/separation and other audio processing strategies.

For text mode, different text processing strategies are configured in the player, such as fonts, embedded/external subtitles and other text processing strategies.

The above-mentioned data processing strategy for the data stream of the currently playing video may be the processing strategy set by the user in the settings page of the player, such as selecting an image special effect, or it may be set by the user in the operation bar on the video playback page. , such as setting double-speed playback, resolution and other operations.

The above audiovisual quality test result can reflect the quality of the data processing strategy adopted by the player to play the video. Therefore, strategy adjustment information can be generated according to the above audiovisual quality test result to adjust the data processing strategy adopted by the player to play the video. For example, after the user sets multi-speed playback, the video playback quality may be low due to the large data calculation pressure. Therefore, based on the above audio-visual quality test results, strategy adjustment information can be generated to prompt the user to set normal multi-speed playback.

The player can also automatically adjust the currently used data processing strategy according to the above strategy adjustment information.

In the scenario of online video service, the above-mentioned test process can be performed synchronously during the video playback. Based on the above-mentioned multimodal quality test model, real-time evaluation of online video playback quality can be realized, so as to find problems in time. After obtaining the above video playback test result, the terminal may upload the above video playback test result to the server to help the developer to analyze. Developers can make related optimizations based on video playback test results, such as technical architecture design, encoding selection, streaming media protocols, adaptive algorithms, connection and freeze logic, client software design, etc.

To sum up, the technical solutions provided by the embodiments of the present application, by acquiring the visual information and auditory information perceivable by the user in the process of playing the video by the player, and the data stream information reflecting the data processing quality of the target player, it can be determined that the video is in at least two The playback quality features on various information modalities, and by using the playback quality features on at least two information modalities above to generate audiovisual quality test results that can characterize the quality of audiovisual information dissemination, the playback quality of media content in different information modalities can be fully explored. information to ensure that the test results conform to the subjective evaluation of users, avoid the problem of inaccurate test results caused by a single information dimension, improve the accuracy of audio and video quality testing, and determine the video playback quality without manual observation, reducing the audio and video quality. Labor costs required for testing.

The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

Please refer to FIG. 10 , which shows a block diagram of a video playback test apparatus provided by an embodiment of the present application. The device has the function of implementing the above-mentioned video playback test method, and the function can be implemented by hardware or by executing corresponding software by hardware. The apparatus may be computer equipment, or may be provided in computer equipment. The apparatus 1000 may include: a playback information acquisition module 1010 , a quality feature determination module 1020 and a test result generation module 1030 .

The playback information acquisition module 1010 is configured to acquire video playback information corresponding to the target video, where the video playback information includes visual information, auditory information corresponding to the target video, and data stream information corresponding to the target video being played by the target player. The data stream information is used to characterize the processing quality of the data stream during video playback.

The quality feature determination module 1020 is configured to determine, according to the visual information, the auditory information and the data stream information, the playback quality feature information corresponding to the target video in at least two information modalities.

The test result generation module 1030 is configured to generate the audiovisual quality test result corresponding to the target player playing the target video based on the corresponding playback quality feature information on the at least two information modalities. It is used to characterize the corresponding audiovisual information dissemination quality of the target video on the target player.

In an exemplary embodiment, the playback information acquisition module 1010 includes: a data stream acquisition unit, a video frame stream processing unit, and an audio stream processing unit.

A data stream acquisition unit, configured to acquire at least two data streams corresponding to the target video in response to a video playback instruction, where the at least two data streams include an original video frame data stream and an original audio data stream.

A video frame stream processing unit, configured to perform image display processing on the original video frame data stream to obtain a video frame sequence displayed on the target page and data stream information corresponding to the video frame sequence, and the video frame sequence is used to represent Visual information on image modalities.

An audio stream processing unit, configured to perform audio playback processing on the original audio data stream, and generate an audio signal and data stream information corresponding to the audio signal, where the audio signal is used to represent the auditory information.

In an exemplary embodiment, the at least two data streams further include original text data streams, and the playback information acquisition module 1010 further includes: a text stream processing unit.

A text stream processing unit, configured to perform text display processing on the original text data stream to obtain text content displayed in the target page and data stream information corresponding to the text content, the text content being used to represent a text model corresponding visual information.

In an exemplary embodiment, the audiovisual quality test result includes a corresponding playback quality attribution score of the target video on each of the at least two information modalities, and the playback quality attribution score is used for Characterizing the propagation quality of the single-modal information in the target video; the test result generation module 1130 includes: an attribution score determination unit.

An attribution score determination unit, configured to determine, based on the playback quality feature information corresponding to each of the at least two information modalities, the playback quality attribution score of the target video corresponding to each of the information modalities .

In an exemplary embodiment, the audio-visual quality test result further includes an overall audio-visual quality score corresponding to the target video, and the overall audio-visual quality score is used to overall characterize that the target video corresponds to the target video on the target player The audio-visual information dissemination quality; the test result generation module 1130 further includes: an overall score determination unit.

The overall score determination unit is configured to perform fusion processing on the corresponding playback quality attribution scores on each of the information modalities to obtain the overall audiovisual quality score.

In an exemplary embodiment, the visual information includes a video frame sequence corresponding to the target video, the auditory information includes an audio signal corresponding to the target video, and the data stream information includes data corresponding to the video frame sequence Stream information and data stream information corresponding to the audio signal;

The quality feature determination module 1020 includes: a video modality feature determination unit, an image modality feature determination unit, and an audio modality feature determination unit.

A video modality feature determining unit, configured to determine, based on the data stream information corresponding to the video frame sequence, the playback quality feature information corresponding to the target video on the video modality.

An image modality feature determination unit, configured to determine, based on the video frame sequence, the playback quality feature information corresponding to the target video on the image modality.

An audio modality feature determining unit, configured to determine, based on the audio signal and data stream information corresponding to the audio signal, the playback quality feature information corresponding to the target video in the audio modality.

In an exemplary embodiment, the visual information further includes text content corresponding to the target video, the data stream information further includes data stream information corresponding to the text content, and the quality feature determining module 1020 further includes: a text model Modal feature determination unit, graphic modal feature determination unit.

a text modality feature determining unit, configured to determine the playback quality feature information corresponding to the target video on the text modality based on the text content and the data stream information corresponding to the text content;

The graphic and text modal feature determination unit is configured to perform feature information fusion processing on the corresponding playback quality feature information on the image modality and the corresponding playback quality feature information on the text modality to obtain the target video in the graphic and text format. The corresponding playback quality feature information on the modality, the graphic modality refers to the fusion information modality corresponding to the image modality and the text modality.

In an exemplary embodiment, the apparatus 1000 further includes: a playback strategy adjustment module.

A playback strategy adjustment module, configured to generate strategy adjustment information for the data processing strategy information configured for the target player based on the audiovisual quality test result;

The data processing policy information includes at least one data processing policy for the data stream, and the policy adjustment information is used to adjust the data processing policy.

To sum up, the technical solutions provided by the embodiments of the present application, by acquiring the visual information and auditory information perceivable by the user in the process of playing the video by the player, and the data stream information reflecting the data processing quality of the target player, it can be determined that the video is in at least two The playback quality features on various information modalities, and by using the playback quality features on at least two information modalities above to generate audiovisual quality test results that can characterize the quality of audiovisual information dissemination, the playback quality of media content in different information modalities can be fully explored. information to ensure that the test results conform to the subjective evaluation of users, avoid the problem of inaccurate test results caused by a single information dimension, improve the accuracy of audio and video quality testing, and determine the video playback quality without manual observation, reducing the audio and video quality. Labor costs required for testing.

It should be noted that, when implementing the functions of the device provided in the above-mentioned embodiments, only the division of the above-mentioned functional modules is used as an example. The internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process thereof is detailed in the method embodiments, which will not be repeated here.

Please refer to FIG. 11 , which shows a structural block diagram of a computer device provided by an embodiment of the present application. The computer device may be a terminal. The computer device is used to implement the video playback test method provided in the above embodiment. Specifically:

Generally, computer device 1100 includes: processor 1101 and memory 1102 .

The processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1101 may be implemented in at least one hardware form among DSP (Digital Signal Processing, digital signal processing), FPGA (FieldProgrammable Gate Array, field programmable gate array), and PLA (Programmable Logic Array, programmable logic array). The processor 1101 may also include a main processor and a coprocessor. The main processor is a processor used to process data in a wake-up state, also called a CPU (Central Processing Unit, central processing unit); A low-power processor for processing data in a standby state. In some embodiments, the processor 1101 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing the content that needs to be displayed on the display screen. In some embodiments, the processor 1101 may further include an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is used to process computing operations related to machine learning.

Memory 1102 may include one or more computer-readable storage media, which may be non-transitory. Memory 1102 may also include high-speed random access memory, as well as non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, a non-transitory computer-readable storage medium in memory 1102 is used to store at least one instruction, at least one program, code set or instruction set, the at least one instruction, at least one program, code set or instruction set and configured to be executed by one or more processors to implement the above-described testing method for video playback.

In some embodiments, the computer device 1100 may also optionally include: a peripheral device interface 1103 and at least one peripheral device. The processor 1101, the memory 1102 and the peripheral device interface 1103 may be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1103 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1104 , a touch display screen 1105 , a camera assembly 1106 , an audio circuit 1107 , a positioning assembly 1108 and a power supply 1109 .

Those skilled in the art can understand that the structure shown in FIG. 11 does not constitute a limitation on the computer device 1100, and may include more or less components than those shown, or combine some components, or adopt different component arrangements.

In an exemplary embodiment, a computer-readable storage medium is also provided, wherein the storage medium stores at least one instruction, at least one piece of program, code set or instruction set, the at least one instruction, the at least one piece of program . When the code set or the instruction set is executed by the processor, the above-mentioned testing method for video playback is implemented.

Optionally, the computer-readable storage medium may include: ROM (Read Only Memory, read only memory), RAM (Random Access Memory, random access memory), SSD (Solid State Drives, solid state hard disk), or an optical disc. The random access memory may include ReRAM (Resistance Random Access Memory, resistive random access memory) and DRAM (Dynamic Random Access Memory, dynamic random access memory).

In an exemplary embodiment, there is also provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned video playback test method.

It should be understood that references herein to "a plurality" means two or more. "And/or", which describes the association relationship of the associated objects, means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects are an "or" relationship. In addition, the numbering of the steps described in this document only exemplarily shows a possible execution sequence between the steps. In some other embodiments, the above steps may also be executed in different order, such as two different numbers. The steps are performed at the same time, or two steps with different numbers are performed in a reverse order to that shown in the figure, which is not limited in this embodiment of the present application. The above are only exemplary embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection of the present application. within the range.

Claims (10)

1. A method for testing video playing, the method comprising:

acquiring video playing information corresponding to a target video, wherein the video playing information comprises visual information and auditory information corresponding to the target video and data stream information corresponding to the target video played by a target player, and the data stream information is used for representing the processing quality of a data stream in the video playing process;

according to the visual information, the auditory information and the data stream information, determining corresponding playing quality characteristic information of the target video on at least two information modalities;

and generating an audio-visual quality test result corresponding to the target video played by the target player based on the playing quality characteristic information corresponding to the at least two information modes, wherein the audio-visual quality test result is used for representing the audio-visual information transmission quality of the target video corresponding to the target player.

2. The method according to claim 1, wherein the obtaining video playing information corresponding to the target video includes:

responding to a video playing instruction, and acquiring at least two data streams corresponding to the target video, wherein the at least two data streams comprise an original video frame data stream and an original audio data stream;

performing image display processing on the original video frame data stream to obtain a video frame sequence displayed on a target page and data stream information corresponding to the video frame sequence, wherein the video frame sequence is used for representing visual information on an image modality;

and carrying out audio playing processing on the original audio data stream to generate an audio signal and data stream information corresponding to the audio signal, wherein the audio signal is used for representing the auditory information.

3. The method according to claim 2, wherein the at least two data streams further include an original text data stream, and after the at least two data streams corresponding to the target video are obtained in response to the video playing instruction, the method further includes:

and performing text display processing on the original text data stream to obtain text content displayed in the target page and data stream information corresponding to the text content, wherein the text content is used for representing visual information corresponding to a text mode.

4. The method according to claim 1, wherein the audiovisual quality test result comprises a corresponding playback quality attribute of the target video in each of the at least two information modalities, and the playback quality attribute is used for characterizing the propagation quality of single-modality information in the target video;

generating an audio-visual quality test result corresponding to the target video played by the target player based on the playing quality characteristic information corresponding to the at least two information modalities, including:

and determining the corresponding play quality attribution of the target video on each information modality based on the corresponding play quality characteristic information on each information modality of the at least two information modalities.

5. The method according to claim 4, wherein the audiovisual quality test result further includes an audiovisual quality overall score corresponding to the target video, and the audiovisual quality overall score is used to represent audiovisual information propagation quality corresponding to the target video on the target player as a whole;

after determining the playback quality attribution of the target video corresponding to each information modality based on the playback quality characteristic information corresponding to each information modality of the at least two information modalities, the method further includes:

and carrying out fusion processing on the corresponding play quality attribution points on each information modality to obtain the integral audio-visual quality points.

6. The method of claim 1, wherein the visual information comprises a sequence of video frames corresponding to the target video, wherein the auditory information comprises an audio signal corresponding to the target video, and wherein the data stream information comprises data stream information corresponding to the sequence of video frames and data stream information corresponding to the audio signal;

the determining, according to the visual information, the auditory information, and the data stream information, the playing quality feature information corresponding to the target video in at least two information modalities includes:

determining corresponding playing quality characteristic information of the target video on a video modality based on data stream information corresponding to the video frame sequence;

determining corresponding playing quality characteristic information of the target video on an image modality based on the video frame sequence;

and determining corresponding playing quality characteristic information of the target video on an audio modality based on the audio signal and the data stream information corresponding to the audio signal.

7. The method according to claim 6, wherein the visual information further includes text content corresponding to the target video, the data stream information further includes data stream information corresponding to text content, and wherein determining the corresponding play quality characteristic information of the target video in at least two information modalities according to the visual information, the auditory information and the data stream information further comprises:

determining playing quality characteristic information corresponding to the target video on a text mode based on the text content and the data stream information corresponding to the text content;

and performing feature information fusion processing on the playing quality feature information corresponding to the image modality and the playing quality feature information corresponding to the text modality to obtain the playing quality feature information corresponding to the target video in the image-text modality, wherein the image-text modality refers to a fusion information modality corresponding to the image modality and the text modality.

8. The method according to any one of claims 1 to 7, further comprising:

generating policy adjustment information for the data processing policy information configured for the target player based on the audiovisual quality test result;

wherein the data processing policy information includes at least one data processing policy for the data flow, and the policy adjustment information is used to adjust the data processing policy.

9. A device for testing video playback, the device comprising:

the playing information acquisition module is used for acquiring video playing information corresponding to a target video, wherein the video playing information comprises visual information and auditory information corresponding to the target video and data stream information corresponding to the target video played by a target player, and the data stream information is used for representing the processing quality of a data stream in the video playing process;

the quality characteristic determining module is used for determining corresponding playing quality characteristic information of the target video on at least two information modalities according to the visual information, the auditory information and the data stream information;

and the test result generation module is used for generating an audio-visual quality test result corresponding to the target video played by the target player based on the playing quality characteristic information corresponding to the at least two information modalities, wherein the audio-visual quality test result is used for representing the audio-visual information transmission quality corresponding to the target video on the target player.

10. A computer device comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by the processor to implement a method of testing video playback as claimed in any of claims 1 to 8.

Images