WO2022179306A1

WO2022179306A1 - Audio/video playing method and apparatus, and electronic device

Info

Publication number: WO2022179306A1
Application number: PCT/CN2021/143557
Authority: WO
Inventors: 罗诚; 李刚; 周凡; 向宇
Original assignee: 华为技术有限公司
Priority date: 2021-02-26
Filing date: 2021-12-31
Publication date: 2022-09-01
Also published as: CN114979783A; CN114979783B

Abstract

Provided are an audio/video playing method and apparatus, and an electronic device. The audio/video playing method comprises: recognizing the current application scenario; and according to the current application scenario, configuring a playing policy for audio playing and/or video image playing, including: when the current application scenario is an interactive application scenario, configuring a catch-up policy as the playing policy for audio playing and/or video image playing. According to the method of the embodiments of the present application, an audio/video playing scenario is recognized, and a corresponding playing policy is selected according to a specific application scenario, such that the user experience of audio/video playing can be greatly improved.

Description

A kind of audio and video playback method, device and electronic equipment

technical field

The present application relates to the technical field of intelligent terminals, and in particular, to an audio and video playback method, device, and electronic device.

Background technique

With the development of distributed technology, there are more and more distributed cross-device services centered on smartphones. In the current distributed cross-device business, a common application scenario is the audio and video separation application scenario. Different collection devices are used to collect audio data and video image data respectively, and different collection devices are used to play audio data and video image data respectively. . For example, the mobile phone cooperates with the surrounding large screen, camera, microphone, and speaker to realize the video telephony service with the remote end.

In the audio and video separation application scenario, the audio data and video image data are respectively transmitted from the data output end to the audio playback end (eg, a smart speaker) and the video image playback end (eg, a large-screen TV) for playback. In the transmission of audio data and video image data, there are unfavorable factors such as transmission errors, unstable transmission signal strength, transmission delay, etc. These unfavorable factors will cause playback freezes and playback delays in audio playback and video image playback. This greatly reduces the user experience.

SUMMARY OF THE INVENTION

Aiming at the problem of poor user experience in the application scenario of audio and video separation in the prior art, the present application provides an audio and video playback method, device and electronic device; the present application also provides an audio playback method, device and electronic device; The present application also provides a video image playback method, apparatus and electronic device; the present application also provides a computer-readable storage medium.

The embodiment of the present application adopts the following technical solutions:

In a first aspect, the present application provides a method for playing audio and video, including:

Identify the current application scenario;

Configure a playback strategy for audio playback and/or video image playback according to the current application scenario, including:

When the current application scenario is an interactive application scenario, the chasing strategy is configured as a playback strategy for audio playback and/or video image playback, wherein the chasing strategy includes:

Adjust the audio playback and/or the video image playback, so that the playback progress of the audio playback and/or the video image playback catches up with the audio and video data output progress, so that the audio playback and/or the video playback progress The playback delay of the image playback compared to the audio and video data output is less than or equal to the preset interactive scene playback delay threshold.

According to the method of the embodiment of the present application, the audio and video playback scenarios are identified, and the corresponding playback strategy is selected according to the specific application scenarios, which can greatly improve the user experience of audio and video playback.

Further, according to the method of the embodiment of the present application, using a catch-up strategy for audio and video playback in interactive application scenarios can ensure that the playback delay of audio and video playback meets the application requirements of interactive application scenarios, thereby greatly improving the user experience of interactive application scenarios. experience.

In a feasible application scheme of the above-mentioned first aspect, the method further includes:

When the current application scenario is a non-interactive application scenario, a smooth playback strategy is configured as a playback strategy for the audio playback and/or the video image playback.

According to the method of the embodiment of the present application, the catch-up strategy is used to play audio and video for non-interactive application scenarios, which can ensure that the smoothness of audio and video playback meets the application requirements of non-interactive application scenarios, thereby greatly improving the user experience of interactive application scenarios. .

In a feasible application solution of the above first aspect, the configuration catch-up strategy is a playback strategy for audio playback and/or video image playback, including:

The chasing strategy is configured as a playing strategy of the audio playing and the video image playing.

Configuring the chasing strategy to be the playback strategy of the audio playback;

The configuration synchronization strategy is the playback strategy of the video image playback, wherein the synchronization strategy includes:

Based on the playback progress of the audio playback, the video image playback is adjusted so that the playback progress of the video image playback is synchronized with the playback progress of the audio playback.

Configuring the chasing strategy to be a playback strategy of the video image playback;

The configuration synchronization strategy is the playback strategy of the audio playback, wherein the synchronization strategy includes:

The audio playback is adjusted based on the playback progress of the video image playback, so that the playback progress of the audio playback is synchronized with the playback progress of the video image playback.

In a second aspect, the present application provides an audio playback method, comprising:

When the playback strategy of audio playback is the catch-up strategy, obtain the preset interactive scene playback delay threshold;

The audio playback is performed based on the interactive scene playback delay threshold, and during the execution of the audio playback, the audio playback is adjusted so that the playback progress of the audio playback catches up with the output progress of the audio and video data, so that the audio playback progresses The playback delay of audio playback compared to the audio and video data output is less than or equal to the preset interactive scene playback delay threshold.

In a feasible application solution of the second aspect, the audio playback based on the interactive scene playback delay threshold includes:

Monitoring the playback delay of the audio playback relative to the audio and video data output;

When the playback delay of the audio playback exceeds the interactive scene playback delay threshold, the audio playback is adjusted so that the playback delay of the audio playback is less than or equal to the interactive scene playback delay threshold.

Monitor the data buffer amount of the unplayed data of the audio playback;

When the data buffer amount of the unplayed data of the audio playback exceeds the preset data buffer threshold, the data buffer amount of the unplayed data of the audio playback is adjusted so that the data buffer amount of the unplayed data of the audio playback is less than Equal to the preset data cache threshold.

In a feasible application solution of the above second aspect, the adjusting the audio playback includes:

All or part of the unplayed data in the audio buffer is deleted, so that the audio playback skips the deleted unplayed data, so that the playback progress of the audio playback catches up with the output progress of the audio and video data.

In a feasible application solution of the above second aspect, the deletion of all or part of the unplayed data in the audio buffer includes:

Monitor the waveform and frequency of the audio data in the audio buffer, and when it is necessary to delete unplayed data in the audio buffer, preferentially delete the audio data that is not sensitive to human ears.

In a third aspect, the present application provides a video image playback method, comprising:

When the playback strategy of video image playback is the catch-up strategy, obtain the preset interactive scene playback delay threshold;

The video image playback is performed based on the interactive scene playback delay threshold, and during the execution of the video image playback, the video image playback is adjusted so that the playback progress of the video image playback catches up with the audio and video data output progress, So that the playback delay of the video image compared to the audio and video data output is less than or equal to a preset interactive scene playback delay threshold.

In a feasible application solution of the third aspect, the video image playback based on the interactive scene playback delay threshold includes:

Monitoring the playback delay of the video image playback relative to the audio and video data output;

When the playback delay of the video image playback exceeds the interactive scene playback delay threshold, the video image playback is adjusted so that the playback delay of the video image playback is less than or equal to the interactive scene playback delay threshold.

Monitoring the data buffer amount of the unplayed data played by the video image;

When the data cache amount of the unplayed data played by the video image exceeds the preset data cache threshold, adjust the data cache size of the unplayed data played by the video image so that the data of the unplayed data played by the video image is The cache amount is less than or equal to the preset data cache threshold.

In a feasible application solution of the above third aspect, the adjusting the video image playback includes:

All or part of the unplayed data in the video image buffer is deleted, so that the video image playback skips the deleted unplayed data, so that the playback progress of the video image playback catches up with the audio and video data output progress.

In a fourth aspect, the present application provides an audio and video playback device, the device comprising:

a scene identification module, which is used to identify the current application scene;

A playback strategy configuration module, configured to configure the chasing strategy as a playback strategy for audio playback and/or video image playback when the current application scenario is an interactive application scenario, wherein the chasing strategy includes:

In a fifth aspect, the present application provides an audio playback device, the device comprising:

a threshold acquisition module, which is used to acquire a preset interactive scene playback delay threshold when the playback strategy of audio playback is a catch-up strategy;

A playback adjustment module, which is used to adjust the audio playback in the process of performing the audio playback, so that the playback progress of the audio playback catches up with the progress of the audio and video data output, so that the audio playback is compared with the audio and video data. The output playback delay is less than or equal to the preset interactive scene playback delay threshold.

In a sixth aspect, the present application provides a video image playback device, the device comprising:

a threshold acquisition module, which is used to acquire a preset interactive scene playback delay threshold when the playback strategy of video image playback is a catch-up strategy;

A playback adjustment module, which is used to adjust the playback of the video image in the process of playing the video image, so that the playback progress of the video image playback catches up with the progress of the audio and video data output, so that the video image playback is relatively The playback delay of the audio and video data output is less than or equal to the preset interactive scene playback delay threshold.

In a seventh aspect, the present application provides an audio and video playback device, the device comprising:

A playback strategy configuration module, which is used to configure a playback strategy for audio playback and/or video image playback according to the current application scenario, including: when the current application scenario is an interactive application scenario, configuring a catch-up strategy for audio playback and/or Or a playback strategy for video image playback, wherein the chasing strategy includes: adjusting the audio playback and/or the video image playback, so that the playback progress of the audio playback and/or the video image playback chases the audio and video data The progress of the output, so that the playback delay of the audio playback and/or the video image playback compared to the audio and video data output is less than or equal to a preset interactive scene playback delay threshold;

a threshold acquisition module, configured to acquire a preset interactive scene playback delay threshold when the playback strategy of the audio playback and/or the video image playback is a catch-up strategy;

A playback adjustment module, which is used to adjust the audio playback and/or the video image playback in the process of performing the audio playback and/or the video image playback, so that the audio playback and/or the video image playback The playback progress of playing catches up with the progress of the audio and video data output, so that the playback delay of the audio playback and/or the video image playback compared to the audio and video data output is less than or equal to a preset interactive scene playback delay threshold.

In an eighth aspect, the application provides an audio and video playback system, the system comprising:

An audio and video output device, which is used to output audio data and video data respectively; and, identify the current application scene, when the current application scene is an interactive application scene, configure the chasing strategy to play audio playback and/or video image playback strategy, wherein the chasing strategy includes: adjusting the audio playback and/or the video image playback, so that the playback progress of the audio playback and/or the video image playback catches up with the audio and video data output progress, so that The playback delay of the audio playback and/or the video image playback compared to the audio and video data output is less than or equal to a preset interactive scene playback delay threshold;

An audio playback device, configured to receive the audio data, and play the audio data according to a playback strategy of the audio playback configured by the audio and video output device;

A video image playback device, configured to receive the video image data, and play the video image data according to a playback strategy of the video image playback configured by the audio and video output device.

In a ninth aspect, the application provides an audio and video playback system, the system comprising:

an audio output device for outputting audio data;

a video image output device for outputting video image data;

Audio and video playback device, which is used for:

receiving the audio data and the video image data;

Identifying the current application scenario, when the current application scenario is an interactive application scenario, configure a catch-up strategy as a playback strategy for audio playback and/or video image playback, wherein the catch-up strategy includes: adjusting the audio playback and/or The video image is played, so that the playback progress of the audio playback and/or the video image playback catches up with the progress of the audio and video data output, so that the audio playback and/or the video image playback is compared with the audio and video data. The output playback delay is less than or equal to the preset interactive scene playback delay threshold;

The audio data and the video image data are played according to the playback strategy of the audio playback and the playback strategy of the video image playback.

In a tenth aspect, the present application provides an electronic device, the electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, triggering The electronic device performs the method steps as described in the first aspect above.

In an eleventh aspect, the present application provides an electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, The electronic device is triggered to perform the method steps described in the second aspect above.

In a twelfth aspect, the present application provides an electronic device comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, The electronic device is triggered to perform the method steps described in the third aspect above.

According to a thirteenth aspect, the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when it runs on a computer, the computer causes the computer to execute the above-mentioned first to third aspects. 's method.

Description of drawings

Fig. 1 is a schematic diagram of an application scenario of audio and video data transmission respectively;

Figure 2 is a schematic diagram of an application scenario of audio and video data transmission respectively;

Fig. 3 shows the flow chart of audio and video data playing respectively;

FIG. 4 is a schematic diagram of an application scenario where audio and video data are respectively transmitted;

5 is a flowchart of an audio and video playback method according to an embodiment of the present application;

FIG. 6 shows a structural block diagram of an audio and video playback system according to an embodiment of the present application;

FIG. 7 is a structural block diagram of an audio and video playback system according to an embodiment of the present application;

FIG. 8 is a schematic diagram of an application scenario where audio and video data are transmitted respectively;

FIG. 9 shows a flowchart of an audio playback method according to an embodiment of the present application;

FIG. 10 shows a flowchart of an audio playback method according to an embodiment of the present application;

Figure 11 shows a schematic diagram of the AAC audio data decoding process;

FIG. 12 is a flowchart of a video image playback method according to an embodiment of the present application;

FIG. 13 shows a flowchart of a video image playback method according to an embodiment of the present application;

FIG. 14 shows a flowchart of an audio and video playback method according to an embodiment of the present application;

FIG. 15 shows a flowchart of an audio and video playback method according to an embodiment of the present application;

FIG. 16 is a partial flowchart of a method for playing audio and video according to an embodiment of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The terms used in the embodiments of the present application are only used to explain specific embodiments of the present application, and are not intended to limit the present application.

In the audio and video separation application scenario, the main disadvantage of audio and video data transmission is the fluctuation of transmission delay. Fluctuations in transmission delay can cause audio and video playback to be stuck, thus degrading the user experience. For example, FIG. 1 shows a schematic diagram of an application scenario of transmitting audio and video data respectively. As shown in FIG. 1 , the drone 11 collects video images for the residential building, and transmits the collected video image data to the mobile phone 13 ; the microphone 12 performs audio collection for the community (for example, to collect the current environment of the shooting area of the drone 11 ) audio, collect the on-site explanation voice of the tour guide for the shooting area of the drone 11 ), and transmit the collected audio data to the mobile phone 13 . The mobile phone 13 plays the audio data collected by the microphone 12 while playing the video image data collected by the UAV 11 , so that the user of the mobile phone 13 can take a virtual tour of the cell.

Generally, when the drone 11 continuously collects video image data (for example, A11, A12, A13...) and sends it to the mobile phone 13 in sequence; the microphone 12 continuously collects audio data (for example, B11, B12, B13) synchronized with the video image data ...) and sent to the mobile phone 13 in turn.

The mobile phone 13 plays video image data and audio data in the order in which the data is received. During this process, if the transmission delay of the video image data is stable, when the drone 11 continuously transmits the video image data, the mobile phone 13 will also continuously receive the video image data, so as to perform continuous video image playback; however, if If the video image data transmission delay is unstable, the mobile phone 13 cannot continuously receive the video image data, and the video image playback of the mobile phone 13 may freeze or skip frames. When the audio data is sent, the mobile phone 13 will also continuously receive the audio data, thereby performing continuous audio playback; however, if the audio data transmission delay is unstable, the mobile phone 13 cannot continuously receive the audio data, and the audio of the mobile phone 13 Stuttering, skipping, etc. may occur during playback.

For another example, FIG. 2 shows a schematic diagram of an application scenario of transmitting audio and video data respectively. As shown in FIG. 2 , the drone 21 collects video images for the residential building, and transmits the collected video image data to the mobile phone 23 ; the microphone 22 performs audio collection for the community (for example, to collect the current environment of the shooting area of the drone 21 ) The audio data collected by the tour guide is transmitted to the mobile phone 23 . The mobile phone 23 transmits video image data to the large-screen TV 25 for playback; at the same time, the mobile phone 23 also transmits audio data to the smart speaker 24 for playback. In this way, the user can take a virtual tour of the community through the cooperation of the large-screen TV 25 and the smart speaker 24 .

Generally, the mobile phone 23 continuously transmits video image data (eg, A21, A22, A23 . . . ) to the large-screen TV 25 in sequence. The mobile phone 23 continuously sends the audio data (eg, B21, B22, B23 . . . ) synchronized with the video image data to the smart speaker 24 in sequence.

The large-screen television 25 plays video image data in the order in which the data is received. During this process, if the transmission delay of the video image data is stable, when the mobile phone 23 continuously transmits the video image data, the large-screen TV 25 will also continuously receive the video image data, thereby performing continuous video image playback; however, if The video image data transmission delay is unstable, the large-screen TV 25 cannot continuously receive video image data, and the video image playback of the large-screen TV 25 may freeze or skip frames.

Likewise, the smart speaker 24 plays audio data in the order in which the data is received. If the audio data transmission delay is stable, when the mobile phone 23 continuously transmits the audio data, the smart speaker 24 will also continuously receive the audio data, thereby performing continuous audio playback; however, if the audio data transmission delay is unstable, the smart speaker 24 If the audio data cannot be received continuously, the audio playback of the smart speaker 24 will be stuck, skipped, and the like.

Aiming at the problem caused by unstable data transmission delay, an embodiment of the present application provides a method for playing audio and video based on a cache. Specifically, after receiving the audio data and video data, the receiving device for audio data and video data does not play the audio data and video data immediately, but temporarily buffers the audio data and video data, and after buffering a certain amount of data, plays the buffered data. data. FIG. 3 shows a flow chart of playing audio and video data respectively. As shown in Figure 3, in the process of playing video image data:

Step 311, the data output terminal outputs video image data;

Step 312, the playback terminal receives video image data;

Step 313, the player buffers the video image data;

Step 314: The playback end plays the video image data in the buffer according to the buffer order in the buffer.

During playback of audio data:

Step 321, the data output terminal outputs audio data;

Step 322, the player receives audio data;

Step 323, the player buffers the audio data;

Step 324: The playback end plays the audio data in the buffer according to the buffer order in the buffer.

In the above process, the data output terminal for audio playback and the data output terminal for video image playback may be the same device or different devices; the playback terminal for audio playback and the playback terminal for video image playback may be the same device, or Can be for different devices. When buffering audio data and/or video image data, the data buffering amount can be dynamically changed according to the data transmission delay fluctuation; or, a maximum data buffering amount can be set according to the maximum value of the data delay fluctuation.

In the embodiment shown in FIG. 3 , because the player plays the video image data or audio data in the cache according to the cache order in the cache, the player receives the time node of a certain video image data or a certain audio data, and the player plays a The time nodes of the video image data or the audio data may be different time nodes. Theoretically, before playing, the larger the data volume of buffered audio data or video image data, the less interference caused by fluctuations in data transmission delay during subsequent playing. For example, if the audio data and video image data of a certain video are all cached and then played, then in the playback process, there will be no problems of freezing and frame skipping caused by fluctuations in data transmission delay.

However, since the data is first cached after being received, the previously cached data is called during playback, not the data that has just been received currently. Therefore, there is a playback delay between when the data is output by the data output until the data is played. For example, in the application scenario of the embodiment shown in FIG. 2 , after the audio data B23 is output by the mobile phone 23 , it is received and buffered by the smart speaker 24 . When the audio data B23 is buffered, other audio data (eg, B21 and B22 ) that have been buffered before and have not been played still exist in the buffer queue. The smart speaker 24 performs audio playback according to the arrangement order in the cache queue (in the order of B21, B22, and B23). Before being played, the larger the amount of buffered data, the greater the playback delay.

In some application scenarios that require real-time interaction, the increase in playback delay will greatly reduce the user experience. For example, in the embodiment shown in FIG. 1 , if the flying attitude of the drone 11 is assumed, the user of the mobile phone 13 controls the drone according to the image captured by the drone displayed on the mobile phone 13 . Then, if the time difference between outputting the video image from the drone 11 and receiving and playing the video image on the mobile phone 13 is too large, then the user confirms the drone according to the drone photographed image displayed on the mobile phone 13 . There is a deviation between the flight attitude and the real-time flight attitude of the UAV 11, and the user cannot control the flight attitude smoothly.

For another example, in the embodiment shown in FIG. 2 , if the flying attitude of the drone 21 is assumed, the user of the mobile phone 23 controls the drone according to the image captured by the drone displayed on the large-screen TV 25 . Then, even if the delay between the video image output by the drone 21 and the reception of the video image by the mobile phone 23 is not considered, if the video image is output from the mobile phone 23 and the video image is received and displayed by the large-screen TV 25, the difference between If the time difference is too large, then, there will be a deviation between the flight attitude of the drone confirmed by the user according to the image captured by the drone displayed on the large-screen TV 25 and the real-time flight attitude of the drone 21, and the user will not be able to smoothly Control the flight attitude.

For another example, FIG. 4 shows a schematic diagram of an application scenario of transmitting audio and video data respectively. As shown in FIG. 4 , user A and user B use their respective mobile phones (mobile phone 43 and mobile phone 44 ) to implement a video call with each other. In order to optimize the effect of the video call, user B sends the video image of the video call to the large-screen TV 41 for display through the mobile phone 43, and sends the voice of the video call to the smart speaker 42 for playback. If the mobile phone 43 outputs a video image to the large-screen TV 41 and the mobile phone 43 outputs a voice to the smart speaker 42, the playback delay between the display of the video image on the large-screen TV 41 and the playback of the voice on the smart speaker 42 increases, and the delay between user A and user B increases. The interaction delay will also increase, which will greatly affect the user experience of the video call between user A and user B.

Therefore, an embodiment of the present application proposes an audio and video playback method based on a catch-up strategy, which identifies audio and video playback scenarios, and selects a corresponding playback strategy according to specific application scenarios.

Further, for application scenarios with real-time interaction requirements (interactive application scenarios), an embodiment of the present application proposes an audio and video playback method based on a catch-up strategy. Specifically, according to the real-time interaction requirement of the application scenario, the playback delay threshold of the interaction scenario is preset. The interactive scene playback delay threshold is the maximum value of playback delay between the audio data and/or video image data output by the output device and the playback end of the audio data and/or video image data on the basis of satisfying the expected user interaction experience. . For example, in the application scenario shown in FIG. 4 , at user B, when the increase of the video image and voice lag time does not exceed 150ms, the user experience of the video call between user A and user B will not be affected. Therefore, the playback delay thresholds of the interaction scenes between the mobile phone 43 and the large-screen TV 41 and between the mobile phone 43 and the smart speaker 42 are set to be 150 ms.

When the application scenario is an interactive application scenario, audio playback and/or video image playback is performed based on the catch-up strategy. In the catch-up strategy, the preset interactive scene playback delay threshold is called, and in the process of audio playback and/or video image playback, the audio playback and/or video image playback are adjusted to make the audio playback and/or video image playback playback. The delay is less than or equal to the interactive scene playback delay threshold.

FIG. 5 is a flowchart of a method for playing audio and video according to an embodiment of the present application. The audio and video playback method is executed by one or more devices in the audio and video playback system (including audio and video data output devices, audio playback terminals, and video image playback terminals). As shown in Figure 5:

Step 500, identifying the current application scenario;

Step 501, judging whether the current application scene is an interactive application scene;

When the current application scenario is an interactive application scenario, step 502 is performed;

Step 502, calling the catch-up strategy;

Step 520, using a catch-up strategy to play audio and/or video images, including:

Step 521, calling a preset interactive scene playback delay threshold;

Step 522, adjust audio playback and/or video image playback, so that the playback progress of audio playback and/or video image playback catches up with the progress of audio and video data output, so that audio playback and/or video image playback is compared to audio and video data output. The playback delay is less than or equal to the interactive scene playback delay threshold.

Specifically, an embodiment of the present application also proposes an audio and video playback system. FIG. 6 is a structural block diagram of an audio and video playback system according to an embodiment of the present application. As shown in FIG. 6 , an audio output device 601 (eg, a sound capture device) outputs audio data; a video image output device 602 (eg, an image capture device) outputs video image data. The audio data and video image data are output to a playback device 603 (eg, a mobile phone), and the playback device 603 plays the received audio data and video image data synchronously. The user realizes the viewing and listening of the video through the playback device 603 .

In an actual application scenario, for the application scenario of multiple output devices (capture devices) and single playback devices shown in FIG. 6 ,

steps

500 , 501 and 502 can be performed by the playback device 603 (for example, a mobile phone). When the current application scene is an interactive application scene, it invokes the catch-up strategy (calls the preset interactive scene playback delay threshold), and performs playback based on the catch-up strategy.

For example, in the application scenario shown in FIG. 1 , the mobile phone 13 determines that the current application scenario is an interactive application scenario (real-time control of the flight attitude of the drone 11 ). The mobile phone 13 invokes the preset interactive scene playback delay threshold, implements a catch-up strategy based on the interactive scene playback delay threshold, and plays audio data from the microphone 12 and video image data from the drone 11 based on the catch-up strategy.

Specifically, an embodiment of the present application also proposes an audio and video playback system. FIG. 7 is a structural block diagram of an audio and video playback system according to an embodiment of the present application. As shown in FIG. 7 , an audio and video output device 701 (eg, a mobile phone) synchronously outputs audio data and video image data of the same video (eg, a video of a video call), wherein the audio data is output to the audio playback device 702 (eg, smart speakers), the video image data is output to the video image playback device 703 (eg, a large-screen display device). The audio playing device 702 plays the received audio file, and the video image playing device 703 plays the received video image file. The user realizes the viewing and listening of the video by watching the video image playing device 703 and listening to the audio playing device 702 .

In an actual application scenario, for the application scenario of a single output device and multiple playback devices shown in FIG. 7 ,

steps

500 , 501 and 502 can be performed by an audio and video output device 701 (eg, a mobile phone). When the audio and video output device 701 When determining that the current application scene is an interactive application scene, it invokes the catch-up strategy (including the interactive scene playback delay threshold), and sends the catch-up strategy (including the interactive scene playback delay threshold) to the audio playback device 702 and/or video image playback The device 703; the audio playback device 702 and/or the video image playback device 703 implements a catch-up strategy based on the acquired interactive scene playback delay threshold.

The following embodiments are mainly described based on the audio and video playback system with a single output device and multiple playback devices shown in FIG. 7 . It should be noted here that the steps in the following embodiments can also be applied to the multiple output device shown in FIG. 6 . Equipment (collection equipment), audio and video playback system of a single playback device.

For example, in the application scenario shown in FIG. 2 , the mobile phone 23 determines that the current application scenario is an interactive application scenario (real-time control of the flight attitude of the drone 21 ). The mobile phone 23 invokes the preset interactive scene playback delay threshold, and sends the interactive scene playback delay threshold to the large-screen TV 25; the large-screen TV 25 implements a catch-up strategy based on the acquired interactive scene playback delay threshold, and plays the data from the mobile phone based on the catch-up strategy. 23 video image data.

For another example, in the application scenario shown in FIG. 4 , the mobile phone 43 determines that the current application scenario is an interactive application scenario (making a video call with the mobile phone 44). The mobile phone 43 calls the preset interactive scene playback delay threshold, and sends the interactive scene playback delay threshold to the large-screen TV 41 and the smart speaker 42; the large-screen TV 41 implements a catch-up strategy based on the acquired interactive scene playback delay threshold, and based on the catch-up strategy The strategy plays the video image data from the mobile phone 23 ; the smart speaker 42 implements a catch-up strategy based on the acquired interactive scene playback delay threshold, and plays the audio data from the mobile phone 23 based on the catch-up strategy.

Further, in some application scenarios without real-time interaction requirements, the impact of playback delay on user experience is not obvious, and playback delay does not need to be maintained at a low level.

For example, in the application scenario shown in FIG. 1 , the user only uses the mobile phone 13 to perform a virtual tour of the cell with the designated route, and does not control the drone 11 . Therefore, the playback delay between the audio output from the microphone 12 and the playback of the audio from the mobile phone 13, and the playback delay between the video image output from the drone 11 and the playback of the video image from the mobile phone 13 will not significantly affect the user's ability to perform virtual tours. user experience.

For another example, FIG. 8 is a schematic diagram of an application scenario of transmitting audio and video data respectively. As shown in FIG. 8 , the mobile phone 63 is connected to the large-screen TV 61 and the smart speaker 62 . If the user uses the mobile phone 63 to watch a video (for example, in the embodiment shown in FIG. 1 , a video generated by integrating the video image collected by the drone 11 and the audio collected by the microphone 12 ). The mobile phone 63 plays the video. In order to optimize the playing effect, the mobile phone 63 does not output audio and video images through its own speakers and screen, but sends the video image of the played video to the large-screen TV 61 for display, and the video image of the played video is displayed. The audio is sent to the smart speaker 62 for playback. The playback delay between the video image and audio output by the mobile phone 63 and the video image displayed by the large-screen TV 61 and the audio playback by the smart speaker 62 will not significantly affect the user experience of video viewing.

Therefore, for application scenarios without real-time interaction requirements (non-interactive application scenarios), an embodiment of the present application proposes an audio and video playback method based on a smooth playback strategy.

Specifically, the data cache value is set according to the data transmission delay fluctuation situation of the application scenario. Specifically, the data cache value matches the maximum value of the data transmission delay fluctuation, or the data cache value is adjusted in real time according to the data transmission delay fluctuation. When the application scenario is a non-interactive application scenario, audio playback and/or video image playback is performed based on the smooth playback strategy. In the smooth playback strategy, the audio data and video image data are cached and played based on the data cache value to ensure that the audio and video data can be played smoothly when the data transmission delay fluctuates.

As shown in Figure 5:

When the current application scenario is not an interactive application scenario (non-interactive scenario), perform step 503;

Step 503, calling the smooth playback strategy;

Step 510, using a smooth playback strategy to play audio and/or video images.

In an actual application scenario, for an application scenario of multiple output devices (capture devices) and a single playback device, step 503 may be performed by a playback device (eg, a mobile phone). When the playback determines that the current application scenario is a non-interactive application scenario, the Call the smooth playback strategy and play based on the smooth playback strategy.

For example, in the application scenario shown in FIG. 1 , the mobile phone 13 determines that the current application scenario is a non-interactive application scenario (simple video viewing, without controlling the flight attitude of the drone 11 ). The mobile phone 13 invokes the smooth playback strategy, and plays the audio data from the microphone 12 and the video image data of the drone 11 based on the smooth playback strategy.

In an actual application scenario, for the application scenario of a single output device and multiple playback devices, step 503 can be performed by an audio and video output device (for example, a mobile phone). When the audio and video output device determines that the current application scenario is an interactive application scenario, its Invoke the interactive scene playback delay threshold, and deliver the interactive scene playback delay threshold to the audio player and/or the video image player; the audio player and/or the video image player implement a catch-up strategy based on the acquired interactive scene playback delay threshold .

For example, in the application scenario shown in FIG. 8 , the mobile phone 63 determines that the current application scenario is a non-interactive application scenario (simple video playback scenario). The mobile phone 63 invokes the smooth playback strategy and delivers the smooth playback strategy to the large-screen TV 61 and the smart speaker 62; the large-screen TV 61 plays the video image data from the mobile phone 63 based on the smooth playback strategy; the smart speaker 62 plays the video data from the mobile phone 63 based on the smooth playback strategy. Video image data of the mobile phone 63 .

In practical application scenarios, those skilled in the art can implement the smooth playback strategy by adopting various implementation manners. For example, in an implementation manner of the smooth playback strategy, the data cache value is set according to the fluctuation situation of the data transmission delay in the application scenario. During the playback process, the amount of data buffering is monitored to implement the catch-up strategy, and when the current amount of buffered data reaches the data buffering value, it is negotiated again to increase the buffering. For example, when the buffered amount of video data reaches the data buffered value of video data, the video playback device reports the central device (audio and video output device); the central device (audio and video output device) monitors the change and renegotiates to the video playback device, The audio playback device issues a new data cache value.

In practical application scenarios, those skilled in the art can also adopt various implementation manners to implement the catch-up strategy.

Taking audio playback as an example, an embodiment of the present application proposes an audio playback method, including: when the playback strategy of audio playback is a catch-up strategy, obtaining a preset interactive scene playback delay threshold; Play, during the execution of audio playback, adjust the audio playback so that the playback progress of the audio playback catches up with the progress of the audio and video data output, so that the playback delay of the audio playback compared to the audio and video data output is less than or equal to the preset interactive scene playback delay threshold.

Specifically, in an implementation of the catch-up strategy for audio playback, during the audio playback process, the playback delay of the audio playback relative to the audio and video data output is directly monitored; when the playback delay of the audio playback exceeds the playback delay of the interactive scene When the threshold is reached, adjust the audio playback so that the playback progress of the audio playback terminal based on the audio cache data catches up with the audio and video data output progress, so as to reduce the playback delay of the audio playback, so that the playback delay of the audio playback is less than or equal to the interactive scene playback delay threshold. .

FIG. 9 is a flowchart of an audio playback method according to an embodiment of the present application. In an implementation of step 522, the audio player executes the following process as shown in Figure 9:

Step 910, monitor the playback delay of audio playback relative to the audio and video data output;

Step 920, judging whether the playback delay of audio playback exceeds a preset interactive scene playback delay threshold;

When the playback delay of audio playback does not exceed the preset interactive scene playback delay threshold, return to step 910;

When the playback delay of audio playback exceeds the preset interactive scene playback delay threshold, perform step 930;

Step 930, adjust the audio playback, so that the playback progress of the audio playback terminal based on the audio buffer data to play catches up with the audio and video data output progress, to reduce the playback delay of the audio playback, so that the playback delay of the audio playback is less than or equal to the interactive scene playback delay threshold. .

In practical application scenarios, those skilled in the art can implement step 910 by adopting various solutions. For example, when outputting audio data, the output device adds a time stamp to the audio data, and when playing the audio data, the playback end compares the time stamp in the audio data being played with the current moment to calculate the playback delay.

Further, the audio playback terminal plays according to the storage order of the audio buffer data in the audio buffer. If the audio data in the audio buffer is moved out of the buffer after being played, then the amount of data buffered in the audio buffer can be output on behalf of the data output terminal. The difference between the progress of the audio data and the progress of playing the audio data by the audio player. Therefore, in order to simplify the operation process and reduce the difficulty of implementing the solution, in an implementation manner of the catch-up strategy, the buffered amount of audio data is monitored to implement the catch-up strategy.

Specifically, the corresponding buffered data volume threshold is determined according to the preset interactive scene playback delay threshold, and when the current audio buffered data volume exceeds the buffered data volume threshold, it can be determined that the audio playback delay exceeds the interactive scene playback delay threshold. For example, assuming that the interactive scene playback delay threshold is 150ms, according to the audio playback speed, the buffered data volume threshold corresponding to the audio can be calculated to be 4 frames.

FIG. 10 is a flowchart of an audio playback method according to an embodiment of the present application. In an implementation of step 522, the audio player executes the following process as shown in Figure 10:

Step 1010, monitor the data buffer amount of unplayed data of audio playback;

Step 1020, judging whether the audio data buffering amount exceeds a preset audio data buffering threshold, wherein the data buffering threshold is a value determined according to the interactive scene playback delay threshold;

When the audio data buffering amount does not exceed the preset audio data buffering threshold, return to step 1010;

When the audio data buffering amount exceeds the preset audio data buffering threshold, perform step 1030;

Step 1030: Adjust the audio data buffering amount of the unplayed audio data, so that the audio data buffering amount is less than or equal to a preset audio data buffering threshold.

In practical application scenarios, those skilled in the art can implement step 522 by adopting various solutions. For example, when the audio playback end plays based on the audio buffer data, the audio data is not read and played in sequence according to the storage order of the data in the buffer, but the audio data is read and played in a skip mode.

Further, in order to simplify the operation process and reduce the difficulty of implementing the scheme, in an embodiment of step 522, when the playback progress of the audio player needs to be made to catch up with the progress of the audio and video data output, delete all or part of the cache of the audio player. The audio data is not played, so that the playback of the player side skips the deleted data, so that the audio playback progress of the audio player side catches up with the output progress of the audio and video data. The deletion amount of the audio data corresponds to the playback progress of the current audio playback, so that the playback progress of the audio playback terminal satisfies the interactive scene playback delay threshold.

For example, in one implementation of step 930, the deletion amount of unplayed audio data corresponds to a value where the playback delay exceeds a playback delay threshold of the interactive scene.

For another example, in an embodiment of step 1030, the data in the audio buffer is deleted, so that the audio data buffer volume is less than or equal to the audio buffer data volume threshold.

Further, in the specific implementation process of step 522, those skilled in the art may adopt various different schemes to delete the unplayed audio data in the cache. For example, delete the earliest unplayed audio data stored in the audio buffer; another example, randomly select the data to be deleted from the unplayed audio data in the audio buffer; Select the data to be deleted from the audio data.

Further, in order to keep the information carried in the audio as much as possible, to avoid information loss caused by deleting the audio data in the cache, in an embodiment of step 522, the waveform and frequency of the audio data in the audio cache are monitored, and when necessary When deleting audio data in the cache, preferentially delete audio data that is not sensitive to human ears.

For example, the sound frequency range that normal people can feel is 20-20000Hz, but the human ear is most sensitive to sound in the frequency range of 1000-3000Hz. In order to minimize the distortion caused by the audio frame dropping during the synchronization process, the audio frames are selectively dropped according to the sensitivity priority of the human ear.

In a distributed scenario, the format of the transmitted audio frame is generally Advanced Audio Coding (AAC). In the process of decoding AAC audio data, the frequency domain information of the frame data is extracted (usually through fast Fourier transform), and the sensitivity of this frame of audio to the human ear is analyzed (for example, 3000Hz is the most sensitive, and the sensitivity is represented by 100). ; 20000Hz is the least sensitive, and the sensitivity is represented by 0).

Specifically, after the main control module of audio processing starts to run, the main control module puts a part of the AAC bit stream (AAC Stream) into the input buffer, and obtains the start of a frame by searching for the synchronization word. The syntax described in IEC 13818-7 starts decoding. The main task of the main control module is to operate the input and output buffers and call other modules to work together. The input and output buffers are provided with interfaces by a digital signal processing (digital signal processing, DSP) control module. The data stored in the output buffer is the decoded Pulse Code Modulation (PCM) data, which represents the amplitude of the sound. It consists of a fixed-length buffer. By calling the interface function of the DSP control module, the head pointer is obtained. After the output buffer is filled, the interrupt processing is called to output the audio analog/digital converter (Analog) connected to the I2S interface. -to-Digital Converter, ADC) chip (stereo audio digital/analog converter (Digital to analog converter, DAC) and direct drive (DirectDrive) headphone amplifier) output analog sound.

FIG. 11 is a schematic diagram showing the decoding process of AAC audio data. As shown in Figure 11, AAC Stream goes through:

Step 1101, noiseless decoding (Noisless Decoding), noiseless coding is Huffman coding, and its function is to further reduce the redundancy of the scale factor and the quantized spectrum, that is, to perform Huffman coding on the scale factor and the quantized spectrum information. ;

Step 1102, Dequantize;

Step 1103, joint stereo (Joint Stereo), joint stereo is a certain rendering work performed on the original sample, so that the sound is more "good";

Step 1104, perceptual noise substitution (PNS);

Step 1105, transient noise shaping (temporary noise shaping, TNS);

Step 1106, inverse discrete cosine transform (IMDCT);

Step 1107, frequency band replication (Spectral Band Replication, SBR);

After that, the PCM code stream of the left and right channels is obtained, and then the main control module puts it into the output buffer and outputs it to the sound playback device.

When the audio data in the cache needs to be deleted, the sensitive distribution of each audio frame in the current audio buffer queue is calculated first, and then which frames are lost. For example, frames with a sensitivity level below 60 account for 50%. If half of the audio frames need to be discarded at present, the frames with a sensitivity level below 60 are discarded.

Further, the catch-up strategy for video image playback can be implemented in a manner similar to the above-mentioned implementation manner of the catch-up strategy for audio playback. For example, an embodiment of the present application also proposes a video image playback method, including: when the playback strategy of video image playback is a catch-up strategy, acquiring a preset interactive scene playback delay threshold; Play, during the execution of the video image playback, adjust the video image playback, so that the playback progress of the video image playback catches up with the progress of the audio and video data output, so that the playback delay of the video image playback compared to the audio and video data output is less than or equal to the preset. The interactive scene playback delay threshold.

Specifically, in an implementation of the catch-up strategy for video playback, during the video image playback process, the playback delay of the video image playback relative to the audio and video data output is directly monitored; when the playback delay of the video image playback exceeds the interaction When the scene playback delay threshold is set, adjust the video image playback so that the playback progress of the video image playback terminal based on the video image cache data catches up with the audio and video data output progress, so as to reduce the playback delay of the video image playback and make the playback of the video image playback. The delay is less than or equal to the interactive scene playback delay threshold.

FIG. 12 is a flowchart of a method for playing a video image according to an embodiment of the present application. In an implementation of step 522, the video image player performs the following process as shown in Figure 12:

Step 1110, monitor the playback delay of the video image playback relative to the audio and video data output;

Step 1120, judging whether the playback delay of the video image playback exceeds a preset interactive scene playback delay threshold;

When the playback delay of the video image playback does not exceed the preset interactive scene playback delay threshold, return to step 1110;

When the playback delay of the video image playback exceeds the preset interactive scene playback delay threshold, perform step 1130;

Step 1130, adjust the video image playback, so that the playback progress of the video image playback terminal based on the video image cache data to catch up with the audio and video data output progress, to reduce the playback delay of the video image playback, so that the playback delay of the video image playback is less than or equal to Interaction scene playback delay threshold.

In practical application scenarios, those skilled in the art can implement step 1110 by adopting various solutions. For example, when outputting video image data, the output device adds a time stamp to the video image data, and the video image player compares the time stamp in the video image data being played with the current moment when playing the video image data, so as to calculate the playback delay .

Further, the video image player plays according to the storage order of the video image cache data in the video image cache. If the video image data in the video image cache is moved out of the cache after being played, then the amount of data cached in the video image cache can be It represents the difference between the progress of outputting video image data at the data output end and the progress of playing video image data at the video image playback end. Therefore, in order to simplify the operation process and reduce the difficulty of implementing the solution, in an implementation manner of the catch-up strategy, the buffered amount of video image data is monitored to implement the catch-up strategy.

Specifically, according to the preset interactive scene playback delay threshold, the corresponding video image buffer data volume threshold is determined, and when the current video image buffer data volume exceeds the video image buffer data volume threshold, it can be determined that the playback delay exceeds the interactive scene playback delay threshold. For example, assuming that the playback delay threshold of the interactive scene is 150ms, according to the playback speed of the video image, the buffered data volume threshold corresponding to the video image can be calculated to be 3 frames.

FIG. 13 is a flowchart of a method for playing a video image according to an embodiment of the present application. In an implementation of step 522, the video image player performs the following process as shown in Figure 13:

Step 1210, monitor the data buffer amount of unplayed data of audio playback and/or video image playback;

Step 1220, judging whether the video image data buffering amount exceeds a preset data buffering threshold, wherein: the data buffering threshold is a value determined according to the interactive scene playback delay threshold; when the execution device is an audio player, the data buffering threshold is audio data Cache threshold; when the execution device is the video image player, the data cache threshold is the video image data cache threshold;

When the data cache amount does not exceed the preset data cache threshold, return to step 1210;

When the data cache amount exceeds the preset data cache threshold, perform step 1230;

Step 1230: Adjust the data buffering amount of the unplayed video image data, so that the video image data buffering amount is less than or equal to a preset data buffering threshold.

In practical application scenarios, those skilled in the art can implement step 522 by adopting various solutions. For example, the playback speed of video image playback is accelerated (for example, the original playback speed of 10 frames/second is adjusted to 15 frames/second); for another example, when the video image playback end plays based on the video image cache data, it does not follow the cached data. In the data storage order, the data is read and played in sequence, but the data is read and played in a skip mode (for example, for video image buffer data, the mode of reading and playing every other frame is used for reading and playing).

Further, in order to simplify the operation process and reduce the difficulty of implementing the scheme, in an embodiment of step 522, when it is necessary to make the playback progress of the video image player catch up with the progress of the audio and video data output, delete all the buffers in the buffer of the video image player. Or part of the video image data is not played, so that the playback of the video image player skips the deleted data, so that the playback progress of the video image player catches up with the audio and video data output progress. The deletion amount of the video image data corresponds to the playback progress of the current video image playback, so that the playback progress of the video image playback terminal satisfies the interactive scene playback delay threshold.

For example, in one implementation of step 1130, the deletion amount of the unplayed video image data corresponds to the value of the video image playback delay exceeding the interactive scene playback delay threshold.

For another example, in an embodiment of step 1230, the data in the video image buffer is deleted, so that the video image data buffer amount is less than or equal to the video image buffer data amount threshold.

Further, in the specific implementation process of step 522, those skilled in the art may adopt various solutions to delete the unplayed video image data in the video image cache. For example, delete the earliest unplayed video image data stored in the video image cache; another example, randomly select the data to be deleted from the unplayed video image data in the video image cache; The data to be deleted is selected from the unplayed video image data in the cache.

Further, in order to ensure the smooth playback of the video image and try to avoid the effect of frame skipping in the image playback, in an embodiment of step 522, when the video image data in the cache needs to be deleted, the video frames to be deleted are selected according to fixed intervals. .

Further, in the application scenario of audio and video separation, the fluctuation of transmission delay will also cause the audio and video playback to be unsynchronized, thereby reducing the user experience. For example, in the audio and video data collection link, independent equipment is used to collect and transmit audio data and video image data respectively. The simultaneously captured audio data and video image data are transmitted separately. Due to the fluctuation of transmission delay, the transmission delay of audio data and video data may be different, then, the audio data and video data received by the player will be out of synchronization, which may lead to out-of-sync audio data and video data playback. .

For example, in the application scenario shown in FIG. 1 , when the drone 11 collects the video image data A1, the microphone 12 will collect the audio data B1 that is synchronized with the video image data A1. When the drone 11 transmits the video image data A1 to the mobile phone 13, the microphone 12 transmits the audio data B1 to the mobile phone 13. If the video image data A1 and the audio data B1 are received synchronously on the mobile phone 13, then, if the mobile phone 13. The video image data A1 and the audio data B1 are played at the same time as the video image data A1 and the audio data B1 are received, and the playback of the video image data A1 and the audio data B1 is synchronized.

However, if the transmission delay of the microphone 12 transmitting the audio data B1 to the mobile phone 13 is greater than the transmission delay of the drone 11 transmitting the video image data A1 to the mobile phone 13, then the reception of the video image data A1 and the audio data B1 on the mobile phone 13 is If it is not synchronized, the mobile phone 13 will first receive the video image data A1. If the mobile phone 13 plays the video image data A1 while receiving the video image data A1, when the mobile phone 13 starts to play the video image data A1, the mobile phone 13 has not yet received the audio data B1, and finally the audio and video playback on the mobile phone 13 will not be synchronized. The user experience of virtual tours in the community will be greatly reduced.

For the application scenario shown in FIG. 1 , a solution for solving the asynchronous playback of audio and video is to integrate the video image data and audio data before playing the video image data and the audio data to generate a video file with audio and video synchronization, and play the audio and video files. The video file to achieve audio and video synchronization. For example, in the embodiment shown in FIG. 1 , the mobile phone 13 receives video image data A1 and audio data B1, integrates them to generate a video file C1, and the mobile phone 13 plays the video file C1 to realize audio and video synchronization.

However, in some application scenarios, the integration of audio and video data cannot be achieved.

For example, in the embodiment shown in FIG. 1 , in the mobile phone 13 , the application (APP) for receiving the video image data of the drone 11 and the application for receiving the audio data of the microphone 12 are applications of different manufacturers. The application that receives the video image data of the drone 11 can realize the simultaneous playback of the video image data, and the application that receives the audio data of the microphone 12 can realize the simultaneous playback of the audio data. However, between the application receiving the video image data of the drone 11 and the application receiving the audio data of the microphone 12 , the integration of the audio and video data cannot be easily realized. For example, video image data and audio data need to be imported into a third-party video production application, and then integrated after synchronizing time tags, to generate video files. In this way, if the audio and video are integrated and then played, the user can only perform a virtual tour after the drone 11 and the microphone 12 have completed the audio and video collection, but not when the drone 11 and the microphone 12 are collecting audio and video. Take a synchronized virtual tour.

Further, in some application scenarios, in order to optimize the audio and video playback effect, different playback terminals are used to play audio data and video image data respectively.

For example, in the embodiment shown in FIG. 2, there is a difference in transmission delay between the transmission of audio data from the microphone 22 to the mobile phone 23 and the transmission of video image data from the drone 21 to the mobile phone 23, and the difference between the transmission of the audio data from the mobile phone 23 to the mobile phone 23 The difference in transmission delay between the audio data transmission of the speaker 24 and the video image data transmission from the mobile phone 23 to the large-screen TV 25 may cause the audio and video playback to be out of synchronization. Although audio and video synchronization can be achieved by integrating the audio data of the microphone 22 and the video image data of the drone 21 on the mobile phone 23, since the smart speaker 24 and the large-screen TV 25 are different playback ends, the audio and video cannot be integrated by integrating the audio and video data. data to synchronize audio and video playback.

For another example, in the embodiment shown in FIG. 4 , the difference in transmission delay between the transmission of audio data from the mobile phone 43 to the smart speaker 42 and the transmission of video image data from the mobile phone 43 to the large-screen TV 41 may cause audio and video playback. out of sync.

For another example, in the embodiment shown in FIG. 8 , the difference in transmission delay between the transmission of audio data from the mobile phone 63 to the smart speaker 62 and the transmission of video image data from the mobile phone 63 to the large-screen TV 41 may cause audio and video playback. out of sync.

Therefore, in view of the problem of unsynchronized audio and video playback in the audio and video separation application scenario, in an embodiment of the present application, for the interactive application scenario, on the basis of using the catch-up strategy for audio and video playback, an audio and video synchronization operation is introduced. .

Specifically, in an implementation manner of step 520, a catch-up strategy is adopted for both audio playback and video image playback to achieve audio and video synchronization. That is, a catch-up strategy is used for audio playback and video image playback. Adjust the audio playback and video image playback so that the playback progress of the audio playback terminal and the video image playback terminal catches up with the progress of the audio and video data output, so that the playback delay of the audio playback and video image playback compared to the audio and video data output is less than or equal to the interactive scene. Playback delay threshold.

Using the catch-up strategy for audio playback and video image playback can control the audio playback delay of audio playback and the video image playback delay of video image playback within the interactive scene playback delay threshold. Since playback delay is the delay between reference data playback and data output, and audio output and video image output are synchronized, the difference in playback progress between audio playback and video image playback is also controlled within The interactive scene playback delay threshold is within the threshold, thus realizing the synchronization of audio playback and video image playback.

Further, in an implementation manner of step 520, a catch-up strategy is only used for one playback process in audio playback and video image playback, so that the playback delay of the playback process using the catch-up strategy is controlled within the interactive scene playback delay threshold. Inside. For the playback process that does not adopt the catch-up strategy, the synchronization strategy is adopted for playback. In the synchronization strategy, based on the playback progress of the playback process using the catch-up strategy, the playback process using the synchronization strategy is adjusted so that the playback progress of the playback process using the synchronization strategy is synchronized with the playback progress of the playback process using the catch-up strategy.

In some application scenarios, it is necessary to prioritize the smoothness of audio playback and minimize the number of times to adjust the audio playback progress. For example, in daily video call application scenarios, voice is the main way to carry information for communication. The video image jam often does not lead to the lack of communication information, while the voice jam often leads to the lack of communication information. Therefore, it is necessary to Priority is given to ensuring the smoothness of voice playback, and the number of times to adjust the progress of voice playback is minimized.

Therefore, in one embodiment, for audio playback, a catch-up strategy based on audio and video output progress is adopted; for video image playback, a synchronization strategy based on audio playback progress is adopted.

FIG. 14 is a flowchart of a method for playing audio and video according to an embodiment of the present application. The audio and video playback method is executed by one or more devices in the audio and video playback system (including audio and video data output devices, audio playback terminals, and video image playback terminals). As shown in Figure 14:

Step 1300, identifying the current application scenario;

Step 1301, judging whether the current application scene is an interactive application scene;

When the current application scenario is not an interactive application scenario, perform step 1310;

Step 1310, using a smooth playback strategy to play audio and/or video images;

When the current application scenario is an interactive application scenario, perform step 1120;

Step 1320, using the catch-up strategy to play audio;

Step 1330, determine whether the playback progress of the video image is synchronized with the playback progress of the audio;

When the playback progress of the video image is synchronized with the playback progress of the audio, return to step 1320;

When the playback progress of the video image is not synchronized with the playback progress of the audio, perform step 1340;

Step 1340: Adjust the playback of the video image based on the playback progress of the audio, so that the playback progress of the video image is synchronized with the playback progress of the audio.

In some application scenarios, it is necessary to give priority to ensuring the smoothness of video image playback, and minimize the number of times to adjust the video image playback progress. For example, in the application scenario of real-time control of a remote device through a video image, the video image is used to reflect the status of the remote device, and the live sound is only used to assist, and the video image freeze can easily lead to control errors, so it is necessary to give priority to ensuring the video The smoothness of image playback, minimize the number of times to adjust the video image playback progress.

Therefore, in an embodiment, for video image playback, a catch-up strategy based on audio and video output progress is adopted; for audio playback, a synchronization strategy based on video image playback progress is adopted.

FIG. 15 is a flowchart of a method for playing audio and video according to an embodiment of the present application. The audio and video playback method is executed by one or more devices in the audio and video playback system (including audio and video data output devices, audio playback terminals, and video image playback terminals). As shown in Figure 15:

Step 1400, identifying the current application scenario;

Step 1401, judging whether the current application scene is an interactive application scene;

When the current application scenario is not an interactive application scenario, perform step 1410;

Step 1410, using a smooth playback strategy to play audio and/or video images;

When the current application scenario is an interactive application scenario, perform step 1420;

Step 1420, using a catch-up strategy to play video images;

Step 1430, determine whether the playback progress of the video image is synchronized with the playback progress of the audio;

When the playback progress of the video image is synchronized with the playback progress of the audio, return to step 1420;

When the playback progress of the video image is not synchronized with the playback progress of the audio, perform step 1440;

Step 1440: Adjust the audio playback based on the playback progress of the video image, so that the playback progress of the audio is synchronized with the playback progress of the video image.

Further, in the embodiment shown in FIG. 14 or FIG. 15 , those skilled in the art can adopt a variety of different implementations to determine whether the playback progress of the audio is synchronized with the playback progress of the video image, and adjust the audio Play or video image playback, so that the playback progress of the audio is synchronized with the playback progress of the video image.

For example, in an embodiment of the present application, the judgment based on the synchronous playback delay threshold is adopted to judge whether the playback progress of the audio is synchronized with the playback progress of the video image. Specifically, a synchronous playback delay threshold is preset, and the synchronous playback delay threshold is the maximum value of the difference in playback progress between audio playback and video playback on the basis of satisfying the expected user video viewing experience. For example, when the difference between the playback progress of audio playback and video playback exceeds 150ms, the user will feel obvious inconsistency of audio and video playback. Therefore, the synchronous playback delay threshold is set to 150ms. When the difference value between the playback progress of the audio and the playback progress of the video image exceeds the synchronous playback delay threshold, it is determined that the playback progress of the audio is not synchronized with the playback progress of the video image.

Specifically, in an application scenario, the difference in playback progress can be represented by a timestamp. For example, after the output device outputs the audio data A9 and the video image data B9, when the audio data A9 is played at the audio playback end, the playback time (time stamp T1) is recorded; when the video image data B9 is played at the video image playback end, the playback time is recorded. (time stamp T2); the interval duration between T1 and T2 is the difference in playback progress between audio playback and video image playback.

Further, in the playback process, adjust audio playback or video image playback, so that the playback progress difference between audio playback and video image playback is controlled within the synchronous playback delay threshold, so as to meet the real-time performance of audio playback and video image playback. On this basis, the synchronization of audio and video playback is realized.

Further, in the specific implementation process of step 1340 or step 1440, the way of deleting the unplayed data in the cache can be used to accelerate the progress of playing, or the way of adding transition data to the unplayed data in the cache can be used to delay. The playback progress is advanced, so that the playback progress of the audio is synchronized with the playback progress of the video image.

For example, for video image data, a transition frame is added between two video image frames (the transition frame can be a duplicate frame of adjacent video image frames), so that the playback of the transition frame is added to the playback process of the video image playback end, thereby delaying the The progress of the video image playback progress.

Further, in an implementation manner of step 520, audio and video synchronization is implemented based on monitoring the data transmission delay. Specifically, the playback cache link is set. After the playback device receives the audio data or video image data, before the audio data or video image data is played, the audio data or video image data is additionally buffered for a certain period of time (playback buffering) before playing, and the buffering period is the audio data transmission. The difference between the delay and the transmission delay of the video image data, so that the difference between the transmission delay of the audio data and the transmission delay of the video image data can be compensated to ensure that the audio data and the video image data are played synchronously.

Further, considering that in the playback buffering link, data is buffered before playing the video image data or audio data, the buffering duration will be calculated into the playback delay. Therefore, in an implementation manner of step 520, the playback buffering link is limited. The duration of cached data cannot exceed the interactive scene playback delay threshold. Since the cache duration of the playback cache is the difference between the audio data transmission delay and the video image data transmission delay, when the difference between the audio data transmission delay and the video image data transmission delay exceeds the interactive scene playback delay threshold, Delete the cached data in the cache and add new data in the cache, so as to realize the catch-up strategy on the premise of maintaining the synchronization of audio and video playback.

Taking a specific application scenario as an example, as shown in FIG. 8 , if the user uses the mobile phone 63 to play games. In order to optimize the game effect, users use the game screen projection mode to play the game. In the game screen projection mode, the mobile phone 63 does not output game audio and game video images through its own speakers and screen, but sends the game video images to the large-screen TV 61 for display, and sends the game audio to the smart speaker 62 for playback. . The user obtains the game content through the image output of the large-screen TV 61 and the audio output of the smart speaker 62, and performs corresponding game operations on the mobile phone. Since there is real-time interaction between the user and the mobile phone 63, the game screen projection application scenario is an interactive application scenario. Specifically, after the user activates the game screen projection mode, the mobile phone 63, the large-screen TV 61 and the smart speaker 62 execute the following processes:

After the mobile phone 63 recognizes the game screen projection scene, it establishes connections with the large-screen TV 61 and the smart speaker 62 respectively;

The large-screen TV 61 reports a network delay of 100ms to the mobile phone 63, and the smart speaker 62 reports a network delay of 200ms to the mobile phone 63;

The mobile phone 63 sends a catch-up strategy to the large-screen TV 61 and the smart speaker 62 respectively (the interactive scene playback delay threshold is 150ms), and the mobile phone 63 sends the network delay data of the large-screen TV 61 to the smart speaker 62, and sends the network delay data of the smart speaker 62. to the big screen TV 61;

The mobile phone 63 starts to send video image frames to the large-screen TV 61 and audio frames to the smart speaker 62;

The smart speaker 62 learns that the network delay of the large-screen TV 61 is 100ms, which is less than its own network delay of 200ms, so it directly plays the audio frame;

The large-screen TV 61 knows that the network delay of the smart speaker 62 is 200ms, so the video image frame is buffered for 100ms (<interactive scene playback delay threshold of 150ms) before decoding and playback, thereby synchronizing with the audio frame playback of the smart speaker 62.

Further, if at the next moment, the network delay of the smart speaker 62 further deteriorates to 300ms, the network delay of the large-screen TV 61 is still 100ms, the smart speaker 62 reports the network delay to the mobile phone 63 300ms, and the large-screen TV 61 reports the network delay to the mobile phone 63 100ms ; The mobile phone 63 notifies the large-screen TV 61 and the smart speaker 62 of the updated network delay data respectively;

The large-screen TV 61 updates the smart speaker 62 with a network delay of 300ms. The calculation needs to be cached for 200ms to synchronize with the audio playback, but the threshold of 150ms is exceeded. Therefore, after discarding the first 50ms of video image data that was previously cached for 100ms, and then caching the video image data for 100ms ( 150ms in total), decode and play the video image data, so as to synchronize the video image playback with the audio playback.

Further, if the network of the large-screen TV 61 is interrupted for 50ms and then continues, since the current large-screen TV 61 has a video buffer of 150ms, within 50ms of the flash, there will continue to be video image data available for playback and the audio of the smart speaker 62. Data is played synchronously.

Further, in practical application scenarios, the data transmission delay often fluctuates continuously, but the monitoring of the data transmission delay cannot be performed continuously without interruption. Therefore, in an implementation manner of step 520, the transmission delay of the audio playback device and the video image playback device is periodically refreshed, and playback buffering is performed periodically based on the data transmission delay to synchronize audio playback and video image playback.

Further, in an implementation manner of step 520, the synchronization effect is improved by adopting a combination of a synchronization audio and video playback scheme based on data transmission delay and other audio and video synchronization schemes. For example, in an application scenario where audio and video are played separately, perform a time stamp synchronization operation periodically. In the time stamp synchronization operation, compare the time stamps of the currently playing audio data and video image data to confirm the difference in playback progress and adjust the audio playback. Or video image playback, so that the playback progress difference between audio playback and video image playback is controlled within the synchronous playback delay threshold to achieve audio and video playback synchronization.

Between two time stamp synchronization operations, the audio and video playback scheme based on data transmission delay is used to synchronize audio and video playback. The data transmission delay of audio data and video image data is regularly refreshed, and playback buffering is performed according to the difference between data transmission delays to achieve audio and video playback synchronization. .

Further, in order to ensure smooth playback, the playback device does not play the audio data or video image data immediately after receiving the audio data, but buffers a certain amount of data to cope with fluctuations in transmission delay. When playing audio data or video image data, the playback device extracts data from the audio buffer or video image buffer, so that the extracted data enters the playback link. In an implementation manner of step 520, on the basis of adopting the catch-up strategy for audio playback and/or video image playback, a playback buffer link is added in the playback link. Specifically, after the audio data or the video image data is extracted from the audio buffer or the video image buffer, before the audio data or the video image data is played, the audio data or the video image data are additionally buffered for a certain period of time (playback buffer). During playback, the buffering time is the difference between the audio data transmission delay and the video image data transmission delay, so that the difference between the audio data transmission delay and the video image data transmission delay can be compensated, so as to ensure that the audio data is synchronized with the video image data. play.

Specifically, FIG. 16 shows a partial flowchart of an audio and video playback method according to an embodiment of the present application. In an implementation manner of step 520, the following process shown in Figure 16 is performed to realize audio and video playback synchronization:

Step 1510: Before the transmission of audio data and video data, the initial transmission delay of audio data transmission and video image transmission is obtained, and the data type with high transmission delay is the first type of data, and the data type with low transmission delay is the second type of data. Type data, the initial data transmission delay of the first type of data is the first delay, and the initial data transmission delay of the second type of data is the second delay (when the transmission delays are the same, the first type of data and the second type of data are arbitrarily defined);

Step 1511, determine whether the first delay difference is less than or equal to the interactive scene playback delay threshold, wherein the first delay difference is the difference between the first delay and the second delay;

Step 1512, when the first delay difference is less than or equal to the interactive scene playback delay threshold, transmit the first type of data and the second type of data, and play directly after the first type of data enters the playback link, and after the second type of data enters the playback link. Play the second type of data after buffering the first delay difference;

Step 1513, when the first delay difference is greater than the interactive scene playback delay threshold, it means that in the current application scenario, if the audio and video synchronization is to be ensured, the real-time interactive requirements cannot be met. Therefore, the output delay is too high to remind the user to remind the user of the current data. The transmission link cannot meet the real-time interaction requirements.

In the above process, because the second type of data is buffered and then played, it is equivalent to the second type of data waiting for the first type of data, so it can be ensured that the first type of data and the second type of data are played synchronously.

After step 1512 is performed, the following steps are also performed:

Step 1520, obtaining the current transmission delay of audio data transmission and video image transmission, taking the current data transmission delay of the first type of data as the third delay, and the current data transmission delay of the second type of data as the fourth delay;

Step 1521, determine whether the current transmission delay of the first type of data is higher than that of the second type of data;

Step 1531, when the current transmission delay of the first type of data is higher than that of the second type of data, determine whether the second delay difference is less than or equal to the interactive scene playback delay threshold, where the second delay difference is the third delay and the third delay. The difference between the four delays;

Step 1532, when the second delay difference is less than or equal to the interactive scene playback delay threshold, directly play the first type of data after entering the playback link, and cache the second type of data after the second type of data enters the playback link. The second delay difference after play;

Step 1533, when the second delay difference is greater than the interactive scene playback delay threshold, directly play the first type of data after entering the playback link; delete some data in the cached second type of data, and the deletion amount is the second delay difference The value that exceeds the interactive scene playback delay threshold; increases the cache amount of the second type of data, and starts to play the second type of data when the cache amount of the second type of data reaches the interactive scene playback delay threshold.

Further, after step 1521 is executed, the following steps are also executed:

Step 1541, when the current transmission delay of the second type of data is higher than that of the first type of data, determine whether the third delay difference is less than or equal to the interactive scene playback delay threshold, where the third delay difference is the fourth delay and the third delay. The difference between the three delays;

Step 1542, when the third delay difference is less than or equal to the interactive scene playback delay threshold, directly play the second type of data after entering the playback link, and cache the first type of data after the first type of data enters the playback link. The third delay difference after play;

Step 1543, when the third delay difference is greater than the interactive scene playback delay threshold, directly play the second type of data after entering the playback link; delete some data in the cached first type of data, and the deletion amount is the third delay difference The value that exceeds the interactive scene playback delay threshold; increases the buffer amount of the first type of data, and starts to play the first type of data when the buffered amount of the first type of data reaches the interactive scene playback delay threshold.

Further, in an embodiment of the present application, for a non-interactive application scenario, an audio and video synchronization operation is introduced based on a smooth playback strategy.

Specifically, in an implementation manner of step 510, a synchronization strategy is used to play a playback process of audio playback and video image playback. In the synchronization strategy, based on the playback progress of the playback process without the synchronization strategy, the playback process with the synchronization strategy is adjusted so that the playback progress of the playback process with the synchronization strategy is the same as the playback progress of the playback process with the smooth playback strategy. Synchronize.

For example, in one embodiment, a smooth playback strategy is adopted for audio playback, and a synchronization strategy based on audio playback progress is adopted for video image playback. For another example, in one embodiment, a smooth playback strategy is adopted for video image playback, and a synchronization strategy based on video image playback progress is adopted for audio playback. For another example, in one embodiment, a smooth playback strategy is adopted for audio playback and video image playback. Further, for video image playback, a synchronization strategy based on audio playback progress is adopted on the basis of smooth playback. For another example, in one embodiment, a smooth playback strategy is adopted for audio playback and video image playback. Further, for audio playback, a synchronization strategy based on video and image playback progress is adopted on the basis of smooth playback.

Further, in the specific implementation process of step 510, those skilled in the art can implement the synchronization strategy by adopting various implementation manners.

For example, the method based on the synchronous playback delay threshold is used to judge whether the playback progress of the audio is synchronized with the playback progress of the video image (for example, the difference of the playback progress is reflected by the timestamp); during the playback process, adjust the audio playback or video playback progress. Image playback, so that the playback progress difference between audio playback and video image playback is controlled within the synchronous playback delay threshold, so that audio and video playback synchronization is achieved on the basis of satisfying the real-time performance of audio playback and video image playback. Specifically, the playback progress can be accelerated by deleting the unplayed data in the cache, or the progress of the playback progress can be delayed by adding transition data to the unplayed data in the cache, so that the playback progress of the audio is different from the playback progress. The playback progress of the video image is synchronized.

For another example, audio and video synchronization is implemented based on monitoring the data transmission delay.

It can be understood that some or all of the steps or operations in the foregoing embodiments are only examples, and other operations or variations of various operations may also be performed in the embodiments of the present application. Furthermore, the various steps may be performed in a different order presented in the above-described embodiments, and may not perform all operations in the above-described embodiments.

Further, in the 1990s, an improvement in a technology can be clearly differentiated between improvements in hardware (for example, improvements to circuit structures such as diodes, transistors, switches, etc.) or improvements in software (improvements in method procedures). ). However, with the development of technology, the improvement of many methods and processes today can be regarded as a direct improvement of the hardware circuit structure. Designers almost get the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be realized by hardware entity modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is an integrated circuit whose logical function is determined by an accessor programming the device. It is programmed by the designer to "integrate" a digital device on a PLD, without the need for a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, today, instead of making integrated circuit chips by hand, this kind of programming is also mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development and writing, and needs to be compiled before compiling. The original code also has to be written in a specific programming language, which is called Hardware Description Language (HDL), and there is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language) , AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., currently the most commonly used The ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. It should also be clear to those skilled in the art that a hardware circuit for implementing the logic method process can be easily obtained by simply programming the method process in the above-mentioned several hardware description languages and programming it into the integrated circuit.

The controller may be implemented in any suitable manner, for example, the controller may take the form of eg a microprocessor or processor and a computer readable medium storing computer readable program code (eg software or firmware) executable by the (micro)processor , logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers and embedded microcontrollers, examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicon Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that, in addition to implementing the controller in the form of pure computer-readable program code, the controller can be implemented as logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded devices by logically programming the method steps. The same function can be realized in the form of a microcontroller, etc. Therefore, such a controller can be regarded as a hardware component, and the devices included therein for realizing various functions can also be regarded as a structure within the hardware component. Or even, the means for implementing various functions can be regarded as both a software module implementing a method and a structure within a hardware component.

Therefore, based on the audio and video playback method proposed by the embodiment of the present application, an embodiment of the present application proposes an audio and video playback device, which is installed in the audio and video output device 701 as shown in FIG. 7 , and the device includes:

A playback strategy configuration module, which is used to configure the chasing strategy to be a playback strategy of audio playback and/or video image playback when the current application scenario is an interactive application scenario, wherein the chasing strategy includes:

Adjust the audio playback and/or the video image playback so that the playback progress of the audio playback and/or the video image playback catches up with the audio and video data output progress, so that the audio playback and/or the video image playback is compared with the audio and video data output. The playback delay is less than or equal to the preset interactive scene playback delay threshold.

Further, the play strategy configuration module is also used for:

When the current application scenario is a non-interactive application scenario, configure the smooth playback strategy as a playback strategy for audio playback and/or video image playback.

Based on the audio playback method proposed by the embodiment of the present application, an embodiment of the present application proposes an audio playback device. The device is installed in the audio playback device 702 as shown in FIG. 7 , and the audio playback device includes:

The first playback adjustment module is used to adjust the audio playback in the process of audio playback when the playback strategy of the audio playback is a catch-up strategy, so that the playback progress of the audio playback catches up with the output of the audio and video data, so that the audio playback phase is comparable. The playback delay compared to the audio and video data output is less than or equal to the preset interactive scene playback delay threshold.

Further, the audio playback device also includes:

The second playback adjustment module is configured to adjust the audio playback based on the smooth playback strategy during the audio playback when the playback strategy of the audio playback is the smooth playback strategy.

Based on the video image playback method proposed by the embodiment of the present application, an embodiment of the present application proposes a video image playback device. The device is installed in the video image playback device 703 as shown in FIG. 7 , and the video image playback device includes:

a threshold acquisition module, which is used to acquire a preset interactive scene playback delay threshold when the playback strategy for video image playback is a catch-up strategy;

The first playback adjustment module is used to adjust the playback of the video image in the process of playing the video image when the playback strategy of the video image playback is a catch-up strategy, so that the playback progress of the video image playback catches up with the output of the audio and video data. The playback delay of the video image playback compared to the audio and video data output is less than or equal to the preset interactive scene playback delay threshold.

Further, the video image playback device also includes:

The second playback adjustment module is configured to adjust the playback of the video image based on the smooth playback strategy during the process of playing the video image when the playback strategy of the video image playback is the smooth playback strategy.

Further, an embodiment of the present application proposes an audio, video and image playback device, the device is installed in the playback device 603 as shown in FIG. 6 , and the device includes:

A playback strategy configuration module, which is used to configure the catch-up strategy as a playback strategy for audio playback and/or video image playback when the current application scenario is an interactive application scenario; and configure smooth playback when the current application scenario is a non-interactive application scenario The strategy is the playback strategy for audio playback and/or video image playback;

a threshold acquisition module, which is used to acquire a preset interactive scene playback delay threshold when the playback strategy of audio playback and/or video image playback is a catch-up strategy;

The first playback adjustment module, which is used to adjust audio playback and/or video image playback when the playback strategy of audio playback and/or video image playback is a catch-up strategy, so that the playback progress of audio playback and/or video image playback can catch up with the sound. The progress of video data output, so that the playback delay of audio playback and/or video image playback is less than or equal to the preset interactive scene playback delay threshold compared to the audio and video data output;

The second playback adjustment module is configured to adjust audio playback and/or video image playback based on the smooth playback strategy when the playback strategy for audio playback and/or video image playback is a smooth playback strategy.

In the description of the embodiments of the present application, for the convenience of description, when describing the device, functions are divided into various modules/units for description, and the division of each module/unit is only a logical function division. At the same time, the functions of each module/unit may be implemented in one or more software and/or hardware.

Specifically, the apparatuses proposed in the embodiments of the present application may be fully or partially integrated into a physical entity during actual implementation, or may be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also all be implemented in hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in hardware. For example, the detection module may be a separately established processing element, or may be integrated in a certain chip of the electronic device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, and can also be implemented independently. In the implementation process, each step of the above-mentioned method or each of the above-mentioned modules can be completed by an integrated logic circuit of hardware in the processor element or an instruction in the form of software.

For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more specific integrated circuits (Application Specific Integrated Circuit, ASIC), or, one or more digital signal processors ( Digital Singnal Processor, DSP), or, one or more Field Programmable Gate Array (Field Programmable Gate Array, FPGA), etc. For another example, these modules can be integrated together and implemented in the form of an on-chip device (System-On-a-Chip, SOC).

An embodiment of the present application also proposes an electronic device (audio and video data output device), the electronic device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the computer program When the processor executes, the electronic device is triggered to execute the steps of the audio and video playback method described in the embodiments of the present application.

An embodiment of the present application also proposes an electronic device (audio playback device), the electronic device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor During execution, the electronic device is triggered to execute the steps of the audio playback method described in the embodiments of the present application.

An embodiment of the present application also proposes an electronic device (video image playback device), the electronic device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are processed by the When the controller is executed, the electronic device is triggered to execute the steps of the video image playback method described in the embodiments of the present application.

An embodiment of the present application also proposes an electronic device (audio and video playback device), the electronic device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are processed by the When the device is executed, the electronic device is triggered to execute the steps of the audio and video playback method described in the embodiments of the present application.

Specifically, in an embodiment of the present application, the above-mentioned one or more computer programs are stored in the above-mentioned memory, and the above-mentioned one or more computer programs include instructions. When the above-mentioned instructions are executed by the above-mentioned device, the above-mentioned device is made to execute the application. The method steps described in the examples.

Specifically, in an embodiment of the present application, the processor of the electronic device may be an on-chip device SOC, and the processor may include a central processing unit (Central Processing Unit, CPU), and may further include other types of processors. Specifically, in an embodiment of the present application, the processor of the electronic device may be a PWM control chip.

Specifically, in an embodiment of the present application, the involved processor may include, for example, a CPU, a DSP, a microcontroller, or a digital signal processor, and may also include a GPU, an embedded Neural-network Process Units (NPU, NPU) ) and an image signal processor (Image Signal Processing, ISP), the processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of programs in the technical solution of the present application Wait. Furthermore, the processor may have the function of operating one or more software programs, which may be stored in a storage medium.

Specifically, in an embodiment of the present application, the memory of the electronic device may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (random access memory) memory, RAM) or other types of dynamic storage devices that can store information and instructions, also can be electrically erasable programmable read-only memory (electrically erasable programmable read-only memory, EEPROM), compact disc read-only memory, CD-ROM) or other optical disk storage, optical disk storage (including compact disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage medium or other magnetic storage device, or can also be used for portable or Any computer-readable medium that stores desired program code in the form of instructions or data structures and can be accessed by a computer.

Specifically, in an embodiment of the present application, a processor may be combined with a memory to form a processing device, which is more commonly an independent component. The processor is used to execute program codes stored in the memory to implement the method described in the embodiment of the present application. . During specific implementation, the memory can also be integrated in the processor, or be independent of the processor.

Further, the device, apparatus, module or unit described in the embodiments of the present application may be specifically implemented by a computer chip or entity, or implemented by a product having a certain function.

Those skilled in the art should understand that the embodiments of the present application may be provided as a method, an apparatus, or a computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein.

In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

Specifically, an embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when it runs on a computer, the computer executes the method provided by the embodiment of the present application.

An embodiment of the present application further provides a computer program product, where the computer program product includes a computer program that, when running on a computer, causes the computer to execute the method provided by the embodiment of the present application.

The descriptions of the embodiments in the present application are described with reference to flowcharts and/or block diagrams of methods, apparatuses (apparatuses), and computer program products according to the embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

It should also be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And/or", which describes the association relationship of the associated objects, indicates that there can be three kinds of relationships, for example, A and/or B, which can indicate the existence of A alone, the existence of A and B at the same time, and the existence of B alone. where A and B can be singular or plural. The character "/" generally indicates that the associated objects are an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c may represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c may be single, or Can be multiple.

In the embodiments of the present application, the terms "comprising", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes Other elements not expressly listed, or which are inherent to such a process, method, article of manufacture, or apparatus are also included. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article of manufacture, or device that includes the element.

The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

The various embodiments in this application are described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for related parts.

Those of ordinary skill in the art can realize that each unit and algorithm steps described in the embodiments of the present application can be implemented by a combination of electronic hardware, computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the above-described devices, devices and units, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

The above are only specific embodiments of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application. The protection scope of the present application shall be subject to the protection scope of the claims.

Claims

A method for playing audio and video, comprising:

Identify the current application scenario;

Configure a playback strategy for audio playback and/or video image playback according to the current application scenario, including:

When the current application scenario is an interactive application scenario, the chasing strategy is configured as a playback strategy for audio playback and/or video image playback, wherein the chasing strategy includes:

Adjust the audio playback and/or the video image playback, so that the playback progress of the audio playback and/or the video image playback catches up with the audio and video data output progress, so that the audio playback and/or the video playback progress The playback delay of the image playback compared to the audio and video data output is less than or equal to the preset interactive scene playback delay threshold.
The method according to claim 1, wherein the method further comprises:

When the current application scenario is a non-interactive application scenario, a smooth playback strategy is configured as a playback strategy for the audio playback and/or the video image playback.
The method according to claim 1 or 2, wherein the configuration catch-up strategy is a playback strategy for audio playback and/or video image playback, comprising:

The chasing strategy is configured as a playing strategy of the audio playing and the video image playing.
The method according to claim 1 or 2, wherein the configuration catch-up strategy is a playback strategy for audio playback and/or video image playback, comprising:

Configuring the chasing strategy to be the playback strategy of the audio playback;

The configuration synchronization strategy is the playback strategy of the video image playback, wherein the synchronization strategy includes:

Based on the playback progress of the audio playback, the video image playback is adjusted so that the playback progress of the video image playback is synchronized with the playback progress of the audio playback.
The method according to claim 1 or 2, wherein the configuration catch-up strategy is a playback strategy for audio playback and/or video image playback, comprising:

Configuring the chasing strategy to be a playback strategy of the video image playback;

The configuration synchronization strategy is the playback strategy of the audio playback, wherein the synchronization strategy includes:

The audio playback is adjusted based on the playback progress of the video image playback, so that the playback progress of the audio playback is synchronized with the playback progress of the video image playback.
An audio playback method, comprising:

When the playback strategy of audio playback is the catch-up strategy, obtain the preset interactive scene playback delay threshold;

The audio playback is performed based on the interactive scene playback delay threshold, and during the execution of the audio playback, the audio playback is adjusted so that the playback progress of the audio playback catches up with the output progress of the audio and video data, so that the audio playback progresses The playback delay of audio playback compared to the audio and video data output is less than or equal to the preset interactive scene playback delay threshold.
The method according to claim 6, wherein the performing the audio playback based on the interactive scene playback delay threshold comprises:

Monitoring the playback delay of the audio playback relative to the audio and video data output;

When the playback delay of the audio playback exceeds the interactive scene playback delay threshold, the audio playback is adjusted so that the playback delay of the audio playback is less than or equal to the interactive scene playback delay threshold.
The method according to claim 6, wherein the performing the audio playback based on the interactive scene playback delay threshold comprises:

Monitor the data buffer amount of the unplayed data of the audio playback;

When the data buffer amount of the unplayed data of the audio playback exceeds the preset data buffer threshold, the data buffer amount of the unplayed data of the audio playback is adjusted so that the data buffer amount of the unplayed data of the audio playback is less than Equal to the preset data cache threshold.
The method according to any one of claims 6 to 8, wherein the adjusting the audio playback comprises:

All or part of the unplayed data in the audio buffer is deleted, so that the audio playback skips the deleted unplayed data, so that the playback progress of the audio playback catches up with the output progress of the audio and video data.
The method according to claim 9, wherein the deleting all or part of the unplayed data in the audio buffer comprises:

Monitor the waveform and frequency of the audio data in the audio buffer, and when it is necessary to delete unplayed data in the audio buffer, preferentially delete the audio data that is not sensitive to human ears.
A method for playing video images, comprising:

When the playback strategy of video image playback is the catch-up strategy, obtain the preset interactive scene playback delay threshold;

The video image playback is performed based on the interactive scene playback delay threshold, and during the execution of the video image playback, the video image playback is adjusted so that the playback progress of the video image playback catches up with the audio and video data output progress, So that the playback delay of the video image compared to the audio and video data output is less than or equal to a preset interactive scene playback delay threshold.
The method according to claim 11, wherein the performing the video image playback based on the interactive scene playback delay threshold comprises:

Monitoring the playback delay of the video image playback relative to the audio and video data output;

When the playback delay of the video image playback exceeds the interactive scene playback delay threshold, the video image playback is adjusted so that the playback delay of the video image playback is less than or equal to the interactive scene playback delay threshold.
The method according to claim 11, wherein the performing the video image playback based on the interactive scene playback delay threshold comprises:

Monitoring the data buffer amount of the unplayed data played by the video image;

When the data cache amount of the unplayed data played by the video image exceeds the preset data cache threshold, adjust the data cache size of the unplayed data played by the video image so that the data of the unplayed data played by the video image is The cache amount is less than or equal to the preset data cache threshold.
The method according to any one of claims 11 to 13, wherein the adjusting the video image playback comprises:

All or part of the unplayed data in the video image buffer is deleted, so that the video image playback skips the deleted unplayed data, so that the playback progress of the video image playback catches up with the audio and video data output progress.
An audio and video playback device, characterized in that the device comprises:

a scene identification module, which is used to identify the current application scene;

A playback strategy configuration module, which is used to configure a playback strategy for audio playback and/or video image playback according to the current application scenario, including:

When the current application scenario is an interactive application scenario, the chasing strategy is configured as a playback strategy for audio playback and/or video image playback, wherein the chasing strategy includes:

Adjust the audio playback and/or the video image playback, so that the playback progress of the audio playback and/or the video image playback catches up with the audio and video data output progress, so that the audio playback and/or the video playback progress The playback delay of the image playback compared to the audio and video data output is less than or equal to the preset interactive scene playback delay threshold.
An audio playback device, characterized in that the device comprises:

a threshold acquisition module, which is used to acquire a preset interactive scene playback delay threshold when the playback strategy of audio playback is a catch-up strategy;

A playback adjustment module, which is used to adjust the audio playback in the process of performing the audio playback, so that the playback progress of the audio playback catches up with the progress of the audio and video data output, so that the audio playback is compared with the audio and video data. The output playback delay is less than or equal to the preset interactive scene playback delay threshold.
A video image playback device, characterized in that the device comprises:

a threshold acquisition module, which is used to acquire a preset interactive scene playback delay threshold when the playback strategy for video image playback is a catch-up strategy;

A playback adjustment module, which is used to adjust the playback of the video image in the process of playing the video image, so that the playback progress of the video image playback catches up with the progress of the audio and video data output, so that the video image playback is relatively The playback delay of the audio and video data output is less than or equal to the preset interactive scene playback delay threshold.
An audio and video playback device, characterized in that the device comprises:

a scene identification module, which is used to identify the current application scene;

A playback strategy configuration module, which is used to configure a playback strategy for audio playback and/or video image playback according to the current application scenario, including: when the current application scenario is an interactive application scenario, configuring a catch-up strategy for audio playback and/or Or a playback strategy for video image playback, wherein the chasing strategy includes: adjusting the audio playback and/or the video image playback, so that the playback progress of the audio playback and/or the video image playback chases the audio and video data The progress of the output, so that the playback delay of the audio playback and/or the video image playback compared to the audio and video data output is less than or equal to a preset interactive scene playback delay threshold;

a threshold acquisition module, configured to acquire a preset interactive scene playback delay threshold when the playback strategy of the audio playback and/or the video image playback is a catch-up strategy;

A playback adjustment module, which is used to adjust the audio playback and/or the video image playback in the process of performing the audio playback and/or the video image playback, so that the audio playback and/or the video image playback The playback progress of playing catches up with the progress of the audio and video data output, so that the playback delay of the audio playback and/or the video image playback compared to the audio and video data output is less than or equal to a preset interactive scene playback delay threshold.
An audio and video playback system, characterized in that the system includes:

An audio and video output device, which is used to output audio data and video data respectively; and, identify the current application scene, when the current application scene is an interactive application scene, configure the chasing strategy to play audio playback and/or video image playback strategy, wherein the chasing strategy includes: adjusting the audio playback and/or the video image playback, so that the playback progress of the audio playback and/or the video image playback catches up with the audio and video data output progress, so that The playback delay of the audio playback and/or the video image playback compared to the audio and video data output is less than or equal to a preset interactive scene playback delay threshold;

An audio playback device, configured to receive the audio data, and play the audio data according to a playback strategy of the audio playback configured by the audio and video output device;

A video image playback device, configured to receive the video image data, and play the video image data according to the playback strategy of the video image playback configured by the audio and video output device.
An audio and video playback system, characterized in that the system includes:

an audio output device for outputting audio data;

a video image output device for outputting video image data;

Audio and video playback device, which is used for:

receiving the audio data and the video image data;

Identifying the current application scenario, when the current application scenario is an interactive application scenario, configure a catch-up strategy as a playback strategy for audio playback and/or video image playback, wherein the catch-up strategy includes: adjusting the audio playback and/or The video image is played, so that the playback progress of the audio playback and/or the video image playback catches up with the progress of the audio and video data output, so that the audio playback and/or the video image playback is compared with the audio and video data. The output playback delay is less than or equal to the preset interactive scene playback delay threshold;

The audio data and the video image data are played according to the playback strategy of the audio playback and the playback strategy of the video image playback.
An electronic device, characterized in that the electronic device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered The apparatus performs the method steps of any one of claims 1-5.
An electronic device, characterized in that the electronic device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered The apparatus performs the method steps of any one of claims 6-10.
An electronic device, characterized in that the electronic device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered The apparatus performs the method steps of any one of claims 11-14.
A computer-readable storage medium, characterized in that, a computer program is stored in the computer-readable storage medium, and when it runs on a computer, the computer executes the method according to any one of claims 1-14 .