WO2022252909A1

WO2022252909A1 - Screen casting method and apparatus in wireless terminal, wireless terminal, and storage medium

Info

Publication number: WO2022252909A1
Application number: PCT/CN2022/091090
Authority: WO
Inventors: 许超杰
Original assignee: Oppo广东移动通信有限公司
Priority date: 2021-06-03
Filing date: 2022-05-06
Publication date: 2022-12-08
Also published as: CN115442777A

Abstract

A screen casting method and apparatus in a wireless terminal, a wireless terminal, and a storage medium, relating to the technical field of screen casting. The method comprises: in response to a screen casting instruction, establishing a data communication link with a screen casting device (201); when a Bluetooth link is established between an audio output terminal and the wireless terminal, transmitting audio data in screen casting data to the audio output terminal by means of the Bluetooth link, such that the audio output terminal outputs sound on the basis of the audio data (202); and transmitting image data in the screen casting data to the screen casting device by means of the data communication link, such that the screen casting device outputs a picture on the basis of the image data (203). By adopting the solution provided in the embodiments of the present application, audio output terminals except for the screen casting device can also be used for audio playback during a screen casting process, such that the degree of freedom of audio playback during the screen casting process is improved, and the impact caused by switching of the audio playback source during the screen casting process is avoided.

Description

Screen projection method and device in wireless terminal, wireless terminal and storage medium

This application claims the priority of the Chinese patent application with the application number 202110619639.8 and the invention title "screen projection method, device, wireless terminal and storage medium in wireless terminal" filed on June 3, 2021, the entire contents of which are incorporated by reference incorporated in this application.

technical field

The embodiments of the present application relate to the technical field of screen projection, and in particular, to a screen projection method and device in a wireless terminal, a wireless terminal, and a storage medium.

Background technique

Screen projection is a technology that transmits the audio and video data on the terminal side to the screen projection device, so as to display the terminal screen and play the terminal sound through the screen projection device.

For example, when the mobile phone and the TV are connected to the same wireless network, the user can turn on the screen projection function of the mobile phone, and the mobile phone transmits the audio and video data to the TV through the wireless network, so that the screen of the mobile phone is displayed on the TV side, and the sound of the mobile phone is played.

Contents of the invention

Embodiments of the present application provide a screen projection method and device in a wireless terminal, a wireless terminal, and a storage medium. Described technical scheme is as follows:

On the one hand, an embodiment of the present application provides a screen projection method in a wireless terminal, the method comprising:

Establishing a data communication link with the screen projection device in response to the screen projection instruction;

In the case that the audio output terminal establishes a Bluetooth link with the wireless terminal, transmit the audio data in the projection data to the audio output terminal through the Bluetooth link, so that the audio output terminal outputs sound based on the audio data ;

The image data in the screen projection data is transmitted to the screen projection device through the data communication link, so that the screen projection device outputs a picture based on the image data.

On the other hand, an embodiment of the present application provides a screen projection device, and the device includes:

A data link establishment module, configured to establish a data communication link with the screen projection device in response to the screen projection instruction;

The first transmission module is configured to transmit the audio data in the projection data to the audio output terminal through the Bluetooth link when the audio output terminal and the wireless terminal establish a Bluetooth link, so that the audio output terminal outputting sound based on the audio data;

The second transmission module is configured to transmit the image data in the screen projection data to the screen projection device through the data communication link, so that the screen projection device outputs a picture based on the image data.

On the other hand, an embodiment of the present application provides a wireless terminal, the wireless terminal includes a processor and a memory, at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement The method for projecting a screen in a wireless terminal according to the above aspects.

On the other hand, an embodiment of the present application provides a computer-readable storage medium, where at least one program is stored in the computer-readable storage medium, and the at least one program is loaded and executed by a processor to implement the above-mentioned The screen projection method in the wireless terminal.

On the other hand, an embodiment of the present application provides a computer program product, the computer program product includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; the Bluetooth chip of the electronic device reads the computer-readable storage medium from the computer-readable storage medium. Computer instructions, the processor executes the computer instructions, so that the electronic device executes the screen projection method in the wireless terminal provided in the above aspect.

Description of drawings

FIG. 1 shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

FIG. 2 shows a flow chart of a screen projection method provided by an exemplary embodiment of the present application;

Fig. 3 is an implementation schematic diagram of a screen projection process shown in an exemplary embodiment of the present application;

Fig. 4 is a flowchart of the audio data transmission process shown in an exemplary embodiment of the present application;

Fig. 5 is an implementation schematic diagram of a screen projection process shown in another exemplary embodiment of the present application;

Fig. 6 is an implementation schematic diagram of a handover process shown in an exemplary embodiment of the present application;

FIG. 7 is a flow chart of an image data and audio data transmission synchronization process shown in an exemplary embodiment of the present application;

FIG. 8 shows an interactive flowchart of the screen projection process provided by an embodiment of the present application;

FIG. 9 shows a structural block diagram of a screen projection device provided by an embodiment of the present application;

Fig. 10 shows a structural block diagram of a wireless terminal provided by an exemplary embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the implementation manners of the present application will be further described in detail below in conjunction with the accompanying drawings.

Please refer to FIG. 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The implementation environment includes a wireless terminal 110 , an audio output terminal 120 and a screen projection device 130 .

The wireless terminal 110 is an electronic device with bluetooth and screen projection functions. The electronic device may be a smart phone, a tablet computer, a personal computer, etc. In FIG. limited.

The audio output terminal 120 is an electronic device with Bluetooth and audio output functions. The electronic device can be a Bluetooth headset, a Bluetooth speaker, etc. In FIG. 1, the audio output terminal 120 is a Bluetooth headset as an example for illustration, but this is not limited .

After the wireless terminal 110 establishes a Bluetooth link with the audio output terminal 120, the wireless terminal 110 transmits audio data to the audio output terminal 120 through the first Bluetooth link, and the audio output terminal 120 outputs sound based on the audio data.

The screen projection device 130 is an electronic device with a display function. The electronic device may be a smart TV, a monitor, a personal computer, or a projector, etc. In FIG. 1 , the screen projection device 130 is a smart TV as an example for illustration. In a possible implementation manner, the wireless terminal 110 and the screen projection device 130 are located in the same wireless network. When receiving the screen projection instruction, the wireless terminal 110 and the screen projection device 130 establish a data communication link through the wireless network, and the screen projection During the process, the wireless terminal 110 transmits screen projection data (picture data and audio data) to the screen projection device 130 through the data communication link, and the screen projection device 130 outputs the screen and outputs the audio through its own speaker.

Of course, in other possible implementation manners, the wireless terminal 110 and the screen projection device 130 may also establish a data communication link based on a wired connection. For example, the wireless terminal 110 and the screen projection device 130 are connected through a data cable. When the screen command is issued, the wireless terminal 110 and the screen projection device 130 establish a data communication link through the data cable, which is not limited in this embodiment.

Optionally, during the screen projection process, if a Bluetooth link is established between the wireless terminal 110 and the audio output terminal 120, after the wireless terminal 110 establishes a Bluetooth link with the audio output terminal 120, the wireless terminal 110 will use the Bluetooth link to The audio data is transmitted to the audio output terminal 120, and the audio output terminal 120 outputs sound based on the audio data. At this time, the wireless terminal 110 separates audio and video data from the projected data, and on the one hand, transmits the image data to the screen projection device 130 through a data communication link for the screen projection device 130 to output images; The audio data is transmitted to the audio output terminal 120 for the audio output terminal 120 to output sound, that is, the audio and video of the wireless terminal can be displayed and played by different devices during the screen projection process.

Optionally, during the screen projection process, the wireless terminal 110 also provides an audio channel switching function. Through this function, the user can switch the audio channel from the audio output terminal 120 to the screen projection device 130 as required, so as to output sound through the screen projection device 130 , or switch the audio path from the screen projection device 130 to the audio output terminal 120 , and then output sound through the audio output terminal 120 .

In related technologies, during the screen projection process, the wireless terminal sends the screen projection data (including image data and audio data) to the screen projection device, and the screen projection device performs screen display and sound playback, even if there is a gap between the terminal and the audio output terminal at this time. There is a bluetooth connection and no terminal sound is played through the audio output terminal.

However, in the solution provided by the embodiment of the present application, if there is a Bluetooth connection between the wireless terminal and the audio output terminal to perform screen projection, the wireless terminal can send the screen projection data to the screen projection device, and the screen projection device will display the screen. As well as sound playback, it is also possible to separate audio and video data from the projection data, and then transmit image data to the projection device through the data communication link and transmit audio data to the audio output terminal through the Bluetooth link, so that the projection process can be passed through While the screen projection device is displaying the screen, it can also play sound through the audio output terminal to avoid the impact caused by switching the audio playback source from the audio output terminal to the screen projection device.

In a possible application scenario, a Bluetooth connection is established between a smartphone and a Bluetooth headset. When a user uses the smartphone to watch a video, the smartphone transmits audio data to the Bluetooth headset, and the Bluetooth headset plays the audio to avoid causing interference to others. When the user wants to continue watching videos through the smart TV, the smart phone can establish a data communication link with the smart TV by triggering the screen projection function of the smart phone.

Using the solution provided by related technologies, during the screen projection process, the sound of the smart phone will be played through the smart TV, and the sound played by the smart TV will cause interference to others.

With the solution provided by the embodiment of the present application, after the data communication link is established, the smart phone transmits the image data to the smart TV, and transmits the audio data to the Bluetooth headset, so that the smart TV displays the screen of the mobile phone based on the received image data. The earphone plays the sound of the mobile phone based on the received audio data, so as to prevent the external sound from the smart TV from causing interference to others during screen projection.

Please refer to FIG. 2, which shows a flow chart of a screen projection method in a wireless terminal provided by an exemplary embodiment of the present application. In this embodiment, the method is used in the wireless terminal 110 in the implementation environment shown in FIG. 1 as an example To illustrate, the method includes:

Step 201, in response to the screen projection instruction, establish a data communication link with the screen projection device.

Optionally, in response to the screen projection instruction, the wireless terminal and the screen projection device establish a wireless data communication link through a wireless network, or, the wireless terminal and the screen projection device establish a wired data communication link through a wired connection. For the convenience of description, the following embodiments are described by taking a wireless data communication link as an example.

In some embodiments, when there are multiple candidate screen projection devices in the wireless network, when receiving a screen projection instruction, the wireless terminal displays the device identification of each candidate screen projection device, and when receiving a device selection operation, communicates with the selected The projection device establishes a data communication link.

During the screen projection process, the wireless terminal uses the image data of its own display screen and the audio data of its own voice as the projection data, and the data communication link is used to transmit the projection data (all or part of the projection data).

Step 202, when the audio output terminal establishes a bluetooth link with the wireless terminal, transmit the audio data in the projection data to the audio output terminal through the bluetooth link, so that the audio output terminal outputs sound based on the audio data.

In a possible implementation, when a data communication link is established between the wireless terminal and the screen projection device, it is detected whether a Bluetooth link is established with the audio output terminal, and if no Bluetooth link is established, the wireless terminal adopts the conventional The screen projection process transmits the screen projection data including audio and video data to the screen projection device; if a Bluetooth link is established, the wireless terminal separates the audio and video data from the projection data.

Wherein, the Bluetooth link between the wireless terminal and the audio output terminal can be established before the screen projection, or can be established during the screen projection process (if the Bluetooth link is not established during the screen projection process, the screen projection device will display the wireless terminal screen and play terminal sound).

Further, for the separated audio data, the wireless terminal transmits the audio data to the audio output terminal through the bluetooth link, so that the terminal sound can continue to be played through the audio output terminal during the screen projection process.

Schematically, as shown in Figure 3, before the screen projection, a Bluetooth link is established between the mobile phone 31 and the Bluetooth headset 32, and audio data is transmitted to the Bluetooth headset 32 through the Bluetooth link; and after the screen projection, the mobile phone 31 Separate the screen projection data from audio and video, and continue to transmit the audio data to the Bluetooth headset 32 through the Bluetooth link with the Bluetooth headset 32 .

Step 203, transmit the image data in the screen projection data to the screen projection device through the data communication link, so that the screen projection device outputs a picture based on the image data.

While the wireless terminal transmits audio data to the audio output terminal, it transmits image data to the screen projection device through the established data communication link, and the screen projection device analyzes the image data and displays the terminal screen.

It should be noted that the above steps 202 and 203 are executed synchronously, so as to ensure that the output sound of the audio output terminal and the image output by the screen projection device are kept in sync.

Schematically, as shown in FIG. 3 , during the screen projection process, the mobile phone 31 transmits the image data to the smart TV 33 , and the smart TV 33 and the mobile phone 31 synchronously display the picture of the mobile phone.

It can be seen that with the solution provided by the embodiment of the present application, the sound of the wireless terminal is played through the audio output terminal before and after screen projection, so as to avoid interference to others during the screen projection process due to the terminal sound being directly released through the screen projection device during the screen projection process.

To sum up, in the embodiment of this application, during the process of the wireless terminal projecting the screen to the screen projection device, if the wireless terminal establishes a Bluetooth link with the audio output terminal, the wireless terminal performs the image data and audio data in the projection data. Separation, and transmit audio data to the audio output terminal through the Bluetooth link, and transmit image data through the data communication link with the screen projection device, so as to realize the separation of screen display and sound playback devices during the screen projection process; this application is implemented In the solution provided by the example, you can continue to use audio output terminals other than the projection device for audio playback during the screen projection process. For example, you can continue to play audio through the Bluetooth headset connected to the wireless terminal before the screen projection process, which improves the projection. The degree of freedom of audio playback during the screen projection process can avoid the impact caused by the switching of audio playback sources during the screen projection process.

Optionally, before transmitting the audio data in the projection data to the audio output terminal through the Bluetooth link, the method includes:

Based on the link states of the data communication link and the Bluetooth link, the image data and the audio data are transmitted synchronously.

Optionally, based on the link status of the data communication link and the Bluetooth link, the image data and audio data are transmitted synchronously, including:

determining a first transmission delay of the data communication link and a second transmission delay of the bluetooth link;

determining a transmission time difference based on the first transmission delay and the second transmission delay;

Transmission synchronization of image data and audio data is performed based on a transmission time difference.

Optionally, when a Bluetooth link is established between the audio output terminal and the wireless terminal, transmit the audio data in the projection data to the audio output terminal through the Bluetooth link, including:

Establish a Bluetooth link with the audio output terminal;

displaying a first switching prompt, the first switching prompt is used to confirm whether to output sound through the audio output terminal;

In response to the confirmation command, audio data is transmitted to the audio output terminal through the Bluetooth link.

Optionally, after the audio channel switching prompt is displayed, the method further includes:

In response to the cancel instruction, image data and audio data are transmitted to the screen projection device through the data communication link.

Optionally, the Bluetooth link is established before receiving the screen projection instruction;

In the case where a Bluetooth link is established between the audio output terminal and the wireless terminal, transmit the audio data in the projection data to the audio output terminal through the Bluetooth link, including:

In the case that the audio output terminal establishes a bluetooth link with the wireless terminal, obtain the device status of the audio output terminal;

When the audio output terminal is in use, the audio data is transmitted to the audio output terminal through a Bluetooth link.

Optionally, the method also includes:

In response to the switching instruction, image data and audio data are transmitted to the screen projection device through the data communication link.

Optionally, in response to the switching instruction, the image data and audio data are transmitted to the projection device through a data communication link, including:

Display the audio channel switching interface;

In response to the audio channel switching operation in the audio channel switching interface, a switching instruction is triggered, and the image data and audio data are transmitted to the screen projection device through the data communication link.

Determining the link quality of the Bluetooth link;

When the link quality does not meet the transmission requirements, the second switching prompt is displayed, and the second switching prompt is used to confirm whether to output sound through the screen projection device;

In response to the confirmation operation of the second switching prompt, the switching instruction is triggered, and the image data and audio data are transmitted to the screen projection device through the data communication link.

Optionally, after triggering the switching instruction in response to the confirmation operation of the second switching prompt, and transmitting the image data and audio data to the screen projection device through the data communication link, the method further includes:

In the case that the link quality meets the transmission requirement, a third switching prompt is displayed, and the third switching prompt is used to confirm whether to output sound through the audio output terminal;

In response to the confirmation operation of the third switching prompt, the audio data is transmitted to the audio output terminal through the Bluetooth link.

In a possible application scenario, before performing screen projection, the wireless terminal has established a Bluetooth link with the audio output terminal (that is, the Bluetooth link is established before receiving the screen projection instruction), and transmits data to the audio output terminal through the Bluetooth link. Transmit audio data. When the user triggers the wireless terminal to perform screen projection in this situation, in response to the screen projection command, the wireless terminal separates audio and video data for the screen projection data, so that the audio output terminal can continue to play the terminal sound during the screen projection process.

In this application scenario, although the wireless terminal establishes a Bluetooth connection with the audio output terminal, the user may not use the audio output terminal. In order to avoid continuing to play the terminal sound through the audio output terminal in this case, in a possible implementation In this method, when a Bluetooth link is established between the wireless terminal and the audio output terminal, the terminal obtains the device status of the audio output terminal, and transmits audio data to the audio output terminal through the Bluetooth link when the audio output terminal is in use. If the audio output terminal is not in use, the terminal transmits screen projection data including audio and video data to the screen projection device through the data communication link.

Optionally, the device status is detected by the audio output terminal through its own sensor, and reported to the wireless terminal through the Bluetooth link. For example, when the audio output terminal is a bluetooth earphone, the bluetooth earphone determines the attitude of the earphone through the built-in sensor, and when the attitude of the earphone is in the ear state, it is determined that it is in the use state.

In this embodiment, the wireless terminal determines whether to perform separate transmission of audio data and image data based on the device status of the audio output terminal, so as to avoid playing the terminal sound through the audio output terminal when the audio output terminal is not in use, resulting in the need for manual switching again Audio playback source (manually switch to the screen mirroring device) problem.

In another possible application scenario, before the screen projection, the wireless terminal does not establish a Bluetooth link with the audio output terminal. If the screen projection is performed at this time, both image data and audio data will be transmitted to the screen projection device. Correspondingly, the screen projection device displays the terminal screen and plays the terminal sound. If the wireless terminal establishes a Bluetooth connection to the audio output terminal during screen projection, the terminal can determine whether to play the terminal sound through the audio output terminal according to user requirements. As shown in FIG. 4, the above step 202 may include the following steps.

Step 202A, establishing a Bluetooth link with the audio output terminal.

Optionally, during the screen projection process, when a Bluetooth connection instruction is received, the wireless terminal establishes a Bluetooth link with the audio output terminal. Wherein, the audio output terminal may be a Bluetooth device paired with the wireless terminal for the first time, or a Bluetooth device previously paired with the wireless terminal.

In step 202B, a first switching prompt is displayed, and the first switching prompt is used to confirm whether to output sound through the audio output terminal.

When detecting that a Bluetooth connection is established with the audio output terminal during screen projection, in order to determine whether the audio needs to be switched from the screen projection device to the audio output terminal, the wireless terminal displays a first switching prompt to prompt the user to confirm whether to play through the audio output terminal terminal sound.

If the confirmation instruction is received, the wireless terminal determines that the device for playing the terminal sound needs to be switched, and performs step 202C; if a cancel instruction is received, the wireless terminal determines to continue playing the terminal sound through the screen projection device.

Schematically, as shown in FIG. 5 , during the screen projection process, the mobile phone 31 transmits screen projection data to the TV 33 , and the TV 33 displays the screen of the mobile phone according to the image data therein, and plays the sound of the mobile phone according to the audio data therein. When the mobile phone 31 establishes a Bluetooth connection with the Bluetooth headset 32, the mobile phone 31 displays a first switching prompt 311, prompting whether to switch the Bluetooth headset 32 to play the mobile phone sound.

Step 202C, in response to the confirmation instruction, transmit the audio data to the audio output terminal through the Bluetooth link.

When receiving the confirmation instruction, the terminal will separate the audio and video data from the projected screen data, and transmit the audio data to the audio output terminal through the Bluetooth link for playback by the audio output terminal. Wherein, the confirmation instruction is triggered by a confirmation operation, and the confirmation operation is received through a wireless terminal (such as clicking a confirmation option), or is received through an audio output terminal (such as tapping a Bluetooth headset).

Schematically, as shown in FIG. 5 , after the user clicks the confirmation option, the mobile phone 31 sends audio data to the Bluetooth earphone 32 through the Bluetooth link, thereby playing the sound of the mobile phone through the Bluetooth earphone 32 .

Optionally, in response to the cancel instruction, the wireless terminal continues to transmit image data and audio data to the screen projection device through the data communication link.

In order to further improve the freedom of audio playback during the screen projection process, the user can switch the audio playback source of the phone's voice as needed during the screen projection process. In a possible implementation manner, in response to the switching instruction, the wireless terminal transmits image data and audio data to the screen projection device through the data communication link, so that the screen projection device synchronously plays the terminal sound while displaying the terminal screen.

In a possible implementation, the terminal displays the audio channel switching interface during the screen projection process, and when receiving the audio channel switching operation in the audio channel switching interface, triggers the switching instruction, and sends the audio channel switching instruction to the screen projection device through the data communication link. Transfer image data and audio data.

Schematically, as shown in Figure 6, during the screen projection process, the TV 33 first displays the screen of the mobile phone, and plays the sound of the mobile phone through the Bluetooth headset 32; when receiving the switching instruction, the mobile phone 31 stops transmitting audio data to the Bluetooth headset 32, and The screen projection data including image data and audio data is transmitted to the TV 33, so that the sound of the mobile phone is played through the built-in speaker of the TV 33.

Optionally, when receiving the switching instruction triggered again, the terminal transmits the audio data to the audio output terminal again through the bluetooth link.

During the screen projection process, if the link quality of the Bluetooth link between the wireless terminal and the audio output terminal deteriorates (for example, the terminal is too far away from the audio output terminal, or the Bluetooth link is interfered), the transmission of audio data will be affected. This in turn affects the playback of terminal sound on the audio output terminal side. In this case, in order to improve the synchronization of audio and video during screen projection, the wireless terminal may prompt the user to switch the audio to the screen projection device.

In a possible implementation manner, during the screen projection process, the wireless terminal determines the link quality of the Bluetooth link. Wherein, the link quality can be determined by measuring the received signal strength indication (Received Signal Strength Indication, RSSI), transmission rate, transmission delay or packet loss rate of the Bluetooth signal. limited.

Further, the wireless terminal detects whether the link quality of the Bluetooth link meets the transmission requirements, and if the transmission requirements are met, the audio data will continue to be transmitted through the Bluetooth link; if the transmission requirements are not met, a second switching prompt will be displayed, and the second switching The prompt is used to confirm whether to play the terminal sound through the screen mirroring device. In response to the confirmation operation of the second switching prompt, the wireless terminal triggers the switching instruction, and transmits the image data and audio data to the screen projection device through the data communication link.

Optionally, the transmission requirements may include at least one of the following: RSSI requirements, transmission rate requirements, transmission delay requirements, and packet loss rate requirements. The embodiment of the present application does not limit the specific content of the transmission requirement.

In an illustrative example, when the wireless terminal detects that the packet loss rate of the Bluetooth link with the audio output terminal is greater than a packet loss rate threshold (such as 5%), it is determined that the link quality of the Bluetooth link does not meet the transmission requirements , so as to display a second switching prompt, prompting the user to switch the terminal audio to the screen projection device.

In some embodiments, after switching the audio data transmission path from the Bluetooth link to the data communication link, the wireless terminal continues to monitor the link quality of the Bluetooth link. When the link quality meets the transmission requirements, the wireless terminal displays a third switching prompt for the user to confirm whether to switch the audio data transmission path to the Bluetooth link, so as to play the terminal sound through the audio output terminal again.

In an illustrative example, the wireless terminal switches and transmits audio data to the screen projection device because the RSSI of the Bluetooth link is lower than the RSSI threshold, and when the RSSI of the subsequent Bluetooth link is greater than the RSSI threshold, the wireless terminal displays a third switching prompt , prompting the user to play the terminal sound through the audio output terminal again.

In response to the confirmation operation of the third switching prompt, the wireless terminal transmits audio data to the audio output terminal through the Bluetooth link, thereby resuming the playback of the terminal sound at the audio output terminal.

In this embodiment, during the screen projection process, when receiving the switching instruction, the wireless terminal switches the transmission link of the audio data, so as to realize the switching of sound playback between the screen projection device and the audio output terminal, which further improves the performance of the screen projection process. The freedom of sound playback.

In addition, in this embodiment, the wireless terminal monitors the link quality of the Bluetooth link, so as to perform switching prompts based on the link quality. When the channel quality is restored, it prompts to switch back to the audio output terminal to play the terminal sound, so as to improve the audio playback quality during screen mirroring.

When both the terminal picture and the terminal sound are displayed and played through the screen projection device, since the image data and audio data are transmitted to the screen projection device through the same link, the terminal picture at the screen projection device and the terminal sound are kept in sync; and when the terminal screen Displayed by the screen projection device, when the terminal sound is played by the audio output terminal, since the terminal image data and audio data are transmitted through different links, and the transmission delay between different links may be different, there may be a problem of out-of-sync audio and video .

In order to improve the synchronization of audio and video during the screen projection process, in a possible implementation manner, during the screen projection process, when the terminal sound is played through the audio output terminal, the wireless terminal is based on the link between the data communication link and the Bluetooth link. State, to synchronize the transmission of image data and audio data, to ensure that in the projection data under the same timestamp, the display time of image data is synchronized with the playback time of audio data.

As shown in Figure 7, the synchronization of the transmission of image data and audio data by the wireless terminal may include the following steps:

Step 701, determine a first transmission delay of a data communication link and a second transmission delay of a Bluetooth link.

Regarding the specific method of determining the transmission delay corresponding to the link, in a possible implementation manner, the wireless terminal sends the first delay measurement instruction to the screen projection device through the data communication link, and the screen projection device receives the first delay measurement instruction After the command, the first response command is sent through the data communication link. The wireless terminal determines the first transmission delay of the data communication link based on the time interval between the first delay measurement instruction and the first response instruction.

Similarly, the wireless terminal sends the second delay measurement instruction to the audio output terminal through the Bluetooth link, and the audio output terminal sends the second response instruction through the Bluetooth link after receiving the second delay measurement instruction. The wireless terminal determines the second transmission delay of the Bluetooth link based on the time interval between the second delay measurement instruction and the second response instruction.

Schematically, when the sending time of the first delay measurement command is t1, and the receiving time of the first response command is t2, the first transmission delay of the data communication link is (t2-t1)/2; When the sending time of the second delay measurement command is t1', and the receiving time of the second response command is t2', the second transmission delay of the Bluetooth link is (t2'-t1')/2.

Step 702: Determine a transmission time difference based on the first transmission delay and the second transmission delay.

Further, the wireless terminal determines the difference between the first transmission delay and the second transmission delay as the transmission time difference.

Combining the examples in the above steps, the transmission time difference is (t2-t1)/2-(t2'-t1')/2.

Step 703: Synchronize the transmission of the image data and the audio data based on the transmission time difference.

In a possible implementation, the wireless terminal can delay the transmission of image data or audio based on the transmission time difference, so as to use the delay to offset the timing difference caused by the transmission of audio and video data in different links, ensuring that the projection device Synchronization of image decoding in the middle and audio decoding in the audio output terminal. For example, when the transmission time difference is 20ms (the transmission delay of the data communication link is higher than the transmission delay of the Bluetooth link), the wireless terminal delays sending audio data to the audio output terminal by 20ms.

In this embodiment, when the terminal screen is displayed through the screen projection device, and the terminal sound is played through the audio output terminal, the wireless terminal synchronizes the transmission of image data and audio data based on the transmission time difference between the data communication link and the Bluetooth link. Improved the synchronization of audio and video during screencasting.

In an exemplary embodiment, as shown in FIG. 8 , when a wireless terminal realizes screen projection through a screen projection device and an audio output terminal, the following steps are included.

Step 801, the wireless terminal establishes a bluetooth link with the audio output terminal.

Step 802, the wireless terminal transmits audio data to the audio output terminal through the Bluetooth link.

Step 803, the audio output terminal outputs sound based on the audio data.

Step 804, in response to the screen projection instruction, the wireless terminal establishes a data communication link with the screen projection device.

Step 805, the wireless terminal synchronizes the transmission of the image data and audio data in the screen projection data based on the link states of the data communication link and the Bluetooth link.

Step 806, the wireless terminal transmits the audio data in the projection data to the audio output terminal through the bluetooth link.

Step 807, the audio output terminal outputs sound based on the audio data.

Step 808, the wireless terminal transmits the image data in the screen projection data to the screen projection device through the data communication link.

Step 809, the screen projection device outputs a picture based on the image data.

Step 810, in response to the switching instruction, the wireless terminal transmits image data and audio data to the screen projection device through the data communication link.

Step 811, the screen projection device outputs a picture based on the image data, and outputs a sound based on the audio data.

In each of the foregoing embodiments, the screen projection device and the audio output terminal are independent of each other, and there is no data communication. In a possible implementation manner, when the screen projection device does not have the Bluetooth function, the wireless terminal can use the solutions provided in the above embodiments to realize the screen projection; when the screen projection device has the Bluetooth function, the wireless terminal and the audio output terminal establish In the case of a Bluetooth link, the wireless terminal sends a Bluetooth link establishment request to the screen projection device, so that the screen projection device establishes a second Bluetooth link with the audio output terminal based on the Bluetooth link establishment request. During the screen projection process, the wireless terminal transmits the screen projection data to the screen projection device through the data communication link, and the screen projection device displays the terminal screen based on the image data, and transmits the screen projection data to the audio output terminal through the second Bluetooth link with the audio output terminal. The audio data is transmitted, and then the audio output terminal plays the terminal sound based on the audio data.

Correspondingly, when the audio data is transmitted to the audio output terminal through the screen projection device, the screen projection device needs to determine the transmission delay of the second Bluetooth link with the audio output terminal, so as to process the image data and audio data based on the transmission delay. Transmission is synchronized. For the process of determining the transmission delay and performing transmission synchronization based on the transmission delay, reference may be made to the above steps 701 to 703, which will not be repeated in this embodiment.

Optionally, during the screen projection process, in response to the switching instruction, the wireless terminal sends a switch request to the screen projection device, so that the screen projection device stops transmitting audio data to the audio output terminal through the second Bluetooth link, and the screen projection device transmits audio data based on the audio output. Data output sound.

Optionally, in response to the screen projection end instruction, the wireless terminal sends a screen projection end request to the screen projection device, so that the screen projection device disconnects the data communication link and the second Bluetooth link with the audio output terminal based on the screen projection end request road.

At the same time, when the audio output terminal supports a single Bluetooth link, the screen projection device will also send the Bluetooth information of the audio output terminal to the wireless terminal after receiving the screen projection end request, so that the wireless terminal can communicate with the audio output terminal based on the Bluetooth information of the audio output terminal. The audio output terminal re-establishes the Bluetooth link to ensure the continuity of audio playback at the audio output terminal before and after the end of screen projection.

It should be noted that, in each of the above-mentioned embodiments, the steps with the wireless terminal as the execution subject can be independently implemented as a screen projection method on the wireless terminal side, and the steps with the screen projection device as the execution subject can be independently implemented as a projection method on the screen projection device side. screen method, this embodiment will not repeat it here.

The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

Please refer to FIG. 9 , which shows a structural block diagram of a screen projection device provided by an embodiment of the present application. The device can include:

A data link establishment module 901, configured to establish a data communication link with the screen projection device in response to the screen projection instruction;

The first transmission module 902 is configured to transmit the audio data in the projection data to the audio output terminal through the Bluetooth link when the audio output terminal establishes a Bluetooth link with the wireless terminal, so that the audio output the terminal outputs sound based on the audio data;

The second transmission module 903 is configured to transmit the image data in the screen projection data to the screen projection device through the data communication link, so that the screen projection device outputs a picture based on the image data.

Optionally, the device also includes:

A synchronization module, configured to synchronize the transmission of the image data and the audio data based on the link states of the data communication link and the Bluetooth link.

Optionally, the synchronization module is specifically used for:

The image data and the audio data are transmitted synchronously based on the transmission time difference.

Optionally, the first transmission module 902 includes:

A Bluetooth link establishing unit, configured to establish the Bluetooth link with the audio output terminal;

The first display unit is configured to display a first switching prompt, and the first switching prompt is used to confirm whether to output sound through the audio output terminal;

The first transmission unit is configured to transmit the audio data to the audio output terminal through the Bluetooth link in response to the confirmation instruction.

Optionally, the device also includes:

A third transmission module, configured to transmit the image data and the audio data to the screen projection device through the data communication link in response to a cancel instruction.

The first transmission module 902 includes:

A state acquiring unit, configured to acquire the device state of the audio output terminal when the Bluetooth link is established between the audio output terminal and the wireless terminal;

The second transmission unit is configured to transmit the audio data to the audio output terminal through the Bluetooth link when the audio output terminal is in use.

Optionally, the device includes:

The fourth transmission module is configured to transmit the image data and the audio data to the screen projection device through the data communication link in response to a switching instruction.

Optionally, the fourth transmission module includes:

a first display unit, configured to display an audio channel switching interface;

A first switching transmission unit, configured to trigger the switching instruction in response to an audio channel switching operation in the audio channel switching interface, and transmit the image data and the image data to the screen projection device through the data communication link. described audio data.

Optionally, the fourth transmission module includes:

a link quality determining unit, configured to determine the link quality of the Bluetooth link;

The second display unit is configured to display a second switching prompt when the link quality does not meet the transmission requirement, and the second switching prompt is used to confirm whether to output sound through the screen projection device;

A second switching transmission unit, configured to trigger the switching instruction in response to a confirmation operation of the second switching prompt, and transmit the image data and the audio to the screen projection device through the data communication link data.

Optionally, the device also includes:

A prompt module, configured to display a third switching prompt when the link quality meets the transmission requirements, and the third switching prompt is used to confirm whether to output sound through the audio output terminal;

A fifth transmission module, configured to transmit the audio data to the audio output terminal through the Bluetooth link in response to a confirmation operation of the third switching prompt.

Please refer to FIG. 10 , which shows a structural block diagram of a wireless terminal provided by an exemplary embodiment of the present application. The wireless terminal 1000 in this application may include one or more of the following components: a processor 1010 , a memory 1020 and a Bluetooth chip 1030 .

Processor 1010 may include one or more processing cores. The processor 1010 uses various interfaces and lines to connect various parts in the entire wireless terminal 1000, and executes by running or executing instructions, programs, code sets or instruction sets stored in the memory 1020, and calling data stored in the memory 1020. Various functions and processing data of the wireless terminal 1000. Optionally, the processor 1010 may adopt at least one of Digital Signal Processing (Digital Signal Processing, DSP), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), and Programmable Logic Array (Programmable Logic Array, PLA). implemented in the form of hardware. The processor 1010 may integrate one or more of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU), a neural network processor (Neural-network Processing Unit, NPU) and a modem, etc. The combination. Among them, the CPU mainly handles the operating system, user interface and application programs, etc.; the GPU is used to render and draw the content that needs to be displayed on the display screen; the NPU is used to realize artificial intelligence (Artificial Intelligence, AI) functions; the modem is used to process wireless communication. It can be understood that, the above-mentioned modem may not be integrated into the processor 1010, but may be realized by a single chip.

The memory 1020 may include random access memory (Random Access Memory, RAM), and may also include read-only memory (Read-Only Memory, ROM). Optionally, the memory 1020 includes a non-transitory computer-readable storage medium. The memory 1020 may be used to store instructions, programs, codes, sets of codes or sets of instructions. The memory 1020 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), Instructions and the like for implementing the following method embodiments; the storage data area can store data created according to the use of the wireless terminal 1000 (such as audio data, phone book) and the like.

The Bluetooth chip 1030 is a component for realizing the Bluetooth function. Wherein, the Bluetooth chip 1030 includes two parts, Host and Controller (corresponding to different Bluetooth protocol stacks), and the Host and Controller can run on the same chip (single-chip architecture), or can run on different chips (dual-chip architecture). For example, the Host runs on the processor, and the Controller runs on the Bluetooth module; or, both the Host and the Controller run on the Bluetooth chip 1030 .

In addition, those skilled in the art can understand that the structure of the wireless terminal 1000 shown in the above drawings does not constitute a limitation on the electronic device, and the electronic device may include more or less components than those shown in the figure, or combine certain components, or a different arrangement of components. For example, the wireless terminal 1000 also includes components such as a display screen, a sensor, a speaker, a microphone, a power supply, and a WiFi component, which will not be repeated here.

The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores at least one program, and at least one program is loaded and executed by a processor to realize the projection in the wireless terminal as described in each of the above embodiments. screen method.

The embodiment of the present application also provides a computer program product, where the computer program product includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The bluetooth chip of the electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device executes the screen projection method in the wireless terminal provided in various optional implementation manners of the above aspect.

It should be understood that the "plurality" mentioned herein refers to two or more than two. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B may indicate: A exists alone, A and B exist simultaneously, and B exists independently. The character "/" generally indicates that the contextual objects are an "or" relationship. In addition, the numbering of the steps described herein only exemplarily shows a possible sequence of execution among the steps. In some other embodiments, the above-mentioned steps may not be executed according to the order of the numbers, such as two different numbers The steps are executed at the same time, or two steps with different numbers are executed in the reverse order as shown in the illustration, which is not limited in this embodiment of the present application.

The above are only optional embodiments of the application, and are not intended to limit the application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the application shall be included in the protection of the application. within range.

Claims

A screen projection method in a wireless terminal, the method comprising:

Establishing a data communication link with the screen projection device in response to the screen projection instruction;

In the case that the audio output terminal establishes a Bluetooth link with the wireless terminal, transmit the audio data in the projection data to the audio output terminal through the Bluetooth link, so that the audio output terminal outputs sound based on the audio data ;

The image data in the screen projection data is transmitted to the screen projection device through the data communication link, so that the screen projection device outputs a picture based on the image data.
The method according to claim 1, wherein, before transmitting the audio data in the projection data to the audio output terminal through the Bluetooth link, the method comprises:

The transmission of the image data and the audio data is synchronized based on the link states of the data communication link and the Bluetooth link.
The method according to claim 2, wherein said synchronizing the transmission of the image data and the audio data based on the link state of the data communication link and the Bluetooth link comprises:

determining a first transmission delay of the data communication link and a second transmission delay of the Bluetooth link;

determining a transmission time difference based on the first transmission delay and the second transmission delay;

The image data and the audio data are transmitted synchronously based on the transmission time difference.
The method according to any one of claims 1 to 3, wherein, when the audio output terminal establishes a Bluetooth link with the wireless terminal, the screen projection data is transmitted to the audio output terminal through the Bluetooth link audio data, including:

Establishing the Bluetooth link with the audio output terminal;

displaying a first switching prompt, the first switching prompt is used to confirm whether to output sound through the audio output terminal;

Transmitting the audio data to the audio output terminal via the Bluetooth link in response to an acknowledgment instruction.
The method according to claim 4, wherein, after displaying the first switching prompt, the method further comprises:

In response to a cancel instruction, transmitting the image data and the audio data to the screen projection device via the data communication link.
The method according to any one of claims 1 to 3, wherein the Bluetooth link is established before receiving the screen projection instruction;

In the case where the audio output terminal and the wireless terminal have a bluetooth link, transmitting the audio data in the projection data to the audio output terminal through the bluetooth link includes:

When the Bluetooth link is established between the audio output terminal and the wireless terminal, acquire the device status of the audio output terminal;

When the audio output terminal is in use, transmit the audio data to the audio output terminal through the Bluetooth link.
The method according to any one of claims 1 to 3, wherein the method further comprises:

In response to a switching instruction, transmitting the image data and the audio data to the screen projection device through the data communication link.
The method according to claim 7, wherein the transmitting the image data and the audio data to the screen projection device through the data communication link in response to the switching instruction comprises:

Display the audio channel switching interface;

In response to an audio channel switching operation in the audio channel switching interface, the switching instruction is triggered, and the image data and the audio data are transmitted to the screen projection device through the data communication link.
The method according to claim 7, wherein the transmitting the image data and the audio data to the screen projection device through the data communication link in response to the switching instruction comprises:

determining a link quality of the bluetooth link;

In the case that the link quality does not meet the transmission requirement, displaying a second switching prompt, the second switching prompt is used to confirm whether to output sound through the screen projection device;

In response to a confirmation operation on the second switching prompt, the switching instruction is triggered, and the image data and the audio data are transmitted to the screen projection device through the data communication link.
The method according to claim 9, wherein the switching instruction is triggered in response to the confirmation operation of the second switching prompt, and the image is transmitted to the screen projection device through the data communication link After the data and the audio data, the method also includes:

When the link quality meets the transmission requirement, displaying a third switching prompt, where the third switching prompt is used to confirm whether to output sound through the audio output terminal;

In response to a confirmation operation of the third switching prompt, the audio data is transmitted to the audio output terminal through the Bluetooth link.
A screen projection device, the device comprising:

A data link establishment module, configured to establish a data communication link with the screen projection device in response to the screen projection instruction;

The first transmission module is configured to transmit the audio data in the projection data to the audio output terminal through the Bluetooth link when the audio output terminal and the wireless terminal establish a Bluetooth link, so that the audio output terminal outputting sound based on the audio data;

The second transmission module is configured to transmit the image data in the screen projection data to the screen projection device through the data communication link, so that the screen projection device outputs a picture based on the image data.
The device according to claim 11, wherein the device further comprises:

A synchronization module, configured to synchronize the transmission of the image data and the audio data based on the link states of the data communication link and the Bluetooth link.
The device according to claim 12, wherein the synchronization module is specifically used for:

determining a first transmission delay of the data communication link and a second transmission delay of the Bluetooth link;

determining a transmission time difference based on the first transmission delay and the second transmission delay;

The image data and the audio data are transmitted synchronously based on the transmission time difference.
The device according to any one of claims 11 to 13, wherein the first transmission module includes:

A Bluetooth link establishing unit, configured to establish the Bluetooth link with the audio output terminal;

The first display unit is configured to display a first switching prompt, and the first switching prompt is used to confirm whether to output sound through the audio output terminal;

The first transmission unit is configured to transmit the audio data to the audio output terminal through the Bluetooth link in response to the confirmation instruction.
The device according to claim 14, wherein after the display of the first switching prompt, the device further comprises:

A third transmission module, configured to transmit the image data and the audio data to the screen projection device through the data communication link in response to a cancel instruction.
The device according to any one of claims 11 to 13, wherein the Bluetooth link is established before receiving the screen projection instruction;

The first transmission module includes:

A state acquiring unit, configured to acquire the device state of the audio output terminal when the Bluetooth link is established between the audio output terminal and the wireless terminal;

The second transmission unit is configured to transmit the audio data to the audio output terminal through the Bluetooth link when the audio output terminal is in use.
The device according to any one of claims 11 to 13, wherein the device comprises:

The fourth transmission module is configured to transmit the image data and the audio data to the screen projection device through the data communication link in response to a switching instruction.
A wireless terminal, the wireless terminal includes a processor and a memory, at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement any one of claims 1 to 10 The screen projection method in the wireless terminal.
A computer-readable storage medium, at least one program is stored in the computer-readable storage medium, and the at least one program is loaded and executed by a processor to implement the wireless terminal according to any one of claims 1 to 10. Screencasting method.
A computer program product, the computer program product includes computer instructions, the computer instructions are stored in a computer-readable storage medium; the bluetooth chip of the electronic device reads the computer instructions from the computer-readable storage medium, the The processor executes the computer instructions, so that the electronic device executes the screen projection method in the wireless terminal according to any one of claims 1 to 10.