CN113890932A - Audio control method and system and electronic equipment - Google Patents

Abstract

The application provides an audio control method, an audio control system and electronic equipment, relates to the field of terminals, and can flexibly switch audio data of a certain terminal to other terminals for playing. The terminal can select other equipment according to the input of the user, process the audio data to be switched by combining the capabilities of the other equipment, send the processed audio data to the other equipment, and directly play the audio data by the other equipment or play the audio data after further processing.

Description

Audio control method and system and electronic equipment

The present application claims the following priority from 9 chinese patent applications. Wherein, these 9 chinese patent applications include: the Chinese patent application with the application number of 202010628962.7, which is submitted by the national intellectual property office on the year 2020, month 07 and 02; submitting a Chinese patent application with the application number of 202011149031.5 by the national intellectual property office on 23/10/2020; the Chinese patent application with the application number of 202011058010.2 is filed in 29/09/29/2020 by the national intellectual property office; submitting a Chinese patent application with the application number of 202010847900.5 by the national intellectual property office on 21/08/2020; the Chinese patent application with the application number of 202010718939.7 is filed on 23.07.23.2020; the Chinese patent application with the application number of 202010700459.8 is submitted to the national intellectual property office in 20/07/2020; submitting a Chinese patent application with the application number of 202010774729.X by the national intellectual property office on 04.08 month in 2020; the Chinese patent application with the application number of 202010701720.6 is submitted to the national intellectual property office in 20/07/2020; the chinese patent application with application number 202011360717.9, the national intellectual property office, was filed on 27/11/2020. The entire contents of the above-mentioned 9 chinese patent applications are incorporated by reference in the present application.

Technical Field

The present application relates to the field of terminals, and in particular, to an audio control method, system and electronic device.

Background

Intelligent terminals such as mobile phones generally have audio input and output functions. For example, a mobile phone may play audio through its own speaker (spaker) or record audio through its own microphone (mic). For another example, the mobile phone may output audio through an audio output device such as a wired headset, a bluetooth headset, or a bluetooth speaker.

Due to the rapid development of smart terminal technology in recent years, a plurality of terminal devices are often provided in one user or home. For example, a user's home may include an interconnected system of terminals such as a number of cell phones, televisions, speakers, and PCs. In such an internet system, there is an increasing demand for users to deliver audio from a certain terminal to one or more other terminals for playing. How to flexibly play and control the audio data of a certain terminal on one or more other terminals becomes an urgent problem to be solved.

Disclosure of Invention

The application provides an audio control method, an audio control system and electronic equipment, which can flexibly switch the audio of a certain terminal to one or more other terminals for playing, and improve the audio use experience of a user.

In order to achieve the purpose, the technical scheme is as follows:

in a first aspect, the present application provides an audio control method, including: in response to a first operation input by a user, the first device can display a candidate device list, wherein the candidate device list comprises at least one candidate device with an audio output function; if the first device detects that the user selects the second device from the candidate device list, indicating that the user wishes to switch the audio function of the first device to the second device for implementation, the first device is a master device, and the second device is a slave device (the second device may be an electronic device with an operating system, such as a sound box, a television, or a mobile phone); furthermore, the first device may obtain an audio capability parameter of the second device, where the audio capability parameter is used to indicate a hardware capability of the second device to play audio and a software capability of the second device to play audio; in this way, the first device may determine the first audio playing policy according to the audio capability parameter of the second device, so as to perform the first processing on the first audio data from the first audio application according to the first audio playing policy, and obtain the second audio data; furthermore, the first device can send the second audio data to the second device for playing.

It can be seen that the first device may obtain the audio capability parameter of the slave device (i.e., the second device) when performing audio switching, and since the audio capability parameter can reflect the software processing capability and the hardware processing capability of the second device when playing audio, the first device can determine the first audio playing policy matched with the audio capability of the second device according to the audio capability parameter of the second device. The software processing capability of the second device when playing audio is mainly determined by software characteristics (for example, sound effect processing algorithm), and the hardware processing capability of the second device when playing audio is mainly determined by hardware characteristics (for example, devices of a speaker, etc.), which will be described in detail in the detailed description.

In addition, the first processing may include sound effect processing, mixing according to the number of channels, and the like, and will be described in detail below with reference to possible embodiments.

In this way, after the first device processes the first audio data from the first audio application according to the first audio playing policy, the second audio data obtained after the processing can be sent to the second device, so that the slave device can play the audio data from the first device matching with the audio capability of the slave device. For example, the second device may directly play the second audio data after receiving the second audio data, or may process the second audio data and then play the second audio data. Therefore, the first device can flexibly switch the audio playing function of the first device to one or more other slave devices, and control the audio playing process of the slave devices according to the audio capabilities of the slave devices, so that better audio using experience is provided for users, and meanwhile, the audio processing capability of the slave devices can be utilized to complete the processing and playing of audio data in cooperation with the master device.

In a possible implementation manner, the audio capability parameter may specifically include a sound effect parameter, where the sound effect parameter is used to indicate whether the second device starts a sound effect mode; at this time, the first device determines a first audio playing policy according to the audio capability parameter of the second device, including: if the second device starts the sound effect mode, the first device can set the first audio data not to be subjected to sound effect processing in the first audio playing strategy; if the second device does not start the audio mode, the first device may set the first audio playing strategy to perform audio processing on the first audio data.

Correspondingly, if the first audio playing policy sets sound effect processing on the first audio data, the first processing performed on the first audio data by the first device may specifically include sound effect processing, such as adding a subwoofer sound effect. If the first audio playing strategy sets that the sound effect processing is not carried out on the first audio data, the first equipment does not need to carry out the sound effect processing on the first audio data, and the sound effect processing can be carried out on the received audio data by the second equipment subsequently. Therefore, the sound effect superposition phenomenon caused by sound effect processing on the audio data by the first equipment and the second equipment can be avoided, and the sound effect missing phenomenon caused by sound effect processing on the audio data by the first equipment and the second equipment can also be avoided, so that the audio use experience of a user is improved, and the efficiency of processing the audio data by the equipment is also improved.

In a possible implementation manner, the audio capability parameter may specifically include a sound effect parameter, where the sound effect parameter is used to indicate a sound effect mode supported by the second device, for example, a dolby sound effect, a histen sound effect, and the like; in addition, the sound effect mode of audio playing is set to be the first sound effect mode in the first audio application; at this time, the first device determines a first audio playing policy according to the audio capability parameter of the second device, including: if the first sound effect mode is a sound effect mode supported by the second equipment, which indicates that the second equipment can complete the first sound effect, the first equipment can set the first audio data not to be subjected to sound effect processing in the first audio playing strategy; if the first sound effect mode is not the sound effect mode supported by the second device, which indicates that the second device has no capability of finishing the first sound effect, the first device may set sound effect processing on the first audio data according to the first sound effect mode in the first audio playing strategy.

Or, if the first audio mode is an audio mode supported by both the first device and the second device, the first device may default to the first audio playing policy to perform audio processing by the first device or the second device.

If the first audio playing strategy sets that sound effect processing is performed on the first audio data according to the first sound effect mode, the first processing performed on the first audio data by the first device may specifically be adding a first sound effect to the first audio data; if the first audio playing strategy sets that the first audio data is not subjected to sound effect processing according to the first sound effect mode, the first equipment does not need to add a first sound effect into the first audio data, and the second equipment can subsequently add the first sound effect into the received audio data.

In a possible implementation manner, the audio capability parameter may specifically include a target sampling rate, where the target sampling rate is a sampling rate supported by the second device (e.g., 48KHz or 96 KHz); at this time, the first device determines a first audio playing policy according to the audio capability parameter of the second device, including: the first device sets sampling of the first audio data according to a target sampling rate in a first audio playing strategy.

Accordingly, the first processing performed by the first device on the first audio data may specifically be resampling according to the target sampling rate. For example, 96KHz first audio data is down-sampled to 48KHz second audio data. As another example, the first audio data of 48KHz is up-sampled to the second audio data of 96 KHz.

In a possible implementation manner, the audio capability parameter may specifically include a target number of channels, where the target number of channels is the number of channels supported by the second device (e.g., two channels, 4 channels, etc.); the method for determining the first audio playing strategy by the first device according to the audio capability parameter of the second device includes: the first device sets the first audio data to be mixed according to the target channel number in the first audio playing strategy.

Accordingly, the first processing performed by the first device on the first audio data may specifically be performing mixing according to the target number of channels. For example, 8-channel first audio data is downmixed into 4-channel second audio data. For another example, the first audio data of two channels is upmixed into the second audio data of 8 channels.

In a possible implementation manner, the audio capability parameter may specifically include a delay parameter, where the delay parameter is used to indicate whether the second device supports a low-delay audio processing capability; at this time, the first device determines a first audio playing policy according to the audio capability parameter of the second device, including: if the second device supports the audio processing capability with low time delay, the first device may set, in the first audio playing policy, to process the first audio data according to the low time delay mode; if the second device does not support the audio processing capability with low delay, the first device may set the first audio data to be processed according to the non-low delay mode in the first audio playing policy. In the low-latency mode, the first device may set the duration of each frame of audio data to be smaller, so as to shorten the latency of the second device in processing the frame of audio data, compared to the non-low-latency mode.

In a possible implementation manner, the audio capability parameter may include a delay parameter, where the delay parameter is used to indicate an audio playing delay of the second device, that is, a time required for the second device to process one frame of audio data, for example, 10ms or 5 ms; at this time, the first device determines a first audio playing policy according to the audio capability parameter of the second device, including: if the audio playing time delay of the second device is smaller than the preset value, which indicates that the second device supports the audio processing capability of low time delay, the first device may set a first audio playing strategy for processing the first audio data according to a low time delay mode; and if the audio playing time delay of the second equipment is greater than or equal to the preset value and the second equipment does not support the audio processing capacity with low time delay, the first equipment sets the first audio data to be processed according to the non-low time delay mode in the first audio playing strategy.

Correspondingly, if the low delay mode is set in the first audio playing strategy, the first processing performed on the first audio data by the first device may specifically be dividing the first audio data into each frame of audio data according to a first duration; if the non-low-latency mode is set in the first audio playing policy, the first processing performed on the first audio data by the first device may specifically be dividing the first audio data into each frame of audio data according to a second duration, where the second duration is longer than the first duration.

In one possible implementation, the application layer of the first device is installed with a first preset application, such as a DV APP; the method for acquiring the audio capability parameter of the second device by the first device includes: the first device obtains the audio capability parameters of the second device from the second device by running the first preset application, namely, obtains the audio capability of the slave device through the installed DV APP. Accordingly, an application adapted to the first preset application may be installed in the second device to provide the audio capability parameters of the second device to the first device and receive the second audio data. Because the operating system of the second equipment does not need to be changed, the first equipment and the second equipment can be provided by different manufacturers, and the first equipment can be matched with the second equipment to execute the scheme only by installing the corresponding application, so that the scheme can be adapted to equipment provided by various manufacturers and can be widely applied.

In a possible implementation manner, after the first device obtains the audio capability parameter of the second device, the method further includes: a first preset application (e.g. DV APP) may create a corresponding hardware abstraction module (e.g. DMSDP Audio HAL) at the HAL according to the Audio capability parameters of the second device; at this time, the first device transmits the second audio data to the second device, including: the first device calls the DMSDP Audio HAL to send the second Audio data to the second device. That is, the first device may dynamically create a corresponding DMSDP Audio HAL according to the Audio capability parameters of the slave device when performing Audio switching, so as to send Audio data to the slave device using the DMSDP Audio HAL.

In one possible implementation, the application framework layer of the first device includes an audio policy module (e.g., AudioPolicy); at this time, the first device determines a first audio playing policy according to the audio capability parameter of the second device, including: the first preset application sends the audio capacity parameter of the second device to the audio strategy module, and the audio strategy module determines a first audio playing strategy according to the audio capacity parameter of the second device.

In one possible implementation, the application framework layer of the first device contains an audio processor (e.g., an AudioFlinger); at this time, the first device performs first processing on first audio data from a first audio application according to a first audio playing policy to obtain second audio data, including: the audio processor may receive first audio data from a first audio application; and the audio processor can acquire a first audio playing strategy from the audio strategy module, and further, the audio processor can perform first processing on the first audio data according to the first audio playing strategy and output second audio data obtained after the processing.

It should be noted that what the first processing includes depends on the specific content of the first audio playing policy, for example, the first audio playing policy requires the first device to add the first sound effect, and the first processing may include a sound effect processing procedure of adding the first sound effect. For another example, the first audio playing strategy requires the first device to sample according to a sampling rate of 48KHz, and the sampling process of 48KHz is performed according to the sampling rate of 48 KHz. In some cases, the first audio playing policy may also be null, that is, the first device is not required to perform any processing on the first audio data from the first audio APP, and at this time, the first device is not required to perform the first processing on the first audio data and may directly send the first audio data to the second device.

In a possible implementation manner, the method may further include: when the audio capability parameter of the second device is updated, for example, the sound effect mode of the second device is adjusted from the dolby sound effect to the subwoofer sound effect, and at this time, the first preset application can acquire the updated audio capability parameter of the second device; the first preset application can send the updated audio capacity parameter to the audio strategy module, so that the audio strategy module updates the first audio playing strategy into a second audio playing strategy according to the updated audio capacity parameter; furthermore, the audio policy module may output the second audio playing policy to the audio processor, and the audio processor performs a second processing on the received audio data according to the updated second audio playing policy. Similar to the first process described above, the second process includes what depends on the specific content of the second audio playback policy.

In addition, after the first preset application acquires the updated Audio capability parameter of the second device, the first preset application may also update the DMSDP Audio HAL according to the updated Audio capability parameter, so that the updated DMSDP Audio HAL can support sending the Audio data subjected to the second processing to the second device.

That is to say, when the Audio capability of the slave device (i.e., the second device) connected to the first device changes, the first device may dynamically update the Audio playing policy for processing the Audio data, and may also dynamically update the corresponding DMSDP Audio HAL, that is, keep the match between the DMSDP Audio HAL and the Audio playing policy with the capability of the slave device in time, so that the first device can perform Audio switching with the second device according to the updated Audio capability of the second device.

In one possible implementation manner, after the first device sends the second audio data to the second device, the method further includes: the first device may receive a second operation of setting the volume level by the user, such as an operation of clicking a volume + button or a volume-button; the audio policy module may set a volume level in the first audio playing policy in response to the second operation, for example, the volume is set from 100 to 120, and the set audio playing policy is a third audio playing policy; furthermore, the audio policy module may output the third audio playing policy to the audio processor, so that the audio processor modifies the gain of the received audio data according to the volume level set in the third audio playing policy, thereby modifying the volume of the audio data subsequently played by the second device.

That is, when the volume of the audio data changes, the first device may also dynamically update the audio playing policy for processing the audio data, i.e., re-customize the audio playing policy matching the audio capability of the second device.

In one possible implementation, the application layer of the first device is installed with a second audio application; after the first device transmits the second audio data to the second device, the method further comprises the following steps: the audio processor receiving audio data from a second audio application; when the type of the audio data of the second audio application is different from that of the audio data of the first audio application, the audio policy module may update the first audio playing policy to a fourth audio playing policy; furthermore, the audio strategy module outputs a fourth audio playing strategy to the audio processor, and the audio processor carries out third processing on the received audio data according to the fourth audio playing strategy; and the audio processor transmits the audio data subjected to the third processing to the second device through the hardware abstraction module. Similar to the first process described above, the specific content of the third process depends on the specific content of the fourth audio playback policy.

That is to say, in the process of audio switching, if the audio data output by the first device changes, the first device may also dynamically update the audio playing policy for processing the audio data, so that the audio data received by the second device matches with its own audio capability.

In one possible implementation manner, after the first device sends the second audio data to the second device, the method further includes: the first equipment receives a third operation that the user selects the third equipment to play the audio, namely, the user changes the slave equipment of the first equipment from the second equipment to the third equipment; in response to the third operation, the first preset application may acquire an audio capability parameter of the third device from the third device; the first preset application sends the audio capacity parameter of the third device to the audio strategy module, so that the audio strategy module updates the first audio playing strategy to a fifth audio playing strategy according to the audio capacity parameter of the third device; and the audio strategy module outputs the fifth audio playing strategy to the audio processor, so that the audio processor performs fourth processing on the received audio data according to the fifth audio playing strategy. Similar to the first process described above, the specific content of the fourth process depends on the specific content of the fifth audio playback policy.

In addition, after the first preset application acquires the Audio capability parameter of the third device from the third device, the first preset application may further update the DMSDP Audio HAL according to the Audio capability parameter of the third device, so that the updated DMSDP Audio HAL supports sending of the fourth processed Audio data to the third device.

That is, when the slave device connected to the first device changes, the first device may dynamically update the Audio playing policy for processing the Audio data, and may also dynamically update the corresponding DMSDP Audio HAL, that is, keep the match between the DMSDP Audio HAL and the Audio playing policy and the capability of the slave device in time, so that the first device can perform Audio switching with the third device according to the Audio capability of the new slave device (i.e., the third device).

In another possible implementation manner, the application program layer of the first device may install a second preset application, and the second preset application may be the same as or different from the first preset application; wherein, first equipment sends second audio data to second equipment, includes: and the first equipment sends the second audio data to a second preset application, and the second preset application sends the second audio data to the second equipment. At this point, the first default application does not need to create a corresponding DMSDP Audio HAL at the HAL.

In one possible implementation manner, before the first device displays the candidate device list, the method further includes: the method comprises the steps that a first device displays an application interface of a first audio application, and an audio switching button can be arranged in the application interface; at this time, the first operation is an operation in which the user clicks an audio switching button. That is, the user may enter the audio switch function from an audio switch button provided in the audio application during the running of the audio application.

In one possible implementation manner, before the first device displays the candidate device list, the method further includes: the first equipment displays a control center, and an audio switching button can be arranged in the control center; at this time, the first operation is an operation in which the user clicks an audio switching button. That is, the user may enter the audio switch function at the audio switch button of the control center.

In one possible implementation, the audio capability parameter of the second device may include a recording parameter, where the recording parameter is used to indicate an audio input capability, i.e., a recording capability, of the second device; similar to the process of audio playing, the method further includes: the first equipment determines an audio recording strategy according to the recording parameters; after receiving the first recording data recorded by the second equipment, the first equipment processes the first recording data according to the audio recording strategy to obtain second recording data; and the first device can report the second recording data to the audio application starting the recording function. That is, the first device may distribute audio input functions to the slave devices in addition to distributing audio output functions to the slave devices.

In a second aspect, the present application provides an audio control method, comprising: responding to a first operation input by a user, displaying a candidate device list by a first device, wherein the candidate device list comprises at least one candidate device with an audio output function; responding to the operation that a user selects a second device in the candidate device list, and acquiring an audio capability parameter of the second device by the first device, wherein the audio capability parameter is used for indicating the hardware capability of the second device for playing audio and the software capability of the second device for playing audio; the first device may determine from the audio capability parameter of the second device that no processing of audio data from the first audio application is required; for example, since the audio capability parameter of the second device indicates that the second device has the capability of sampling, sound effect processing, etc., then the first device may determine that the first device does not need to sample, add sound effect, etc., to the audio data from the first audio application, but the second device processes the audio data from the first audio application; then, the first device may directly transmit the audio data of the first audio application to the second device. After receiving the audio data sent by the first device, the second device may perform corresponding processing (e.g., sampling, adding sound effects, etc.) on the audio data according to its own audio capability, and play the processed audio data. Of course, the second device may also directly play the audio data after receiving the audio data sent by the first device, which is not limited in this embodiment of the application.

For example, if the audio capability parameter of the second device indicates that the second device turns on dolby sound, the first device may determine that the first device is not required to perform the sound processing of dolby sound on the audio data. When the second device receives the audio data sent by the first device, the sound effect processing of the Dolby sound effect can be carried out on the audio data. For another example, if the audio capability parameter of the second device indicates that the second device has mixing capability, the first device may determine that the first device is not required to perform mixing processing on the audio data. When the second device receives the audio data sent by the first device, the audio data can be mixed. In this way, the first device can fully utilize the audio processing capability of the slave device when performing audio switching, and the processing process of the audio data is completed by the first device and the slave device in a more efficient and flexible cooperation manner.

That is, the above-described processing refers to a processing operation related to the audio capability parameter of the second device. The first device may still perform conventional processing operations, such as parsing, encoding, decoding, encapsulating or decapsulating, on the audio data to be transmitted, which processing operations are typically not directly associated with the audio capability parameters of the second device.

Still alternatively, the first device may send, to the second device, in addition to the audio data from the first audio application, an audio policy when the second device processes the audio data. For example, if the first device does not turn on the sound effect mode and the second device turns on the sound effect mode, the first device may set sound effect processing by the second device on the received audio data in the audio policy. Then, after receiving the audio policy and the audio data from the first audio application, the second device may perform sound effect processing on the audio data according to the audio policy, thereby reducing the operation load of the first device.

In a third aspect, the present application provides an audio control method, including: responding to a first operation input by a user, displaying a candidate device list by a first device, wherein the candidate device list comprises at least one candidate device with an audio output function; responding to the operation that a user selects a second device and a third device from a candidate device list, wherein the first device respectively acquires an audio capability parameter of the second device and an audio capability parameter of the third device, at this time, a slave device of the first device comprises the second device and the third device, the audio capability parameter of the second device is used for indicating the hardware capability of the second device for playing audio and the software capability of the second device for playing audio, and the audio capability parameter of the third device is used for indicating the hardware capability of the third device for playing audio and the software capability of the third device for playing audio; furthermore, the first device can determine a multi-device audio playing strategy according to the audio capability parameter of the second device and the audio capability parameter of the third device; the first device may perform first processing on first audio data from the first audio application according to the multi-device audio playing policy to obtain second audio data corresponding to the second device and third audio data corresponding to the third device; furthermore, the first device may send the second audio data to the second device for playing, and send the third audio data to the third device for playing.

Similar to the first aspect or the second aspect in which the first device switches the audio data to the second device for playing, after the first device sends the second audio data to the second device, the second device may directly play the second audio data, or may process the second audio data and then play the second audio data. Similarly, after the first device sends the third audio data to the third device, the third device may directly play the third audio data, or may process and play the third audio data. Therefore, the first device can simultaneously switch the audio playing function of the first device to the plurality of slave devices, and the distributed audio capability of the cross-device is realized.

In a possible implementation manner, when the audio capability parameter of the second device is different from the audio capability parameter of the third device, the multi-device audio playing policy includes a first policy and a second policy; the method for processing the first audio data by the first device according to the multi-device audio playing strategy to obtain the second audio data corresponding to the second device and the third audio data corresponding to the third device includes: the first equipment copies the first audio data to obtain first data and second data (the first data is the same as the first audio data, and the second data is the same as the first audio data); furthermore, the first device may process the first data stream according to a first policy to obtain second audio data; and, the first device may process the second data stream according to a second policy to obtain third audio data.

Illustratively, the application layer of the first device installs a first preset application (e.g., DV APP); after the first device obtains the audio capability parameter of the second device and the audio capability parameter of the third device, respectively, because the audio capability parameter of the second device is different from the audio capability parameter of the third device, the first preset application may create a first hardware abstraction module in the HAL according to the audio capability parameter of the second device, and create a second hardware abstraction module in the HAL according to the audio capability parameter of the third device; at this time, the first device may transmit the second audio data to the second device through the first hardware abstraction module, and the first device may transmit the third audio data to the third device through the second hardware abstraction module.

In a possible implementation manner, when the audio capability parameter of the second device is the same as the audio capability parameter of the third device, the first device performs a first process on the first audio data from the first audio application according to the multi-device audio playing policy to obtain second audio data corresponding to the second device and third audio data corresponding to the third device, including: the first equipment carries out first processing on the first audio data according to the multi-equipment audio playing strategy to obtain second audio data; and the first equipment copies the second audio data to obtain third audio data.

For example, when the audio capability parameter of the second device is the same as the audio capability parameter of the third device, the first preset application installed on the first device may create a hardware abstraction module at the HAL according to the audio capability parameter of the second device (or the audio capability parameter of the third device), and at this time, only one hardware abstraction module needs to be created; the first device may send the second audio data to the second device through the hardware abstraction module, and the first device may send the third audio data to the third device through the hardware abstraction module. That is, the hardware abstraction modules created by the first preset application HAL correspond to the audio capability parameters acquired by the first device, each hardware abstraction module corresponding to a set of uniquely determined audio capability parameters.

In a fourth aspect, the present application provides an audio control method, comprising: responding to a first operation input by a user, displaying a candidate device list by a first device, wherein the candidate device list comprises at least one candidate device with an audio input function; in response to an operation that a user selects a second device (the second device is a sound box, or a television, or a mobile phone) from the candidate device list, the first device acquires an audio capability parameter of the second device, wherein the audio capability parameter is used for indicating hardware capability of the second device for recording audio and software capability of the second device for recording audio; the first device may determine a first audio recording policy according to the audio capability parameter of the second device; subsequently, when the first device receives the first recording data sent by the second device, the first device may perform first processing on the first recording data according to the first audio recording policy to obtain second recording data.

It can be seen that, similar to the audio control method in the first aspect described above, in the audio control method in the fourth aspect, the first device (i.e., the master device) may determine the corresponding first audio recording policy according to the capability related to the audio output function in the audio capability parameters of the second device (i.e., the slave device). Therefore, the first device can flexibly switch the self audio input function to the slave device according to the first audio recording strategy, and can utilize the audio recording capability of the slave device to cooperate with the master device to complete the recording process of the audio data.

In addition, what the first processing includes depends on the specific content of the first audio recording policy, for example, the first audio recording policy requires the first device to perform echo cancellation using the 3A algorithm, and the first processing may include a processing procedure of performing echo cancellation using the 3A algorithm. In some cases, the first audio recording policy may also be null, that is, the first device does not need to perform any processing on the first audio recording data sent from the second device, and at this time, the first device does not need to perform the first processing on the first audio recording data.

In a possible implementation manner, after the first device obtains the audio capability parameter of the second device, the method further includes: when the audio capability parameter of the second device is updated, the first device can acquire the updated audio capability parameter of the second device; furthermore, the first device may update the first audio recording policy to a second audio recording policy according to the updated audio capability parameter; subsequently, when the first device receives the third recording data sent by the second device, the first device may perform second processing on the third recording data according to the second audio recording policy to obtain fourth recording data.

That is, when the audio recording capability of the slave device (i.e., the second device) to which the first device is connected changes, the first device may dynamically update the audio recording policy for processing the recorded data. In this way, the processing (i.e., the second processing) of the subsequently received recording data by the first device may also be adjusted accordingly, so that the first device can perform audio switching with the second device according to the updated audio recording capability of the second device. Similar to the first process described above, the second process includes what depends on the specific content of the second audio playback policy.

In a possible implementation manner, after the first device obtains the audio capability parameter of the second device, the method further includes: the first device can receive a second operation that the user selects the third device to record the audio, namely, the user changes the audio input device of the first device from the second device to the third device; in response to the second operation, the first device may acquire an audio capability parameter of the third device from the third device; furthermore, the first device may determine a third audio recording strategy according to the audio capability parameter of the third device; subsequently, when the first device receives fifth recording data sent by the third device, the first device performs third processing on the fifth recording data according to a third audio recording strategy to obtain sixth recording data.

That is, when the audio output device connected to the first device is changed, the first device may dynamically update the audio playing policy for processing the recording data. In this way, the processing (i.e., the third processing) of the subsequently received recording data by the first device may also be adjusted accordingly, so that the first device can perform audio switching with the third device according to the audio recording capability of the third device. Similar to the first process described above, the third process includes what depends on the specific content of the third audio playback policy.

In a possible implementation manner, after the first device obtains the audio capability parameter of the second device, the method further includes: the method comprises the steps that a first device creates a corresponding hardware abstraction module in a HAL according to audio capability parameters of a second device; at this time, the receiving, by the first device, the first sound recording data sent by the second device includes: the first device receives first recording data sent by the second device through the hardware abstraction module. That is, the first device may dynamically create a corresponding DMSDP Audio HAL according to the Audio capability parameters of the slave device when performing Audio switching, so as to receive the recording data sent by the slave device by using the DMSDP Audio HAL.

Similar to the Audio playing process, when the Audio capability parameter of the second device is updated or the Audio capability parameter of the new slave device is acquired, the first device may further update the DMSDP Audio HAL according to the latest Audio capability parameter, so that the updated DMSDP Audio HAL can perform data transceiving with the current slave device.

Certainly, similar to the audio playing process, the first device may also include a first preset application (e.g., DV APP), an audio policy module (e.g., AudioPolicy), an audio processor (e.g., AudioFlinger), and the like, and these functional modules are similar to the audio playing process in the audio recording process, and therefore are not described herein again.

In a fifth aspect, the present application provides an audio control method, including: the method comprises the steps that a second device sends a first audio capability parameter to a first device, wherein the first audio capability parameter is used for indicating the hardware capability of the second device for playing audio and the software capability of the second device for playing audio, and the second device is a sound box, a television or a mobile phone; subsequently, the second device may receive first audio data sent by the first device, where the first audio data is obtained after the first device processes according to the first audio capability parameter; furthermore, the second device may play the first audio data, or the second device may process the first audio data and then play the processed first audio data.

In a possible implementation manner, the application program layer in the second device is installed with a preset application; wherein the second device sends the first audio capability parameter to the first device, comprising: the second equipment sends the first audio capability parameter to the first equipment by running a preset application; the second device receives the first audio data sent by the first device, and the method comprises the following steps: the second device receives the first audio data sent by the first device by running a preset application.

That is to say, for the second device (i.e. the slave device of the first device), after the second device only needs to install the preset application (which may be referred to as an agent APP), the second device may report its own audio capability parameter to the first device through the agent APP, and acquire the audio data matching its own audio capability from the first device to play. Because the operating system of the second equipment does not need to be changed, the first equipment and the second equipment can be provided by different manufacturers, and the first equipment can be matched with the second equipment to execute the scheme only by installing the corresponding application, so that the scheme can be adapted to equipment provided by various manufacturers and can be widely applied.

In a possible implementation manner, after the second device plays the first audio data, or after the second device processes and plays the first audio data, the method further includes: the second device sets the third device as an audio output device (the third device is different from the second device, and the third device is different from the first device); the second equipment sends a second audio capacity parameter to the first equipment, wherein the second audio capacity parameter is used for indicating the hardware capacity of the third equipment for playing the audio and the software capacity of the third equipment for playing the audio; the second equipment receives second audio data sent by the first equipment, wherein the second audio data are obtained after the first equipment is processed according to the second audio capacity parameter; and the second audio data is played by the second equipment, or the second audio data is played after being processed by the second equipment.

That is to say, when the second device is connected to another audio output device (for example, a third device), the second device may be used as a master device of the third device to report its own audio capability parameter (i.e., a second audio capability parameter), where the second audio capability parameter reflects the audio capability of the third device, so that the first device can process audio data according to the new audio capability parameter, thereby implementing an audio switching function.

In a possible implementation manner, after the second device plays the first audio data, or after the second device processes and plays the first audio data, the method further includes: when the audio capacity of the second device changes, the second device updates the first audio capacity parameter to a third audio capacity parameter; the second device sends a third audio capability parameter to the first device; the second equipment receives third audio data sent by the first equipment, wherein the third audio data are obtained after the first equipment is processed according to the third audio capacity parameter; and the second equipment plays the third audio data, or the second equipment processes the third audio data and then plays the third audio data.

That is to say, when the audio capability of the second device changes, the second device may report its own audio capability parameter (i.e., the third audio capability parameter) again, so that the first device can process the audio data according to the new audio capability parameter, thereby implementing the audio switching function.

In a possible implementation manner, the first audio capability parameter may further indicate a hardware capability of the second device for recording audio and a software capability of the second device for recording audio, that is, the first audio capability parameter may further include a parameter that may reflect the recording capability of the second device, for example, an echo cancellation algorithm used by the second device; then, after the second device sends the first audio capability parameter to the first device, the method further includes: the second equipment starts to record audio to obtain recording data; and the second equipment sends the recording data to the first equipment so that the first equipment processes the recording data according to the first audio capability parameter.

In a possible implementation manner, the method further includes: the second device receives an audio policy sent by the first device, wherein the audio policy is determined by the first device according to the first audio capability parameter, and the audio policy may include an audio playing policy and/or an audio recording policy; at this time, the second device may process the first audio data according to the audio playing policy, and play the processed first audio data. That is to say, the first device as the master device may indicate which processes the second device specifically needs to perform when switching audio according to the audio capability parameter of the second device, so as to fully exert the audio capability of the second device and provide a better audio use experience for the user.

In a sixth aspect, the present application provides an audio control method, including: the second equipment sends a first audio capability parameter to the first equipment, wherein the first audio capability parameter is used for indicating the hardware capability of the second equipment for recording audio and the software capability of the second equipment for recording audio, and the second equipment can be a sound box, a television or a mobile phone; furthermore, the second device can call a microphone to start recording the audio to obtain first recording data; and the second equipment sends the first sound recording data to the first equipment so that the first equipment processes the first sound recording data according to the first audio capacity parameter.

In a possible implementation manner, the application program layer in the second device is installed with a preset application; wherein the second device sends the first audio capability parameter to the first device, comprising: the second equipment sends the first audio capability parameter to the first equipment by running a preset application; wherein, the second equipment sends first recording data to first equipment, includes: and the second equipment sends the first sound recording data to the first equipment by running a preset application.

That is to say, for the second device (i.e. the slave device of the first device), after the second device only needs to install the preset application (which may be referred to as the proxy APP), the second device may report the audio capability parameter related to its own audio recording capability to the first device through the proxy APP, and send the recorded data to the first device. Because the operating system of the second equipment does not need to be changed, the first equipment and the second equipment can be provided by different manufacturers, and the first equipment can be matched with the second equipment to execute the scheme only by installing the corresponding application, so that the scheme can be adapted to equipment provided by various manufacturers and can be widely applied.

In a possible implementation manner, after the second device sends the first sound recording data to the first device, the method further includes: the second device sets the third device as an audio input device (the third device is different from the second device and the third device is different from the first device); the second equipment sends a second audio capacity parameter to the first equipment, wherein the second audio capacity parameter is used for indicating the hardware capacity of the third equipment for recording the audio and the software capacity of the third equipment for recording the audio; and the second equipment sends the second recording data recorded by the third equipment to the first equipment, so that the first equipment processes the second recording data according to the second audio capacity parameter.

That is to say, when the second device is connected to another audio input device (for example, a third device), the second device may serve as a master device of the third device, and report its own audio capability parameter (i.e., a second audio capability parameter) to the first device again, where the second audio capability parameter reflects an audio recording capability of the third device, so that the first device can process recording data according to the new audio capability parameter, thereby implementing an audio switching function.

In a possible implementation manner, after the second device sends the first sound recording data to the first device, the method further includes: when the audio recording capability of the second equipment is changed, the second equipment updates the first audio capability parameter into a third audio capability parameter; the second device sends a third audio capability parameter to the first device; and the second equipment sends the recorded third recording data to the first equipment, so that the first equipment processes the third recording data according to the third audio capability parameter.

That is to say, when the audio recording capability of the second device changes, the second device may report its own audio capability parameter (i.e., the third audio capability parameter) again, so that the first device can process the recording data reported by the second device according to the new audio capability parameter, thereby implementing the audio switching function.

In a possible implementation manner, the method further includes: the second equipment receives an audio recording strategy sent by the first equipment, wherein the audio recording strategy is determined by the first equipment according to the first audio capacity parameter; at this time, the second device may record the audio according to the audio playing policy to obtain the first recording data. That is to say, the first device as the master device may indicate which processing needs to be performed by the second device when recording according to the audio capability parameter of the second device, so as to fully utilize the audio capability of the second device and provide a better audio use experience for the user.

In a seventh aspect, the present application provides an audio control method, including: after the first device establishes network connection with the second device, the first device can take the second device as a slave device and acquire a device selection strategy corresponding to the second device; subsequently, when the first device acquires first audio data to be played (the type of the first audio data is a first type), the first device may determine whether the first audio data is allowed to be played in the second device according to the first type and the device selection policy, that is, determine whether an audio output device of the first audio data is the second device; if the first audio data is allowed to be played in the second device, the first device can send the first audio data to the second device for playing.

It can be seen that when the first device (i.e., the master device) outputs the audio data to the second device (i.e., the slave device) for audio switching, the specific type of audio data can be sent to the second device for playing according to the corresponding device selection policy, so that some audio data which are not suitable for being played on the second device are filtered out for a user, different audio output devices can play corresponding audio to the user in a targeted manner, the adaptability between the audio data and the audio output devices is higher, and the audio use experience of the user is improved.

In a possible design method, the device selection policy may specifically include: the type of audio data that the second device allows to be played and the type of audio data that the second device does not allow to be played; at this time, the first device determining whether the first audio data is allowed to be played in the second device according to the first type and the device selection policy includes: the first equipment inquires whether the second equipment allows playing the audio data of the first type in the equipment selection strategy; if the second equipment allows the first type of audio data to be played, the first equipment determines that the first audio data is allowed to be played in the second equipment; if the second device does not allow the first type of audio data to be played, the first device determines that the first audio data is not allowed to be played in the second device.

Therefore, when the first device switches the audio to the second device, the multiple types of audio data allowed to be played by the second device can be projected to the second device for playing according to the device selection strategy, and the multiple types of audio data not allowed to be played by the second device in the device selection strategy cannot be projected to the second device for playing, so that some audio data which are not suitable for being played on the second device are filtered out for a user.

In a possible design method, the device selection policy may specifically include: the type of the audio data which the second device allows playing from, and the type of the audio data which the second device does not allow playing from; that is, the device selection policy may set the audio data that the second device allows or does not allow to play in the dimension of the application.

At this time, the first device determining whether the first audio data is allowed to be played in the second device according to the first type and the device selection policy includes: the first device determines that the first audio data is from a first application; further, the first device may query the device selection policy whether the second device allows playing of the first type of audio data from the first application; if the second device allows the first type of audio data from the first application to be played, the first device determines that the first audio data is allowed to be played in the second device; if the second device does not allow the first type of audio data from the first application to be played, the first device determines that the first audio data is not allowed to be played in the second device.

Therefore, when the first device switches the audio to the second device, the first device can output different types of audio data generated by one application to the corresponding audio output device according to the device selection strategy for playing, so that different audio output devices can play specific types of audio data in specific applications in a targeted manner, and the audio use experience of a user is improved.

In one possible design approach, the device selection strategy further includes audio output devices for different types of audio data; generally, the audio output device allowing the audio data to be played by the second device is the second device itself, and the audio output device not allowing the audio data to be played by the second device may be the first device or other electronic devices.

Then, if the first device determines that the first audio data is not allowed to be played in the second device according to the device selection policy, the first device may query, according to the first type of the first audio data, that the specific audio output device of the first audio data is the third device in the device selection policy (the third device is different from the second device); furthermore, the first device can send the first audio data to be played to the third device for playing. Therefore, the first audio data which are not suitable for being played in the second device can be shunted to the third device for playing, and the audio data to be played in the first device are prevented from being omitted.

In a possible design method, the third device may be an audio output device that has recently accessed the first device in addition to the second device. That is, when the first audio data is not suitable for playing on the second device, the first device may select an audio output device for the first audio data following a post-access priority principle.

In a possible design method, the device selection policy may specifically include: correspondence between different types of audio data and different audio output devices; for example, audio output devices for audio data of music type are televisions, speakers, and cell phones. At this time, the first device determining whether the first audio data is allowed to be played in the second device according to the first type and the device selection policy includes: the first device may query whether the audio output device corresponding to the first type of audio data includes the second device in the device selection policy; if the first equipment contains the second equipment, the first equipment determines that the first audio data are allowed to be played in the second equipment; if the second device is not included, the first device determines that the first audio data is not allowed to be played in the second device.

In a possible design method, the device selection policy may specifically include: correspondence between different applications, different types of audio data, and different audio output devices; for example, audio output devices for audio data of the music type generated within WeChat applications are televisions, speakers, and cell phones. At this time, the first device determining whether the first audio data is allowed to be played in the second device according to the first type and the device selection policy includes: the first device determines that the first audio data is from a first application; further, the first device may query whether the audio output device corresponding to the first type and the first application includes the second device in the device selection policy; if the first equipment contains the second equipment, the first equipment determines that the first audio data are allowed to be played in the second equipment; if the second device is not included, the first device determines that the first audio data is not allowed to be played in the second device.

For example, when the second device is a speaker type electronic device, the permission of the second device to play audio data of a music type and the prohibition of the second device to play audio data of a notification type may be set in the device selection policy; therefore, the first equipment can project the audio data of the music type to the sound box for playing, the audio data of the notification type cannot be projected to the sound box for playing, and the user is prevented from being disturbed by the notification sound in the first equipment when the sound box is used for playing the audio.

For example, when the second device is an electronic device of a vehicle-mounted device type, the permission of the second device to play the audio data of the navigation type may be set in the device selection policy, and the permission of the second device to play the audio data of the notification type may not be set; therefore, the first device can project the navigation type audio data to the vehicle-mounted device for playing, the notification type audio data cannot be projected to the vehicle-mounted device for playing, and the user is prevented from being disturbed by the notification sound in the first device when the vehicle-mounted device is used for playing the audio.

For example, when the second device is an electronic device of a large screen device type, the second device may be set in the device selection policy to allow the audio data from the first application to be played, and the second device may not allow the audio data from the second application to be played. Therefore, the first device can project the audio data of the first application to the large-screen device for playing, the audio data of the second application cannot be projected to the large-screen device for playing, and the user is prevented from being disturbed by the fact that a certain application in the first device generates audio when the large-screen device is used for playing the audio.

For example, whether the second device is a sound box type, a vehicle-mounted device type, a large-screen device type or another type of electronic device, the first device may specifically be an electronic device with strong operation and processing capabilities, such as a mobile phone, a tablet computer or a notebook computer.

In one possible design method, a preset application, such as a DV APP, is installed in an application program layer of a first device, and an application framework layer of the first device includes an audio policy module, such as AudioPolicy; the method for acquiring the device selection policy corresponding to the second device by the first device includes: after the first device establishes network connection with the second device, the preset application can determine the device type of the second device; furthermore, the preset application can obtain a corresponding equipment selection strategy according to the equipment type of the second equipment; and issues the device selection policy to the audio policy module. That is, the first device may pre-store device selection policies of multiple audio output devices, and the first device may dynamically issue the corresponding device selection policy to the AudioPolicy according to the device type of the currently connected second device, so as to adjust the audio output device of the audio data in time according to the device selection policy.

In one possible design approach, the application framework layer of the first device further comprises an audio processor, such as an AudioFlinger; the method for determining whether the first audio data is allowed to be played in the second device or not by the first device according to the first type and the device selection strategy comprises the following steps: after receiving the first audio data, the audio processor can send a query request to the audio policy module, wherein the query request comprises a first type identifier; then, in response to the query request, the audio policy module may determine whether the first audio data of the first type is allowed to be played in the second device according to the device selection policy.

In one possible design method, after the first device establishes a network connection with the second device, the method further includes: the first device creates a first hardware abstraction module corresponding to the second device, for example DMSDP Audio HAL, at the hardware abstraction layer HAL; wherein, if the first audio data is allowed to be played in the second device, the first device sends the first audio data to the second device for playing, including: the audio processor receives a first indication sent by the audio policy module, wherein the first indication is used for indicating that the audio output device of the first audio data is a second device; in response to the first indication, the audio processor may invoke the first hardware abstraction module to send the first audio data to the second device.

In one possible design approach, the HAL further includes a second hardware abstraction module corresponding to a third device, such as a Primary HAL, A2dp HAL, or the like; the method further comprises the following steps: the audio processor receives a second indication sent by the audio policy module, wherein the second indication is used for indicating that the audio output device of the first audio data is a third device, that is, the first audio data is not allowed to be played on the second device; then, in response to the second indication, the audio processor may invoke the second hardware abstraction module to send the second audio data to the third device.

In one possible design method, after the first device obtains the device selection policy corresponding to the second device, the method further includes: the first equipment displays a setting interface of the audio output equipment; the method comprises the steps that a first device receives a device selection strategy corresponding to a second device, which is set in a setting interface by a user; in response to the user's settings, the first device updates the device selection policy. In this way, the user can set or modify the audio types allowed to be played on each audio output device in the setting interface according to the own needs, that is, a corresponding device selection policy is set for each audio output device, so that various types of audio data can be distributed to the corresponding audio output devices by the first device according to the setting of the user and played.

In one possible design method, the method further includes: when the second device plays the first audio data, the first device acquires second audio data to be played (the type of the second audio data is a second type), namely the audio data to be played simultaneously comprises the first audio data and the second audio data; furthermore, the first device may determine whether the second audio data is allowed to be played in the second device according to the second type and the device selection policy; if the second audio data is allowed to be played in the second device, the first device can send the first audio data and the second audio data to the second device for playing after mixing; or, the first device may send the unmixed first audio data and second audio data to the second device for playing, which is not limited in this embodiment of the present application.

In one possible design method, the method further includes: when the first device stops sending the first audio data to the second device, the first device acquires second audio data to be played (the type of the second audio data is a second type), namely the audio data to be played is changed from the first audio data to the second audio data; further, similar to the first audio data, the first device may determine whether the second audio data is allowed to be played in the second device according to the second type and the device selection policy; and if the second audio data are allowed to be played in the second equipment, the first equipment sends the second audio data to the second equipment for playing. In this way, when different types of audio data are played, the first device may dynamically determine the audio output device for each channel of audio data according to the device selection policy, so that each channel of audio data may be played on the most appropriate audio output device.

Of course, if the slave device connected to the first device is updated from the second device to the third device, the first device may also select an appropriate audio output device for the audio data to be played at this time according to the device selection policy of the third device (i.e., the new slave device) similarly to the process in which the first device switches the audio data to the second device for playing.

In an eighth aspect, the present application provides an audio control method, including: after the first device establishes network connection with the second device, the first device can take the second device as a slave device to acquire a sound mixing strategy corresponding to the second device; subsequently, when the first device acquires first audio data (the type of the first audio data is a first type) and second audio data (the type of the second audio data is a second type) to be played, it is indicated that multiple paths of audio data need to be played simultaneously, and at this time, the first device can determine whether to mix the first audio data and the second audio data according to the first type, the second type and a mixing strategy; if the first audio data and the second audio data do not need to be mixed, the first equipment can send the unmixed first audio data and second audio data to the second equipment for playing; correspondingly, if the first audio data and the second audio data need to be mixed, the first device may mix the first audio data and the second audio data into third audio data, and then audio characteristics (such as loudness, frequency, and the like) of the first audio data and/or the second audio data in the mixed third audio data may be changed; furthermore, the first device can send the mixed third audio data to the second device for playing.

It can be seen that, when a first device (i.e. a master device) outputs multiple audio data to a second device (i.e. a slave device) for audio switching, some of the audio data can be sent to the second device without mixing according to a corresponding mixing strategy. Therefore, the first audio data and the second audio data acquired by the second device from the first device are not subjected to audio mixing processing, so that the audio characteristics of the first audio data and the second audio data can be reflected really, and the first audio data and the second audio data to be played, which are obtained by the second device, do not have distortion phenomenon caused by the audio mixing processing of the first device, so that the distortion phenomenon generated when the second device plays the first audio data and the second audio data is correspondingly reduced, thereby reducing the distortion condition of multi-channel audio data during cross-device playing and improving the audio output quality of an audio switching scene.

In a possible implementation manner, the mixing strategy may specifically include: a type of audio data that requires mixing and/or a type of audio data that does not require mixing;

the method for determining whether the first audio data and the second audio data need to be mixed according to the first type, the second type and the mixing strategy by the first device includes: the method comprises the steps that a first device inquires whether audio data of a first type needs sound mixing or not and whether audio data of a second type needs sound mixing or not in a sound mixing strategy; the first device may determine that mixing of the first audio data and the second audio data is not required if at least one of the first type of audio data and the second type of audio data does not require mixing; if both the first type of audio data and the second type of audio data require mixing, the first device may determine that mixing of the first audio data and the second audio data is required.

In a possible implementation manner, the mixing strategy may specifically include: the type of the audio data to be mixed and the application from which the audio data to be mixed comes, and/or the type of the audio data not to be mixed and the application from which the audio data not to be mixed comes;

the method for determining whether the first audio data and the second audio data need to be mixed according to the first type, the second type and the mixing strategy by the first device includes: the first device determines that the first audio data is from a first application and the second audio data is from a second application; further, the first device may query whether the first type of audio data from the first application requires mixing and whether the second type of audio data from the second application requires mixing in the mixing policy; the first device may determine that mixing of the first audio data and the second audio data is not required if at least one of the first type of audio data from the first application and the second type of audio data from the second application does not require mixing; the first device may determine that mixing of the first audio data and the second audio data is required if both the first type of audio data from the first application and the second type of audio data from the second application require mixing.

In a possible implementation manner, the mixing strategy may specifically include: correspondence between different types of audio data and audio output devices that do not allow mixing; for example, audio output devices that do not allow mixing corresponding to audio data of a music type include a cellular phone, a sound box, and a television.

The method for determining whether the first audio data and the second audio data need to be mixed according to the first type, the second type and the mixing strategy by the first device includes: the method comprises the steps that a first device inquires whether a first audio output device corresponding to first type audio data comprises a second device in a sound mixing strategy; the first device inquires whether a second audio output device corresponding to the second type of audio data comprises a second device in the mixing strategy; if the first audio output device comprises a second device and/or the second audio output device comprises a second device, that is, at least one of the first audio data and the second audio data does not need to be mixed, the first device may determine that the first audio data and the second audio data do not need to be mixed; if the first audio output device does not include the second device and the second audio output device does not include the second device, that is, both the first audio data and the second audio data require mixing, the first device may determine that mixing of the first audio data and the second audio data is required.

That is, the first device may inquire whether the second device allows mixing of each audio data in the above-described mixing strategy according to the type of the current multiple audio data and the type of the current slave device (i.e., the second device). If one path of audio data does not allow mixing, the first device can independently transmit the path of audio data to the second device, and distortion of the audio data caused by mixing is reduced.

It is understood that the correspondence between different types of audio data and audio output devices that allow mixing may also be set in the mixing strategy; for example, audio output devices allowing mixing corresponding to the notification type audio data include headphones and televisions.

At this time, corresponding to the above method, the first device may query, in the sound mixing policy, whether the first audio output device corresponding to the first type of audio data includes the second device; the first device inquires whether a second audio output device corresponding to the second type of audio data comprises a second device in the mixing strategy; if the first audio output device comprises a second device and the second audio output device comprises a second device, which indicates that the first audio data and the second audio data both need to be mixed, the first device may determine that the first audio data and the second audio data need to be mixed; if the first audio output device does not include the second device and/or the second audio output device does not include the second device, indicating that at least one of the first audio data and the second audio data does not require mixing, the first device may determine that mixing of the first audio data and the second audio data is not required.

In a possible implementation manner, the mixing strategy may specifically include: correspondence between different types of audio data, different applications, and audio output devices that do not allow mixing; for example, audio output devices that do not allow mixing corresponding to audio data of a music type from the WeChat APP include a cellular phone, a sound box, and a television.

The method for determining whether the first audio data and the second audio data need to be mixed according to the first type, the second type and the mixing strategy by the first device includes: the first device determines that the first audio data is from a first application and the second audio data is from a second application; the first device inquires whether the first audio output device corresponding to the first application and the first type comprises a second device in the sound mixing strategy; furthermore, the first device may query whether a second audio output device corresponding to the second application and the second type includes a second device in the sound mixing strategy; if the first audio output device comprises a second device and/or the second audio output device comprises a second device, that is, at least one of the first audio data and the second audio data does not need to be mixed, the first device may determine that the first audio data and the second audio data do not need to be mixed; if the first audio output device does not include the second device and the second audio output device does not include the second device, that is, both the first audio data and the second audio data require mixing, the first device may determine that mixing of the first audio data and the second audio data is required.

Similarly, the corresponding relationship between different types of audio data, different applications and audio output devices allowing mixing may also be set in the mixing strategy; for example, audio output devices allowing mixing corresponding to notification-type audio data from the video APP include headphones and televisions.

At this time, corresponding to the method, the first device may query, in the sound mixing policy, whether the first audio output device corresponding to the first type and the first application includes the second device; the first device inquires whether a second audio output device corresponding to the second type and the second application comprises a second device in the mixing strategy; if the first audio output device comprises a second device and the second audio output device comprises a second device, which indicates that the first audio data and the second audio data both need to be mixed, the first device may determine that the first audio data and the second audio data need to be mixed; if the first audio output device does not include the second device and/or the second audio output device does not include the second device, indicating that at least one of the first audio data and the second audio data does not require mixing, the first device may determine that mixing of the first audio data and the second audio data is not required.

For example, when the second device is a speaker type electronic device, no mixing may be required when sending audio data of a music type to the second device may be set in a mixing strategy; therefore, the first device can independently project the audio data of the music type to the second device for playing without mixing with other audio data, so that the distortion phenomenon of the audio is reduced when the second device plays the audio data of the music type.

For example, when the second device is an electronic device of a vehicle-mounted device type, no mixing may be required when sending audio data of a navigation type to the second device is set in a mixing policy; therefore, the first device can independently project the audio data of the navigation type to the second device for playing without mixing with other audio data, so that the second device can reduce the distortion phenomenon of the audio when playing the audio data of the navigation type.

For example, when the second device is an electronic device of a large screen device type, mixing may be required when transmitting notification type audio data to the second device may be set in a mixing policy. That is, when multiple channels of audio data containing notification types need to be sent to the second device, because the attention of the user on the large-screen device is low, the first device can send the audio data of the notification types to the second device for playing after mixing with other audio data.

Therefore, when the multi-channel audio data needs to be output to the slave device, the master device can select certain audio data in the multi-channel audio data not to be subjected to audio mixing processing according to the audio mixing strategy of the current slave device, and independently transmit the audio data to the slave device for playing, so that the distortion phenomenon caused by the audio mixing processing on certain type of audio data is avoided, and the audio output quality of an audio switching scene is improved.

In one possible implementation, the method for transmitting, by a first device, first audio data and second audio data to a second device includes: the first equipment packs the first audio data to obtain at least one first data packet; the first equipment packs the second audio data to obtain at least one second data packet; the first device sends the first data packet and the second data packet to the second device.

Illustratively, the first data packet includes a first identifier, and the first identifier is used for indicating the first audio data; the second data packet includes a second identifier therein, the second identifier indicating second audio data. Subsequently, the second device may restore the corresponding unmixed first audio data and second audio data according to the first identifier in the first data packet and the second identifier in the second data packet.

Illustratively, the first data packet includes first characteristic information, and the first characteristic information is used for indicating audio characteristics of the first audio data; the second data packet includes second characteristic information therein, the second characteristic information indicating an audio characteristic of the second audio data.

For example, the first feature information may include at least one of an application to which the first audio data belongs, a type of the first audio data, a volume level of the first audio data, a playback format of the first audio data, and track information of the first audio data; the second feature information may include at least one of an application to which the second audio data belongs, a type of the second audio data, a volume level of the second audio data, a playback format of the second audio data, and track information of the second audio data.

The characteristic information can reflect the actual audio characteristics of the audio data, so that the second device receiving the data packet can truly restore the audio data and the related characteristics of the audio data by analyzing the characteristic information in the data packet, the corresponding audio data is played according to the characteristic information of the audio data, and the distortion phenomenon of the audio data in the audio switching process is reduced.

For example, whether the second device is a sound box type, a vehicle-mounted device type, a large-screen device type or another type of electronic device, the first device may specifically be an electronic device with strong operation and processing capabilities, such as a mobile phone, a tablet computer or a notebook computer.

In a possible implementation manner, after the first device establishes a network connection with the second device, the method further includes: the method comprises the steps that a first device obtains a device selection strategy corresponding to a second device; after the first device acquires the first audio data and the second audio data to be played, the method further comprises the following steps: the first device determines the audio output device of the first audio data and the second audio data as the second device according to the first type, the second type and the device selection policy. That is, when there are multiple channels of audio data to be played in the first device, the first device may determine an audio output device for each channel of audio data in the multiple channels of audio data. If the audio output devices for the multiple audio data are the same, the first device may further determine whether to mix the multiple audio data according to a mixing strategy.

In a possible implementation manner, the application layer of the first device is installed with a preset application, such as a DV APP, and the application framework layer of the first device includes an audio policy module, such as AudioPolicy; the method for acquiring the sound mixing strategy corresponding to the second device by the first device includes: the preset application determines the device type of the second device; the preset application acquires a corresponding sound mixing strategy according to the equipment type of the second equipment; and the preset application issues the sound mixing strategy to the audio strategy module. That is, the first device may pre-store the audio mixing strategies of multiple audio output devices, and the first device may dynamically issue the corresponding audio mixing strategies to the AudioPolicy according to the device type of the currently connected second device, so as to adjust whether audio data is mixed in time according to the audio mixing strategies.

In one possible implementation, the application framework layer of the first device further comprises an audio processor, such as an AudioFlinger; the method for acquiring the first audio data and the second audio data to be played by the first device includes: after the audio processor acquires the first audio data and the second audio data from the application program layer, the audio processor may send a query request to the audio policy module, where the query request includes a first type identifier and a second type identifier; then, in response to the query request, the audio policy module may determine whether mixing of the first audio data of the first type and the second audio data of the second type is required according to the mixing policy.

In a possible implementation manner, after the first device establishes a network connection with the second device, the method further includes: the first device creates a hardware abstraction module corresponding to the second device, such as DMSDP Audio HAL, at the hardware abstraction layer HAL; wherein, if the first audio data and the second audio data do not need to be mixed, the method further comprises: the audio processor can receive a first indication sent by the audio policy module, wherein the first indication is used for indicating that the first audio data and the second audio data are not mixed; then, in response to the first indication, the audio processor may encapsulate the first audio data into at least one first packet and store the first packet in a buffer (buffer) of the audio processor; and, in response to the first indication, the audio processor may encapsulate the second audio data into at least one second data packet and store the second data packet in a cache of the audio processor; and further, the audio processor sends the first data packet and the second data packet in the cache of the audio processor to the hardware abstraction module.

At this time, the first device transmitting the first audio data and the second audio data to the second device includes: the hardware abstraction module can send the first data packet and the second data packet in the cache of the audio processor to the second device, so that the second device can restore the first audio data and the second audio data by analyzing the first data packet and the second data packet, and at the moment, the first device does not need to analyze the first data packet and the second data packet; or; the hardware abstraction module can analyze the first data packet and the second data packet, restore corresponding first audio data and second audio data, and then send the first audio data and the second audio data to the second device, and at this time, the second device does not need to analyze the audio data sent by the first device.

In a possible implementation manner, after the first device obtains the mixing policy corresponding to the second device, the method further includes: the first equipment displays a setting interface of the audio output equipment; the method comprises the steps that a first device receives a sound mixing strategy corresponding to a second device, which is set in a setting interface by a user; in response to the user's setting, the first device updates the mixing policy. Therefore, a user can set or modify the audio types which need or do not need mixing when sending the audio data to each audio output device in the setting interface according to the own needs, namely, the corresponding mixing strategies are set for each audio output device, so that the various types of audio data can be mixed or not mixed according to the setting of the user when being switched from the device.

In a possible implementation manner, the method further includes: when the first device acquires the first audio data and the second audio data, the first device may further acquire third audio data to be played (the type of the third audio data is a third type), that is, the audio data to be played simultaneously includes three paths of audio data; furthermore, the first device can determine that the first audio data does not need to be mixed and the second audio data and the third audio data need to be mixed according to the first type, the second type, the third type and the mixing strategy; then, the first apparatus may transmit the first audio data to the second apparatus without mixing alone, and the first apparatus may transmit the second audio data and the third audio data to the second apparatus after mixing.

Of course, the first device may also determine that the first audio data, the second audio data, and the third audio data do not need to be mixed according to the mixing policy, or that the first audio data, the second audio data, and the third audio data all need to be mixed, which is not limited in this embodiment of the present application.

In addition, if the slave device connected to the first device is updated from the second device to the third device, similar to the process in which the first device switches the multiple channels of audio data to the second device for playing, the first device may also determine whether the multiple channels of audio data to be played at this time need to be mixed according to the mixing policy of the third device (i.e., a new slave device).

In a ninth aspect, the present application provides an audio control method, comprising: the second equipment establishes network connection with the first equipment; the method comprises the steps that a second device obtains unmixed first audio data and second audio data from a first device, wherein the type of the first audio data is a first type, and the type of the second audio data is a second type; the second device plays the first audio data and the second audio data.

In one possible implementation manner, the obtaining, by the second device, unmixed first audio data and second audio data from the first device includes: the second device acquires a first data packet corresponding to the first audio data and a second data packet corresponding to the second audio data from the first device; the second device restores the first audio data and the second audio data by analyzing the first data packet and the second data packet.

In a possible implementation manner, the first data packet includes a first identifier, and the first identifier is used for indicating the first audio data; the second data packet comprises a second identifier which is used for indicating second audio data; the second device restores the first audio data and the second audio data by analyzing the first data packet and the second data packet, and includes: the second device restores the audio data in the data packet carrying the first identifier to the first audio data and restores the audio data in the data packet carrying the second identifier to the second audio data by analyzing the first data packet and the second data packet.

In one possible implementation manner, the first data packet includes first feature information, and the first feature information is used for indicating an audio feature of the first audio data; the second data packet comprises second characteristic information, and the second characteristic information is used for indicating the audio characteristics of the second audio data; wherein, the second equipment plays first audio data and second audio data, includes: the second device plays the first audio data according to the first characteristic information, and simultaneously plays the second audio data according to the second characteristic information.

In a tenth aspect, the present application provides a method for playing multiple audio tasks, including: the method comprises the steps that the electronic equipment displays a first window and a second window in a display interface, wherein a first application runs in the first window, a second application runs in the second window, and at the moment, the electronic equipment enters a split screen mode; furthermore, the electronic equipment can receive a first operation input by a user; in response to the first operation, the electronic device may establish an association relationship between the first window and the first audio output device, that is, the user may manually set the audio output device associated with the first window as the first audio output device; similarly, if the electronic device receives a second operation input by the user, the electronic device may establish an association relationship between the second window and the second audio output device, that is, the user may manually set the audio output device associated with the second window as the second audio output device; therefore, when the electronic equipment acquires the first audio data from the first application, the electronic equipment can send the first audio data to the first audio output equipment for playing according to the incidence relation between the first window and the first audio output equipment; similarly, when the electronic device obtains the second audio data from the second application, the electronic device may send the second audio data to the second audio output device for playing according to the association relationship between the second window and the second audio output device.

That is, when multiple windows are running in the electronic device, the audio data output by the audio task in each window may be played using the audio output device associated with that window. Under the scene of multi-window multi-audio task concurrence, the electronic equipment can send the audio data output by the related audio task in each window to the corresponding audio output equipment for playing. Therefore, the multiple audio data which are concurrent by the electronic equipment cannot interfere with each other, the multiple audio data can be played to different users through different audio output equipment, each user can listen to the corresponding audio data by using the audio output equipment which is associated with the related window, the influence of the audio data from other windows is avoided, and the audio using experience of the users is improved.

In one possible implementation manner, after the electronic device displays the first window and the second window in the display interface, the method further includes: the electronic device may display a first dialog box in the first window, the first dialog box including at least one first candidate device having audio playback functionality therein, e.g., the electronic device may display an audio output device located on the same Wi-Fi network as the electronic device in the first dialog box; wherein the first operation is an operation of selecting a first audio output device among at least one first candidate device by a user; likewise, the electronic device may display a second dialog box in the second window, the second dialog box including at least one second candidate device having an audio playback function, e.g., the electronic device may display an audio output device having an audio playback function connected with the electronic device in the second dialog box; wherein the second operation is an operation of the user selecting the second audio output device among the at least one second candidate device.

In a possible implementation manner, the electronic device establishes an association relationship between a first window and a first audio output device, and specifically includes: the electronic equipment can record first audio configuration information of the first window, wherein the first audio configuration information comprises an association relation between the first window and the first audio output equipment; similarly, the electronic device establishes an association relationship between the second window and the second audio output device, and specifically includes: the electronic device may record second audio configuration information for the second window, the second audio configuration information including an association of the second window with a second audio output device.

In a possible implementation manner, the method further includes: when the first application runs in the first window, the electronic device may record a first corresponding relationship between the first window and the first application, that is, application information of the first window, for indicating the application running in the first window; likewise, when the second application is running in the second window, the electronic device may record a second correspondence between the second window and the second application, i.e., application information of the second window, for indicating the application running in the second window.

Illustratively, the recording, by the electronic device, a first corresponding relationship between the first window and the first application specifically includes: the first application can apply for obtaining a first audio focus of the first window; when the first application acquires the first audio focus, the electronic device can record a first corresponding relation between the first window and the first application; similarly, the recording, by the electronic device, a second corresponding relationship between the second window and the second application specifically includes: the second application applies for obtaining a second audio focus of the second window; when the second application acquires the second audio focus, the electronic device may record a second correspondence between the second window and the second application.

That is, the electronic device may set one Audio Focus (Audio Focus) for each window in the split-screen mode. When the application in each window needs to play an audio task, the audio focus of the window needs to be acquired first, and the electronic device may maintain the audio focus application corresponding to each window, so as to determine from which window the audio focus application the audio data comes when acquiring the audio data to be played.

In a possible implementation manner, before the electronic device sends the first audio data to the first audio output device for playing, the method further includes: the electronic device determines that the first audio data is from a first application; furthermore, the electronic equipment can determine a first window corresponding to the first audio data according to the first corresponding relation; furthermore, the electronic device may determine, according to the first audio configuration information, that the audio output device of the first audio data is the first audio output device associated with the first window.

Similarly, before the electronic device sends the second audio data to the second audio output device for playing, the method further includes: the electronic device determines that the second audio data is from a second application; furthermore, the electronic equipment determines a second window corresponding to second audio data according to the second corresponding relation; furthermore, the electronic device may determine, according to the second audio configuration information, that the audio output device of the second audio data is the second audio output device associated with the second window.

In one possible implementation manner, the first audio configuration information may further include: a type of audio data allowed to be played in the first audio output device, and/or a volume level of audio data allowed to be played in the first audio output device; the second audio configuration information may further include: the type of audio data allowed to be played in the second audio output device, and/or the volume level of audio data allowed to be played in the second audio output device. In this way, the electronic device can output the audio data of the specific type to the corresponding audio output device for playing according to the first audio configuration information and the second audio configuration information, and control the volume of the corresponding audio data.

In a possible implementation manner, if the first audio configuration information includes a volume size of audio data allowed to be played in the first audio output device; after the electronic device displays the first window and the second window in the display interface, the method further includes: the electronic equipment displays a first volume adjusting bar in a first window; if it is detected that the user inputs the first volume adjustment operation on the first volume adjustment bar, the electronic device may modify the volume of the first type of audio data in the first audio configuration information, so as to control the volume of the audio output in the first audio output device, where the first type is the type of audio data being played in the first window or a preset type of audio data.

In a possible implementation manner, if the second audio configuration information includes a volume size of audio data allowed to be played in the second audio output device; after the electronic device displays the first window and the second window in the display interface, the method further includes: the electronic equipment displays a second volume adjusting bar in a second window; if it is detected that the user inputs a second volume adjustment operation on the second volume adjustment bar, the electronic device may modify the volume of the second type of audio data in the second audio configuration information, so as to control the volume of the audio output in the second audio output device, where the second type is the type of the audio data being played in the second window or a preset type of the audio data.

In one possible implementation manner, after the electronic device establishes the association relationship between the first window and the first audio output device, the method further includes: and responding to a third operation input by the user, the electronic equipment updates the first audio configuration information, and the updated first audio configuration information comprises the incidence relation between the first window and the third audio output equipment.

In one possible implementation manner, after the electronic device establishes the association relationship between the second window and the second audio output device, the method further includes: and responding to a fourth operation input by the user, the electronic equipment updates the second audio configuration information, and the updated second audio configuration information comprises the incidence relation between the second window and the fourth audio output equipment.

That is, the user can manually update the audio output devices associated with the windows, so that the audio output devices of the windows are set according to the needs of the user by taking the windows as granularity, and the audio data in the windows can be distributed to the corresponding audio output devices to be played according to the settings of the user.

In one possible implementation, the application framework layer of the electronic device includes an audio policy module (e.g., AudioPolicy) and an audio processor (e.g., AudioFlinger), and the method further includes: when the audio processor receives first audio data, the audio processor sends a first query request to the audio policy module; in response to the first query request, the audio policy module may determine that the audio output device of the first audio data is the first audio output device; similarly, when the audio processor receives the second audio data, the audio processor sends a second query request to the audio policy module; in response to the second query request, the audio policy module may determine that the audio output device for the second audio data is the second audio output device.

In one possible implementation, the application framework layer of the electronic device includes a window manager, and the window manager is configured to send a first correspondence between the first window and the first application, a second correspondence between the second window and the second application, first audio configuration information of the first window, and second audio configuration information of the second window to the audio policy module. In this way, the audio policy module may determine the audio output devices for the first audio data and the second audio data based on these parameters.

And when the corresponding relation between the window and the application is updated or the corresponding relation between the window and the audio output device is updated, the window manager can send the updated application information and audio configuration information to the audio policy module, so that the audio policy module can determine the audio output device corresponding to each window according to the latest application information and audio configuration information.

In one possible implementation manner, the HAL of the electronic device includes a first audio abstraction module corresponding to the first audio output device and a second audio abstraction module corresponding to the second audio output device; wherein, electronic equipment sends first audio data to first audio output device, includes: the audio processor sends the first audio data to first audio output equipment through the first audio abstraction module; wherein, electronic equipment sends second audio data to second audio output device, includes: the audio processor sends the second audio data to the second audio output device through the second audio abstraction module.

For example, the first Audio output device may be a sound box, the first Audio abstraction module may be a DMSDP Audio HAL, and the Audio flinger may send the first Audio data to the sound box for playing through the DMSDP Audio HAL. For another example, the second audio output device may be a bluetooth headset, the second audio abstraction module may be A2dp HAL, and the AudioFlinger may send the second audio data to the bluetooth headset via the A2dp HAL for playing.

In one possible implementation, the HAL of the electronic device includes a Display abstraction module (e.g., Display HAL); the method further comprises the following steps: the display abstraction module acquires first display data corresponding to a first window and second display data corresponding to a second window; furthermore, the display abstraction module may synthesize the first display data and the second display data into third display data corresponding to the display interface.

For example, if the display output device of the electronic device is a display screen of the electronic device, after the display abstraction module synthesizes the first display data and the second display data into third display data, the method further includes: and the display abstraction module sends the third display data to a display screen of the electronic equipment for display.

For example, if the display output device of the electronic device is a first display device (e.g., a television), after the display abstraction module synthesizes the first display data and the second display data into third display data to be displayed, the method further includes: and the display abstraction module sends the third display data to the first display device for display.

That is to say, in the screen-shot scene of the split-screen mode, the distribution process of the display data and the audio data can be respectively controlled by the main device, and the audio data in different windows are distributed to the corresponding audio output devices by the main device for playing.

Or, except that the third display data is sent to the first display device for displaying, when the display output device of the electronic device is the first display device, the electronic device sends the first audio data to the first audio output device for playing according to the association relationship between the first window and the first audio output device, specifically including: the electronic equipment sends the first audio data and the incidence relation between the first window and the first audio output equipment to the first display equipment, so that the first display equipment can send the first audio data to the first audio output equipment for playing according to the incidence relation between the first window and the first audio output equipment; similarly, the electronic device sends the second audio data to the second audio output device for playing according to the association relationship between the second window and the second audio output device, and the method specifically includes: the electronic equipment sends the second audio data and the incidence relation between the second window and the second audio output equipment to the first display equipment, so that the first display equipment can send the second audio data to the second audio output equipment for playing according to the incidence relation between the second window and the second audio output equipment.

That is, in a screen-casting scene of the split screen mode, the master device may transmit both display data and audio data generated in each window to the slave device. And the master device can send the application information and the audio configuration information of each window to the slave device, so that the slave device can distribute the audio data of different windows in the master device to corresponding audio output devices according to the application information and the audio configuration information for playing, and the isolation effect that the audio data among multiple windows are not interfered with each other is achieved.

In one possible implementation, the HAL of the electronic device may include a Display abstraction module (e.g., Display HAL); if the display output device corresponding to the first window is the first display device, the display output device corresponding to the second window is the second display device; the method further comprises: the display abstraction module can acquire first display data corresponding to the first window and send the first display data to first display equipment for display; and the display abstraction module may obtain second display data corresponding to the second window, and send the second display data to the second display device for display. That is, in the split-screen mode, the electronic device may send the display data in different windows to different display output devices (including the mobile phone itself) for display, so as to implement a screen projection function with the window as a granularity. At this time, the display data in each display output device will not interfere with each other, and the audio data in each display output device will not interfere with each other, so as to improve the audio using experience of the user.

In an eleventh aspect, the present application provides a screen recording method, including: in response to an operation of opening a screen recording function in first equipment by a user, the first equipment can display a screen recording equipment list, wherein the screen recording equipment list comprises one or more electronic equipment which are accessed to the same communication network (such as the same Wi-Fi network) with the first equipment; if the user is detected to select the second device in the screen recording device list, on one hand, the first device can send a screen recording instruction to the second device, so that the second device can respond to the screen recording instruction to execute screen recording operation; on the other hand, the first device can execute screen recording operation to obtain first screen data of the first device; moreover, the first device can acquire second screen data obtained by the second device executing screen recording operation; in this way, the first device generates the screen recording file according to the first screen data and the second screen data.

That is, in the screen recording scene, the user may select to record screen data in multiple devices, such as the first device and the second device, simultaneously in the first device. The first device can generate the screen recording file according to the screen data recorded by the two devices, and the screen recording file can restore the audio and the picture of the first device and can also restore the audio and the picture of the second device, so that the service scenes realized on the first device and the second device can be recorded and restored more comprehensively, and the use experience of a user in a distributed system is improved.

In a possible implementation manner, the first screen data may include first display data and first audio data; the first display data includes: display data played by the first device (for example, display data played by a display screen in the first device) and/or display data acquired by the first device (for example, display data acquired by a camera of the first device); the first audio data includes: audio data played by the first device (e.g., audio data played by a speaker or earpiece of the first device) and/or audio data captured by the first device (e.g., audio data captured by a microphone or earpiece of the first device).

That is to say, when the first device records a screen, the source of the recorded display data may be multiple, and the source of the recorded audio data may also be multiple, which is not limited in this embodiment of the present application.

In a possible implementation manner, before the first device performs a screen recording operation to obtain first screen data of the first device, the method further includes: the method comprises the steps that first equipment obtains first screen recording parameters used by the first equipment for executing screen recording operation, wherein the first screen recording parameters comprise display recording parameters and audio recording parameters; the display recording parameters may include resolution used during screen recording, and parameters required to be used during display data recording, such as DPI. For another example, the display recording parameter may also indicate a specific source of the display data recorded this time, for example, recording the display data acquired by the camera, or recording the display data played in the display screen. Similarly, the audio recording parameters may include a sampling rate of the first device to the audio data, sound effect settings, and other parameters that need to be used when recording the audio data. For another example, the audio recording parameter may also indicate a specific source of the audio data recorded this time, for example, recording audio data from a speaker or recording audio data from a microphone.

At this time, the first device performs a screen recording operation to obtain first screen data of the first device, and the method specifically includes: the first equipment records according to the display recording parameters to obtain first display data; and the first equipment records the audio recording parameters to obtain first audio data.

Of course, the first screen recording parameter may only include a display recording parameter or an audio recording parameter. For example, when the first screen recording parameter only includes a display recording parameter, the first device may record the display recording parameter to obtain first display data, where the first display data is first screen data recorded by the first device. For another example, when the first screen recording parameter only includes the display audio parameter, the first device may record the first audio data according to the audio recording parameter, where the first audio data is the first screen data recorded by the first device.

In a possible implementation manner, before the first device sends the screen recording instruction to the second device, the method further includes: the first equipment acquires second screen recording parameters used by the second equipment for executing screen recording operation, wherein the second screen recording parameters are similar to the first screen recording parameters and comprise display recording parameters (such as resolution, DPI and the like) and audio recording parameters (such as sampling rate and the like); wherein, first equipment sends the instruction of recording the screen to second equipment, includes: the first device can send the second screen recording parameter carried in the screen recording instruction to the second device. Therefore, the second device can execute screen recording operation according to the second screen recording parameters carried in the screen recording instruction to obtain second screen data.

In a possible implementation manner, the acquiring, by the first device, the first screen recording parameter and the second screen recording parameter specifically includes: the first equipment can acquire a first screen recording capability parameter of the first equipment, and the first screen recording capability parameter is used for reflecting the screen recording capability of the first equipment; the first device can obtain a second screen recording capability parameter of the second device, and the second screen recording capability parameter is used for reflecting the screen recording capability of the second device; the first device can determine a first screen recording parameter corresponding to the first device and a second screen recording parameter corresponding to the second device according to the first screen recording capability parameter and the second screen recording capability parameter.

That is to say, when multiple devices record screens synchronously, the first device as the master device may determine, by combining the screen recording capabilities of the first device and the second device, the screen recording parameters used when the mobile phone and the slave device record screens synchronously. In general, parameters for processing the display data and the audio data in the first screen recording parameter and the second screen recording parameter are generally the same, for example, parameters such as resolution, DPI, and sampling rate in the first screen recording parameter and the second screen recording parameter are the same. In this way, first screen data recorded by the first device using the first screen recording parameter and second screen data recorded by the second device using the second screen recording parameter can be made into a unified screen recording file.

However, the sources of the screen data set in the first screen capture parameter and the second screen capture parameter may be different. For example, the first screen recording parameter may set that the first device records display data from the display screen, and the second screen recording parameter may set that the second device records acquired display data from the camera.

In a possible implementation manner, when the resolution of the first device in the first screen recording capability parameter is greater than the resolution of the second device in the second screen recording capability parameter, the resolutions of the first screen recording parameter and the second screen recording parameter are the resolutions of the second device, that is, a smaller resolution is selected as the resolutions of the first screen recording parameter and the second screen recording parameter, so as to ensure that both the first device and the second device can record screens according to the resolutions of the corresponding screen recording parameters.

Or when the DPI of the first device in the first screen recording capability parameter is greater than the DPI of the second device in the second screen recording capability parameter, the DPI in the first screen recording parameter and the second screen recording parameter is the DPI of the second device, that is, a smaller DPI is selected as the DPI in the first screen recording parameter and the second screen recording parameter, so as to ensure that the first device and the second device can record screens according to the DPI in the corresponding screen recording parameters.

Or when the sampling rate of the first device in the first screen recording capability parameter is greater than the sampling rate of the second device in the second screen recording capability parameter, the sampling rates of the first screen recording parameter and the second screen recording parameter are the sampling rates of the second device, that is, a smaller sampling rate is selected as the sampling rate of the first screen recording parameter and the second screen recording parameter, so that the first device and the second device can record screens according to the sampling rate of the corresponding screen recording parameter.

Of course, the first device may also determine, in combination with factors such as a current network bandwidth, a service scenario, or a user habit, a first screen recording parameter corresponding to the first device and a second screen recording parameter corresponding to the second device, which is not limited in this application.

In a possible implementation manner, similar to the first screen data, the second screen data recorded by the second device may include second display data and second audio data; the second display data includes: display data played by the second device (for example, display data played by a display screen in the second device) and/or display data acquired by the second device (for example, display data acquired by a camera of the second device); the second audio data includes: audio data played by the second device (e.g., audio data played by a speaker or earpiece of the second device) and/or audio data captured by the second device (e.g., audio data captured by a microphone or earpiece of the second device).

In a possible implementation manner, the generating, by the first device, the screen recording file according to the first screen data and the second screen data includes: the first equipment synthesizes the first display data and the second display data into third display data; and then, the first device makes the third display data, the first audio data and the second audio data into the screen recording file. That is, the first device may make the first screen data and the second screen data as one screen recording file, and a picture in the screen recording file includes the first display data and the second display data.

Alternatively, the first device may make the first screen data and the second screen data as two screen recording files, for example, a first file and a second file, respectively; the first device generates a screen recording file according to the first screen data and the second screen data, and the screen recording file comprises: the first device makes the first display data and the first audio data into a first file; the first device creates the second display data and the second audio data as a second file.

In a possible implementation manner, before the first device displays the list of screen recording devices, the method further includes: the method comprises the steps that a first device detects N electronic devices which are accessed to the same communication network with the first device, wherein N is an integer larger than 0; furthermore, the first device can determine the electronic device with the screen recording function in the N electronic devices; therefore, the first device can display the determined electronic device with the screen recording function in the screen recording device list for the user to select.

In one possible implementation, the application layer of the first device includes a screen recording application, the application framework layer of the first device includes a screen recorder, and the HAL of the first device includes a preset hardware abstraction module; the method includes that a first device executes screen recording operation to obtain first screen data of the first device, and specifically includes: the screen recording application calls a screen recorder to execute screen recording operation to obtain first screen data of the first device; the method includes that a first device acquires second screen data obtained by a second device executing screen recording operation, and specifically includes: and the screen recorder acquires second screen data obtained by the second equipment executing the screen recording operation through the hardware abstraction module.

That is, the screen recorder in the first device may not only record the first screen data of the first device itself, but also instruct a slave device (i.e., a second device) of the first device to record the second screen data of the second device. Furthermore, the screen recorder can generate a corresponding screen recording file according to the first screen data and the second screen data, and the function of synchronously recording screens of the first device and the second device is achieved.

In a twelfth aspect, the present application provides a screen recording method, including: in response to an operation of a user for opening a screen recording function in the first device, the first device may display a screen recording device list including one or more electronic devices accessing the same communication network (e.g., the same Wi-Fi network) as the first device. If it is detected that the user selects the second device in the screen recording device list, it indicates that the user needs to record the screen of the first device and the second device synchronously, at this time, the screen recording application in the first device may obtain the screen recording capability parameter of the second device from the second device, and the screen recording application may obtain the screen recording capability parameter of the first device. Further, the screen recording application determines a first screen recording parameter used by the screen recording of the first equipment and a second screen recording parameter used by the screen recording of the second equipment according to the screen recording capability parameters of the first equipment and the second equipment; the screen recording application can carry the second screen recording parameter in a screen recording instruction and send the screen recording instruction to the second equipment, and the second equipment is triggered to execute screen recording operation according to the second screen recording parameter; meanwhile, the screen recording application can call a screen recorder of the application program framework layer to execute screen recording operation according to the determined first screen recording parameter; for example, the screen recorder may acquire display data (i.e., first display data) played in the display screen from the display module in real time, and the screen recorder may acquire audio data (i.e., first audio data) played by a speaker of the first device from the audiomaker in real time by using the AudioRecord; in this way, the screen recorder can acquire the first screen data (the first screen data includes the first display data and the first audio data) recorded by the first device in real time. The first display data may also be display data acquired by a Camera acquired by a screen recorder from a Camera module, and the first audio data may also be audio data acquired by a microphone of the first device acquired by the screen recorder from an AudioFlinger, which is not limited in this application. The screen recorder can acquire first screen data recorded by the first equipment and can also receive second screen data recorded by the second equipment. Furthermore, the screen recorder can report the first screen data and the second screen data to the screen recording application, the screen recording application can manufacture corresponding screen recording files according to the first screen data and the second screen data, the screen recording files can restore the audio and the picture of the first equipment through the first screen data and can restore the audio and the picture of the second equipment through the second screen data, and therefore the service scenes achieved on the first equipment and the second equipment are recorded and restored more comprehensively.

In a thirteenth aspect, the present application provides an audio delay updating method, including: after the first device establishes network connection with the second device, if the second device is set as an audio output device of the first device, the first device may dynamically acquire a first time delay, a second time delay and a third time delay, wherein the first time delay refers to a time delay generated when audio data is processed in the first device, the second time delay refers to a time delay generated when the audio data is transmitted between the first device and the second device, and the third time delay refers to a time delay generated when the audio data is played in the second device; in this way, the first device may dynamically determine the current audio delay based on the first delay, the second delay, and the third delay, for example, the audio delay is the first delay + the second delay + the third delay; for another example, the audio delay L ═ k1 ═ first delay + k2 ×, second delay + k3 ×, third delay, and k1+ k2+ k3 ═ 1.

That is, when the first device needs to project the audio data into the second device for playing, the first device may dynamically calculate the current audio delay according to the first delay, the second delay, and the third delay. Subsequently, the audio delay acquired by the relevant application or player each time is also the audio delay calculated by the first device, and the audio delay can reflect the time delay of the current audio data more accurately. Therefore, when the related application or player performs synchronous processing such as sound-picture synchronization according to the audio time delay, a more accurate synchronization effect can be achieved, and the use experience of a user is improved.

In a possible implementation manner, after the first device establishes a network connection with the second device, the method further includes: the method comprises the steps that a first device obtains an audio capability parameter of a second device, wherein the audio capability parameter is used for indicating the hardware capability of the second device for playing audio and the software capability of the second device for playing audio; at this time, the first device acquires the third delay, including: and the first equipment acquires a third time delay from the audio capability parameter. That is to say, the second device may calculate its own audio delay (i.e., a third delay), and report the third delay carried in the audio capability parameter to the first device, so that the first device obtains the third delay from the audio capability parameter of the second device.

In a possible implementation manner, after the first device obtains the audio capability parameter of the second device, the method further includes: the first device determines an audio strategy according to the audio capacity parameter (for example, whether to add sound effect, whether to resample and the like in the processing process of the first device on the audio data); wherein, the first equipment obtains the first time delay, including: and the first equipment determines the first time delay according to the audio strategy.

For example, the audio policy includes N processing procedures on the audio data, where N is an integer greater than 0; the first equipment determines a first time delay according to the audio strategy, and comprises the following steps: the first equipment determines the consumed time of each processing procedure in the N processing procedures; further, the first device may calculate the first time delay based on the elapsed time for each process. For example, the first device may add the time consumption of each of the N processing procedures to obtain a first time delay, i.e., the time consumption required for processing the audio data in the first device.

In a possible implementation manner, if the second device is set as the audio output device of the first device, the audio capability parameter of the second device may be dynamically updated, for example, the audio capability such as the sound effect mode of the second device may be updated, and at this time, the first device may further obtain the updated audio capability parameter of the second device; and the first device may update the audio policy according to the updated audio capability parameter; when the audio capability parameter is updated, the first device can be triggered to update the third time delay according to the updated audio capability parameter; when the audio policy is updated, the first device may be triggered to update the first delay according to the updated audio policy. That is to say. The first time delay and the third time delay can be dynamically updated, so that the first device can acquire the latest audio time delay.

In one possible implementation, the first hardware abstraction module for transmitting audio data, e.g. a Wi-Fi HAL; at this time, after the first device establishes a network connection with the second device, the method further includes: the first device creates a second hardware abstraction module in the HAL corresponding to the second device, e.g., DMSDP Audio HAL; wherein, the first device obtains the second time delay, including: the second hardware abstraction module may obtain a second latency, i.e., a time required for audio data to be transmitted between the first device and the second device, from the first hardware abstraction module.

For example, a concrete method for the first hardware abstraction module to obtain the second latency may include: the first hardware abstraction module sends a test data packet to the second device; the first hardware abstraction module receives a response data packet sent by the second device responding to the test data packet; further, the first hardware abstraction module may calculate and store the second time delay according to a time interval between sending the test data packet and receiving the response data packet.

It is to be understood that the first hardware abstraction module may periodically calculate and store the second time delay according to the above method. When the second delay is updated, the first hardware abstraction module may notify the second hardware abstraction module to obtain the latest second delay. Alternatively, the second hardware abstraction module may periodically obtain the latest saved second delay from the first hardware abstraction module.

That is, the first delay, the second delay, or the third delay may be dynamically updated. Then, the first device may recalculate the current audio delay when at least one of the first delay, the second delay, or the third delay is updated. Alternatively, the first device may also periodically obtain the latest first time delay, the latest second time delay, and the latest third time delay, so as to periodically calculate the current latest audio time delay.

In a possible implementation manner, the display data output by the first device is displayed in a display screen of the first device, in this scenario, the first device projects the audio data to the second device for playing, and the display data is kept in the first device for playing; at this time, after the first device determines the current audio delay, the method further includes: a first application running in a first device acquires an audio delay; for example, a first application may call the getLatency () interface to query the AudioService for the current audio latency.

When the first application is a video application, the first application can synchronize audio data and display data of the first application according to audio time delay, namely, perform sound and picture synchronization processing to synchronize video images seen by a user with video sounds heard by the user; when the first application is a music application, the first application may synchronize audio data and lyric data of the first application according to an audio delay, so that lyrics seen by a user are synchronized with a song sound heard; when the first application is a karaoke application, the first application can synchronize the audio data of the first application and the accompaniment music according to the audio time delay, so that the audio data recorded by the user is synchronized with the accompaniment music. Of course, other applications may also acquire the current audio delay to perform audio-related synchronization processing, and this is not limited in this embodiment of the present application.

In a possible implementation manner, the display data output by the first device is displayed in a display screen of the third device, in this scenario, the first device projects the audio data into the second device for playing, and projects the display data into the third device for playing; at this time, the first device may determine, in addition to the current audio delay, a current display delay, where the delay caused by the display data in the first device and the third device is negligible due to the fast processing speed of the display data in the first device and the third device, and then the display delay may include the delay caused by the transmission of the display data between the first device and the third device; furthermore, the first device may determine a current synchronization delay based on the obtained audio delay and display delay; subsequently, the second application running in the first device can synchronize the audio data and the display data of the second application according to the current synchronization delay, so that the synchronization effect of sound and picture synchronization is improved.

For example, the synchronization delay may be a difference between the audio delay and the display delay, so as to reflect a time difference between the audio data and the display data at the same time of playing.

In a possible implementation manner, if the fourth device is set as the audio output device of the first device and the fourth device is connected to the second device, it is described that when the first device is connected to the second device, the second device is connected to play audio by using the fourth device as the audio output device, that is, the first device, the second device, and the fourth device are in a cascade relationship. At this time, the first device may obtain the first time delay and the second time delay according to the above method; when different, the first device further needs to obtain a fourth delay, where the fourth delay includes: the time delay generated when the audio data is processed in the second device, the time delay generated when the audio data is transmitted between the second device and the fourth device, and the time delay generated when the audio data is played in the fourth device; further, the first device may calculate a current audio delay based on the first delay, the second delay, and the fourth delay. For example, the audio delay is a sum of the first delay, the second delay, and the fourth delay.

The fourth delay may be carried by the second device in the audio capability parameter and reported to the first device.

In a fourteenth aspect, the present application provides an audio delay updating method, including: the first equipment and the second equipment establish network connection; if the second device is set as the audio output device and the display output device of the first device, it is described that the first device projects both the audio data and the display data to the second device for playing, in this scenario, the time delays caused by the transmission of the audio data and the display data between the first device and the second device are substantially the same, so that the first device can obtain a first time delay and a second time delay, the first time delay is a time delay generated when the audio data is processed in the first device, and the second time delay is a time delay generated when the audio data is played in the second device; further, the first device may determine a current synchronization delay based on the first delay and the second delay, and similarly, the current synchronization delay is dynamically updated with the first delay and the second delay. For example, the synchronization delay is a sum of the first delay and the second delay.

Subsequently, after the first device determines the current synchronization delay based on the first delay and the second delay, the method may further include: a first application running in first equipment acquires synchronous time delay; further, the first application may synchronize the audio data and the display data of the first application according to the acquired synchronization delay.

Because the first device can dynamically calculate the latest synchronous time delay, the synchronous time delay obtained each time is also the latest calculated synchronous time delay, and the synchronous time delay can accurately reflect the time difference between the current audio data and the display data. Therefore, when synchronous processing such as sound-picture synchronization and the like is carried out according to the synchronous time delay, the audio and the image played in the second equipment can better achieve the sound-picture synchronization effect, and the sound-picture synchronization experience of a user is improved.

In a fifteenth aspect, the present application provides a method for playing an audio/video file, including: the method comprises the steps that a first device receives a first operation input by a user, wherein the first operation is used for projecting a first file (the first file can be an audio file or a video file) played in the first device to a second device to be played; the first device can acquire a data packet of a first file (the first file comprises N data packets, N is greater than 0) when playing the first file, and after receiving the first operation, if the first data packet of the first file is received, the first device can extract a first audio parameter and first audio data in the first data packet, wherein the first audio data is encoded audio data, and the first audio parameter is an encoding parameter used when the first audio data is encoded; further, the first device may transmit the first audio parameter to the second device, so that the second device may determine whether itself has a capability of decoding the first audio data according to the first audio parameter; if the second device has the capability of decoding the first audio data, the first device does not need to decode and re-decode the first audio data, and correspondingly, the first device can directly send the first audio data to the second device, and the second device decodes the first audio data and then plays the first audio data.

Therefore, when the first device projects the audio and video file to the second device, the first device does not need to decode and re-encode the encoded audio data in the data packet, but directly sends the encoded audio data in the audio and video file to the second device, and the audio data is decoded and played by using the decoding capability of the second device, so that the time delay of the whole projection process of the audio and video file is shortened, and the use experience of a user in the cross-device playing scene of the audio and video file is improved.

In addition, because the audio data sent by the first device to the second device is data which is not decoded in the video file, the audio data received by the second device cannot be distorted due to the decoding operation of the first device, so that lossless transmission of the audio and video file between the first device and the second device is realized, and the fidelity of the audio and video file in a cross-device playing scene of the audio and video file is improved.

In a possible implementation manner, the first data packet includes a header (head) and a data (data) portion; the extracting, by the first device, the first audio parameter and the first audio data in the first data packet specifically includes: the first device extracts a first audio parameter from a packet header of a first data packet; the first device extracts first audio data, i.e., encoded audio data, from the data portion of the first data packet.

In a possible implementation manner, the first audio parameter may include: at least one of an encoding format (e.g., AAC format), an encoding specification (e.g., AAC-LC), a version (e.g., version 1), a sampling rate, or a number of channels used when encoding the first audio data. Of course, the first audio parameter may also include other parameters used in encoding the first audio data, which is not limited in this embodiment of the present application.

In a possible implementation manner, if the first file is a video file, the first data packet of the first file includes first display data in addition to first audio data, and the first display data is also encoded data; then, after the first device acquires the first data packet of the first file, the method further includes: the first equipment extracts a first display parameter and first display data in the first data packet, wherein the first display parameter is an encoding parameter used when the first display data is encoded; furthermore, the first device may send the first display parameter to the second device, so that the second device determines whether the second device has the capability of decoding the first display data according to the first display parameter; if the second device has the capability of decoding the first display data, the first device does not need to decode and re-decode the first display data, and correspondingly, the first device can directly send the first display data to the second device, so that the second device can decode the first display data and then play the first display data.

Therefore, when the first device projects the video file to the second device, the first device does not need to decode and re-encode the encoded display data in the data packet, but directly sends the encoded display data in the video file to the second device, and the display data is decoded and played by using the decoding capability of the second device, so that the time delay of the whole projection process of the video file is shortened, and the use experience of a user in the cross-device playing scene of the video file is improved.

In addition, because the display data sent by the first device to the second device is data which is not decoded in the video file, the display data received by the second device is not distorted due to the decoding operation of the first device, so that lossless transmission of the video file between the first device and the second device is realized, and the fidelity of the video file in a cross-device playing scene of the video file is improved.

It should be noted that, the first device may extract the first audio data, the first display data, the first audio parameter, and the first display parameter from the first data packet at a time. In addition, the first device may send the extracted first audio parameter and the first display parameter to the second device together, or the first device may also send the first audio parameter and the first display parameter to the second device respectively, which is not limited in this embodiment of the present application.

Similar to the extracting of the first audio data and the first audio parameter, the extracting, by the first device, the first display parameter and the first display data in the first data packet specifically includes: the first device may extract a first display parameter from a header of the first data packet; the first device may extract the first display data from the data portion of the first data packet.

For example, the first display parameter may include: at least one of an encoding format (e.g., h.264 format), an encoding specification, a version, or a resolution used when encoding the first display data. Of course, the first display parameter may also include other parameters used when encoding the first display data, which is not limited in this embodiment of the application.

In a possible implementation manner, the first device is provided with a first application (e.g., DV APP), a second application (e.g., video APP), and a first audio/video player (e.g., Nuplayer), where the first application is configured to communicate with the second device, and the second application is configured to send a data packet of a first file to be played to the first audio/video player; after the first device receives the first operation of the user input, the method further comprises the following steps: in response to the first operation, the first device may establish a network connection with the second device by running the first application; furthermore, the first application can send a first broadcast message to the second application to inform the second application to project the first file to the second device for playing; then, in response to the first broadcast message, the second application may send a first notification message to the first audiovisual player to notify the first audiovisual player to begin projecting audiovisual files to the second device. Otherwise, the second application may play the first file in the first device according to an existing audio/video playing method.

In one possible implementation, the first device further comprises a multimedia extractor (e.g., MediaExtractor); after the first audio/video player receives the first notification message, if a first data packet of the first file is obtained, the first multimedia extractor can use the multimedia extractor to analyze the first data packet to obtain a first audio parameter; and the first multimedia extractor can be used for de-encapsulating the first data packet by the first multimedia extractor to obtain first audio data.

In a possible implementation manner, since audio parameters used when encoding audio data in each data packet in the same audio/video file are generally uniform, if the second device has the capability of decoding the first audio data, it indicates that the second device has the capability of decoding other audio data in the first file. Then, after the first device transmits the first audio data to the second device, the method further includes: the first audio and video player can acquire a second data packet (namely a data packet subsequent to the first data packet) of the first file from the second application; furthermore, the first audio/video player can extract the second audio data in the second data packet and send the second audio data to the second device, so that the second device decodes the second audio data and plays the second audio data. At this time, the first audio/video player does not need to extract the audio parameters in the second data packet, and also does not need to send the extracted audio parameters to the second device to trigger the second device to determine whether the second device has the capability of decoding the second audio data, so that the time delay of the first file projected to the second device for playing is reduced.

Of course, if the second data packet includes the second display data, the first audio/video player may further extract the second display data in the second data packet, and send the second display data to the second device, so that the second device decodes the second display data and plays the second display data.

In one possible implementation manner, after the first device sends the first audio data to the second device, the method further includes: responding to a second operation input by the user, and establishing network connection between the first application and the third equipment; the first application sends a second broadcast message to the second application, and the second broadcast message is used for informing the second application to project the first file to the third equipment for playing; that is, the user switches the slave device connected to the first device from the second device to the third device through the second operation, and at this time, the first file needs to be continuously played on the third device. Then, after receiving the second broadcast message, the first audio/video player may start to project the first file to the third device.

Illustratively, the process of projecting the first file to the third device is similar to the process of projecting the first file to the second device. For example, the first audio-video player may obtain a third data packet of the first file from the second application; furthermore, the first audio/video player can extract a second audio parameter and third audio data in the third data packet, and similarly, the third audio data is encoded audio data, and the second audio parameter is an encoding parameter used when encoding the third audio data; the first device does not know whether the third device has the capacity of decoding the audio data in the first file, so that the first audio and video player can send the extracted second audio parameter to the third device, and the third device can determine whether the third device has the capacity of decoding the third audio data according to the second audio parameter; if the third device has the capability of decoding the third audio data, the first audio/video player can directly send the third audio data to the third device, so that the third device can decode the third audio data and then play the third audio data.

Similarly, if the third data packet includes third display data, the first audio/video player may further extract the third display data and the display parameter in the third data packet, send the display parameter in the third data packet to the third device, and trigger the third device to determine whether the third device has the capability of decoding the third display data. If the third device has the capability of decoding the third display data, the first audio/video player can directly send the third display data to the third device, so that the third device can decode the third display data and then play the third display data.

Subsequently, after the first audio/video player acquires other data packets in the first file, the first audio/video player can extract display data and/or audio data in the data packets, and send the extracted display data and/or audio data to the third device, so that the third device decodes and plays the received display data and/or audio data by utilizing the decoding capability of the third device.

In one possible implementation manner, after the first device sends the first audio data to the second device, the method further includes: responding to a third operation input by the user, switching the currently played first file into a second file (the second file is an audio file or a video file) by the second application; the second application creates a second audiovisual player (e.g., a new Nuplayer) for receiving data packets from the second file; that is, the user switches the first file being played in the first device to the second file through the third operation, and at this time, the second file needs to be projected onto the second device for playing. Then, the second application may send a third notification message to the second audio/video player, and after receiving the third broadcast message, the second audio/video player may start to project the second file to the second device.

Illustratively, the process of projecting the second file to the second device is similar to the process of projecting the first file to the second device described above. For example, the second audio/video player may obtain a data packet of a second file from a second application, for example, a fourth data packet; furthermore, the second audio/video player can extract a third audio parameter and fourth audio data in a fourth data packet, wherein the fourth audio data is encoded audio data, and the third audio parameter is an encoding parameter used when the fourth audio data is encoded; the first device does not know whether the second device has the capability of decoding the audio data in the second file, so that the second audio and video player can send the third audio parameter to the second device, and the second device can determine whether the second device has the capability of decoding the fourth audio data according to the third audio parameter; and if the second device has the capability of decoding the fourth audio data, the second audio and video player sends the fourth audio data to the second device so that the second device can decode the fourth audio data and then play the fourth audio data.

Similarly, if the fourth data packet includes fourth display data, the second audio/video player may further extract the fourth display data and the display parameter in the fourth data packet, send the display parameter in the fourth data packet to the second device, and trigger the second device to determine whether the fourth display data has the capability of decoding. If the fourth device has the capability of decoding the fourth display data, the second audio/video player can directly send the fourth display data to the second device, so that the second device can decode the fourth display data and then play the fourth display data.

Subsequently, after the second audio/video player acquires other data packets in the second file, the second audio/video player can extract the display data and/or the audio data in the data packets and send the extracted display data and/or audio data to the second device, so that the second device decodes and plays the received display data and/or audio data by utilizing the decoding capability of the second device.

In one possible implementation manner, after the first device sends the first audio data to the second device, the method further includes: the first device runs a third application; the first device plays an audio file or a video file from the third application. That is, while projecting the audio-video file to the second device, the first device may also run another application (e.g., a third application) and play display data and/or audio data in the other application, so that the user may play the audio-video file across devices and also use application functions provided by the other application in the first device.

In a sixteenth aspect, the present application provides a method for playing an audio/video file, including: the second equipment receives a first audio parameter sent by the first equipment, wherein the first audio parameter is an encoding parameter used when encoding first audio data in a first data packet, and the first data packet is a data packet of a first file; in turn, the second device may determine whether it has the capability to decode the first audio data according to the first audio parameters; if the first device has the capability of decoding the first audio data, the second device can send a first response message to the first device to trigger the first device to send the first audio data which is not decoded to the second device; after receiving the first audio data sent by the first equipment, the second equipment can decode the first audio data to obtain decoded second audio data; and then, the second device plays the second audio data to realize the cross-device playing function of the first file.

That is to say, if the second device has the audio decoding capability of the first file played by the first device, the first device can directly send the audio data which is not decoded in the first file to the second device, and the second device decodes and plays the received audio data, so that the processing process of the audio data between the first device and the second device during the projection of the audio/video file is simplified, and the time consumption of the first device and the second device in the projection of the audio/video file is shortened.

In a possible implementation manner, the decoding, by the second device, the first audio data to obtain decoded second audio data specifically includes: the second device may decode the first audio data by using the first decoding parameter, to obtain the decoded second audio data, where the first decoding parameter corresponds to the first audio parameter, for example, if the format used for encoding the first audio data in the first audio parameter is an AAC format, then the format used for decoding the first audio data in the corresponding first decoding parameter is also an AAC format.

In a possible implementation manner, the first data packet may further include first display data; at this time, the second device may further receive a first display parameter sent by the first device, where the first display parameter is an encoding parameter used when encoding the first display data; and, the second device may determine whether or not it has a capability of decoding the first display data according to the first display parameter; if the first display data decoding capability is available, the first response message sent by the second device to the first device can also be used for instructing the first device to send the first display data to the second device; subsequently, after the second device receives the un-decoded first display data sent by the first device, the first display data can be decoded by utilizing the decoding capability of the second device, and the decoded second display data is obtained; further, the second device may play the second display data.

That is to say, if the second device has the capability of decoding the image of the first file played by the first device, the first device may directly send the undecoded display data in the first file to the second device, and the second device decodes and plays the received display data, so that the processing procedure of the display data between the first device and the second device during video file projection is simplified, and the time consumption of the first device and the second device for video file projection is shortened.

In a possible implementation manner, the decoding, by the second device, the first display data to obtain decoded second display data includes: the second device decodes the first display data by using the second decoding parameter to obtain decoded second display data, where the second decoding parameter corresponds to the first display parameter, for example, a format used for encoding the first display data in the first display parameter is h.265 format, and a format used for decoding the first display data in the corresponding second decoding parameter is also h.265 format.

In a possible implementation manner, since parameters used when encoding audio data and display data in each data packet in the same audio/video file are generally uniform, if a second device has the capability of decoding first audio data and first display data, it indicates that the second device has the capability of decoding other audio data and display data in the first file, and if a subsequent second device receives third audio data in the first file, the second device may directly decode and play the third audio data; similarly, if the subsequent second device receives the third display data in the first file, the second device can directly decode the third display data and play the third display data.

In one possible implementation manner, after the second device determines whether the second device has the capability of decoding the first audio data according to the first audio parameter, the method further includes: if the second device has the capability of decoding the first audio data, the second device creates an audio decoder (audio decoder) according to the first audio parameters, wherein the audio decoder is used for decoding the audio data in the first file; after the second device determines whether the second device has the capability of decoding the first display data according to the first display parameters, the method further comprises the following steps: if the second device has the capability of decoding the first display data, the second device creates an image decoder (video decoder) according to the first display parameters, and the image decoder is used for decoding the display data in the first file. That is to say, after the second device determines that the second device has the capability of decoding the audio and video data in the first file, the corresponding audio decoder and image decoder may be created in advance, so that when the second device receives the audio data/display data sent by the first device and not decoded, the second device may use the created audio decoder/image decoder for decoding, and the time consumption of the first device and the second device for projecting the audio and video files is further shortened.

In a seventeenth aspect, the present application provides an apparatus recommendation method, including: the first equipment can receive a first operation of opening a first service by a user; in response to the first operation, the first device may respectively acquire device parameters associated with the first service from N (N is an integer greater than 0) electronic devices located in the same communication network; furthermore, the first device may predict, according to the obtained device parameters, a service quality when each of the N electronic devices executes the first service; subsequently, the first device may display a device list according to the service quality of the first service executed by each of the N electronic devices, where the device list includes a device identifier of at least one of the N electronic devices, so as to recommend a specific device suitable for executing the first service to the user.

Illustratively, the first service may be a service related to an audio function, such as an audio playing service, a recording service, and the like. Alternatively, the first service may be a service related to a display function, such as a video playing service, a video call service, a game service, and the like. Alternatively, the first service may be a service related to a camera function, such as a video call service, a video recording service, and the like.

For example, the device parameters may include one or more of audio parameters related to an audio function, display parameters related to a display function, and camera parameters related to a camera function.

That is, before a certain service is executed, the first device as a master device may obtain device parameters related to slave devices (e.g., the above N electronic devices), so as to predict the service quality when each slave device executes the service. Therefore, the first device can recommend the specific device for executing the service to the user in a mode of displaying the device list based on the predicted service quality when each slave device executes the service, so that the service quality when multiple devices cooperatively work in a distributed scene is improved, and the use experience of the user is improved.

In a possible implementation manner, the N electronic devices may include a second device; the device parameters acquired by the first device from the second device may include M (M is an integer greater than 0) parameters; at this time, the predicting, by the first device, the service quality of the first service executed by the second device according to the device parameter includes: the first equipment determines M scores corresponding to the M parameters one by one; furthermore, the first device can calculate the service quality score of the second device for executing the first service according to the M scores; the quality of service of the first service is higher when the quality of service score is higher.

For example, the first device may score M parameters of the second device one by one to obtain corresponding M scores, and further, the first device may perform weighted average on the M scores according to a preset weight to obtain a quality of service score for the second device to execute the first service. Of course, the service quality score of the first service executed by the other devices in the N electronic devices may also be calculated according to the method, which is not limited in this application.

In a possible implementation manner, the N electronic devices may further include a third device; when the service quality score of the second equipment executing the first service is higher than that of the third equipment executing the first service, the arrangement sequence of the second equipment in the equipment list is before the third equipment. That is, the first device may recommend devices for performing the related service to the user in the displayed device list through the order of precedence between the devices.

In a possible implementation manner, when the service quality score of the second device executing the first service is higher than the service quality score of other electronic devices in the N electronic devices executing the first service, indicating that the second device is the most suitable device for executing the first service in the N electronic devices, the second device may be marked as a recommended device in the device list, so as to prompt the user to execute the first service using the recommended device.

In a possible implementation manner, the M parameters may include a first parameter corresponding to a first function, for example, a display parameter corresponding to a display function; then, when the score of the first parameter of the second device is higher than the scores of the first parameters of the other electronic devices in the N electronic devices, which indicates that the second device has higher quality of service when executing the first function of the first service, the first device may display first prompt information in the device list, where the first prompt information is used to recommend the user to implement the first function using the second device. That is, the first device may recommend a device to the user, in which a certain function is more prominent when the first service is executed.

In a possible implementation manner, the M parameters include a first parameter corresponding to a first function, for example, a display parameter corresponding to a display function; then, when the score of the first parameter of the second device is lower than the scores of the first parameters of the other electronic devices in the N electronic devices, which indicates that the second device has lower quality of service when executing the first function of the first service, the first device may display, in the device list, second prompt information, where the second prompt information includes a reason why the user is not recommended to use the second device to implement the first function. That is, the first device may prompt the user for a specific reason why the execution of the first service using a certain device is not recommended.

In a possible implementation manner, the N electronic devices include a second device; after the first device displays the device list according to the service quality of the first service executed by each of the N electronic devices, the method further includes: the first device receives a second operation that the user selects a second device in the device list; then, in response to the second operation, the first device may switch the first service to be executed in the second device. Therefore, the user can switch the first service to the second device based on the device recommended by the first device in the device list, and a distributed service scene of multi-device cooperative work is realized.

In a possible implementation manner, after the first device switches the first service to the second device for execution, the method further includes: the first device may again obtain device parameters associated with the first service from current slave devices (e.g., N electronic devices located in the same communication network as the mobile phone); similarly, the first device may predict, according to the obtained device parameters, the service quality of the first service executed by each of the current N electronic devices; when the second device is not the electronic device with the highest quality of service for executing the first service among the N electronic devices, which indicates that the first service is currently not suitable for being executed on the second device, the first device may display a first prompt message, where the first prompt message is used to prompt the user to execute the first service using the recommendation device, and the recommendation device is one or more of the N electronic devices.

That is, the first device as a master device may obtain, in real time, relevant device parameters of each slave device in the process of executing a certain service in cooperation with other devices (e.g., the second device), so as to predict the service quality when each slave device executes the service. Therefore, the first device can recommend the recommendation device for executing the service to the user by displaying the prompt message based on the predicted service quality when each slave device executes the service, so that the service quality when multiple devices cooperatively work in a distributed scene is improved, and the use experience of the user is improved.

In a possible implementation manner, the predicting, by the first device, the service quality of the first service executed by each of the N electronic devices according to the device parameter includes: the first device may calculate a quality of service score for each electronic device to execute the first service according to the device parameter acquired from each electronic device. For example, the first device may score each of the device parameters corresponding to each electronic device, and further perform weighted average on all the scores to calculate a service quality score of the corresponding electronic device for executing the first service.

In a possible implementation manner, the N electronic devices include a third device; when the service quality score of the first service executed by the third device is higher than that of other electronic devices in the N electronic devices, the recommendation device is the third device; or, when the service quality score of the first subtask executed by the third device in the first service is higher than that of the other electronic devices in the N electronic devices, the recommendation device is the third device. That is, when a certain device of the N electronic devices except the second device executes the first service or the service quality of a certain subtask in the first service is high, the first device may determine the device as a recommendation device, so as to recommend the user to execute the first service using the recommendation device.

In a possible implementation manner, the first service may include a first subtask and a second subtask, for example, a screen-casting service may include an audio playing task and a display task; the N electronic devices may include a third device and a fourth device. When the service quality score of the third device executing the first subtask is higher than that of other electronic devices in the N electronic devices, and the service quality score of the fourth device executing the second subtask is higher than that of other electronic devices in the N electronic devices, the recommendation device is a device group consisting of the third device and the fourth device. That is, when a higher quality of service can be obtained when a plurality of devices cooperatively execute a first service, the first device may recommend to a user to use the plurality of devices to execute the first service collectively in the form of a device group.

In a possible implementation manner, the method further includes: the first equipment predicts the service quality score of the first equipment for executing the first service; when the service quality score of the first service executed by the first device is higher than that of the N electronic devices, the recommendation device may be the first device. That is, the main device (i.e., the first device) may also be a candidate device for the recommendation device, and when the service quality of the target service (e.g., the first service) executed by the first device is high, the user may also be prompted to execute the first service by using the first device.

In a possible implementation manner, the first prompt message may include a device identifier of the recommendation device; after the first prompting message is displayed by the first equipment, the method further comprises the following steps: the first equipment receives a third operation that the user selects the recommendation equipment in the first prompt message; and responding to the third operation, and switching the first service to the recommending device by the first device for execution. Therefore, the user can quickly switch the first service to the recommendation device with higher service quality for execution through the device identifier of the recommendation device in the first prompt message.

In a possible implementation manner, when the second device is not an electronic device with the highest quality of service for executing the first service among the N electronic devices, the method further includes: the first device may notify the second device of the recommendation device, so that the second device displays a second prompt message, where the second prompt message is used to prompt the user to execute the first service using the recommendation device. Similarly, the user can quickly switch the first service to the recommendation device with higher service quality in the second device through the second prompt message.

In one possible implementation, the device parameters may include one or more of audio parameters, display parameters, camera parameters, or time delays. The audio parameters may include one or more items of sound effect algorithms, the number of channels, sampling rates, mixing strategies, coding and decoding algorithms of audio data, the number and model of microphones, the number and model of loudspeakers, and the like, which are supported by the slave device. The display parameters may include one or more of resolution, frame rate, and codec algorithm of the display data supported by the slave device. The camera parameters may include one or more of an image processing algorithm supported by the slave device, the number of cameras, a resolution of the cameras, or a FOV of the cameras.

In an eighteenth aspect, the present application provides an ultrasonic positioning method, including: when the first device detects that a user opens a screen-casting function (also referred to as a function of content connection, cross-device continuous playing, etc.), the first device may detect K (K is an integer greater than 1) electronic devices that access the same communication network as the first device, for example, the K electronic devices may be electronic devices that log in the same account as the first device; furthermore, the first device can perform ultrasonic positioning on a second device of the K electronic devices to obtain a positioning result (i.e., a first positioning result) of the second device; moreover, the first device can perform ultrasonic positioning on a third device of the K electronic devices to obtain a positioning result (i.e., a second positioning result) of the third device; in this way, the first device may display, in the interactive interface, an identification of the second device and an identification of the third device, where a location of the identification of the second device in the interactive interface is associated with the first positioning result and a location of the identification of the third device in the interactive interface is associated with the second positioning result.

That is to say, when the user triggers the screen projection function of the first device, the first device may be used as a main device to perform ultrasonic positioning on the second device and the third device, so as to obtain positioning results of the second device and the third device. Then, the first device may display the identifiers of the second device and the third device in the interactive interface according to the obtained positioning result, so that the user may visually see the currently detected positional relationship between the second device and the third device in the interactive interface. Therefore, the user can conveniently and accurately select the device which is used for projecting the screen with the first device in the interactive interface according to the position relation, and the use experience of the user in the screen projecting scene is improved.

The interactive interface may be all interfaces displayed on the display screen of the first device, or may be a part of interfaces displayed on the display screen of the first device. For example, the first device may display specific content (e.g., video or images, etc.) waiting to be projected in display area 1 of the display screen, while the first device may display the identities of the second and third devices in display area 2 of the display screen. In this case, the display area 2 is the above-mentioned interactive interface.

In one possible implementation, the first device performs ultrasonic localization on the second device, including: the first device may send a first positioning instruction to the second device, where the first positioning instruction is used to trigger the second device to transmit a first ultrasonic signal; the first device can use N (N is an integer larger than 1) ultrasonic receivers to acquire the first ultrasonic signal while the second device transmits the first ultrasonic signal; furthermore, the first device may locate the second device according to the times at which the second ultrasonic signals respectively reach the N ultrasonic receivers.

For example, the first device may locate the second device based on times at which the second ultrasonic signals respectively arrive at the above-mentioned N ultrasonic receivers using a Time Of Arrival (TOA) or Time Difference Of Arrival (TDOA) algorithm.

The N ultrasonic receivers of the first device may specifically be N microphones in a microphone array in the first device. At this moment, the first device does not need to additionally add an ultrasonic receiver to adapt to the ultrasonic positioning method provided by the embodiment of the application, so that the realization cost and difficulty of indoor positioning are reduced.

Of course, the first device may also perform ultrasonic positioning on the third device according to the above method, which is not limited in this application.

Or, in another possible implementation manner, the ultrasonic positioning of the third device by the first device specifically includes: the first equipment sends a second positioning instruction to the third equipment, and the second positioning instruction is used for triggering the third equipment to transmit a second ultrasonic signal; the first device may acquire first ultrasonic data while the third device transmits the second ultrasonic signal, the first ultrasonic data including times at which the second ultrasonic signal respectively reaches the N ultrasonic receivers in the first device; also, the first device may acquire second ultrasonic data including times at which the second ultrasonic signals respectively reach M (M is an integer greater than 1) ultrasonic receivers in the second device; the first device locates the third device based on the first ultrasound data and the second ultrasound data.

That is to say, the first device can utilize the existing ultrasonic positioning capability of the second device as the master device, and the ultrasonic receivers of the first device and the second device are simultaneously started to receive the ultrasonic signals sent by the third device to be positioned in a multi-device cooperation mode, so that the third device is positioned according to more received ultrasonic signals, the positioning accuracy of the third device is improved, and meanwhile, the operation process and the implementation cost of indoor positioning are simplified.

In a possible implementation manner, before the first device sends the second positioning instruction to the third device, the method further includes: the first equipment determines second equipment as cooperative equipment in the K electronic equipment, and the cooperative equipment is used for performing ultrasonic positioning in cooperation with the first equipment; the first device sends a first co-location instruction to the second device, and the first co-location instruction is used for triggering the second device to detect a second ultrasonic signal by using the M ultrasonic receivers. The second device may turn on its M ultrasonic receivers in response to the first co-location instruction, detect the second ultrasonic signals using the M ultrasonic receivers, and transmit the detected M channels of second ultrasonic signals as the second ultrasonic data to the first device.

Therefore, in a positioning scene, the number of microphones which actually collect ultrasonic signals transmitted by the third equipment to be positioned is increased, and the microphones which collect the ultrasonic signals are positioned at different positions of the first equipment and the second equipment, so that the first equipment can position the third equipment according to more ultrasonic signals and more directions, and the positioning accuracy of the equipment is improved.

In one possible implementation manner, the determining, by the first device, the second device as a cooperative device among the K electronic devices includes: and when the second equipment is the electronic equipment with the highest positioning capability in the K electronic equipment, the first equipment determines the second equipment as the cooperative equipment. For example, when the number of microphones provided on a certain electronic device is larger, the positioning capability of the electronic device can be set to be higher. Alternatively, when the distance between microphones provided on a certain electronic device is larger, the positioning capability of the electronic device may be set to be higher. Alternatively, when the performance of a hardware parameter such as a processor of a certain electronic device is higher, the positioning capability of the electronic device may be set to be higher. And the second equipment with higher positioning capability is used for positioning the third equipment in cooperation with the first equipment, so that the positioning precision of the third equipment can be improved.

In one possible implementation, the first device locates the third device according to the first ultrasonic data and the second ultrasonic data, including: when the first device has obtained the positioning result of the second device, M microphones on the second device can be used as the microphones of the first device, which is equivalent to a part of the microphones (for example, the N microphones) of the first device being arranged on the local device, and another part of the microphones (for example, the M microphones) of the first device being arranged on the second device. Furthermore, the first device may position the third device using a preset positioning algorithm based on the acquired first ultrasonic data and the acquired second ultrasonic data.

In a possible implementation manner, the N ultrasonic receivers may be N microphones in a microphone array of the first device; and/or the M ultrasonic receivers may be M microphones in a microphone array of the second device; at this time, the first device acquires first ultrasonic data including: the first device detects the second ultrasonic signals by using the N microphones to obtain N paths of second ultrasonic signals (namely first ultrasonic data); also, the second device may detect the second ultrasonic signal using the M microphones to obtain M channels of second ultrasonic signals (i.e., second ultrasonic data), and may transmit the second ultrasonic data to the first device.

At this time, the first device locating the third device according to the first ultrasonic data and the second ultrasonic data may include: the first device combines the N second ultrasonic signals and the M second ultrasonic signals into multi-channel ultrasonic data, where the multi-channel ultrasonic data may include: the waveform of each ultrasonic signal in the N second ultrasonic signals and the time for each ultrasonic signal to reach the corresponding microphone, and the waveform of each ultrasonic signal in the M second ultrasonic signals and the time for each ultrasonic signal to reach the corresponding microphone; furthermore, the first device may use a preset positioning algorithm to position the third device according to the multi-channel ultrasound data.

Alternatively, the first device ultrasonically locating the third device may include: the first equipment sends a second positioning instruction to the third equipment, and the second positioning instruction is used for triggering the third equipment to transmit a second ultrasonic signal; while the third device is transmitting the second ultrasonic signal, the first device may determine a first ultrasonic location result from times at which the second ultrasonic signal reaches the N ultrasonic receivers in the first device, and the second device may determine a second ultrasonic location result from times at which the second ultrasonic signal reaches the M ultrasonic receivers in the second device. That is, the first device and the second device may simultaneously ultrasonically locate the third device. And then, the second equipment can send the obtained second ultrasonic positioning result to the first equipment, and the first equipment corrects the first ultrasonic positioning result calculated by the first equipment according to the second ultrasonic positioning result and outputs the final positioning result of the third equipment. Since the finally output positioning result of the third device is obtained by combining the first ultrasonic positioning result and the second ultrasonic positioning result, the positioning accuracy of the third device can be improved.

For example, the N ultrasonic receivers may be N microphones in a microphone array of the first device; at this time, the first device acquires a first ultrasound localization result, including: the first device detects a second ultrasonic signal using the N microphones; and the first equipment positions the third equipment according to the time when each microphone in the N microphones detects the second ultrasonic signal, so as to obtain a first ultrasonic positioning result.

Similarly, the M ultrasonic receivers may be M microphones in a microphone array of the second device; at this time, the second device may detect the second ultrasonic signal using the M microphones; positioning the third equipment according to the time when each microphone in the M microphones detects the second ultrasonic signal to obtain a second ultrasonic positioning result; in turn, the second device may send the second ultrasound localization result to the first device.

In a possible implementation manner, the K electronic devices further include a fourth device; then, in addition to performing ultrasonic positioning on the second device and the third device according to the above method, the first device may also perform ultrasonic positioning on the fourth device according to the above method, so as to obtain a third positioning result of the fourth device; furthermore, the first device may display, in the interactive interface, an identifier of the second device, an identifier of the third device, and an identifier of the fourth device, where a position of the identifier of the second device in the interactive interface is related to the first positioning result, a position of the identifier of the third device in the interactive interface is related to the second positioning result, and a position of the identifier of the fourth device in the interactive interface is related to the third positioning result.

In this way, the user can visually see the currently detected position relationship of the second device, the third device and the fourth device in the interactive interface. Subsequently, the user can conveniently and accurately select the device which is used for projecting the screen with the first device in the interactive interface according to the position relation, and the use experience of the user in the screen projecting scene is improved.

In a possible implementation manner, the process of performing ultrasonic positioning on the fourth device by the first device is similar to the process of performing ultrasonic positioning on the third device by the first device, for example, the first device may send a third positioning instruction to the fourth device, where the third positioning instruction is used to trigger the fourth device to transmit a third ultrasonic signal; the first device may acquire third ultrasonic data while the fourth device transmits the third ultrasonic signal, the third ultrasonic data including times at which the third ultrasonic signal respectively reaches the N ultrasonic receivers in the first device; meanwhile, the first device may acquire fourth ultrasonic data, where the fourth ultrasonic data includes times at which the third ultrasonic signal respectively reaches M ultrasonic receivers in the second device; further, the first device may locate a fourth device based on the third ultrasonic data and the fourth ultrasonic data.

In a possible implementation manner, in the process of performing ultrasonic positioning on the fourth device by the first device, the first device may further use both the second device and the third device as cooperative devices to perform ultrasonic positioning on the fourth device in cooperation with the first device. For example, while the fourth apparatus transmits the third ultrasonic signal, the third apparatus may detect the third ultrasonic signal using its own Z ultrasonic receivers as a cooperative apparatus, and the third apparatus may transmit the detected Z-path third ultrasonic signal as fifth ultrasonic data to the first apparatus. Therefore, the first device can position the fourth device according to the acquired third ultrasonic data, the acquired fourth ultrasonic data and the acquired fifth ultrasonic data, so that more ultrasonic signals are used for positioning the fourth device, and the positioning accuracy of the fourth device is improved.

For example, when the first device sends a third positioning instruction to the fourth device, the first device may also send a second co-positioning instruction to the second device and the third device (i.e., two co-devices), the second co-positioning instruction being used to trigger the second device to detect the third ultrasonic signal using the M ultrasonic receivers, and the second co-positioning instruction being used to trigger the third device to detect the third ultrasonic signal using the Z ultrasonic receivers.

Of course, if the K electronic devices further include other electronic devices such as a fifth device, the first device may still perform ultrasonic positioning on the other electronic devices according to the method described above, which is not limited in this embodiment of the present application.

In one possible implementation manner, after the first device displays the identifier of the second device and the identifier of the third device in the interactive interface, the method further includes: in response to the operation that the user selects the identifier of the second device, the first device can project the content played by the first device to the second device for playing; or, in response to an operation of selecting the identifier of the third device by the user, the first device may project the content played by the first device to the third device for playing. That is, the user can select the device needing to be projected with the first device according to the position relation among the various electronic devices displayed in the interactive interface.

The operation of the user on a certain device in the interactive interface may specifically be a click operation, a slide operation, an air gesture operation, or a corresponding voice instruction input operation, which is not limited in this application.

In a nineteenth aspect, the present application provides an ultrasonic positioning method, including: the second equipment can be used as electronic equipment to be positioned to receive the positioning instruction sent by the first equipment; in response to the positioning instruction, the second device may emit the first ultrasonic signal using its own ultrasonic generator (e.g., a speaker). Subsequently, the first device may perform ultrasonic positioning on the second device according to the time when the first ultrasonic signal reaches each ultrasonic receiver in the first device, so as to obtain a positioning result of the second device.

In a twentieth aspect, the present application provides an ultrasonic positioning method, comprising: the second device can be used as a cooperative device of the first device to receive the cooperative positioning instruction sent by the first device; in response to the co-location instruction, the second device may turn on its own M ultrasonic receivers (e.g., M microphones, M being an integer greater than 1) to detect the ultrasonic signal emitted by the electronic device to be located (e.g., the third device). The second device may send the detected M ultrasonic signals (i.e., the second ultrasonic data in the first aspect) to the first device, and the first device performs ultrasonic positioning on the third device by combining the second ultrasonic data. Or, the second device may locate the third device according to the time when the M ultrasonic signals reach the M microphones, send the obtained ultrasonic location result (the second ultrasonic location result in the first aspect) to the first device, and locate the third device by the first device in combination with the second ultrasonic location result.

In a twenty-first aspect, the present application provides an electronic device (e.g., the first device described above), comprising: a display screen, a communication module, one or more processors, one or more memories, and one or more computer programs; wherein a processor is coupled to the communication module, the display screen and the memory, wherein the one or more computer programs are stored in the memory, and wherein when the first device is running, the processor executes the one or more computer programs stored in the memory to cause the first device to perform the method of any of the first to twentieth aspects.

In a twenty-second aspect, the present application provides an electronic device (e.g., the second device described above) comprising: a communication module, one or more processors, one or more memories, and one or more computer programs; wherein the processor is coupled to both the communication module and the memory, and the one or more computer programs are stored in the memory, and when the second device is running, the processor executes the one or more computer programs stored in the memory to cause the second device to perform the method of any of the first to twentieth aspects.

In a twenty-third aspect, the present application provides an audio system, comprising the above first device and second device, where the first device and the second device may perform the method of any one of the above first to twentieth aspects by interaction.

In a twenty-fourth aspect, the present application provides a computer-readable storage medium, comprising computer instructions, which, when run on the first device or the second device, cause the first device or the second device to perform the method of any of the first to twentieth aspects.

In a twenty-fifth aspect, the present application provides a computer program product for causing a first device or a second device to perform the method of any one of the first to twentieth aspects when the computer program product is run on the first device or the second device.

It is understood that the electronic device, the audio system, the computer-readable storage medium and the computer program product provided in the foregoing aspects are all applied to the corresponding method provided above, and therefore, the beneficial effects achieved by the electronic device, the audio system, the computer-readable storage medium and the computer program product may refer to the beneficial effects of the corresponding method provided above, and are not described herein again.

Drawings

Fig. 1 is a schematic diagram of an audio system according to an embodiment of the present application;

fig. 2 is a schematic diagram 1 of an audio application scenario provided in an embodiment of the present application;

fig. 3 is a schematic diagram of an audio application scenario provided in an embodiment of the present application 2;

fig. 4 is an interaction diagram of an audio control method according to an embodiment of the present disclosure 1;

fig. 5 is a schematic structural diagram 1 of an electronic device according to an embodiment of the present disclosure;

fig. 6 is a schematic diagram 3 of an audio application scenario provided in an embodiment of the present application;

fig. 7 is a schematic diagram of an audio application scenario provided in an embodiment of the present application 4;

fig. 8 is a schematic diagram 5 of an audio application scenario provided in an embodiment of the present application;

fig. 9 is a schematic diagram of an audio application scenario provided in an embodiment of the present application 6;

fig. 10 is a schematic diagram 7 of an audio application scenario provided in an embodiment of the present application;

Fig. 11 is a schematic diagram 8 of an audio application scenario provided in an embodiment of the present application;

fig. 12 is a schematic diagram of an audio application scenario 9 according to an embodiment of the present application;

fig. 13 is a schematic diagram 10 of an audio application scenario provided in an embodiment of the present application;

fig. 14A is a schematic view 11 of an audio application scenario provided in an embodiment of the present application;

fig. 14B is a schematic diagram 12 of an audio application scenario provided in the embodiment of the present application;

fig. 15 is a schematic diagram 13 of an audio application scenario provided in an embodiment of the present application;

fig. 16A is a schematic view 14 of an audio application scenario provided in the embodiment of the present application;

fig. 16B is a schematic diagram 15 of an audio application scenario provided by an embodiment of the present application;

fig. 17 is a schematic diagram 16 of an audio application scenario provided in an embodiment of the present application;

fig. 18A is a schematic diagram 17 of an audio application scenario provided in the embodiment of the present application;

fig. 18B is a schematic diagram 18 of an audio application scenario provided in the embodiment of the present application;

fig. 19 is a schematic diagram 19 of an audio application scenario provided in an embodiment of the present application;

fig. 20 is a schematic diagram 20 of an audio application scenario provided in an embodiment of the present application;

fig. 21 is a schematic view 21 of an audio application scenario provided in an embodiment of the present application;

Fig. 22 is a schematic diagram 22 of an audio application scenario provided in an embodiment of the present application;

fig. 23 is a schematic diagram 23 of an audio application scenario provided in an embodiment of the present application;

fig. 24 is a schematic diagram 24 of an audio application scenario provided in an embodiment of the present application;

fig. 25 is a schematic diagram 25 of an audio application scenario provided in an embodiment of the present application;

fig. 26 is a schematic diagram 26 of an audio application scenario provided in the embodiment of the present application;

fig. 27 is a schematic view 27 of an audio application scenario provided in an embodiment of the present application;

fig. 28 is a schematic diagram 28 of an audio application scenario provided in an embodiment of the present application;

fig. 29 is a schematic view 29 of an audio application scenario provided in an embodiment of the present application;

fig. 30 is a schematic diagram 30 of an audio application scenario provided in the embodiment of the present application;

fig. 31 is a schematic view 31 of an audio application scenario provided in an embodiment of the present application;

fig. 32 is a schematic diagram 32 of an audio application scenario provided in the embodiment of the present application;

fig. 33 is a schematic diagram 33 of an audio application scenario provided in an embodiment of the present application;

fig. 34 is an interaction diagram of an audio control method according to an embodiment of the present application, schematically illustrated in fig. 2;

fig. 35 is a schematic diagram 34 of an audio application scenario provided in the embodiment of the present application;

Fig. 36 is a schematic diagram 35 of an audio application scenario provided in an embodiment of the present application;

fig. 37 is a schematic diagram 36 of an audio application scenario provided in an embodiment of the present application;

fig. 38 is a schematic diagram 37 of an audio application scenario provided in an embodiment of the present application;

fig. 39 is a schematic diagram 38 of an audio application scenario provided in the embodiment of the present application;

fig. 40 is a schematic diagram 39 of an audio application scenario provided in an embodiment of the present application;

fig. 41 is an interaction diagram of an audio control method according to an embodiment of the present application 3;

fig. 42 is a schematic diagram 40 of an audio application scenario provided by an embodiment of the present application;

fig. 43 is a schematic diagram 41 of an audio application scenario provided in an embodiment of the present application;

fig. 44 is a schematic diagram 42 of an audio application scenario provided in an embodiment of the present application;

fig. 45 is an interaction diagram of an audio control method according to an embodiment of the present application 4;

fig. 46 is a schematic view 43 of an audio application scenario provided in an embodiment of the present application;

fig. 47 is a schematic diagram 44 of an audio application scenario provided in an embodiment of the present application;

fig. 48 is a schematic diagram 45 of an audio application scenario provided in the embodiment of the present application;

fig. 49 is a schematic diagram 46 of an audio application scenario provided by an embodiment of the present application;

Fig. 50 is a schematic view 47 of an audio application scenario provided by an embodiment of the present application;

fig. 51 is a schematic diagram 48 of an audio application scenario provided in an embodiment of the present application;

fig. 52 is a schematic diagram 49 of an audio application scenario provided by an embodiment of the present application;

fig. 53 is a schematic diagram 50 of an audio application scenario provided in an embodiment of the present application;

fig. 54 is a schematic diagram 51 of an audio application scenario provided in an embodiment of the present application;

fig. 55 is a schematic diagram 52 of an audio application scenario provided in an embodiment of the present application;

fig. 56 is a schematic diagram 53 of an audio application scenario provided in an embodiment of the present application;

fig. 57 is a schematic diagram 54 of an audio application scenario provided in an embodiment of the present application;

fig. 58 is a schematic diagram 55 of an audio application scenario provided in an embodiment of the present application;

fig. 59 is a schematic diagram 56 of an audio application scenario provided by an embodiment of the present application;

fig. 60 is a schematic diagram 57 of an audio application scenario provided in an embodiment of the present application;

fig. 61 is a schematic view 58 of an audio application scenario provided in an embodiment of the present application;

fig. 62 is a schematic diagram 59 of an audio application scenario provided in an embodiment of the present application;

fig. 63 is a schematic diagram 60 of an audio application scenario provided by an embodiment of the present application;

Fig. 64 is a schematic diagram 61 of an audio application scenario provided in an embodiment of the present application;

fig. 65 is a schematic diagram 62 of an audio application scenario provided in an embodiment of the present application;

fig. 66 is a schematic diagram 63 of an audio application scenario provided in an embodiment of the present application;

fig. 67 is a schematic diagram 64 of an audio application scenario provided by an embodiment of the present application;

fig. 68 is a schematic diagram 65 of an audio application scenario provided in an embodiment of the present application;

fig. 69 is a schematic diagram 66 of an audio application scenario provided by an embodiment of the present application;

fig. 70 is a schematic diagram 67 of an audio application scenario provided in an embodiment of the present application;

fig. 71 is a schematic diagram 68 of an audio application scenario provided in an embodiment of the present application;

fig. 72 is a schematic diagram 69 of an audio application scenario provided by an embodiment of the present application;

fig. 73 is a schematic diagram 70 of an audio application scenario provided by an embodiment of the present application;

fig. 74 is a schematic diagram 71 of an audio application scenario provided in an embodiment of the present application;

fig. 75 is a schematic diagram 72 of an audio application scenario provided by an embodiment of the present application;

fig. 76 is a schematic view 73 of an audio application scenario provided by an embodiment of the present application;

fig. 77 is a schematic diagram 74 of an audio application scenario provided by an embodiment of the present application;

Fig. 78 is a schematic diagram 75 of an audio application scenario provided by an embodiment of the present application;

fig. 79 is a schematic diagram 76 of an audio application scenario provided by an embodiment of the present application;

fig. 80 is a schematic diagram 77 of an audio application scenario provided by an embodiment of the present application;

fig. 81 is a schematic diagram 78 of an audio application scenario provided by an embodiment of the present application;

fig. 82 is a schematic diagram 79 illustrating an audio application scenario provided by an embodiment of the present application;

fig. 83 is a schematic diagram 80 of an audio application scenario provided by an embodiment of the present application;

fig. 84 is a schematic diagram 81 of an audio application scenario provided in an embodiment of the present application;

fig. 85 is an interaction diagram of a screen recording method according to an embodiment of the present application;

fig. 86 is a schematic diagram 82 of an audio application scenario provided by an embodiment of the present application;

fig. 87 is a schematic diagram 83 of an audio application scenario provided by an embodiment of the present application;

FIG. 88 is a schematic diagram 84 of an audio application scenario provided by an embodiment of the present application;

fig. 89 is a schematic diagram 85 of an audio application scenario provided in the embodiment of the present application;

fig. 90 is a schematic diagram 86 of an audio application scenario provided by an embodiment of the present application;

fig. 91 is a schematic view 87 of an audio application scenario provided by an embodiment of the present application;

Fig. 92 is a schematic diagram 88 of an audio application scenario provided by an embodiment of the present application;

fig. 93 is a schematic view 89 of an audio application scenario provided in an embodiment of the present application;

fig. 94 is a schematic diagram 90 of an audio application scenario provided in an embodiment of the present application;

fig. 95 is a schematic diagram 91 of an audio application scenario provided in an embodiment of the present application;

fig. 96 is a schematic diagram 92 of an audio application scenario provided by an embodiment of the present application;

fig. 97 is a schematic view 93 of an audio application scenario provided in an embodiment of the present application;

fig. 98 is a schematic diagram 94 of an audio application scenario provided by an embodiment of the present application;

fig. 99 is a schematic diagram 95 of an audio application scenario provided by an embodiment of the present application;

fig. 100 is a schematic diagram 96 of an audio application scenario provided by an embodiment of the present application;

fig. 101 is a schematic view 97 of an audio application scenario provided in an embodiment of the present application;

fig. 102 is a schematic diagram 98 of an audio application scenario provided by an embodiment of the present application;

fig. 103 is a schematic diagram 99 of an audio application scenario provided in an embodiment of the present application;

fig. 104 is a schematic diagram 100 of an audio application scenario provided in an embodiment of the present application;

FIG. 105 is an interaction diagram of an audio update method according to an embodiment of the present application, which is shown in FIG. 1;

FIG. 106 is an interaction diagram of an audio update method according to an embodiment of the present application, which is shown in FIG. 2;

fig. 107 is a schematic view 101 of an audio application scenario provided in an embodiment of the present application;

fig. 108 is a schematic view 102 of an audio application scenario provided in an embodiment of the present application;

fig. 109 is a schematic diagram 103 of an audio application scenario provided in an embodiment of the present application;

fig. 110 is a schematic view 104 of an audio application scenario provided by an embodiment of the present application;

fig. 111 is a schematic diagram 105 of an audio application scenario provided in an embodiment of the present application;

fig. 112 is a schematic diagram 106 of an audio application scenario provided by an embodiment of the present application;

fig. 113 is a schematic view 107 of an audio application scenario provided in an embodiment of the present application;

fig. 114 is a schematic view 108 of an audio application scenario provided in an embodiment of the present application;

fig. 115 is a schematic diagram 109 of an audio application scenario provided by an embodiment of the present application;

fig. 116 is a schematic view 110 of an audio application scenario provided by an embodiment of the present application;

fig. 117 is a schematic view 111 of an audio application scenario provided in an embodiment of the present application;

fig. 118 is a schematic diagram 112 of an audio application scenario provided by an embodiment of the present application;

fig. 119 is a schematic view 113 of an audio application scenario provided by an embodiment of the present application;

Fig. 120 is a schematic diagram 114 of an audio application scenario provided by an embodiment of the present application;

fig. 121 is a schematic diagram 115 of an audio application scenario provided by an embodiment of the present application;

fig. 122 is an interaction diagram of a device recommendation method according to an embodiment of the present application;

fig. 123 is a schematic view 116 of an audio application scenario provided by an embodiment of the present application;

fig. 124 is a schematic view 117 of an audio application scenario provided in an embodiment of the present application;

fig. 125 is a schematic diagram 118 of an audio application scenario provided by an embodiment of the present application;

fig. 126 is a schematic view 119 of an audio application scenario provided in an embodiment of the present application;

fig. 127 is a schematic diagram 120 of an audio application scenario provided by an embodiment of the present application;

fig. 128 is a schematic diagram 121 of an audio application scenario provided in the embodiment of the present application;

fig. 129 is a schematic diagram 122 of an audio application scenario provided by an embodiment of the present application;

fig. 130 is a schematic view 123 of an audio application scenario provided by an embodiment of the present application;

fig. 131 is a schematic diagram 124 of an audio application scenario provided in an embodiment of the present application;

fig. 132 is a schematic diagram 125 of an audio application scenario provided by an embodiment of the present application;

fig. 133 is a schematic diagram 126 of an audio application scenario provided by an embodiment of the present application;

Fig. 134 is a schematic view 127 of an audio application scenario provided by an embodiment of the present application;

fig. 135 is a schematic diagram 128 of an audio application scenario provided by an embodiment of the present application;

fig. 136 is a schematic view 129 of an audio application scenario provided by an embodiment of the present application;

fig. 137 is a schematic view 130 of an audio application scenario provided by an embodiment of the present application;

fig. 138 is a schematic diagram 131 of an audio application scenario provided by an embodiment of the present application;

fig. 139 is a schematic diagram 132 of an audio application scenario provided by an embodiment of the present application;

fig. 140 is a schematic view 133 of an audio application scenario provided by an embodiment of the present application;

fig. 141 is a schematic diagram 134 of an audio application scenario provided in the embodiment of the present application;

fig. 142 is a schematic diagram 135 of an audio application scenario provided by an embodiment of the present application;

fig. 143 is a schematic view 136 of an audio application scenario provided in an embodiment of the present application;

fig. 144 is a schematic diagram 137 of an audio application scenario provided by an embodiment of the present application;

fig. 145 is a schematic diagram 138 of an audio application scenario provided by an embodiment of the present application;

fig. 146 is a schematic diagram 139 of an audio application scenario provided in the embodiment of the present application;

fig. 147 is a schematic diagram 140 illustrating an audio application scenario provided by an embodiment of the present application;

Fig. 148 is a schematic diagram 141 of an audio application scenario provided by an embodiment of the present application;

fig. 149 is a schematic view 142 of an audio application scenario provided in an embodiment of the present application;

fig. 150 is a schematic view 143 of an audio application scenario provided in an embodiment of the present application;

fig. 151 is a schematic diagram 144 of an audio application scenario provided by an embodiment of the present application;

FIG. 152 is a schematic view 145 of an audio application scenario provided by an embodiment of the present application;

fig. 153 is a schematic view 146 illustrating an audio application scenario provided in an embodiment of the present application;

fig. 154 is a schematic diagram 147 of an audio application scenario provided by an embodiment of the present application;

FIG. 155 is a diagram 148 illustrating an audio application scenario provided by an embodiment of the present application;

FIG. 156 is a diagram 149 of an audio application scenario provided by an embodiment of the present application;

fig. 157 is a schematic structural diagram of an electronic device according to an embodiment of the present application 2;

fig. 158 is a schematic structural diagram 3 of an electronic device according to an embodiment of the present application.

Detailed Description

Embodiments of the present embodiment will be described in detail below with reference to the accompanying drawings.

The audio control method provided by the embodiment of the application can be applied to the audio system 200 shown in fig. 1. As shown in fig. 1, the audio system 200 may include a master device (master)101 and N slave devices (slave)102, where N is an integer greater than 0. The master device 101 and any one of the slave devices 102 may communicate with each other by wire or wirelessly.

Illustratively, a wired connection may be established between the master device 101 and the slave device 102 using a Universal Serial Bus (USB). For another example, the master device 101 and the slave device 102 may establish a wireless connection through a global system for mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), time division code division multiple access (TD-SCDMA), Long Term Evolution (LTE), bluetooth, wireless fidelity (Wi-Fi), NFC, voice over Internet protocol (VoIP), and a communication protocol supporting a network slice architecture.

The master device 101 is configured to provide audio data to be played, and output the audio data to the slave device 102. The slave device 102 is configured to receive the audio data sent by the master device 101 and play the audio data. Alternatively, the slave device 102 may be configured to collect audio data and send the collected audio data to the master device 101 for processing, such as storage. That is, the master device 101 may distribute its own audio input output functionality to one or more slave devices 102, thereby enabling a distributed audio capability across the devices.

The main device 101 may be specifically a mobile phone, a sound box, a tablet Computer, a television (also referred to as a smart television, a smart screen, or a large screen device), a notebook Computer, an Ultra-mobile Personal Computer (UMPC), a handheld Computer, a netbook, a Personal Digital Assistant (PDA), a wearable electronic device, a vehicle-mounted device, a virtual reality device, and other devices having an audio input and output function, which is not limited in this embodiment. For example, the slave device 102 may be a conventional audio output device such as a bluetooth headset or a wired headset, or an intelligent electronic device such as a mobile phone, a television, a sound box or a vehicle-mounted device, which is equipped with an operating system (e.g., an android system). After receiving the audio data sent by the master device 101, the slave device 102 may perform processing such as sampling, mixing, or adding sound effects to the audio data through its operating system.

Taking a mobile phone as the master device 101 for example, a user may select one or more slave devices 102 as audio output devices of the mobile phone through a control center of the mobile phone. For example, as shown in fig. 2, the mobile phone may display the control center 201 (which may also be referred to as opening the control center) in response to a preset operation (e.g., a pull-up operation or a pull-down operation) input by the user in the current display interface, where the control center 201 may also be referred to as a pull-up menu or a pull-down menu. In the control center 201, a user may set some switches for shortcut functions in the mobile phone, for example, a bluetooth function switch, a Wireless Local Area Network (WLAN) function switch, a flashlight switch, and a brightness and volume adjusting switch, which is not limited in this embodiment of the present application.

In the embodiment of the present application, as also shown in fig. 2, the mobile phone may further display a switch button 202 of the audio switching function in the control center 201. When a user wishes to modify the audio output device of a current handset, the clickable switch button 202 queries one or more electronic devices that are currently available as audio output devices.

Illustratively, upon detecting that the user has clicked the switch button 202, the handset may display a detailed menu 301 of audio switching functions in the control center 201, as shown in fig. 3. The details menu 301 includes one or more candidate devices 302 that the current handset searches for that can be switched audio with the handset. For example, the handset may search for electronic devices that are located in the same Wi-Fi network as the handset and display the searched electronic devices as candidate devices in the details menu 301. For another example, the mobile phone may query the server for an electronic device that is logged in to the same account (e.g., hua account) as the mobile phone, and display the queried electronic device as a candidate device in the details menu 301. In this way, the user can visually see one or more electronic devices currently available for audio switching with the mobile phone in the control center 201, which is convenient for the user to select.

As also shown in fig. 3, the candidate devices displayed in the details menu 301 may include the handset itself (i.e., native), headphones, speakers, a television or other handset, or other like electronic device with audio input and output capabilities. If the user selects the mobile phone, the user wants to output the audio of the mobile phone by using a speaker (speaker) of the mobile phone; if the user selects other candidate devices, it indicates that the user wishes to output the audio of the mobile phone through the candidate device selected by the user, and the candidate device selected by the user can be used as the slave device 102 of the mobile phone to play the audio output by the mobile phone.

In the embodiment of the present application, as shown in fig. 4, after the mobile phone detects that the user selects the slave device 102 for audio switching with the mobile phone, the mobile phone may establish a network connection with the slave device 102. Taking the television as the slave device 102 selected by the user for example, the mobile phone can establish network connection with the television after detecting that the user selects the television. For example, the handset may establish a Wi-Fi connection with the television through the router, or the handset may establish a Wi-Fi P2P connection directly with the television. Furthermore, the handset can obtain the audio capability parameter of the television from the television, wherein the audio capability parameter is used for reflecting the specific audio input or output capability supported by the television. Taking the audio output capability supported by the television as an example, the audio capability parameter may include one or more parameters of the television, such as the playing delay, the sound effect parameter, the audio sampling rate, or the number of sound channels. Therefore, the mobile phone can determine an audio playing strategy when audio switching is carried out based on the audio capability parameter of the television, and then output corresponding audio data to the television according to the audio playing strategy, so that the audio on the mobile phone is switched to the television to be played.

For example, the mobile phone acquires audio data indicating that the television supports dual-channel playing in the audio capability parameter of the television. Then, if the audio data output by the running music APP in the mobile phone is 4 channels of audio data, the mobile phone may require that the 4 channels of audio data are merged into two channels of audio data in the audio playing policy. Then, the mobile phone can send the combined dual-channel audio data to the television for playing, so that the mobile phone can flexibly switch the audio to the slave device for playing according to the audio capability of the current slave device, and better audio use experience is provided for a user.

For another example, the user may select a plurality of electronic devices as the slave devices 102 of the mobile phone in the detail menu 301 to perform audio switching. For example, if the user is detected to select headphones and a television in the above-described details menu 301, the handset may obtain the audio capability parameters of the headphones and the television, respectively. Because the earphone has better portability and the television has better display experience, the mobile phone can send the audio data corresponding to the video picture to the television for playing when audio switching is carried out, and send the system audio data of the ring tone, the notification sound and the like of the mobile phone to the earphone for playing. Therefore, the mobile phone can distribute the audio data to be played on the plurality of slave devices according to different audio capabilities of the plurality of slave devices, and provides better audio use experience for users.

It can be seen that in the embodiment of the present application, the master device 101 (e.g., the above-mentioned mobile phone) may output audio data to one or more slave devices 102 for audio switching in response to a selection of a user, so as to implement a distributed audio capability across the devices. Moreover, when performing audio switching, since the slave device 102 may also have a certain audio processing capability, the master device 101 may obtain the audio capability parameter of the slave device 102, and dynamically determine the audio playing policy when performing audio switching according to the audio capability parameter of the slave device 102. In this way, the master device 101 may output customized audio data to the slave device 102 according to the actual audio capability of the slave device 102, so that the slave device 102 may also process the audio data sent by the master device according to its own audio capability, thereby flexibly distributing the audio function of the master device 101 on the slave device 102 to implement, and providing a better audio using experience for a user.

The specific details of the audio switching between the master device and the slave device will be described in detail in the following embodiments, and therefore, the details are not described herein.

Exemplarily, a mobile phone is taken as an example of the master device 201 in the audio 200, and fig. 5 shows a schematic structural diagram of the mobile phone.

The handset may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, and the like.

It is to be understood that the illustrated structure of the embodiment of the present invention is not to be specifically limited to a mobile phone. In other embodiments of the present application, the handset may include more or fewer components than shown, or combine certain components, or split certain components, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units, such as: the processor 110 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors.

A memory may also be provided in processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Avoiding repeated accesses reduces the latency of the processor 110, thereby increasing the efficiency of the system.

The wireless communication function of the mobile phone can be realized by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, the baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals.

The mobile communication module 150 may provide a solution including wireless communication of 2G/3G/4G/5G, etc. applied to a mobile phone. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the same device as at least some of the modules of the processor 110.

The wireless communication module 160 may provide solutions for wireless communication applied to a mobile phone, including Wireless Local Area Networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), and the like.

In some embodiments, the handset antenna 1 is coupled to the mobile communication module 150 and the handset antenna 2 is coupled to the wireless communication module 160 so that the handset can communicate with the network and other devices via wireless communication techniques.

The mobile phone realizes the display function through the GPU, the display screen 194, the application processor and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, video, and the like. The display screen 194 includes a display panel. The display panel may adopt a Liquid Crystal Display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode, AMOLED), a flexible light-emitting diode (FLED), a miniature, a Micro-oeld, a quantum dot light-emitting diode (QLED), and the like. In some embodiments, the cell phone may include 1 or N display screens 194, with N being a positive integer greater than 1.

The mobile phone can realize shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, the application processor and the like.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing element converts the optical signal into an electrical signal, which is then passed to the ISP where it is converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signal in standard RGB, YUV and other formats. In some embodiments, the handset may include 1 or N cameras 193, N being a positive integer greater than 1.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to extend the storage capability of the mobile phone. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music, video, etc. are saved in an external memory card.

The internal memory 121 may be used to store computer-executable program code, which includes instructions. The processor 110 executes various functional applications of the cellular phone and data processing by executing instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The data storage area can store data (such as audio data, a phone book and the like) created in the use process of the mobile phone. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), and the like.

The mobile phone can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playing, recording, etc.

The audio module 170 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be disposed in the processor 110, or some functional modules of the audio module 170 may be disposed in the processor 110.

The speaker 170A, also called a "horn", is used to convert the audio electrical signal into an acoustic signal. The handset can listen to music through the speaker 170A or listen to a hands-free conversation.

The receiver 170B, also called "earpiece", is used to convert the electrical audio signal into an acoustic signal. When the mobile phone receives a call or voice information, the receiver 170B can be close to the ear to receive voice.

The microphone 170C, also referred to as a "microphone," is used to convert sound signals into electrical signals. When making a call or transmitting voice information, the user can input a voice signal to the microphone 170C by speaking the user's mouth near the microphone 170C. The handset may be provided with at least one microphone 170C. In other embodiments, the mobile phone may be provided with two microphones 170C to achieve the noise reduction function in addition to collecting the sound signal. In other embodiments, the mobile phone may further include three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions.

The headphone interface 170D is used to connect a wired headphone. The headset interface 170D may be the USB interface 130, or may be a 3.5mm open mobile electronic device platform (OMTP) standard interface, a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

Certainly, the mobile phone may further include a charging management module, a power management module, a battery, a key, an indicator, 1 or more SIM card interfaces, and the like, which is not limited in this embodiment of the present application.

The software system of the mobile phone can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture or a cloud architecture. The embodiment of the application takes an Android system with a layered architecture as an example, and exemplifies a software structure of a mobile phone. Of course, in other operating systems, as long as the functions implemented by the respective functional modules are similar to the embodiments of the present application.

Fig. 6 is a block diagram of a software structure of a mobile phone according to an embodiment of the present application.

The layered architecture divides the software into several layers, each layer having a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into five layers, which are an application layer, an application framework layer, an Android runtime (Android runtime) and system library (HAL) layer, and a kernel layer from top to bottom.

The application layer may include a series of application packages.

As shown in fig. 6, applications such as calls, memo, browser, contacts, camera, gallery, calendar, map, bluetooth, music, video, and short message may be installed in the application layer.

It should be noted that the applications installed in the application layer may have an audio function. For example, the music APP may play audio, and the camera APP may output shutter sound preset by the system when taking a picture. For another example, the video APP may output audio corresponding to a video picture while playing a video. For another example, the map APP may output a navigation voice after the navigation function is started. In the embodiment of the present application, an application having an audio function may be referred to as an audio APP.

The application framework layer provides an Application Programming Interface (API) and a programming framework for the application program of the application layer. The application framework layer includes a number of predefined functions.

As shown in fig. 6, for example, a music APP is taken as an audio APP, and an audio framework 601 for implementing an audio function for the audio APP is provided in the Android system. In the application framework layer, the audio architecture 601 may include an audio manager (AudioManager), an audio player, and an audio processor.

Illustratively, the audio player may be an AudioTrack or MediaPlayer, etc. player. The audio processor may be an AudioFlinger or the like. When the audio APP needs to play audio, the audio player can be called, and corresponding audio data can be input into the audio player. For example, the audio APP may input original audio data to an audio player, and the audio player performs parsing, decapsulating, or decoding on the original audio data to obtain PCM (pulse code modulation) data of one frame. Alternatively, the audio APP may output the PCM data directly to the audio player. Further, the audio player may send the audio data to an audio processor for audio processing.

Specifically, the audio processor may perform audio processing on the audio data according to an audio playing policy output by the audio manager. Illustratively, the audio manager may include an audio service (AudioService) and an audio policy module (AudioPolicy). The audio APP can call the audio service to inform the audio policy module to set a specific audio playing policy during audio data input and output during operation. Such as volume level, sampling rate of audio data, sound effect settings, mix settings, etc. The audio strategy module can output a corresponding audio playing strategy to the audio processor in real time in the audio playing process, so that the audio processor can perform processing such as audio mixing, resampling, sound effect setting and the like on audio data output by the audio player according to the latest audio playing strategy, and the processed audio data are the audio data to be played.

Still as shown in fig. 6, a HAL (hardware abstraction layer) may also be included between the application framework layer and the kernel layer of the Android system. The HAL is responsible for interacting with each hardware device of the mobile phone, on one hand, the HAL hides the implementation details of each hardware device, and on the other hand, the HAL can provide an interface for calling each hardware device for the Android system. HALs are provided to correspond to different handset hardware devices, such as Audio HALs, Camera HALs, Wi-Fi HALs, and the like.

Among other things, the Audio HAL may also be part of the Audio framework 601 described above. The Audio processor (e.g., AudioFlinger) may directly invoke the Audio HAL, send the processed Audio data to the Audio HAL, and the Audio HAL sends the Audio data to a corresponding Audio output device (e.g., speaker, headphone, etc.) for playback.

Illustratively, the Audio HAL may be further classified into a Primary HAL, A2dp HAL, etc., according to the Audio output device. The AudioFlinger may output audio data to a speaker (speaker) of the handset using a Primary HAL, or the AudioFlinger may output audio data to a bluetooth headset connected to the handset using A2dp HAL. Taking the example of A2dp HAL, when the mobile phone is powered on, the bluetooth APP (also referred to as bluetooth service) in the application layer may create A2dp HAL at the HAL, where the A2dp HAL may run as an idle process, and the A2dp HAL is not connected to a specific bluetooth device.

Subsequently, when the mobile phone is connected to a certain bluetooth device (e.g., a first bluetooth headset), the bluetooth APP may register information such as an identifier and a bluetooth address of the first bluetooth headset in the audio manager, and send related hardware parameters of the first bluetooth headset (e.g., a sampling rate of the first bluetooth headset) to the A2dp HAL, so that the A2dp HAL may support data transceiving with the first bluetooth headset. At this time, A2dp HAL is fixed and unchanged during the process of keeping the mobile phone connected with the first bluetooth headset. When the mobile phone is disconnected from the first bluetooth headset, the bluetooth APP may clear the information related to the first bluetooth headset in the audio manager, and restore A2dp HAL to an idle process to run in the HAL.

When the mobile phone is connected with another bluetooth device (for example, a second bluetooth device, which may be the same as or different from the first bluetooth device), similarly to the above process, the bluetooth APP may register information such as an identifier and a bluetooth address of the second bluetooth headset in the audio manager, and send related hardware parameters of the second bluetooth headset to the A2dp HAL, so that the A2dp HAL may support data transceiving with the second bluetooth headset. When the mobile phone is disconnected from the second bluetooth headset, the bluetooth APP may clear the information about the second bluetooth headset in the audio manager, and restore A2dp HAL to an idle process running in HAL.

In addition, the application framework layer may further include a window manager, a content provider, a view system, a resource manager, a notification manager, and the like, which is not limited in this embodiment of the present application.

For example, the window manager described above is used to manage window programs. The window manager can obtain the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like. The content provider is used to store and retrieve data and make it accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system described above can be used to build a display interface for an application. Each display interface may be comprised of one or more controls. Generally, a control may include an interface element such as an icon, button, menu, tab, text box, dialog box, status bar, navigation bar, Widget, and the like. The resource manager provides various resources, such as localized strings, icons, pictures, layout files, video files, and the like, to the application. The notification manager enables the application program to display notification information in the status bar, can be used for conveying notification type messages, can automatically disappear after a short time of stay, and does not need user interaction. Such as a notification manager used to inform download completion, message alerts, etc. The notification manager may also be a notification that appears in the form of a chart or scroll bar text at the top status bar of the system, such as a notification of a background running application, or a notification that appears on the screen in the form of a dialog window. For example, to prompt text messages in the status bar, to emit a prompt tone, to vibrate, to flash an indicator light, etc.

As shown in fig. 6, the Android runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing an Android system.

The core library comprises two parts: one part is a function which needs to be called by java language, and the other part is a core library of android.

The application layer and the application framework layer run in a virtual machine. And executing java files of the application program layer and the application program framework layer into a binary file by the virtual machine. The virtual machine is used for performing the functions of object life cycle management, stack management, thread management, safety and exception management, garbage collection and the like.

The system library may include a plurality of functional modules. For example: surface managers (surface managers), Media Libraries (Media Libraries), three-dimensional graphics processing Libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), and the like.

Wherein the surface manager is used for managing the display subsystem and providing the fusion of the 2D and 3D layers for a plurality of application programs. The media library supports a variety of commonly used audio, video format playback and recording, and still image files, among others. The media library may support a variety of audio-video encoding formats, such as MPEG4, h.264, MP3, AAC, AMR, JPG, PNG, and the like. The three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like. The 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is located below the HAL and is the layer between hardware and software. The kernel layer at least comprises a display driver, a camera driver, an audio driver, a sensor driver and the like, and the embodiment of the application does not limit the display driver, the camera driver, the audio driver, the sensor driver and the like.

Based on the audio architecture 601 shown in fig. 6, in the embodiment of the present application, as shown in fig. 7, a device virtualization (device virtualization) APP for implementing an audio switching function, which may be referred to as a DV APP in the following, may be installed in an application layer of a mobile phone. The DV APP can be used as system application to reside in a mobile phone to run. Or, the audio switching function realized by the DV APP can be resident in the mobile phone to run in the form of system service. When the mobile phone needs to switch the audio data generated by the audio APP to a certain slave device for playing, or needs to record audio by using the recording function of a certain slave device, the DV APP can trigger the mobile phone to acquire the audio capability of the slave device, so that a corresponding virtual device is created in the mobile phone according to the audio capability of the slave device, and thus, the mobile phone can switch the audio function of the mobile phone to the slave device by controlling the virtual device.

Illustratively, taking a music APP as an audio APP, as shown in (a) of fig. 8, the mobile phone may display a switch button 801 for audio switching when running the music APP. Currently, the mobile phone outputs audio data by using its own speaker, and if the user wishes to continue playing the audio data by using other audio output devices, the user can click the switching button 801 to trigger the audio switching function. After the cell phone detects that the user clicks the switch button 801, the DV APP may trigger the cell phone to start detecting one or more nearby candidate devices that may be subject to audio switching. For example, as shown in fig. 8 (b), the handset may detect a candidate device having an audio output function and located in the same Wi-Fi network as the handset, and display the detected candidate device in the menu 802. Taking the example of three candidate devices including a television 803, a sound box 804 and a headset 805 in the menu 802, the user can select one of the three candidate devices as a slave device for continuously playing audio data in the music APP.

Taking the example where the user selects the television 803 in the menu 802, after detecting that the user clicks on the candidate device of the television 803, the handset may send an access request to the television 803. At this time, as shown in (a) in fig. 9, the television 803 may display a dialog 901 requesting the user to allow the mobile phone to access the television 803. If the user allows the handset to access the television 803, the "allow this time" or "always allow" button in the dialog 901 may be selected, at which point the television 803 may send a response to the handset to allow access. In turn, the DV APP may trigger the cell phone to establish a network connection with the television 803, such as a Wi-Fi P2P connection or the like.

It should be noted that the network connection established between the handset and the slave device (e.g., the television 803) may specifically refer to a network connection (i.e., a service connection) of a service channel. For example, the cell phone may have established a connection with the television 803 via a Wi-Fi network before the user selects the television 803 as a slave device to the cell phone, where the connection is a data channel connection (i.e., a data connection). When the user selects the television 803 as the slave device of the mobile phone, the mobile phone and the television 803 establish a service connection on the basis of the established data connection. For example, the network connection may be a P2P connection based on a TCP (transmission control protocol) or a UDP (user datagram protocol), which is not limited in this embodiment.

If the mobile phone is currently in network connection with the television 803 for the first time, as shown in fig. 9 (b), the television 803 may display a string of randomly generated connection codes 902 as authentication information to perform identity authentication with the mobile phone, and at the same time, the television 803 may send the connection codes 902 to the mobile phone. The DV APP asks the user to enter a connection code 902 displayed on the television 803 in the display of the handset. If the DV APP detects that the connection code input by the user on the mobile phone is the same as the connection code 902 sent by the television 803 to the mobile phone, which indicates that the identity authentication between the mobile phone and the television 803 is passed, the DV APP can trigger the mobile phone to establish network connection with the television 803. Of course, if the identity authentication between the mobile phone and the tv 803 is once completed, the DV APP can directly establish a network connection with the tv 803 after detecting that the user selects the tv 803, and does not need to perform the identity authentication again.

In this embodiment, after the mobile phone establishes a network connection with the television 803, the mobile phone and the television 803 may communicate through the network connection. For example, a DV APP in a cell phone may obtain audio capability parameters of the television 803 from the television 803 based on an established network connection, with the television 803 as a slave to the cell phone. For example, the audio capability parameters may include one or more of audio playback delay, sound effect parameters, audio sampling rate, or number of sound channels of the television 803, which may reflect the specific output capabilities supported by the television 803. Further, as also shown in fig. 7, the DV APP may call a preset interface of the HAL, and input the acquired audio capability parameters into the preset interface, thereby creating the HAL corresponding to the television 803. In the embodiment of the present application, the HAL created by the DV APP to switch the Audio data to the slave device may be referred to as a DMSDP (Distributed Mobile Sensing Development Platform) Audio HAL. Different from the traditional Audio HAL, the DMSDP Audio HAL does not correspond to the actual hardware device of the mobile phone, and the DMSDP Audio HAL can transmit the received Audio data to the slave device of the mobile phone through the established network connection, so as to realize the Audio switching function.

In some embodiments, the television 803 may also send display capability parameters (e.g., screen resolution, codec algorithm of display data, etc.) indicating its own display capability to the mobile phone's DV APP over the established network connection; alternatively, the television 803 (i.e., the slave device) may also send camera capability parameters (e.g., number of cameras, image processing algorithms, etc.) indicating its own camera capabilities to the handset's DV APP over the established network connection. Of course, if the slave device (e.g., the television 803 described above) also has other capabilities (e.g., printing capabilities, etc.), the slave device may also send the relevant capability parameters to the DV APP of the cell phone. Then, the DV APP of the mobile phone may also input these capability parameters and the above-mentioned audio capability parameters into a preset interface to create a corresponding HAL, where the created HAL integrates not only the audio capability of the slave device, but also other capabilities of the slave device. In the following embodiments, the HAL created by the DV APP according to the capability parameters of the slave device may be referred to as DMSDP HAL, that is, the DMSDP Audio HAL may be a functional module in the DMSDP HAL for implementing Audio functions, and in some embodiments, the DMSDP Audio HAL may also be referred to as DMSDP Audio HAL. The mobile phone can implement service functions in various distributed scenes such as audio, display or camera through the DMSDP HAL and the slave device, which is not limited in this embodiment of the present application.

In addition, in the embodiment of the present application, in addition to creating a corresponding DMSDP Audio HAL for a slave device of a mobile phone by a DV APP, the DV APP may call an AudioManager to register the Audio capability of the slave device in AudioPolicy. Illustratively, the DV APP may send the identity of television 803 and the audio capability parameters of television 803 to AudioPolicy for storage. AudioPolicy may determine a corresponding audio policy based on audio capability parameters of television 803 and output the determined audio policy to an audio processor (e.g., AudioFlinger). Thus, the audioFlinger can correspondingly process the Audio data of the music APP output by the Audio player (e.g., audioTrack) according to the Audio policy determined by the audioPolicy, and send the processed Audio data to the television 803 for playing through the DMSDP Audio HAL, so that the Audio on the mobile phone is customized and switched to the slave device (i.e., the television 803) for playing, and the cross-device distributed Audio capability is realized.

The audio capability parameters of the television 803 may include a playing parameter and a recording parameter of the television 803, where the playing parameter is used to reflect the audio playing capability of the television 803, and the recording parameter is used to reflect the audio recording capability of the television 803. For example, the television 803 may encapsulate the playback parameters in a first field and encapsulate the recording parameters in a second field for transmission to the mobile phone. The AudioPolicy of the mobile phone can determine the corresponding audio playing strategy after extracting the playing parameters from the first field, and the AudioPolicy can determine the corresponding audio recording strategy after extracting the recording parameters from the second field. Of course, the first field or the second field may be null, which indicates that the tv 803 does not have the audio playing capability or the audio recording capability.

In addition, the DMSDP Audio HAL created by DV APP at the HAL is dynamically updateable. When the audio capability of the tv 803 changes, for example, if the audio playing delay of the tv 803 changes, or the user manually changes the sound effect parameters of the tv 803, the tv 803 may dynamically send the latest audio capability parameters to the handset. Furthermore, the DV APP of the handset may update the DMSDP Audio HAL according to the latest Audio capability parameters of the television 803, so that the DMSDP Audio HAL matches the Audio capability of the television 803. Meanwhile, the DV APP of the handset may register the latest audio capability parameter of the television 803 in AudioPolicy, so that AudioPolicy may update the current audio policy according to the latest audio capability parameter.

In the above embodiment, the number of the slave devices of the mobile phone is 1 for example when the audio is switched, and in the embodiment of the present application, the mobile phone may further switch its own audio data to a plurality of slave devices for playing. For example, the user may select a plurality of devices among the candidate devices searched by the cell phone as slave devices of the cell phone. Taking the example that the slave device of the mobile phone includes the sound box 804 and the television 803, similar to the above interaction process of the mobile phone and the television 803, the mobile phone may respectively establish a network connection with a plurality of slave devices and obtain the Audio capability parameter of each slave device, thereby creating a DMSDP Audio HAL corresponding to each slave device. For example, as shown in fig. 10, the DV APP may create a DMSDP Audio HAL1 at the HAL in accordance with Audio capability parameter 1 of the loudspeaker 804, and the DV APP may create a DMSDP Audio HAL2 at the HAL in accordance with Audio capability parameter 2 of the television 803. Furthermore, after the audio capabilities of the sound box 804 and the television 803 are registered in AudioPolicy by the mobile phone, AudioPolicy may customize a corresponding audio policy according to the audio capability parameter of each slave device, so that the AudioFlinger may process audio data according to the audio policy determined by AudioPolicy. Subsequently, the audioFlinger can send the first Audio data needing to be played by the sound box 804 to the sound box 804 through the DMSDP Audio HAL1, and send the second Audio data needing to be played by the television 803 to the television 803 through the DMSDP Audio HAL2, so that the cross-device distributed Audio capability is realized. Wherein the first audio data and the second audio data may be the same or different.

It can be seen that, in the audio architecture provided in the embodiment of the present application, the DV APP can respond to the audio switching requirement of the user and establish network connection with the corresponding slave device, thereby obtaining the audio capability parameter of the slave device. Further, the DV APP may create a corresponding DMSDP Audio HAL at the HAL in accordance with the Audio capability parameters and register the Audio capabilities of the slave device in the AudioPolicy, enabling the AudioPolicy to output Audio policies to the AudioFlinger that match the Audio capabilities of the slave device, customized according to the Audio capabilities of the slave device. Subsequently, after the audioFlinger processes the Audio data provided by the mobile phone according to the Audio policy, the created DMSDP Audio HAL can be called to send the processed Audio data to the slave device, so that the slave device can play the Audio data from the mobile phone (namely, the master device) matched with the Audio capability of the slave device. Therefore, the master device can switch the audio function of the master device to one or more other slave devices according to the audio capability of the slave devices, the audio processing capability of the slave devices is fully utilized, and the master device and the slave devices can cooperate to complete the audio switching process more efficiently and flexibly, so that better audio using experience is provided for users.

Still taking a mobile phone as an example of the main device, an audio control method provided by the embodiment of the present application will be described below with reference to specific audio capability parameters.

Still taking the music APP as the audio APP for example, when the mobile phone runs the music APP and plays song a with its own speaker, as shown in fig. 11, the music APP can call an audio player (e.g., AudioTrack) to send audio data 1 of song a to an audio processor (e.g., AudioFlinger). The audio flinger performs processing such as sampling, sound mixing, setting of sound effects, or channel conversion on the audio data 1. At this time, the mobile phone is not connected to other audio output devices, so as shown by a dotted arrow in fig. 11, the AudioFlinger can call the Primary HAL in the HAL, and output the processed audio data 1 to a speaker of the mobile phone for playing, thereby implementing a local audio playing function of the mobile phone.

In addition, the music APP can call AudioManager at runtime to register the relevant information of the audio data 1 being played in AudioPolicy. For example, the music APP may buffer information on the current sound effect mode, the number of channels, the sampling rate, the volume level, and the like of the audio data 1 in AudioPolicy.

In the embodiment of the present application, as also shown in (a) in fig. 8, the cell phone may display a switch button 801 for audio switching while running the music APP. Meanwhile, the mobile phone can run DV APP for realizing the audio switching function in the background. After detecting that the user clicks the switching button 801, the music APP can communicate with the DV APP to trigger the DV APP to search for one or more candidate devices which can be subjected to audio switching nearby. As shown in fig. 8 (b), the DV APP may display three candidate devices, a television 803, a speaker 804, and a headset 805, in the same Wi-Fi network as the handset, in a menu 802.

For example, when the user selects the sound box 804 as the slave device of the mobile phone, after detecting that the user selects the sound box 804 as the candidate device, the DV APP may establish a network connection with the sound box 804 using the sound box 804 as the slave device of the mobile phone. Alternatively, after detecting that the user clicks the switching button 801, the music APP may trigger a preset communication APP (e.g., a hyperlink APP) to search for one or more candidate devices that are nearby and may perform audio switching. When detecting that a user selects a certain candidate device (e.g., the sound box 804), the DV APP may be triggered to establish a network connection with the sound box 804 using the sound box 804 as a slave device of the mobile phone. For example, the network connection may be a TCP (transmission control protocol) or UDP (user datagram protocol) based P2P connection.

It should be noted that, if the network connection is established between the mobile phone and the sound box 804 for the first time, the mobile phone and the sound box 804 also need to perform identity authentication. The specific process of the identity authentication may refer to the description of the identity authentication between the mobile phone and the television 803 in (a) and (b) in fig. 9, and is not described herein again.

After the network connection between the mobile phone and the sound box 804 is established, the DV APP may use the sound box 804 as a slave device of the mobile phone to obtain the audio capability parameters of the sound box 804 from the sound box 804. As shown in fig. 12, an audio architecture in the sound box 804 is similar to that in the mobile phone, and an agent APP for implementing an audio switching function may be installed in the sound box 804. After establishing network connection with the sound box 804, the mobile phone can send an acquisition request of the audio capability parameters to the agent APP of the sound box 804, and then, in response to the acquisition request, the agent APP of the sound box 804 can call its AudioManager to acquire the audio capability parameters of the sound box 804 itself from AudioPolicy or AudioFlinger of the sound box 804. In this way, agent APP of speaker 804 may send audio capability parameters of speaker 804 to DV APP of the handset. Or, also can set up agent APP in the cell-phone, the agent APP of audio amplifier 804 can send the audio frequency ability parameter of audio amplifier 804 to the agent APP of cell-phone, and then sends the audio frequency ability parameter of audio amplifier 804 to the DV APP by the agent APP of cell-phone to alleviate DV APP's operating load.

Still as shown in fig. 11, after the DV APP in the mobile phone acquires the Audio capability parameter of the sound box 804, a DMSDP Audio HAL 1101 corresponding to the sound box 804 may be created according to the Audio capability parameter. Also, DV APP may call AudioManager to register the audio capabilities of speaker 804 in AudioPolicy. For example, the DV APP may send the identification of the loudspeaker 804 and the audio capability parameters of the loudspeaker 804 to Audio policy. In this way, AudioPolicy may set an audio playback strategy when audio is switched to loudspeaker 804 according to the audio capability parameters of loudspeaker 804.

The audio capability parameters of the speaker 804 may be used to reflect the specific audio input or output capabilities supported by the speaker 804. In this embodiment, the audio capability parameters of the slave device (e.g., the speaker 804) may include two types of parameters, one type is an audio capability parameter (which may be referred to as a hardware capability parameter) mainly related to the hardware capability of the speaker 804, such as the sampling rate supported by the speaker 804, the number of channels, and so on, and such parameters may reflect the hardware capability of the speaker 804 when inputting or outputting audio; another type is audio capability parameters (which may be referred to as software capability parameters) that are primarily related to the software capabilities of the enclosure 804, such as sound effect modes supported by the enclosure 804, codec capabilities, etc., which may reflect the software capabilities of the enclosure 804 when inputting or outputting audio. Thus, AudioPolicy may set a corresponding audio playing policy in combination with the hardware capability and the software capability of the sound box 804 when playing audio.

Illustratively, the audio capability parameters (i.e., hardware capability parameters) associated with the hardware capabilities of the enclosure 804 are primarily determined by the hardware characteristics of the enclosure 804, e.g., the hardware type, model interface type, etc., of the processor, speaker, or microphone in the enclosure 804. When the slave device of the mobile phone is not changed, for example, when the slave device of the mobile phone is the sound box 804, the hardware capability parameter reported by the sound box 804 in the audio capability parameter is generally fixed and unchanged. Of course, in some extreme cases, for example, when some hardware in the sound box 804 is faulty or damaged, the hardware capability parameter in the audio capability parameters reported by the sound box 804 to the mobile phone may also be changed.

Accordingly, the audio capability parameters (i.e., software capability parameters) associated with the software capabilities of the loudspeaker 804 are primarily determined by the software characteristics of the loudspeaker 804, such as the sound effect algorithm, echo cancellation algorithm, encoding or decoding algorithm, etc. used by the loudspeaker 804. When the slave device of the mobile phone is not changed, for example, when the slave device of the mobile phone is the sound box 804, the software capability parameter reported by the sound box 804 in the audio capability parameter may be dynamically updated. For example, after the user turns on the sound effect switch of the sound box 804, the sound box 804 may report that the sound effect mode of the sound box 804 is in an on state in the audio capability parameter. When the user turns off the sound effect switch of the sound box 804, the sound box 804 may update its audio capability parameters, and report that the sound effect mode of the sound box 804 is in an off state in the audio capability parameters.

It should be noted that, when any device implements an audio input or output function, hardware and software of the device need to support each other, so that when the software capability parameter and the hardware capability parameter are divided, it is impossible to simply define that the software capability parameter is completely determined by the software characteristic of the device, and the hardware capability parameter is completely determined by the hardware characteristic of the device. For example, the audio capability parameter reported by the sound box 804 may include a parameter of audio playing delay, where the audio playing delay is related to software factors such as a processing algorithm of the sound box 804 on audio data, a network environment, and the like, and is also related to a hardware processing capability of the sound box 804. In the embodiment of the present application, since the audio playing delay reported by the sound box 804 can be dynamically updated, the audio playing delay can be used as a software capability parameter. Of course, those skilled in the art may also divide the software capability parameter and the hardware capability parameter according to practical application scenarios or practical experience, which is not limited in this application.

In some embodiments, the audio capability parameters of the loudspeaker 804 may include sound effects capabilities (which may be referred to as sound effects parameters) supported by the loudspeaker 804. For example, the speaker 804 supports a subwoofer effect, a 3D surround effect, a dolby effect, a histen effect, and the like.

Generally, the sound effect mode of the sound box 804 turned on at a certain time is fixed. For example, the sound box 804 may default to turning on the dolby sound effect after being turned on. If the user wishes to modify the sound effects of the audio box 804 when playing audio, the user can manually set the current sound effect mode on the audio box 804. For example, the sound box 804 may be set from the sound effect mode of the dolby sound effect to the subwoofer sound effect. Then, when the sound box 804 reports the audio capability parameter to the DV APP of the mobile phone, the sound effect mode supported by the sound box 804 and the currently turned on sound effect mode may be added as sound effect parameters to the audio capability parameter. Of course, if the speaker 804 does not have the audio mode enabled, the speaker 804 may also indicate in the audio capability parameter that the speaker 804 does not have the audio mode enabled.

Illustratively, when the handset runs the music APP, the music APP may call AudioTrack to send audio data 1 to AudioFlinger. The AudioFlinger may communicate with AudioPolicy before processing audio data 1, requesting AudioPolicy for the audio playback strategy that this time processes audio data 1. AudioPolicy, in response to the AudioFlinger's request, may query the audio capability parameters of loudspeaker 804 whether loudspeaker 804 currently has the sound effect mode turned on. If the audio mode is turned on, AudioPolicy may set no audio processing to audio data 1 in the audio playback policy. Thus, after the AudioPolicy outputs the audio playback policy to the AudioFlinger, the AudioFlinger does not need to perform sound effect processing on the audio data 1. After the audioFlinger calls the DMSDP Audio HAL 1101 to send the Audio data 1 to the sound box 804, the sound box 804 can perform corresponding sound effect processing on the Audio data 1 and then play the Audio data.

If the sound box 804 does not turn on the sound effect mode, AudioPolicy may set no sound effect processing on the audio data 1 in the audio playing policy. Alternatively, if the audio enclosure 804 does not have the audio mode turned on, in order to improve the playing effect of the audio on the audio enclosure 804, AudioPolicy may set audio processing on the audio data 1 in the audio playing policy. For example, the audio data 1 is subjected to subwoofer processing. Thus, after AudioPolicy outputs the Audio playing policy to AudioFlinger, AudioFlinger may perform subwoofer processing on Audio data 1 in response to the Audio playing policy, and send the processed Audio data 1 to sound box 804 through DMSDP Audio HAL 1101. At this time, although the sound box 804 does not turn on any sound effect, the sound box 804 may present a sound effect of heavy bass to the user when playing the audio from the mobile phone.

Thus, the audio play strategy set by the audio policy according to the audio parameters of the sound box 804 can avoid the sound effect superposition phenomenon caused by the sound effect processing of the audio data by both the mobile phone (i.e. the master device) and the sound box 804 (i.e. the slave device), and also can avoid the sound effect deletion phenomenon caused by the sound effect processing of the audio data by both the mobile phone (i.e. the master device) and the sound box 804 (i.e. the slave device), thereby improving the audio use experience of the user.

In some embodiments, AudioPolicy may also determine audio playback strategies related to sound effects in conjunction with the device type of the slave device. For example, the current slave device of the mobile phone is a wearable watch, and after the mobile phone establishes a network connection with the wearable watch, the identifier of the wearable watch and the audio capability parameter of the wearable watch can be acquired. The mobile phone can determine that the device type of the current slave device is the wearable device through the identification of the wearable watch. If the Dolby sound effect is indicated to be turned on by the wearable watch in the audio capability parameter of the wearable watch, the Audio policy of the mobile phone can set the Dolby sound effect to be used for processing the audio data needing to be played in the audio playing strategy. Subsequently, the AudioFlinger may add dolby sound to the Audio data in response to the Audio playing policy, and send the processed Audio data to the wearable watch through the corresponding DMSDP Audio HAL. Like this, wearable wrist-watch need not to carry out the audio processing again when playing this audio data, not only can reduce the operation load of the lower equipment of this kind of audio processing ability of wearable wrist-watch, can guarantee simultaneously that wearable wrist-watch can demonstrate better dolby audio when playing audio data to improve user's audio frequency and use experience.

In other embodiments, when the music APP runs in the mobile phone, the mobile phone may also call the AudioManager to provide the music APP with the sound effect modes supported by the mobile phone and the sound effect modes supported by the slave device (e.g., the speaker 804), and the music APP selects which sound effect mode is specifically used to play the current audio data. For example, the music APP may show the sound effect modes supported by the mobile phone and the sound box 804 to the user in a list form, and the user manually selects a desired sound effect mode. Then, if the sound effect mode selected by the user is a sound effect mode supported by both the mobile phone and the sound box 804, the AudioPolicy may set that the mobile phone does not perform sound effect processing on the audio data in the audio playing policy, so that the sound box 804 performs corresponding sound effect processing on the audio data sent by the mobile phone and plays the audio data. If the sound effect mode selected by the user is the sound effect mode supported by the mobile phone, the AudioPolicy can set the mobile phone to perform corresponding sound effect processing on the audio data in the audio playing strategy, so that the sound box 804 can still present the sound effect required by the user after playing the audio data. Correspondingly, if the sound effect mode selected by the user is the sound effect mode supported by the sound box 804, the AudioPolicy may set the mobile phone to perform no sound effect processing on the audio data in the audio playing policy, so that the sound box 804 performs corresponding sound effect processing on the audio data sent by the mobile phone and plays the audio data.

In other embodiments, the audio capability parameters of loudspeaker 804 may include the sound production capabilities (which may be referred to as audio quality parameters) supported by loudspeaker 804, such as the sampling rate supported by loudspeaker 804 (i.e., the number of samples that loudspeaker 804 extracts from a continuous audio signal and composes a discrete signal per second), the number of sound channels (i.e., the number of sound channels), and so forth.

Illustratively, the maximum sampling rate that is obtained by AudioPolicy and supported by the sound box 804 in the audio capability parameters is 96KHz, and further, AudioPolicy may query the sampling rate of the audio data 1 received by the current AudioFlinger. If the sampling rate of the audio data 1 received by the current AudioFlinger is greater than 96KHz, the AudioPolicy may set resampling on the audio data 1 in the audio playing policy, where the resampling sampling rate is 96 KHz. Thus, after AudioPolicy outputs the audio playing strategy to AudioFlinger, AudioFlinger may resample audio data 1 at a sampling rate of 96KHz, so that the resampled audio data 1 matches the sound playing capability of the sound box 804.

For another example, AudioPolicy may acquire that the number of sound channels in the audio capability parameter is 2 channels, and further, AudioPolicy may query whether the number of sound channels of audio data 1 received by current AudioFlinger is 2 channels. If not, AudioPolicy may set the number of sound channels to 2 channels in the audio playback policy. Thus, after the AudioPolicy outputs the audio playing policy to the AudioFlinger, the AudioFlinger may mix the audio data 1 according to the number of the 2-channel audio channels, so that the audio data 1 after mixing is matched with the audio playing capability of the sound box 804.

In other embodiments, some audio APPs have low latency playback requirements when playing audio. For example, when the karaoke application records the singing voice of a user, accompanying audio needs to be output in time to ensure the recording experience of the user. For another example, when the video application outputs the display picture, the corresponding audio needs to be output in time to ensure the viewing experience of the user. These audio APPs that require low latency require the audio output device to process audio as fast as possible when outputting audio.

For example, the audio capability parameter obtained by the handset from the speaker 804 may also include an audio playing delay (latency) of the speaker 804. The audio playing delay may specifically refer to a time required for the sound box 804 to process one frame of audio data. Generally, audio data is generated in the form of audio streams, and for convenience of audio algorithm processing or transmission, continuous audio data may be divided into audio data of one frame and one frame according to a certain time unit (also referred to as a certain time duration). Illustratively, it is possible to take 2.5ms to 60ms as the unit of audio data as a frame of audio data, and one frame of audio data is the minimum granularity when the device performs audio processing.

AudioPolicy can determine whether the sound box 804 supports the low-delay audio processing capability according to the audio playing delay in the audio capability parameters. For example, it may be determined that loudspeaker 804 supports low-latency audio processing capabilities when the audio playback latency of loudspeaker 804 is less than or equal to 5ms (i.e., the time required for loudspeaker 804 to process a frame of audio data is less than or equal to 5 ms). AudioPolicy may then set a low latency mode in the audio playback policy so that AudioFlinger may process audio data 1 in the low latency mode. Alternatively, the sound box 804 may indicate whether the sound box 804 supports the low-latency audio processing capability in the audio capability parameter, for example, if the field of the preset delay parameter in the audio capability parameter of the sound box 804 is 0, it indicates that the sound box 804 supports the low-latency audio processing capability. At this time, AudioPolicy may set a low latency mode in the audio playback strategy. Correspondingly, if the field of the preset delay parameter in the audio capability parameter is 1, it indicates that the sound box 804 does not support the low-delay audio processing capability. At this time, AudioPolicy may set a non-low latency mode in the audio playback strategy.

Low latency audio processing capabilities generally require the device to have a high processing capability for the audio data. For example, in the non-low latency mode, the AudioFlinger may set the duration of one frame of audio data to 10ms, and the AudioFlinger needs to buffer 10 frames of audio data each time to ensure continuous consistency of audio output. That is, the time granularity for the handset to process each frame of audio data is 10ms, and the one cycle for the handset to process audio data at a time is 100ms (10ms 10). In the low latency mode, the AudioFlinger may reduce the duration of one frame of audio data to 5ms, and the buffer of the mobile phone still needs to store 10 frames of audio data, so that, in the low latency mode, the time granularity for processing each frame of audio data by the mobile phone is 5ms, and one period for processing audio data by the mobile phone each time is 50ms (5ms × 10).

If a low-delay mode is set in the audio playing strategy output by AudioPolicy, the AudioFlinger may send audio data 1 to the sound box 804 for a duration of one frame of audio data in 5 ms. That is, the handset may set the duration of each frame of audio data to be small in the low latency mode compared to the non-low latency mode, for example, the duration of one frame of audio data is 5 ms. Subsequently, the speaker 804 may process each frame of audio data with a time granularity of 5ms according to the low latency mode, so as to reduce the processing time of each frame of audio data and improve the audio using experience of the user.

Illustratively, AudioPolicy may also query whether the currently running music APP has a low-latency playing requirement. If the music APP has a low-delay playing requirement and the sound box 804 supports a low-delay playing capability, the AudioPolicy may set a low-delay mode in the audio playing policy. If the music APP has a low-latency playing requirement but the sound box 804 does not support the low-latency playing capability, the AudioPolicy may reduce the processing flow of the audio data 1 as much as possible in the audio playing policy. For example, AudioPolicy may set no sound effect processing to the audio data 1. Therefore, the time delay of sending the audio data to the sound box 804 by the mobile phone can be reduced, and the low-time-delay playing requirement of the audio APP can be met as much as possible. Correspondingly, if the music APP has no low-delay playing requirement, the AudioPolicy may set a low-delay mode in the audio playing policy, or may set a non-low-delay mode in the audio playing policy, which is not limited in this embodiment of the present application.

In other embodiments, the audio capability parameters of enclosure 804 may include EQ (Equalizer) parameters of enclosure 804. The EQ parameters generally include three parameters, namely frequency (frequency), gain (gain) and quantization value (quantization), and the frequency is used for setting parameters of frequency points needing to be adjusted in audio data; the gain is used for adjusting the parameters of gain or attenuation of audio data of a certain frequency; quantize is used to set the parameters of the "width" of the band in which the gain or attenuation is performed. One or more frequency bands in the audio data can be gained or attenuated through the EQ parameters, so that the aim of adjusting the tone color is fulfilled.

For example, after AudioPolicy acquires the EQ parameters in the audio capability parameters, AudioPolicy may set the EQ parameters of audio data 1 to the acquired EQ parameters from sound box 804 in an audio playing policy. Thus, after AudioPolicy outputs the audio playing strategy to AudioFlinger, AudioFlinger can adjust the tone of audio data 1 according to the EQ parameter of sound box 804, so that the audio data 1 sent to the sound box by subsequent mobile phones matches with the audio playing capability of sound box 804.

In some embodiments, after the audio capability parameter of the sound box 804 is acquired by AudioPolicy, the audio capability parameter of the sound box 804 may be stored in the mobile phone. As shown in table 1, AudioPolicy may store the device name, identifier, and corresponding audio capability parameter of the slave device, which are acquired by the DV APP each time, in the mobile phone. The identifier may specifically be an identifier capable of uniquely identifying the device, such as an IMEI (international mobile equipment identity) or a mac (media Access control) address of the slave device. The audio capability parameters may include sound effect parameters, audio quality parameters, audio playback delay, EQ parameters, and the like. In this way, AudioPolicy may not only customize an audio playing policy matching with the audio capability of the sound box 804 according to the audio capability parameter of the sound box 804 in table 1, but also subsequently, after the mobile phone establishes a network connection with the mobile phone by using the sound box 804 as a slave device of the mobile phone again, the mobile phone may search in table 1 whether the audio capability parameter of the sound box 804 is stored by obtaining the identifier of the sound box 804. If the audio capability parameters of the sound box 804 are stored, the mobile phone can determine the corresponding audio playing strategy without additionally acquiring the audio capability parameters of the sound box 804, so that the switching speed of the subsequent audio switching process is increased.

TABLE 1

In the above embodiment, the audio capability parameters of the sound box 804 acquired by the mobile phone include an audio effect parameter, an audio quality parameter, an audio playing delay, and an EQ parameter, which can be understood that the audio capability parameters may further include other parameters related to the audio capability of the sound box 804, such as a signal-to-noise ratio, a distortion ratio, and other parameters, which is not limited in this embodiment. Audio policy in the mobile phone can customize an audio playing strategy matched with the audio capability of the sound box 804 according to the audio capability parameters of the sound box 804, so that the sound box 804 (i.e., the slave device) can play audio data from the mobile phone (i.e., the master device) matched with the audio capability of the sound box 804.

Similar to the above embodiment in which the mobile phone stores the audio capability parameters of the sound box 804, in other embodiments, after the AudioPolicy determines the corresponding audio playing policy according to the audio capability parameters of the sound box 804, the audio playing policy of the sound box 804 may also be stored in the mobile phone. As shown in table 2, AudioPolicy may store the audio playback policy determined for different slave devices each time in the handset. Thus, when the mobile phone switches the audio frequency by using the sound box 804 as the slave device of the mobile phone again, the DV APP can search whether the audio playing policy of the sound box 804 is stored in table 2 by obtaining the identifier of the sound box 804. If the audio playing strategy of the sound box 804 is stored, the DV APP can directly issue the audio playing strategy of the sound box 804 to the AudioPolicy, and the mobile phone does not need to additionally acquire the audio capability parameters of the sound box 804 and determine the corresponding audio playing strategy again, so that the switching speed of the subsequent audio switching process is increased.

TABLE 2

Of course, if the audio capability parameters of subsequent speakers 804 change, the AudioPolicy of the handset may re-determine the audio playing policy of the speakers 804 according to the latest audio capability parameters of the speakers 804. At this time, AudioPolicy may update the audio playing policy of sound box 804 in table 2 and store it in the mobile phone.

It should be noted that the audio playing policy of the sound box 804 may also be null, that is, the mobile phone does not need to perform processing related to the audio capability parameter on the audio data that needs to be sent to the sound box 804. At this point, the first device may send the audio data from the audio APP directly to the speaker 804. Of course, before the mobile phone sends the audio data to the sound box 804, the mobile phone may still perform conventional processing operations such as parsing, encoding, decoding, encapsulating or decapsulating on the audio data to be sent, which is not limited in this embodiment of the present application.

For example, as shown in fig. 11, after the AudioFlinger in the mobile phone processes the Audio data 1 according to the Audio playing policy output by AudioPolicy, the DMSDP Audio HAL 1101 may be called to send the processed Audio data 1 to the agent APP of the sound box 804. In some embodiments, when the AudioFlinger sends the Audio data 1 to the sound box 804 through the DMSDP Audio HAL 1101, the Audio data 1 may be further processed according to a certain rule or protocol, such as parsing, packaging, or encoding, which is generally not directly related to the Audio capability parameters of the sound box 804. For example, DMSDP Audio HAL 1101 may package Audio data 1 output by the Audio flinger into a packet of one frame and one frame, and add related information (e.g., type of Audio data 1, Audio format, etc.) of the Audio data 1 to the packet header file and send the packet of the Audio data 1 to the proxy APP of the speaker 804.

Of course, other applications besides the agent APP, such as a music APP, may be installed in the application layer of the sound box 804, and this is not limited in this embodiment of the present application. As also shown in fig. 11, after agent APP of loudspeaker 804 receives audio data 1 from the handset, its audio player (e.g., AudioTrack) may be invoked to send audio data 1 to the audio processor (e.g., AudioFlinger) of loudspeaker 804. And processing the received Audio data 1 by the audioFlinger, calling an Audio HAL in the HAL, outputting the processed Audio data 1 to a loudspeaker of the sound box 804 for playing, and completing the process of switching the Audio to the sound box 804 by the mobile phone to continue playing.

It can be seen that, for the slave device of the mobile phone (for example, the sound box 804), after the slave device only needs to install the agent APP, the audio capability parameter of the slave device may be reported to the mobile phone through the agent APP, and the audio data matched with the audio capability of the slave device is acquired from the mobile phone and played, so that the audio function of the master device 101 is flexibly distributed on the slave device 102, and a better audio use experience is provided for a user.

Further, after the mobile phone successfully switches the audio data 1 of the currently played song a to the sound box 804 for playing, if it is detected that the currently played audio data changes, or it is detected that the audio capability of the slave device changes, or it is detected that the user switches a new slave device, the mobile phone may further dynamically adjust the current audio playing policy, so that the slave device can play the audio data from the mobile phone (i.e., the master device) that matches with its own audio capability.

For example, after the mobile phone switches the audio data 1 of the playing song a to the sound box 804, the mobile phone still runs the music APP. At this time, the user may modify the playing volume of audio data 1 on the speaker 804 through the volume button of the mobile phone. Wherein, multiple types of audio streams are predefined in the audio architecture of the Android system. For example, the types of audio STREAMs include STREAM _ ALARM, STREAM _ MUSIC, STREAM _ RING, STREAM _ SYSTEM, and STREAM _ VOICE _ CALL.

As shown in fig. 13, a volume adjustment control 1302 may be displayed when the handset detects that the user presses the volume button 1301. Meanwhile, since the audio STREAM type of the audio data 1 output by the MUSIC APP is STREAM _ MUSIC, the mobile phone may call the AudioManager to reset the volume in the AudioPolicy according to the audio STREAM type of STREAM _ MUSIC.

For example, after detecting that the user presses the volume button 1301, the mobile phone may call the AudioManager to multiply the current volume of the audio data 1 by a preset volume coefficient in AudioPolicy to obtain the adjusted volume. For example, when the user presses the volume plus button, AudioPolicy may multiply the current volume (e.g., 100) by a volume factor of 120%, resulting in an adjusted volume of 120. For another example, when the user presses the "volume-" button, AudioPolicy may multiply the current volume (e.g., 100) by a volume factor of 80%, resulting in an adjusted volume level of 80. Furthermore, AudioPolicy may output the adjusted volume level to the AudioFlinger, and the AudioFlinger processes the audio data 1 according to the adjusted volume level. For example, the AudioFlinger may call the stream volume () function to modify the gain of the audio data 1, thereby modifying the volume size of the audio data 1. Furthermore, the AudioFlinger may call the DMSDP Audio HAL 1101 to send the processed Audio data 1 to the sound box 804, and the Audio data 1 played by the sound box 804 is the Audio data with the adjusted volume. Therefore, after the user switches the audio in the mobile phone to the sound box 804, the volume of the audio played on the sound box 804 can be conveniently controlled through the mobile phone.

In other embodiments, after the mobile phone switches the audio data 1 of song a being played to the sound box 804, the mobile phone still runs the music APP. As shown in fig. 14A, the music APP may also modify the sound effect mode of the song a being played. For example, the user can modify the sound effect mode of song a from the subwoofer sound effect 1402 to the ballad sound effect 1403 in the setting menu 1401 of the music APP. After the music APP detects that the user clicks the ballad sound 1403 in the setting menu 1401, the AudioManager can be called to modify the sound mode of the audio data 1 played on the mobile phone in AudioPolicy.

Furthermore, AudioPolicy may find whether the sound box 804 currently supports the playback mode of the ballad sound effect according to the audio capability parameter obtained from the sound box 804. If the sound box 804 does not support the playback mode of ballad sound effects, AudioPolicy may update the current audio playback strategy in which audio data 1 is processed using ballad sound effects. Thus, after AudioPolicy outputs an audio playback policy to AudioFlinger, AudioFlinger can perform a ballad sound effect process on the audio data 1. Subsequently, after the AudioFlinger calls the DMSDP Audio HAL 1101 to send the processed Audio data 1 to the sound box 804, the Audio data 1 played by the sound box 804 may present ballad sound effects to the user.

Or, if the sound box 804 supports the playback mode of the ballad sound effect, the AudioPolicy may also update the current audio playback strategy, and set the audio data 1 not to be processed using the ballad sound effect in the audio playback strategy. Thus, after AudioPolicy outputs an audio playback policy to AudioFlinger, AudioFlinger does not need to process balladry sound effect for audio data 1. Subsequently, after the AudioFlinger calls the DMSDP Audio HAL 1101 to send the processed Audio data 1 to the sound box 804, the sound box 804 can play the Audio data 1 after the balladry sound effect processing, so that the balladry sound effect can be presented to the user.

In some embodiments, as shown in fig. 14B, the mobile phone may further identify devices corresponding to various sound effect modes in a setting menu 1401 of the music APP. For example, both cell phone and speakers 804 support subwoofer and dolby sound. The balladry sound effect is only supported by a mobile phone, and the Histen sound effect is only supported by a sound box 804. Therefore, the user can intuitively know the sound effect modes supported by the current master equipment and the current slave equipment, and accordingly the corresponding sound effect mode is selected for the audio data 1. For example, if the sound effect mode selected by the user is only supported by the sound box 804 (for example, Histen sound effects), the AudioPolicy of the mobile phone may update the current audio playing policy, and set that the mobile phone does not perform audio processing on the audio data 1 in the audio playing policy, and then the sound box 804 may perform Histen sound effect processing on the audio data 1 and then play the audio data. If the sound effect mode selected by the user is only supported by the mobile phone (for example, ballad sound effect), the AudioPolicy of the mobile phone can update the current audio playing strategy, the mobile phone is set in the audio playing strategy to process the audio data 1 by using the ballad sound effect, and then after the mobile phone sends the processed audio data 1 to the sound box 804, the audio data 1 played by the sound box 804 can present ballad sound effect for the user. If the user selects a sound effect mode (e.g., dolby sound) supported by both the handset and the speaker, the handset may default to processing audio data 1 by the handset (or speaker 804) using dolby sound in the audio playback strategy. Alternatively, the mobile phone may prompt the user to select to process the audio data using the dolby sound effects provided by the mobile phone or using the dolby sound effects provided by the speaker 804 by displaying a dialog box. Furthermore, the mobile phone can set the audio data 1 to be processed by the mobile phone or the sound box 804 in the audio playing strategy according to the selection of the user.

Or, the mobile phone may show the sound effect modes supported by other audio output devices to the user in addition to the sound effect modes supported by the current master device (i.e., the mobile phone) and the slave device (i.e., the sound box 804). For example, if the mobile phone previously performs audio switching with the television as a slave device, the mobile phone may acquire one or more sound effect modes supported by the television. The mobile phone can store one or more sound effect modes supported by the television. When the mobile phone switches the audio frequency of the sound box 804 as a slave device, one or more audio modes supported by the television can be displayed in the setting menu 1401. Then, if the user selects the sound effect mode supported by the television, which indicates that the user wishes to process the audio data currently output by the mobile phone using the sound effect mode supported by the television, the mobile phone may disconnect the network connection with the sound box 804, and establish a network connection with the television as a slave device. And then, the mobile phone can output the audio data to the television, and the television performs sound effect processing on the audio data according to the sound effect mode selected by the user and plays the audio data. Of course, before the handset changes the slave device, the user may also be asked through a dialog box whether to update the slave device from the speaker 804 to the television. If the user confirms that the slave device of the mobile phone is updated to the television, the mobile phone can disconnect the network connection with the sound box 804 and establish the network connection with the television.

In other embodiments, after the mobile phone switches the audio data 1 of the running music APP to the sound box 804, if the mobile phone quits the music APP, the mobile phone may end the audio switching. At this time, the AudioFlinger in the mobile phone may stop calling the DMSDP Audio HAL 1101 to send the Audio data generated by the subsequent mobile phone to the sound box 804, and call the Primary HAL in the HAL to output the Audio data generated by the subsequent mobile phone to the speaker of the mobile phone for playing.

Or, after the mobile phone quits the music APP, the mobile phone may continue to send the audio data generated by the mobile phone to the sound box 804 for playing, that is, the audio switching is not finished this time. For example, after the user triggers the mobile phone to exit the music APP by clicking a home key or the like, if the user opens the communication APP at this time to perform an audio call (for example, a VoIP call) with the contact Mike, as shown in fig. 15, the mobile phone may display a call interface 1501 with the contact Mike. At this time, similar to the above-described process in which the music APP switches the audio data 1 to the speaker 804, the communication APP may call AudioTrack to transmit the audio data 2 from the contact Mike to the AudioFlinger, as shown in fig. 16A. The audioFlinger can process the Audio data 2 according to the Audio playing strategy output by audioPolicy, and call the DMSDP Audio HAL 1101 to send the processed Audio data 2 to the sound box 804 for playing.

And, when the mobile phone runs the communication APP, the voice of the user needs to be recorded, and the voice of the user is sent to the contact Mike. Then, after the mobile phone starts to operate the communication APP, a recording instruction may be sent to the sound box 804 (i.e., the slave device), so that the sound box 804 may start its own audio recording function to collect audio data input by the user in response to the recording instruction, and send the collected audio data to the mobile phone. Therefore, the mobile phone can switch the audio output function to other slave devices to realize the audio output function, and simultaneously can switch the audio input function to other slave devices to realize the audio output function. The specific process of the mobile phone acquiring the audio data input by the user through the sound box 804 to realize the audio input function can refer to the related description in fig. 26, and therefore, the detailed description is omitted here.

In contrast, after the communication APP receives the audio data 2 from the contact Mike, the audio STREAM type of the audio data 2 is changed to STREAM _ VOICE _ CALL. At this time, the communication APP may call AudioManager to register the relevant information of the audio data 2 in AudioPolicy. For example, the communication APP may buffer information such as the current sound effect mode, the number of channels, the sampling rate, the volume level, and the like of the audio data 2 in AudioPolicy.

At this time, the slave device of the mobile phone is not changed, and is still the sound box 804, but the audio data (i.e. sound source) played by the mobile phone is changed. Then, as also shown in fig. 16A, AudioPolicy may update the corresponding audio playing strategy according to the related information of audio data 2 and the audio capability parameters of loudspeaker 804.

For example, if audio data 2 output by communication APP is 8-channel audio data, and loudspeaker 804 only supports 2-channel and 4-channel audio data, AudioPolicy may instruct the conversion of the 8-channel audio data to 4-channel audio data in the updated audio playback policy so that the converted audio data 2 matches the sound playback capability of loudspeaker 804. For another example, audio data 2 output by communication APP is of STREAM _ VOICE _ CALL type, and speaker 804 only supports STREAM _ MUSIC type audio STREAMs. AudioPolicy may then instruct the handset to play audio data 2 as described above in the updated audio play policy. At this time, the AudioFlinger may call the Primary HAL according to the updated audio playback policy, and output the audio data 2 to the speaker of the cellular phone to play back using the Primary HAL. That is to say, in the process of audio switching, if the audio data output by the mobile phone changes, the mobile phone may also be triggered to re-customize the audio playing strategy matching with the audio capability of the slave device.

In some embodiments, as shown in fig. 16B, when the mobile phone runs the call APP, the audio data 3 sent by the opposite device (e.g. another mobile phone) may be received through a network device such as a base station, and unlike a general audio APP, the mobile phone may directly receive the audio data 3 through the modem HAL of the HAL without processing the audio data 3 through the AudioTrack and the AudioFlinger. In this embodiment of the application, as shown in fig. 16B, the DV APP creates a DMSDP Audio HAL 1101 according to the Audio capability parameters of the sound box 804, and after registering the Audio capability parameters of the sound box 804 in the Audio policy, the Audio policy may indicate to the modem HAL that the output device of the current Audio data is the sound box 804 corresponding to the DMSDP Audio HAL 1101. Furthermore, after receiving the Audio data 3, the modem HAL can send the Audio data 3 to the DMSDP Audio HAL 1101, and the DMSDP Audio HAL 1101 sends the Audio data 3 to the sound box 804 for playing, so as to realize the Audio switching function in the call scene.

In other embodiments, after the mobile phone switches the audio data 1 of the running music APP to the sound box 804, if the audio capability of the sound box 804 changes, the sound box 804 may dynamically report the latest audio capability parameter to the DV APP of the mobile phone. Further, the DV APP may update the DMSDP Audio HAL 1101 according to the latest Audio capability parameters of the loudspeaker 804, and update the registered Audio capability parameters of the loudspeaker 804 in AudioPolicy.

For example, when the network speed of the Wi-Fi network in which the handset and the speaker 804 are located decreases, the audio playing delay of the speaker 804 may increase. At this time, the sound box 804 can report the latest audio playing delay to the DV APP of the mobile phone. The DV APP may update the DMSDP Audio HAL 1101 in the HAL in accordance with the latest Audio playback delay. And, DV APP can update audio playback delay of speaker 804 in AudioPolicy. If the updated audio playback delay is greater than the predetermined time (e.g., 20ms), AudioPolicy may determine that audio enclosure 804 does not support low-delay playback capability at this time. Furthermore, AudioPolicy may update the audio playing policy, and set a non-low-latency mode in the updated audio playing policy, so that the AudioFlinger processes the audio data 1 according to the non-low-latency mode. That is, when the audio capability of the slave device changes, the mobile phone can be triggered to re-customize the audio playing strategy matched with the audio capability of the slave device.

In other embodiments, after the mobile phone switches the audio data 1 of the running music APP to the sound box 804, the user may also manually modify the slave device currently performing audio switching with the mobile phone on the mobile phone.

For example, as shown in (a) of fig. 17, after the mobile phone switches the audio data 1 to the sound box 804, the application interface 1701 of the music APP is continuously displayed, and if the user needs to switch the slave device currently performing audio switching with the mobile phone, the control center of the mobile phone may be turned on. For example, a user may enter a pull-up operation in the application interface 1701, and the cell phone may display the control center 1702 in response to the user's pull-up operation. Control center 1702 includes a toggle button 1703 for audio toggle functions. When the user wishes to modify the slave device of the current handset, clickable toggle button 1703 queries one or more candidate devices that may currently be slave devices of the handset.

As shown in (b) of fig. 17, after the mobile phone detects that the user clicks the switch button 1703, the mobile phone may detect a candidate device having an audio output function and located in the same Wi-Fi network as the mobile phone, and display the detected candidate device in the control center 1702. For example, the control center 1702 may include a sound box 804 currently establishing a network connection with a mobile phone, may further include a mobile phone native 1704, and may further include a television 803 and an earphone 805 which are detected by the mobile phone and do not establish a network connection with the mobile phone. The sound box 804 is displayed in a selected state, which prompts a user that the audio output by the current mobile phone is switched to the sound box 804 for playing.

If the user wishes to switch the audio output by the handset from the audio box 804 to the television 803 for playback, the user can click on the television 803 in the control center 1702. In response to the user's operation of clicking on the television 803, the DV APP may disconnect the network connection with the loudspeaker 804, and the cell phone may establish a network connection with the television 803 as a current slave of the cell phone. Further, as shown in fig. 18A, similar to the process of switching audio data to the audio box 804 by the mobile phone in the above embodiment, the DV APP may obtain the audio capability parameter' of the television 803 from the television 803 by using the television 803 as the latest slave device of the mobile phone. Further, as shown in fig. 18A, the DV APP may create a DMSDP Audio HAL1801 corresponding to the television 803 according to the Audio capability parameter' call-related interface, and close the process of the DMSDP Audio HAL 1101 created for the sound box 803. Alternatively, the DV APP may update the DMSDP Audio HAL 1101 in accordance with the Audio capability parameters of the television 803, such that the updated DMSDP Audio HAL 1101 matches the Audio capability of the television 803.

Also, as shown in fig. 18A, DV APP may call AudioManager to register the audio capabilities of television 803 in AudioPolicy. For example, the DV APP may send the identity of television 803 and the audio capability parameter' of television 803 to AudioPolicy. Thus, AudioPolicy can set a new audio playback policy according to the audio capability parameter' of the television 803 and output the set new audio playback policy to AudioFlinger. The AudioFlinger may process the audio data 1 output by the music APP according to the new audio playback strategy output by AudioPolicy. Subsequently, the AudioFlinger may call the newly created DMSDP Audio HAL 1801, and send the processed Audio data 1 to the television 803, thereby switching the Audio function of the mobile phone from the sound box 804 to the television 803.

In other embodiments, if the user wishes to switch the audio output by the handset to be played in the headphones 805, the user can click on the headphones 805 in the control center 1702. Since the existing Android system already has an Audio HAL corresponding to the headset 805, for example, A2dp HAL. Therefore, after the mobile phone detects that the user clicks the earphone 805 in the control center 1702, the mobile phone may disconnect the network connection with the audio box 804, and the AudioFlinger calls A2dp HAL to output the audio data 1 of the music APP to the earphone 805 according to the existing audio architecture, so as to switch the audio function of the mobile phone from the audio box 804 to the earphone 805.

In some embodiments, the DV APP may not create a corresponding DMSDP Audio HAL at the HAL when the handset performs Audio switching. Still taking the slave device of the mobile phone as the tv 803 for example, as shown in fig. 18B, after acquiring the Audio capability parameters of the tv 2003, the DV APP of the mobile phone can register the Audio capability parameters of the tv 2003 in AudioPolicy without creating a corresponding DMSDP Audio HAL in the HAL. Then, after AudioPolicy outputs a corresponding audio playing policy to AudioFlinger according to the audio capability parameters of television 2003, AudioFlinger may process audio data 1 from music APP according to the audio playing policy. Unlike the above embodiment, the AudioFlinger may send the processed audio data 1 to an agent APP in the application layer in the mobile phone, and the agent APP sends the processed audio data 1 to the television 2003 for playing. Or, the agent APP in the mobile phone may also run in the mobile phone in the form of a system service, which is not limited in this embodiment of the present application.

Similar to fig. 18A, the agent APP in the handset may send the processed audio data 1 to the agent APP in the television 2003, which may call its audio player (e.g., AudioTrack) to send the audio data 1 from the handset to the audio processor (e.g., AudioFlinger). And processing the received Audio data 1 by the audioFlinger, calling an Audio HAL in the HAL, outputting the processed Audio data 1 to a loudspeaker of the television 2003 for playing, and completing the process of switching the Audio to the television 2003 by the mobile phone for continuous playing.

In addition, the above embodiments are exemplified by switching the audio output device of the mobile phone from the control center of the mobile phone or the audio application in which the mobile phone is running by the user, and it is understood that those skilled in the art may also set the entry for switching the audio at other positions of the mobile phone. For example, as shown in (a) of fig. 19, the mobile phone may display the above-described switching button 1703 in the pull-down menu 1901. The mobile phone may also prompt the user in the notification center 1902 of the pull-down menu 1901 in the form of a notification that the current audio of the mobile phone is switched to a certain device for playing. For another example, as shown in (b) of fig. 19, the mobile phone may further display the above-mentioned switch button 1703 in the negative one-screen menu 1903, which is not limited in this embodiment of the present application.

In the above embodiment, a music APP is taken as an example of the audio APP, and a process of switching audio data 1 in the music APP to a slave device for audio switching and dynamically modifying an audio playing policy in an audio switching process is specifically described. In other embodiments, the user may need to switch the audio and image on the handset to be played on another device at the same time. For example, when a mobile phone runs a video APP to play a video, a user may need to switch each frame of image in the video and audio corresponding to the image to a television for playing at the same time.

Currently, some screen projection technologies are used to project images and audio on a mobile phone to a large-screen device (such as a television) for playing. For example, the mobile phone may send display data output by the display screen in real time to the television in a Miracast screen projection manner, and simultaneously, the mobile phone may send audio data output by the AudioFlinger in real time to the television, so as to project images and audio on the mobile phone to the television for playing. For another example, the mobile phone may send a URL (Uniform Resource Locator) of the video to the television in a screen-casting manner of a Digital Living Network Alliance (DLNA), and the television obtains a corresponding image and audio according to the URL to play.

In the embodiment of the present application, as shown in fig. 20, when the mobile phone runs the video APP, if the user needs to use the screen projection function, the control center 2001 of the mobile phone may be opened. A card 2002 may be provided in control center 2001, where card 2002 is used to display one or more devices in the vicinity that are detected by the cell phone. For example, the cell phone may display detected electronic devices located within the same Wi-Fi network as the cell phone in the card 2002 for selection by the user. For another example, the mobile phone may display, on the card 2002, a plurality of electronic devices bound to the mobile phone under the same account (for example, hua is an account), and if the mobile phone detects that a certain electronic device accesses a Wi-Fi network where the mobile phone is located, the mobile phone may light the electronic device on the card 2002, thereby prompting the user that the electronic device may serve as a slave device of the mobile phone to implement a screen projection function.

If it is detected that the user selects a certain electronic device (e.g., television 2003) in the card 2002, the television 2003 has both a display function and an audio playing function, indicating that the user wishes to project images and audio of the video APP running on the mobile phone to the television 2003. At this time, the mobile phone may establish a network connection with the television 2003, and as shown in fig. 21, the mobile phone may send display data output by the video APP in the mobile phone to the television 2003 for displaying according to the existing screen projection method.

In contrast, the mobile phone may switch the audio data in the mobile phone to the television 2003 according to the audio switching process in the above embodiment, instead of switching the audio data to the television 2003 according to the DLNA or Miracast screen projection manner.

Illustratively, after the cell phone detects that the user clicks on the television 2003 in the card 2002, as shown in fig. 21, on the one hand, the cell phone may send display data output by the video APP to the television 2003. On the other hand, the DV APP of the handset may obtain the audio capability parameters of the television 2003 from the television 2003. Further, the DV APP may create a DMSDP Audio HAL2101 corresponding to the television 2003 in the HAL in accordance with the Audio capability parameters of the television 2003, and register the Audio capability parameters of the television 2003 in AudioPolicy. In this way, AudioPolicy may set an audio playback policy matching the audio capability of television 2003 during screen projection according to the audio capability parameters of television 2003, and output the audio playback policy to AudioFlinger. The audioFlinger can process the Audio data generated by the video APP according to the Audio playing strategy and call the DMSDP Audio HAL2101 to send the processed Audio data to the television 2003 for playing. Or, the AudioFlinger may also send the processed audio data to the video APP, and the video APP sends the audio data to the television 2003 for playing.

In addition, the handset can add a time stamp to the Audio data when sending the Audio data to the television 2003 through the DMSDP Audio HAL 2101. Meanwhile, the handset may also add a timestamp to the display data from the video APP when sending the display data to the television 2003. Thus, the television 2003 can play video with pictures synchronized with audio according to the time stamp in the display data and the time stamp in the audio data, so that the audio and the pictures played by the television 2003 are kept synchronized during the screen projection process.

It can be seen that, compared with the conventional screen projection method in which the audio data output by the AudioFlinger in real time is directly sent to the slave device (for example, the television 2003), in the screen projection method provided in the embodiment of the present application, the mobile phone can customize an audio playing policy matched with the audio capability of the slave device by obtaining the audio capability parameter of the slave device, so that the AudioFlinger can output corresponding audio data according to the audio playing policy and send the audio data to the slave device, and thus, the slave device can play the audio data from the mobile phone (i.e., the master device) matched with the audio capability of the slave device, and the playing effect of the audio in the screen projection scene is improved.

In the above embodiments, the example is given by switching the audio data to a certain slave device by using a mobile phone to implement the audio switching function, and in other embodiments of the present application, the mobile phone may also switch the audio data to a plurality of slave devices to perform audio switching.

Illustratively, still taking the example of running a music APP with a mobile phone, as shown in fig. 8 (a), if it is detected that the user clicks a switch button 801 in the music APP, the mobile phone may be triggered to search for one or more nearby candidate devices that may be audio-switched. Unlike the above embodiment, as shown in fig. 22, after the mobile phone displays a plurality of candidate devices searched by the mobile phone in the menu 2201, the user can select a plurality of candidate devices as the slave devices of the mobile phone at the same time for audio switching.

For example, a user selects the first sound box 2202 and the second sound box 2203 in the menu 2201 as slave devices of the mobile phone, and the mobile phone can establish network connection with the first sound box 2202 and the second sound box 2203 respectively. Furthermore, as shown in fig. 23, the mobile phone can obtain the audio capability parameters of the first sound box 2202 and the second sound box 2203 through DV APP. For example, the audio capability parameter of the first sound box 2202 is audio capability parameter 1, and the audio capability parameter of the second sound box 2203 is audio capability parameter 2. Then, the DV APP may register in Audio policy with Audio capability parameter 1 for first sound box 2202 and audio capability parameter 2 for second sound box 2203. Also, the DV APP may create a DMSDP Audio HAL 2301 at the HAL corresponding to the first sound box 2202 according to the Audio capability parameter 1, and create a DMSDP Audio HAL2302 corresponding to the second sound box 2203 according to the Audio capability parameter 2. Furthermore, AudioPolicy may set an audio playing policy according to audio capability parameter 1 of first sound box 2202 and audio capability parameter 2 of second sound box 2203, so that AudioFlinger may process audio data from music APP according to the audio playing policy determined by AudioPolicy.

Illustratively, the audio playback strategy determined by AudioPolicy may include audio playback strategy 1 corresponding to first sound box 2202, and audio playback strategy 2 corresponding to second sound box 2203. The process of AudioPolicy determining the audio playing policy 1 according to the audio capability parameter 1 of the first sound box 2202 is similar to the process of AudioPolicy determining the audio playing policy according to the audio capability parameter of the sound box 803 in the above embodiment, and similarly, the process of AudioPolicy determining the audio playing policy 2 according to the audio capability parameter 2 of the second sound box 2203 is also similar to the process of AudioPolicy determining the audio playing policy according to the audio capability parameter of the sound box 803 in the above embodiment, and therefore, the description is omitted here.

In contrast, AudioPolicy may instruct AudioFlinger to copy audio data 1 from music APP into 2 copies, for example, audio data 11 and audio data 12. Thus, as also shown in fig. 23, after AudioPolicy outputs audio playback policy 1 to AudioFlinger, AudioFlinger may process audio data 11 according to audio playback policy 1 to obtain audio data matching the audio capabilities of first sound box 2202. Similarly, after AudioPolicy outputs audio playback policy 2 to AudioFlinger, AudioFlinger may process audio data 12 according to audio playback policy 2 to obtain audio data matching with the audio capability of second speaker 2203. Subsequently, as also shown in fig. 23, the AudioFlinger may call the DMSDP Audio HAL2301 to send the processed Audio data 11 to the first sound box 2202 for playing, and at the same time, the AudioFlinger may call the DMSDP Audio HAL 2302 to send the processed Audio data 12 to the second sound box 2203 for playing. Therefore, the mobile phone can simultaneously switch the audio playing function of the mobile phone to a plurality of slave devices, and the distributed audio capability of the cross-device is realized.

In some embodiments, AudioPolicy may also customize a corresponding audio playback policy in conjunction with information such as the audio capabilities or device location of each slave device. For example, the first sound box 2202 and the second sound box 2203 may also carry their own location information in the reported audio capability parameters. For example, the first sound box 2202 is positioned to the left of the user with respect to the user's pose, and the second sound box 2203 is positioned to the right of the user with respect to the user's pose. Then, AudioPolicy may set the audio data 11 processed according to the left channel sound effect in the audio playback policy 1 and set the audio data 12 processed according to the right channel sound effect in the audio playback policy 2, according to the position information. Thus, when the AudioFlinger calls the corresponding DMSDP Audio HAL to send the processed Audio data 11 and Audio data 12 to the first sound box 2202 and the second sound box 2203 respectively for playing, a stereo playing effect can be presented to the user.

For another example, the slave device that performs audio switching with the mobile phone is a television and a sound box, for example, the AudioPolicy may determine that the sound effect processing capability of the sound box is stronger than that of the television by combining audio capability parameters respectively reported by the television and the sound box. Then, AudioPolicy may set, in the audio playing policy, to output the audio data that needs to be subjected to sound effect processing to the sound box for playing, and output the audio data that does not need to be subjected to sound effect processing to the television for playing. Furthermore, the audioFlinger can shunt the audio data from the audio APP according to the audio playing strategy output by the audioPolicy to obtain the audio data A needing audio processing and the audio data B not needing audio processing. Subsequently, the audioFlinger can call the DMSDP Audio HAL corresponding to the sound box to send the processed Audio data A to the sound box for playing, and call the DMSDP Audio HAL corresponding to the television to send the processed Audio data B to the television for playing.

In other embodiments, if the Audio capability parameter 1 of the first sound box 2202 and the Audio capability parameter 2 of the second sound box 2203 are the same, for example, the first sound box 2202 and the second sound box 2203 are two sound boxes of the same model, the DV APP may create a DMSDP Audio HAL 2401 at the HAL corresponding to the Audio capability parameter 1 (i.e., the Audio capability parameter 2), as shown in fig. 24. That is, the DMSDP Audio HAL created by the DV APP at the HAL corresponds to the Audio capability parameters. Since the audio capability parameter 1 of the first sound box 2202 is the same as the audio capability parameter 2 of the second sound box 2203, the audio playing strategy determined by AudioPolicy according to the audio capability parameter is also the same. As shown in fig. 24, AudioPolicy may output the determined audio playback policy to AudioFlinger, and the AudioFlinger processes audio data from the audio APP according to the audio playback policy. The difference is that the AudioFlinger may divide the processed audio data into 2 parts, for example, the AudioFlinger may copy the processed audio data 1 to obtain the audio data 11 'and the audio data 12' (in this case, the audio data 11 'and the audio data 12' are the same). Still alternatively, the AudioFlinger may extract audio data 11 'corresponding to the left channel and audio data 12' corresponding to the right channel (in this case, the audio data 11 'and the audio data 12' are different) from the processed audio data 1. Furthermore, the AudioFlinger may invoke DMSDP Audio HAL 2401 to send Audio data 11 'and Audio data 12' to the first sound box 2202 and the second sound box 2203, respectively, to achieve the cross-device distributed Audio capability.

In the above embodiment, it is exemplified that the mobile phone switches its own Audio data to one or more slave devices for playing, for example, the mobile phone creates a DMSDP Audio HAL by acquiring the Audio capability of the television, and switches the Audio data from the Audio APP to the television for playing through the DMSDP Audio HAL. At this time, the television is a slave device of the mobile phone.

In other embodiments, the slave device (e.g. a television) of the mobile phone may also switch the corresponding audio data to other devices for audio switching when playing audio. At this time, the slave device of the mobile phone may serve as a master device, and the audio data sent from the mobile phone is switched to other slave devices again for playing according to the audio switching method in the above embodiment, so as to implement a continuous audio switching function between the multiple levels of devices.

For example, as shown in fig. 25, after the tv 2501 establishes a bluetooth connection with the speaker 2502, the tv 2501 may obtain Audio capability parameters of the speaker 2502, thereby creating a corresponding DMSDP Audio HAL. At this time, the speaker 2502 can play audio data from the television 2501 as a slave device of the television 2501. Then, after the mobile phone sends the audio data to the television 2501, the television 2501 may serve as a master device to set a corresponding audio playing policy according to the audio capability parameter of the sound box 2502, and process the audio data sent by the mobile phone according to the audio playing policy. Subsequently, the tv 2501 may call the DMSDP Audio HAL corresponding to the speaker 2502 to send the Audio data to the speaker 2502 for playing. Therefore, the mobile phone finally switches the audio data to the sound box 2502 for playing, the sound box 2502 can play the audio data matched with the audio capability of the mobile phone, and the main device (i.e., the television 2501) of the sound box 2502 can also manage and control the audio playing process of the sound box 2502 according to the audio capability of the sound box 2502, so that better audio using experience is provided for the user.

Or, when the mobile phone obtains the audio capability parameter of the television 2501, if the television 2501 has already established the bluetooth connection with the sound box 2502, that is, the audio output device of the television 2501 is the sound box 2502 at this time, then the television 2501 can directly report the audio capability parameter of the sound box 2502 to the mobile phone. Therefore, the mobile phone can directly customize the corresponding audio playing strategy according to the audio capability parameters of the sound box 2502, process the audio data according to the audio playing strategy and then send the audio data to the television 2501, and then the television 2501 transmits the audio data sent by the mobile phone to the sound box 2502 for playing.

Or, if the mobile phone obtains that the current slave device of the television 2501 is the sound box 2502 from the television 2501, the mobile phone may directly establish a network connection with the sound box 2502, and obtain the audio capability parameter of the sound box 2502 from the sound box 2502. In this way, the mobile phone can also directly customize the corresponding audio playing strategy according to the audio capability parameters of the sound box 2502, process the audio data according to the audio playing strategy and then send the audio data to the sound box 2502 for playing, and the television 2501 does not need to transmit the audio data sent by the mobile phone to the sound box 2502.

It can be understood that the specific process of the television 2501 serving as the master device switching the audio data to the bluetooth sound box 2502 is similar to the specific process of the mobile phone switching the audio data to the television 2501 serving as the slave device, and therefore, the detailed description is omitted here.

In addition, in the above embodiments, it is exemplified that the mobile phone switches the audio data to one or more slave devices for playing, that is, the audio output function of the mobile phone is distributed to one or more slave devices for implementation, and it is understood that the mobile phone may also distribute the audio input function (also referred to as an audio recording or recording function) to one or more slave devices for implementation. That is, the handset may input audio data using one or more recording functions of the slave device and transmit the input audio data to the handset.

For example, similar to the audio playing process in the foregoing embodiment, the mobile phone may further include, in addition to the audio playing capability (e.g., sound effect parameter, audio quality parameter, or audio playing delay) of the slave device, the audio recording capability of the slave device from the audio capability parameters acquired from the slave device.

For example, taking a sound box as a slave device of a mobile phone, as shown in table 3, the audio capability parameters of the sound box may include a playing parameter and a recording parameter. The playing parameters may specifically include the sound effect parameters, the audio quality parameters, or the audio playing time delay described in the above embodiments, and the playing parameters are used to reflect the audio output capability of the sound box; the recording parameters may specifically include a sampling rate and a voice algorithm supported by the device, and the recording parameters are used to reflect the audio input capability of the speaker. The 3A algorithm in the speech algorithm may specifically include an AEC (acoustic echo cancellation) algorithm, an AGC (automatic gain control) algorithm, and an ANC (active noise control) algorithm, and is used to improve the audio quality of the recorded audio.

TABLE 3

As shown in fig. 26, for example, when a mobile phone runs a communication APP to perform an audio call, it is necessary to play audio data from a contact and record audio data input by a user and send the audio data to the contact when running the communication APP. Then, in the process of running the communication APP, if the user selects to switch the audio function of the mobile phone to the sound box, as shown in fig. 27, the mobile phone may display a prompt 2602 for prompting the user to play and record sound using the speaker and the microphone of the sound box when running the communication APP. Also, as shown in fig. 26, the mobile phone may establish a network connection with the speaker and obtain audio capability parameters of the speaker, where the audio capability parameters may include a playing parameter and a recording parameter as shown in table 2. Further, the DV APP may create a DMSDP Audio HAL 2601 at the HAL in accordance with the Audio capability parameters. The created DMSDP Audio HAL 2601 can support both sending Audio data to the sound box and receiving Audio data recorded by the sound box. And after the DV APP registers the audio capability parameters of the sound box to AudioPolicy, AudioPolicy may customize a corresponding audio playing strategy according to the playing parameters in the audio capability parameters, and simultaneously, AudioPolicy may customize a corresponding audio recording strategy according to the recording parameters in the audio capability parameters.

Of course, if the audio capability parameters reported by the slave device (e.g., a sound box) do not include the recording parameter, which indicates that the slave device does not have the audio recording capability, the mobile phone may select to use its own microphone or other devices having the audio recording capability to record the audio data, which is not limited in this embodiment of the present application.

Illustratively, if the audio capability parameters of the sound box acquired by the mobile phone include both the playing parameter and the recording parameter, it is indicated that the sound box has both audio output and input capabilities. Then, after the mobile phone switches the audio function to the sound box according to the audio capability parameter of the sound box, as shown in (a) of fig. 28, the mobile phone may display a first button 2801 corresponding to the audio output capability and a second button 2802 corresponding to the audio input capability in the control center. The user can control the switching of the audio output and input functions of the handset on the slave device through the first button 2801 and the second button 2802, respectively. For example, if it is detected that the user clicks the first button 2801, as shown in (b) of fig. 28, the handset may display a device list 2803 of candidate devices having an audio output function, where the handset's current audio output device is a loudspeaker. The user may switch the audio output device of the handset in the device list 2803. Similarly, if the user is detected to click the second button 2802, then as shown in (c) of fig. 28, the handset may display a list 2804 of candidate devices with audio input functionality, where the handset's current audio input device is also a speaker. The user may switch the audio input device of the handset in the device list 2804. Therefore, the user can respectively switch the audio output and input capabilities of the mobile phone to the corresponding devices according to the requirements of the user, the audio output and input functions of the mobile phone are separately controlled, and the audio use experience of the user is improved.

Similar to the AudioPolicy customized audio playing strategy of the mobile phone in the above embodiment, AudioPolicy may set an audio recording strategy according to the sampling rate and the voice algorithm in the above recording parameters. For example, if the speaker supports a sampling rate of 16KHz when recording audio data 1, AudioPolicy may set the audio recording strategy to sample audio data 1 at a sampling rate of 16 KHz. For another example, if the sound box supports the 3A algorithm when recording the audio data 1, the AudioPolicy may set not to use the 3A algorithm to process the audio data 1 in the audio recording policy, so as to prevent the mobile phone and the sound box from repeatedly using the 3A algorithm to process the recorded audio data 1.

After the AudioPolicy of the mobile phone determines the current Audio recording policy, as shown in fig. 26, the Audio recording policy may be output to the AudioFlinger, and the AudioFlinger may receive the Audio data recorded from the sound box by calling the DMSDP Audio HAL 2601. Illustratively, as also shown in fig. 26, the sound box may record the audio data 4 through a hardware device such as a microphone in the hardware layer, and output the recorded audio data 4 to the audiomaker of the sound box through the AudioHAL for processing, for example, processing the audio data 4 by using a 3A algorithm. Furthermore, the AudioFlinger can output the processed audio data 4 to an audio recorder (AudioRecord) of the sound box, the AudioRecord reports the recorded audio data 4 to an agent APP installed in the sound box, and finally the agent APP sends the recorded audio data 4 to the mobile phone. After receiving the Audio data 4 sent from the sound box, the mobile phone can output the Audio data 4 to the audioFlinger of the mobile phone through the DMSDP Audio HAL 2601. At this time, the AudioFlinger may process the audio data 4 according to the audio recording policy output by AudioPolicy, so that the processed audio data 4 matches the audio recording capability of the sound box. Subsequently, the AudioFlinger may output the processed audio data 4 to the communication APP through an audio recorder (AudioRecord). Meanwhile, the audio data generated during the operation of the communication APP can be switched to the sound box for playing according to the audio switching method in the above embodiment. Therefore, the mobile phone can flexibly switch the audio input and output functions of the mobile phone to other slave devices by combining the audio capability of the slave devices, and a cross-device distributed audio architecture is realized.

Similar to the above embodiment in which the mobile phone switches the audio data to a plurality of slave devices for playing, the mobile phone may also record the audio data through the plurality of slave devices. For example, as shown in fig. 29, the slave devices of the mobile phone may include a sound box 2701 and a sound box 2702, and both the sound box 2701 and the sound box 2702 have a recording function. Then, after the DV APP of the mobile phone acquires the Audio capability parameters of the sound boxes 2701 and 2702, the corresponding DMSDP Audio HAL 2703 and DMSDP Audio HAL2704 may be created in the HAL. In addition, DV APP can register audio capability parameters of audio box 2701 and audio box 2702 in AudioPolicy, so that AudioPolicy can determine an audio recording policy according to the audio capability parameters of audio box 2701 and audio box 2702. For example, the audio recording policy may instruct the AudioFlinger to mix audio data collected by the sound boxes 2701 and 2702, respectively, so as to achieve a stereo mixing effect.

Then, after the AudioFlinger receives the Audio data 5 recorded by the sound box 2701 through the DMSDP Audio HAL 2703 and receives the Audio data 6 recorded by the sound box 2702 through the DMSDP Audio HAL2704, the AudioFlinger may perform Audio mixing processing on the Audio data 5 and the Audio data 6 according to the Audio recording policy output by AudioPolicy. For example, the AudioFlinger may filter the noise and the echo in the audio data 5 to obtain the audio data 5 as left channel data, and the AudioFlinger may filter the noise and the echo in the audio data 6 to obtain the audio data 6 as right channel data, and further, the AudioFlinger may combine the left channel data and the right channel data into one audio file. Subsequently, the AudioFlinger may output the merged audio file to the communication APP through an audio recorder (AudioRecord), so that the audio finally acquired by the communication APP can restore the real sound situation of the user as truly as possible.

Similar to the process of switching the audio data to the slave device for playing according to the audio playing policy in the foregoing embodiment, after the mobile phone switches the audio recording function to the slave device for implementation, if it is detected that the currently recorded audio data changes, or it is detected that the audio recording capability of the slave device changes, or it is detected that the user switches a new slave device, the mobile phone may further dynamically adjust the current audio recording policy. Therefore, the mobile phone can dynamically adjust the audio recording strategy to manage and control the audio recording process of the slave device, and better audio use experience is provided for users.

In a distributed audio scenario, a master device may interact with different types of audio data with slave devices. Still take the mobile phone as the main device for example, the song played by the mobile phone when running the MUSIC APP is MUSIC type audio data, and the short message prompt tone played by the mobile phone when receiving the short message is NOTIFICATION type audio data. When the handset is connected to a slave device (e.g., a speaker), the handset may send the MUSIC type audio data being played to the speaker for playing. At this time, if the mobile phone receives a new short message, the mobile phone may also send the NOTIFICATION type audio data to the sound box for playing. Therefore, when the user uses the sound box to listen to music, the user can be disturbed by the short message prompt tone, and the audio use experience of the user is reduced.

In the embodiment of the present application, different types of audio data may be defined according to the service function specifically implemented by the audio data. For example, audio data sent by an opposite terminal and received by a mobile phone in a call service for realizing a call function can be defined as call sound; for example, audio data generated by a mobile phone in an audio or video service to implement an audio/video playing function may be defined as media sound; for example, audio data generated by a mobile phone in a navigation service to implement a navigation function may be defined as a navigation sound; for example, audio data generated when a user clicks a key when the mobile phone realizes the key function may be defined as a key tone or the like. One or more types of audio data may be generated or received in an application. For example, the wechat APP may generate audio data of a media sound type when playing a song, and the audio data received by the wechat APP when performing an audio call service is audio data of a communication voice type.

For example, in the android system, the types of audio data can be divided into: ALARM, MUSIC, RING, SYSTEM, NOTIFICATION, DTMF, COMMUNICATIONs, and VOICE CALL. The mobile phone can respectively control different types of audio data. For example, as shown in fig. 30, the setting application of the cellular phone may provide the user with an option of setting the volume size of each type of audio data in the sound setting interface 3001. For example, a volume adjustment slider 3002 corresponding to MUSIC type may be included in the setting interface 3001. The user can set the volume level of MUSIC (MUSIC) type audio data by sliding the volume adjustment slider 3002. Similarly, the user can set the volume level of audio data of the type of NOTIFICATION, ALARM, and VOICE _ CALL in the setting interface 3001. Therefore, when the mobile phone plays different types of audio data, the corresponding audio data can be played according to the volume set by the user.

Since audio data is generally present in the form of a data stream, the type of audio data may be referred to as a type of audio stream. In the android system, as an example of the type of RING audio data, the type of RING audio data may also be represented as STREAM _ RING. Of course, in android systems of different versions, the specific division of the types of the audio data may not be consistent, and this is not limited in this embodiment of the application. In addition, in other operating systems, the audio data may be divided into multiple types according to different service scenes.

For different types of audio data, the mobile phone can shunt different audio data according to a set equipment selection strategy. For example, RING-type audio data can be set in the device selection policy in advance to be played by connected headphones and local speakers at the same time, MUSIC-type audio data is preferentially played by using the headphones.

Then, after the mobile phone is connected to the earphone, when the mobile phone plays MUSIC type audio data, for example, when the mobile phone runs MUSIC APP to play a song, the mobile phone may send the MUSIC type audio data to the earphone for playing according to the above-mentioned device selection policy. If the handset receives an incoming call from a contact, the handset needs to play a RING type RING tone. At this time, the mobile phone can simultaneously send RING tones of RING types to the headset and the local speaker according to the device selection policy, and the headset and the local speaker play RING tones of RING types simultaneously.

In the embodiment of the application, the mobile phone as a master device can switch the generated audio data to one or more slave devices for playing. The mobile phone can set corresponding equipment selection strategies for different types of audio data by combining the equipment characteristics of the current slave equipment, so that shunting is carried out according to the equipment selection strategies when audio switching is carried out.

For example, when the slave device of the mobile phone is a sound box, since the sound box has good sound effect but poor portability when playing audio data, a user generally listens to music or audio programs by using the sound box. Therefore, the loudspeaker box can be arranged in the equipment selection strategy in advance to be used for receiving audio data of a MUSIC type when being used as the slave equipment of the mobile phone, and audio data of a NOTIFICATION type is not received. Subsequently, as shown in fig. 31, when the user selects the loudspeaker as the slave device for the audio switching of the mobile phone this time, the mobile phone may send audio data (e.g., a song) of MUSIC type to the loudspeaker for playing according to the above device selection policy. If the mobile phone generates the NOTIFICATION type audio data, the mobile phone does not send the NOTIFICATION type audio data (such as a short message prompt tone) to the sound box for playing. Therefore, when the mobile phone performs audio switching by using the sound box as the slave device, a user can listen to MUSIC type audio data from the mobile phone on the sound box without being disturbed by other audio data (such as short message prompt tones) in other mobile phones.

For example, when the slave device of the mobile phone is an in-vehicle device (also referred to as a car machine), the user needs to keep more concentrated attention in a driving scene when the in-vehicle device plays audio data. Therefore, when the in-vehicle device is set as a slave device of the mobile phone in the device selection policy in advance, audio data of MUSIC type and navigation voice type can be received, and audio data of other types (such as NOTIFICATION type and SYSTEM) is not received. Subsequently, as shown in fig. 32, when the user selects the in-vehicle device as the slave device for the audio switching of the mobile phone this time, the mobile phone may send audio data of MUSIC type (e.g., song) or navigation voice type to the in-vehicle device according to the device selection policy for playing. If the mobile phone receives or generates other types of audio data, the mobile phone does not send the audio data to the vehicle-mounted equipment for playing. In this way, when the mobile phone performs audio switching with the vehicle-mounted device as a slave device, the user can listen to audio data of MUSIC type or VOICE _ CALL type from the mobile phone on the vehicle-mounted device without being disturbed by other audio data (such as short message prompt tone) in other mobile phones.

In addition, when the mobile phone prohibits receiving certain type of audio data from the device in the device selection policy, for example, prohibits receiving NOTIFICATION type audio data by the in-vehicle device, the mobile phone may further set an audio output device for the NOTIFICATION type audio data in the device selection policy, that is, a specific device that is allowed to play the NOTIFICATION type audio data. For example, the handset may set itself as an audio output device for audio data of the NOTIFICATION type. Then, when the mobile phone switches the audio of the vehicle-mounted device as a slave device, if the audio data of the NOTIFICATION type is generated, the mobile phone does not send the audio data to the vehicle-mounted device, but uses its own speaker to play the audio data, so as to prevent the audio data generated by the mobile phone from being leaked. Of course, the mobile phone may also select another device as the audio output device for NOTIFICATION-type audio data during audio switching, which is not limited in this embodiment of the present application.

It can be seen that, in the embodiment of the present application, when the master device (for example, the above-mentioned mobile phone) outputs the audio data to one or more slave devices for audio switching, the audio data matched with the device characteristics of the slave devices may be sent to the slave devices for playing, so as to filter out some audio data that are not suitable for playing on the slave devices for a user, so that the adaptability of the audio data and the audio output device is higher, and a better audio use experience is provided for the user.

In some embodiments, as shown in fig. 33, after the DV APP of the mobile phone creates a DMSDP Audio HAL for the slave device, the DV APP may also issue a device selection policy corresponding to the current slave device to the AudioPolicy. The device selection policy records the playback capabilities of the slave devices for different types of audio data. Taking the slave device of the mobile phone as the speaker for example, the device selection policy of the speaker may be as shown in table 4, where the playback capability of the speaker for each type of audio data is recorded in the device selection policy. For example, audio data of MUSIC type and RING type are allowed to be played on the speaker, and audio data of SYSTEM type and NOTIFICATION type are not allowed to be played on the speaker.

TABLE 4

Generally, as shown in fig. 33, after the audio APP generates a certain type of audio data, an audio player may be called to play. And, the audio APP can identify the specific type of the audio data in the audio player by setting a preset parameter (e.g., mode parameter). For example, when the audio APP sets the mode parameter to 01, it indicates that the type of audio data input by the audio APP to the audio player is the MUSIC type; when the audio APP sets the mode parameter to 11, it indicates that the type of audio data input by the audio APP to the audio player is a RING type or the like.

Then, as also shown in fig. 33, after the AudioFlinger receives audio data from the audio APP generation through the audio player, the AudioFlinger can determine the type of audio data received at this time by reading the mode parameter in the audio player. Further, as also shown in fig. 33, the AudioFlinger may request AudioPolicy to query whether this type of audio data is allowed to be played on the current slave (i.e., loudspeaker). AudioPolicy may respond to the request, and determine whether the audio output device of the current audio data is a sound box according to a device selection policy issued by the DV APP and shown in table 4, that is, select a suitable audio output device for the current audio data.

If the audio enclosure allows the current type of audio data to be played, AudioPolicy may indicate to the AudioFlinger that the audio output device for the current audio data is the audio enclosure. Further, as shown by the solid arrow in fig. 33, after the Audio flinger processes the Audio data, the processed Audio data may be transmitted to the sound box through the DMSDP Audio HAL, and the sound box may play the Audio data.

Accordingly, if the speaker does not allow the current type of audio data to be played, AudioPolicy may reselect an audio output device for the current audio data, e.g., AudioPolicy may set its speaker as the audio output device for the current audio data. Further, AudioPolicy may indicate to the AudioFlinger that the audio output device for the current audio data is a local speaker. Then, as shown by the dotted arrow in fig. 33, after the AudioFlinger processes the audio data, the processed audio data may be transmitted to the speaker of the mobile phone through the Primary HAL, and the audio data may be played by the speaker of the mobile phone.

It is to be understood that only the types of audio data that the sound box allows to play may be set in the above-described device selection policy (e.g., the device selection policy of the sound box shown in table 4). At this time, other types of audio data may default to audio data that the speaker does not allow to be played. Alternatively, only the types of audio data that the speaker does not allow to be played may be set in the device selection policy shown in table 4. At this time, other types of audio data may be the audio data allowed to be played by the sound box by default, and this is not limited in this embodiment of the present application.

In other embodiments, as shown in table 5, the audio output devices corresponding to each type of audio data may also be set in the device selection policy. For example, for MUSIC type audio data, the audio data is allowed to be played on a cell phone, a sound box, and a television. For NOTIFICATION type audio data, the audio data is allowed to be played on a cell phone and a watch.

TABLE 5

Type (B)	Audio output device
		MUSIC (MUSIC)	Mobile phone, sound box and television
NOTIFICATION (Notification)	Mobile phone and watch
		……	……

Then, when the AudioFlinger receives a certain type of audio data, it may still request AudioPolicy to determine the specific audio output device for that type of audio data. Taking MUSIC type audio data as an example, AudioPolicy may determine, according to the device selection policy shown in table 5, that audio output devices capable of playing the audio data include a mobile phone, a sound box, and a television. If the slave device connected with the current mobile phone is a sound box, the AudioPolicy can determine that the audio output device of the MUSIC type audio data is the sound box, and further instruct the audioFlinger to send the MUSIC type audio data to the sound box for playing. For example, by using the audio data of NOTIFICATION type, AudioPolicy may determine, according to the device selection policy shown in table 5, that the audio output device capable of playing the audio data includes a mobile phone and a watch, that is, a sound box connected to the current mobile phone does not allow the audio data to be played, and then AudioPolicy may determine the mobile phone as the audio output device of the audio data of NOTIFICATION type, and further instruct the AudioFlinger to send the audio data of NOTIFICATION type to a speaker of the mobile phone for playing.

When the type of audio data output by the audio APP changes, the audio APP can modify the mode parameters in the audio player. Furthermore, when the AudioFlinger reads that the mode parameters change, the type of the current audio data can be determined according to the latest mode parameters. In this way, the AudioFlinger may request AudioPolicy again to inquire whether the new type of audio data is allowed to be played on the current slave device (i.e., loudspeaker), so as to dynamically switch the different types of audio data to be played on the corresponding audio output devices.

Or, when the slave device of the mobile phone changes (for example, the slave device is updated from the loudspeaker box to the television), the DV APP may be triggered to issue a device selection policy corresponding to the television to the AudioPolicy. Furthermore, AudioPolicy can select a suitable audio output device for the current type of audio data in real time according to the device selection policy of the television, so that different types of audio data can be dynamically switched to the corresponding audio output devices for playing.

It can be seen that based on the audio architecture shown in fig. 33, the master device may set a corresponding device selection policy for the slave devices in advance according to the device characteristics of each slave device. Therefore, in the process of audio switching, the master device can send the audio data matched with the device characteristics of the slave device to the slave device for playing according to the device selection strategy of the current slave device, so that some audio data which are not suitable for playing on the slave device are filtered out for a user, the master device can selectively switch various audio data to corresponding audio output devices for playing, and better audio use experience is provided for the user.

Still take a mobile phone as an example of a main device, an audio control method provided by the embodiments of the present application will be described below with reference to the accompanying drawings.

As shown in fig. 34, an audio control method provided in an embodiment of the present application includes the following steps:

and S3401, the mobile phone establishes network connection with the first slave device by running the DV APP.

For example, the user can switch the audio output device when the mobile phone outputs audio in the mobile phone. For example, as shown in (a) in fig. 35, the cellular phone may display a switch button 3501 for audio switching when running the music APP. If it is detected that the user clicks the above switching button 3501, the handset can search for one or more candidate devices nearby for audio switching by running the DV APP. As shown in fig. 35 (b), the DV APP may display candidate devices such as a television, a sound box, and a headphone, which are located on the same Wi-Fi network as the cellular phone, in the menu 3502. The user can select one of the candidate devices as an audio output device for the subsequent audio output of the mobile phone, namely, the slave device of the mobile phone.

For another example, the user may open the control center of the mobile phone and switch the audio output device in the control center when the mobile phone outputs audio. In response to an operation of opening the control center input to the cellular phone by the user, the cellular phone may display a control center 3601 as shown in (a) of fig. 36. A switching button 3602 for audio switching is provided in the control center 3601. If it is detected that the user clicks the above switching button 3602, the handset can also search for one or more candidate devices nearby that can be used for audio switching by running the DV APP. As shown in fig. 36 (b), the DV APP may display the searched candidate devices, such as a television, a sound box, and a headset, which are located on the same Wi-Fi network as the mobile phone, in the control center 3601. Similarly, the user can select one of the candidate devices as an audio output device for the subsequent audio output of the mobile phone, namely, a slave device of the mobile phone.

For example, the sound box selected by the user is taken as the slave device, and after the situation that the user selects the candidate device, namely the sound box, from the candidate devices is detected, the DV APP can establish network connection between the sound box and the sound box by taking the sound box as the slave device of the mobile phone. For example, the network connection may be a P2P connection based on a TCP (transmission control protocol) or a UDP (user datagram protocol), which is not limited in this embodiment.

It should be noted that, in step S3401, the network connection established between the handset and the slave device may specifically refer to a network connection of a service channel (i.e., a service connection). For example, the handset may have established a connection with the speaker over the Wi-Fi network before the user selects the speaker as a slave device of the handset, where the connection refers to a connection of a data channel (i.e., a data connection). After the user selects the sound box as the slave device of the mobile phone, the mobile phone and the sound box can establish service connection on the basis of the established data connection.

After the network connection between the mobile phone and the sound box is established, as shown in fig. 37, the DV APP can obtain the audio capability parameter of the sound box based on the network connection, and the audio capability parameter can reflect the audio output function specifically supported by the sound box. Further, the DV APP may create a corresponding DMSDP Audio HAL 3701 at the HAL according to the Audio capability parameters of the loudspeaker. Subsequently, the mobile phone can transmit Audio data to the sound box through the DMSDP Audio HAL 3701.

In some embodiments, the user may also select multiple slave devices for outputting the handset's audio. For example, the user may select the sound box and the television as slave devices of the mobile phone at the same time from the candidate devices, and at this time, the DV APP may establish network connection with the sound box and the television box respectively and obtain corresponding audio capability parameters. Further, the DV APP may create a first DMSDP Audio HAL corresponding to the loudspeaker at the HAL according to the Audio capability parameters of the loudspeaker, and the DV APP may create a second DMSDP Audio HAL corresponding to the television at the HAL according to the Audio capability parameters of the television. Subsequently, the mobile phone may send corresponding first Audio data to the sound box through the first DMSDP Audio HAL, and send corresponding second Audio data to the television through the second DMSDP Audio HAL (the second Audio data may be the same as or different from the first Audio data).

And S3402, the DV APP in the mobile phone issues the equipment selection strategy of the first slave equipment to the AudioPolicy.

In some embodiments of the present application, the device selection policies of a plurality of audio output devices may be pre-stored in the mobile phone. The device selection policy for each audio output device is used to indicate the playback capabilities of this type of device for different types of audio data. For example, the mobile phone may store device selection strategies for four types of audio output devices, namely, a speaker type device, a large screen type device, a vehicle type device and a wearable device in advance.

Then, after the handset establishes a network connection with the first slave device (e.g. loudspeaker 1), the DV APP may obtain the identification of loudspeaker 1. And DV APP can be according to audio amplifier 1's sign discernment audio amplifier 1 is audio amplifier class equipment. Further, as also shown in fig. 37, the DV APP may send a pre-stored device selection policy (i.e., device selection policy 1) for the enclosure class device to AudioPolicy. AudioPolicy may associate device selection policy 1 issued by DV APP with sound box 1, and at this time, device selection policy 1 may be bound with sound box 1 as a device selection policy of sound box 1.

For example, device selection policy 1 of sound box 1 (i.e., device selection policy of sound box type device) may be as shown in table 6, where the playback capability of sound box 1 for each type of audio data is recorded in device selection policy 1. For example, the device selection policy 1 allows audio data of MUSIC type and RING type to be played on the speaker 1, and in this case, the audio output device for the audio data of MUSIC type and RING type is the speaker 1. For another example, audio data of the SYSTEM type and NOTIFICATION type is not allowed to be played on the sound box 1 in the device selection policy 1. For a certain type of audio data that is not allowed to be played, the device selection policy 1 may further set an audio output device for the type of audio data, that is, a specific audio output device that is allowed to play the type of audio data. For example, the SYSTEM type audio data is not allowed to be played on the sound box 1, and the audio output device of the audio data is a mobile phone, that is, the SYSTEM type audio data is allowed to be played by using the speaker of the mobile phone itself.

TABLE 6

In addition, the user can also manually modify the device selection strategy 1 of the sound box 1 through the DV APP. For example, as shown in fig. 38, the DV APP may display a setting interface 3801 of an audio switching function to the user. A switch button 3802 for an audio switching function is provided in the setting interface 3801. If the user turns on the switch button 3802, which indicates that the user wishes to turn on the audio switching function, the mobile phone may allow the audio in the mobile phone to be switched to other audio output devices for playing. Also, the user may further set a device selection policy of the current audio output device in the setting interface 3801. For example, the user may select a specific type of audio data that is allowed to be played on loudspeaker 1 and a specific type of audio data that is not allowed to be played on loudspeaker 1 in setup interface 3801. For audio data that is not allowed to be played on the sound box 1, the user may further set an audio output device for this type of audio data by clicking the button 3803.

The DV APP may update the device selection policy 1 of the sound box 1 shown in table 6 according to a selection of the user in the setting interface 3801, and issue the updated device selection policy 1 to AudioPolicy. And determining whether the current audio data is allowed to be played on the sound box 1 or not by AudioPolicy according to the updated equipment selection strategy 1.

In this way, the user can set or modify the audio types allowed to be played on each audio output device in the setting interface 3801 according to the needs of the user, that is, a corresponding device selection policy is set for each audio output device, so that various types of audio data can be distributed to the corresponding audio output devices by the mobile phone according to the settings of the user for playing.

Of course, the mobile phone may also store the device selection policy set by the user for the sound box 1. Subsequently, when the mobile phone establishes network connection with the sound box 1 again, the mobile phone may find the stored device selection policy of the sound box 1 according to the identifier of the sound box 1, and determine whether to output the current audio data to the sound box 1 for playing by using the device selection policy.

In some application scenarios, the handset may have multiple audio output devices at the same time, i.e., there may be multiple current slave devices of the handset. For example, the mobile phone may establish a network connection with the speaker 1 and the speaker 2 at the same time. At this time, the DV APP in the mobile phone may issue both the device selection policy of the sound box 1 and the device selection policy of the sound box 2 to the AudioPolicy according to the above method. For example, since loudspeaker 1 and loudspeaker 2 are both loudspeaker-type devices, the device selection strategy for loudspeaker 1 may be the same as the device selection strategy for loudspeaker 2. Subsequently, the user may further set a device selection policy for sound box 1 and/or sound box 2 in setting interface 3801 shown in fig. 38. In some embodiments, when audio enclosure 1 and audio enclosure 2 support playing a certain type of audio data simultaneously, AudioPolicy may send the current audio data to both audio enclosure 1 and audio enclosure 2, or AudioPolicy may send the current audio data to one of audio enclosure 1 and audio enclosure 2. For example, AudioPolicy may send the current audio data to one of sound box 1 and sound box 2 that is closer to the user for playing, which is not limited in this embodiment of the present application.

In other embodiments of the present application, the handset may default to allowing each audio output device to play any type of audio data. Then, after the handset establishes a network connection with the first slave device (e.g., loudspeaker 1), the DV APP may send the device selection policy shown in table 7 to AudioPolicy. In the device selection strategy shown in table 7, loudspeaker 1 allows any type of audio data to be played. Subsequently, the mobile phone may update the device selection policy of the sound box 1 shown in table 7 according to the selection of the user in the setting interface 1201, and issue the updated device selection policy of the sound box 1 to AudioPolicy. And determining the specific audio output equipment of the current audio data by the AudioPolicy according to the updated equipment selection strategy of the loudspeaker box 1.

TABLE 7

Of course, the device selection policy of the audio output device may also be obtained by the mobile phone from the server, or the device selection policy of the audio output device may also be set in advance in the mobile phone by a developer, which is not limited in this embodiment of the present application.

And S3403, when the mobile phone runs the audio APP, determining whether the first slave device allows playing of the first audio data from the audio APP by the AudioPolicy according to the device selection policy of the first slave device.

It should be noted that, the mobile phone may start to run the audio APP before establishing the network connection with the first slave device, or may start to run the audio APP after establishing the network connection with the first slave device, that is, there is no sequential relationship between the process of running the audio APP by the mobile phone and steps S3401 to S3402. If the mobile phone runs the audio APP before the network connection with the first slave device is established, audio data is generated, and the mobile phone can play the audio data by using a loudspeaker of the mobile phone since the mobile phone is not connected with other audio output devices.

In step S3403, as also shown in fig. 37, the handset may call an audio player (e.g., AudioTrack) while running the audio APP, and input currently generated audio data (e.g., audio data 1) into an audio processor (e.g., AudioFlinger). Also, the audio APP may set the type of audio data 1 in the AudioTrack. For example, the audio APP may set a mode parameter to 01 in the AudioTrack to indicate that the type of audio data 1 currently input to the AudioTrack is MUSIC type audio data.

After the AudioFlinger receives the audio data from the audio APP, the specific type of audio data 1 can be identified by reading the above mode parameters in the AudioTrack. Further, the AudioFlinger may send a query request to AudioPolicy, where the query request may include a type identifier (e.g., 01) of the audio data 1, so as to request the AudioPolicy to query whether the slave (i.e., the first slave) currently connected to the handset allows playing of the audio data 1 identified as 01 type.

After receiving the query request sent by the audioFlinger, the audioPolicy may query, according to the type identifier of the audio data in the query request, whether the first slave device allows to play the audio data 1 in the device selection policy of the first slave device issued by the DV APP. If the first slave device allows the audio data 1 to be played, the handset may continue to perform the following step S3404, and if the first slave device does not allow the audio data 1 to be played, the handset may continue to perform the following steps S3405 to S3406.

In other embodiments, the DV APP of the mobile phone may also issue the device selection policies of various audio output devices to the AudioPolicy in advance. After the DV APP establishes a network connection with the first slave device, the DV APP may indicate the identity of the first slave device to AudioPolicy. Thus, AudioPolicy may determine the device type of the first slave device based on the identity of the first slave device. Furthermore, the AudioPolicy may find a device selection policy corresponding to the device type of the first slave device among the device selection policies of the various audio output devices, and determine the found device selection policy as the device selection policy of the first slave device. Or, the device selection policy of each audio output device may also be stored in the AudioPolicy in advance, and there is no need for the DV APP to issue the device selection policy of the audio output device to the AudioPolicy, which is not limited in this embodiment of the present application.

And S3404, if the first slave device allows the first audio data to be played, the AudioPolicy instructs the AudioFlinger to send the first audio data to the first slave device.

Still taking the first slave device as the speaker 1 for example, referring to the device selection policy 1 shown in table 6, if the type of the audio data 1 generated by the current audio APP is MUSIC or RING, the AudioPolicy may determine that the speaker 1 allows playing the audio data 1. For example, when the audio APP is MUSIC APP, the type of the audio data 1 generated when the mobile phone runs the MUSIC APP to play a song is MUSIC. The AudioFlinger, upon receiving the audio data 1, may request AudioPolicy to query whether to play MUSIC type audio data 1 on the loudspeaker 1. At this time, AudioPolicy may determine that audio data 1 may be played by the slave device (i.e., speaker 1) of the handset according to the device selection policy shown in table 6. Further, AudioPolicy may send a first indication message to AudioFlinger, instructing AudioFlinger to send currently generated audio data 1 to the sound box.

Then, as also shown in fig. 37, after receiving the first indication message sent by AudioPolicy, the AudioFlinger may call the DMSDP audiohal 3701 corresponding to the sound box 1 in the HAL, and send the Audio data 1 to the sound box 1 through the DMSDP audiohal 3701, so that the sound box 1 plays the Audio data 1 of MUSIC type.

Of course, before the Audio flinger sends the Audio data 1 to the sound box 1 through the DMSDP Audio HAL 3701, the Audio flinger may also perform processing operations such as parsing, sampling, encoding, decoding, encapsulating, or decapsulating on the Audio data 1, and send the processed Audio data 1 to the sound box 1. Similarly, after the sound box 1 receives the audio data 1 sent by the mobile phone, the audio data 1 may also be played after being subjected to processing operations such as parsing, sampling, encoding, decoding, encapsulating or decapsulating, and the like.

And S3405, if the first slave device does not allow the first audio data to be played, determining that the first audio data is played by the first device according to the device selection strategy of the first slave device by the Audio policy.

S3406, AudioPolicy instructs the AudioFlinger to transmit the first audio data to the first device.

Corresponding to step S3404, in steps S3405 to S3406, still taking the first slave device as the sound box 1 for example, see the device selection policy 1 of the sound box 1 shown in table 6, if the type of the audio data 1 generated by the current audio APP is SYSTEM or NOTIFICATION, the AudioPolicy may determine that the sound box 1 does not allow playing of the audio data 1.

For example, as shown in fig. 39, when the mobile phone runs the short message APP, if the short message APP receives a new short message, the short message APP may be used as an audio APP to input a generated short message prompt tone (i.e., audio data 2) to the AudioFlinger through an AudioTrack, and the short message APP may set the type of the audio data 2 in the AudioTrack to be NOTIFICATION. After receiving the audio data 2, the AudioFlinger may request the AudioPolicy to inquire whether the sound box 1 allows the audio data 2 of the NOTIFICATION type to be played. Further, AudioPolicy may determine that audio enclosure 1 does not allow playback of NOTIFICATION type audio data 2 according to device selection policy 1 shown in table 6. Also, AudioPolicy may further determine an audio output device for audio data 2, i.e., a first device that is allowed to play a NOTIFICATION type, in device selection policy 1 shown in table 6.

For example, the audio output device in which the NOTIFICATION type audio data is set in the device selection policy 1 shown in table 6 is a cellular phone. Then, AudioPolicy may send a second indication message to AudioFlinger, instructing AudioFlinger to send audio data 2 from short message APP to the mobile phone, that is, the first device playing audio data 2 is the mobile phone. Still as shown in fig. 39, after receiving the second indication message sent by AudioPolicy, the AudioFlinger may call a Primary HAL corresponding to a speaker of the mobile phone in the HAL, and send the audio data 2 to the speaker of the mobile phone through the Primary HAL for playing.

It can be seen that, after the audio function of the mobile phone is switched to the sound box, the mobile phone does not send all the audio data generated by itself or received indiscriminately to the sound box 1 (i.e. the slave device) for playing, and the mobile phone can send one or more types of audio data suitable for playing on the sound box 1 to the sound box 1 for playing according to the device selection policy of the sound box 1, and send other audio data to other audio output devices for playing. Therefore, when the mobile phone switches the audio frequency, the output flow direction of each audio data can be accurately controlled by combining the equipment characteristics of the slave equipment, and the audio data can be selectively switched to the corresponding audio output equipment for playing, so that some audio data which are not suitable for being played on the slave equipment of the current mobile phone can be filtered out for a user.

In some embodiments, if the device selection policy 1 of the sound box 1 (i.e., the slave device) sets that the audio data 2 of the NOTIFICATION type is not allowed to be played, the AudioPolicy may further dynamically determine the audio output device of the audio data 2 at this time according to a certain selection policy when determining the first device playing the audio data 2. For example, AudioPolicy may determine, according to a principle of priority after back access, an audio output device that is recently accessed to a mobile phone except for a sound box as an audio output device of the audio data 2, that is, a first device playing the audio data 2.

For example, as shown in fig. 40, the audio output device of the cellular phone is the cellular phone itself from the time T0, that is, the audio is played with the speaker of the cellular phone itself. At time T1(T1 > T0), if the handset is connected to the bluetooth headset, the audio output device at that time of the handset is updated to the bluetooth headset. At time T2(T2 > T1), if the user inserts the wired headset into the headset interface of the cellular phone again, the audio output device of the cellular phone at this time is updated to the wired headset. At the time of T3(T3 > T2), if the user selects the sound box 1 as the slave device for audio switching of the mobile phone, and triggers the mobile phone to establish network connection with the sound box 1, the audio output device of the mobile phone at this time is updated to the sound box 1.

It can be seen that the later in time the audio output device of the handset is accessed (i.e., the audio output device of the handset that was recently accessed) has the higher priority. Then, when the audio data 2 is not allowed to be played on the sound box 1, the AudioPolicy may select the audio output device with the highest priority except for the sound box 1 as the audio output device for playing the audio data 2, that is, select the wired headset to play the audio data 2. Further, AudioPolicy may instruct AudioFlinger to send audio data 2 to wired headphones for playback. Of course, those skilled in the art may set the device selection policy of the audio output device according to practical experience or practical application scenarios, which is not limited in this application.

By executing steps S3401-S3406, after the mobile phone is used as a master device to establish a network connection with the first slave device, different types of audio data generated by the audio APP in the mobile phone can be dynamically and selectively switched to the first slave device for playing according to a device selection policy of the first slave device, so that some audio data which are not suitable for playing on the first slave device are filtered out for a user, and a better audio use experience is provided for the user.

Subsequently, if the user modifies the first slave device connected to the mobile phone into the second slave device, the mobile phone may also switch the audio data generated by the audio APP at this time to the second slave device for playing according to the device selection policy of the second slave device, similar to the case where the mobile phone switches the audio data to the first slave device for playing.

Illustratively, as shown in FIG. 41, the handset may switch audio data to the second slave device for playback by performing the following steps S4101-S4106.

S4101, the mobile phone disconnects the network connection with the first slave device, and establishes the network connection with the second slave device by running the DV APP.

Still taking the first slave device as the sound box 1 for example, after the mobile phone switches the audio function to the sound box 1, the user may also manually modify the slave device currently performing audio switching with the mobile phone on the mobile phone. For example, as shown in (a) in fig. 42, the cellular phone may display the control center 4201 in response to an operation of the user to turn on the control center. The control center 4201 includes a switch button 4202 for audio switching function. When a user wishes to modify a slave device of a current handset, clickable toggle button 4202 queries one or more candidate devices that may currently be slave devices of the handset. As shown in fig. 42 (b), upon detecting that the user clicks the switch button 4202, the mobile phone may display the detected one or more candidate devices in the control center 4201. At this time, the user may select a slave device (i.e., a second slave device) for audio switching with the handset in the control center 4201.

Or, if the mobile phone detects that the user starts the screen projecting function of the mobile phone, the mobile phone may also modify the slave device performing audio switching with the mobile phone in response to the operation of the user starting the screen projecting function, because the mobile phone is required to simultaneously switch the audio and the image to other devices for playing in the screen projecting scene, that is, the audio switching is also required in the screen projecting scene.

For example, as shown in fig. 43, when the mobile phone runs the WeChat APP, the application interface 4300 of the WeChat APP may be displayed. In the application interface 4300, the user may open a dialog interface with a certain contact (for example, Sara or Mike) and listen to the voice sent by the user or the contact. If the user needs to use the screen projection function, as shown in fig. 43, the user may turn on the control center 4301 of the mobile phone in the application interface 4300. A card 4302 may be disposed in the control center 4301, and the card 4302 is used to show one or more nearby candidate devices that may be subjected to screen projection and detected by the mobile phone. If the mobile phone has started the screen projection function, the mobile phone may further identify the device currently projecting the screen with the mobile phone in the card 4302 by highlighting or the like. If it is detected that the user selects an electronic device (e.g., a television) in the card 4302, indicating that the user wishes to turn on the screen projection function, the image and audio generated by the mobile phone at this time are projected to the television. At this time, the handset may disconnect the network connection that has been established with the sound box 1 (i.e., the first slave device), and establish a network connection with the television as the second slave device.

Also, similar to step S3401, as shown in fig. 44, the DV APP in the mobile phone may obtain the Audio capability parameters of the television, and create a corresponding DMSDP Audio HAL in the HAL according to the Audio capability parameters of the television. Or, the mobile phone may update the DMSDP Audio HAL created for the sound box in step S3401 according to the Audio capability parameter of the television, so that the updated DMSDP Audio HAL matches the Audio capability of the television. Subsequently, in a screen projection scene, the mobile phone can transmit Audio data to the television through the updated DMSDP Audio HAL. In addition, in a screen projection scene, the mobile phone can send display data (for example, an application interface 4300 of the WeChat APP) in the mobile phone to the television for display according to an existing screen projection mode, and the embodiment of the present application does not limit this.

S4102, the DV APP in the mobile phone issues the device selection policy of the second slave device to the AudioPolicy.

Similar to step S3402, after the mobile phone determines the tv as the current slave device (i.e. the second slave device) and establishes the network connection, as shown in fig. 44, the DV APP may issue a device selection policy (i.e. device selection policy 2) for the tv to AudioPolicy. For example, the DV APP may obtain the identity of the television, thereby determining that the currently connected second slave device is a large screen class device. Furthermore, the DV APP may issue the device selection policy stored in advance for the large-screen device as the device selection policy 2 of the television to the AudioPolicy. Subsequently, AudioPolicy may determine the specific audio output device for the audio data in the current handset according to device selection policy 2 of the television.

For example, the device selection policy 2 of the television may be as shown in table 8, and different from the device selection policy 1 shown in table 6, identifiers of one or more applications, for example, Package names (Package names) of the applications, may also be set in the device selection policy 2 of the television. That is, the mobile phone may further set, in the device selection policy 2, the playing capabilities of different applications for different types of audio data, respectively. For example, for the WeChat APP, the television allows the WeChat APP to play audio data of a MUSIC type and a COMMUNICATION type, and does not allow the WeChat APP to play audio data of a NOTIFICATION type and a DTMF type. For video APPs, the television allows the WeChat APP to play MUSIC type audio data, etc. The specific application included in the device selection policy 2 may be preset by a developer. Or, when a user installs a new application in the mobile phone, the mobile phone may obtain the device selection policy of the application in different audio output devices from the server, so that the mobile phone may dynamically update the device selection policy of each audio output device according to the application installed in the mobile phone, which is not limited in this embodiment of the present application.

TABLE 8

Therefore, when the mobile phone runs the WeChat APP, the voice or media sound generated by the WeChat APP can be switched to the television to be played, the message prompt sound and the key sound generated by the WeChat APP can be shunted to the mobile phone to be played, and the message prompt sound and the key sound generated by the WeChat APP are prevented from disturbing the screen projection process of the mobile phone in the television. Therefore, the mobile phone can output different types of audio data generated by one application to corresponding audio output devices according to the device selection strategy to be played by taking the application as granularity, so that different audio output devices can play specific types of audio data in specific applications in a targeted manner, and the audio use experience of a user is improved.

In other embodiments, a device selection policy as shown in table 9 may also be set in AudioPolicy. The device selection policy sets different types of audio output devices for audio data for different applications. For example, when the wechat APP generates MUSIC type audio data, the audio output device of the audio data may be a mobile phone, a sound box, a television or a headset. For another example, when the music APP generates audio data of a music type, the audio output device of the audio data may be a mobile phone, a sound box, or a television. Subsequently, AudioPolicy may determine the audio output devices for various types of audio data generated by the running audio APP according to the device selection policy shown in table 9.

TABLE 9

In other embodiments, the user may also manually modify the device selection policy of the current slave device in the handset. As shown in fig. 45 (a), when the mobile phone switches audio data using a television as a slave device, a card 4502 for switching audio is displayed in the control center 4501, and a setting button 4503 for switching audio is provided in the card 4502. If it is detected that the user clicks the setting button 4503, the DV APP can display a setting interface 4504 when the television is a slave device, as shown in (b) in fig. 45. Similar to fig. 38, a switch button 4505 for an audio switching function may be provided in the setting interface 4504. When the user turns on the switch button 4505, the user can set which types of audio data the television can receive in the setting interface 4504, or which types of audio data the television can receive when the mobile phone is running different applications. Further, the DV APP can update the device selection policy 2 of the television shown in table 8 according to the selection of the user in the setting interface 4504, so that the user can dynamically specify the type of audio data output in each audio output device.

S4103, when the mobile phone runs the audio APP, the AudioPolicy determines whether the second slave device allows playing of second audio data from the audio APP according to the device selection policy of the second slave device.

Similar to step S3403, as also shown in fig. 44, the AudioFlinger may receive audio data from different audio APP generations. When the slave device of the mobile phone is switched from the sound box 1 to the television, the AudioFlinger may request AudioPolicy to inquire whether the currently received audio data is allowed to be played in the television. Further, AudioPolicy may determine whether the current audio data is allowed to be played in the television according to device selection policy 2 of the television.

In some scenarios, the AudioFlinger may receive multiple audio data generated by the same or different audio APPs at the same time. For example, as shown in fig. 46, at time T1, the user opens a music APP in the mobile phone to start playing a song, at time T2(T2 > T1), the user opens a voice message sent by a contact in the WeChat APP of the mobile phone to start playing a voice, and during the time period T3-T4(T4 > T3 > T2), the user starts playing a key tone when clicking a key with a keyboard in the WeChat APP. Then, the AudioFlinger of the handset can simultaneously receive the audio data a and B of the MUSIC type from the WeChat APP and the audio data C of the MUSIC type from the MUSIC APP during the time period T3-T4. Still as shown in fig. 44, AudioFlinger may query AudioPolicy for whether audio data a, audio data B and audio data C are allowed to be played in the television. Further, AudioPolicy may determine whether the current audio data a, audio data B, and audio data C are allowed to be played in the television according to the device selection policy 2 shown in table 8.

S4104, if the second slave device allows the second audio data to be played, the AudioPolicy instructs the AudioFlinger to send the second audio data to the second slave device.

Similarly to step S3404, as also shown in fig. 44, AudioPolicy may determine that the television is allowed to play MUSIC type audio data a and MUSIC type audio data C, but is not allowed to play DTMF type audio data B, according to device selection policy 2 shown in table 8. AudioPolicy may then send a first indication message to AudioFlinger instructing AudioFlinger to send audio data a and audio data C to the loudspeaker. Furthermore, the AudioFlinger may respond to the first indication message, mix the two Audio data, i.e., the Audio data a and the Audio data C, into Audio data a + C, and send the mixed Audio data a + C to the television through the DMSDP Audio HAL. Or, the AudioFlinger may not perform mixing processing on the Audio data a and the Audio data C, send the Audio data a and the Audio data C that are not mixed to the television through the DMSDP Audio HAL, and play the Audio data a from the WeChat APP and the Audio data C from the music APP in a screen-casting scene by the television.

S4105, if the second slave device does not allow the second audio data to be played, the AudioPolicy determines that the second audio data is played by the second device according to the device selection policy of the second slave device.

S4106, AudioPolicy instructs the AudioFlinger to transmit the second audio data to the second device.

In steps S4105-S4106, AudioPolicy may determine that the television does not allow playing of DTMF-type audio data B according to the device selection policy 2 shown in table 8, as also shown in fig. 44. AudioPolicy may then further determine the particular audio output device allowed to play the DTMF type, i.e., the second device to play audio data B, in the device selection policy shown in table 8.

For example, in the device selection policy shown in table 8, the audio output device that sets DTMF type audio data in the WeChat APP is a mobile phone. AudioPolicy may then send a second indication message to AudioFlinger instructing AudioFlinger to send the currently generated audio data B to the handset. Then, as also shown in fig. 44, after the AudioFlinger responds to the second indication message, the Primary HAL corresponding to the speaker of the mobile phone in the HAL may be called, and the audio data B is sent to the speaker of the mobile phone through the Primary HAL for playing.

By executing steps S4101-S4106, after the mobile phone is used as a master device to establish a network connection with a second slave device, different types of audio data generated by the audio APP in the mobile phone can be dynamically and selectively switched to the second slave device for playing according to a device selection policy of the second slave device, so that some audio data which are not suitable for playing on the second slave device are filtered out for a user, and a better audio use experience is provided for the user.

In a distributed audio scenario, multiple audio data may be simultaneously generated in a host device (e.g., the handset described above). For example, when the mobile phone runs the map APP, the map APP can generate navigation voice in the navigation mode; meanwhile, the mobile phone can also run a music APP to play songs. At this time, the handset can play the navigation voice (i.e., audio data 1) from the map APP and the song (i.e., audio data 2) from the music APP at the same time. Subsequently, if the short message APP of the mobile phone receives a new short message, the mobile phone needs to play an alert tone (i.e. audio data 3) of the short message. At this time, the mobile phone can play the audio data 3 simultaneously in addition to the audio data 1 and the audio data 2.

In the embodiment of the application, the mobile phone serving as the master device can send the generated audio data to one or more slave devices for playing, so that the audio switching function among the multiple devices is realized. Therefore, when the mobile phone needs to output multiple paths of audio data, the mobile phone may mix the multiple paths of audio data and output the mixed audio data to the slave device for playing, but the multiple paths of audio data after mixing may have a distortion phenomenon, which also causes a distortion phenomenon when the mixed audio data is played on the slave device.

Illustratively, the audio output device (i.e., the slave device) of the handset may be a speaker. The handset may generate audio data 1 and audio data 2 simultaneously while in operation. In the prior art, as shown in fig. 47, when a mobile phone projects audio data 1 and audio data 2 to a sound box, the audio data 1 and the audio data 2 need to be mixed first, so that audio signals in the audio data 1 and the audio data 2 are superimposed to obtain audio data 3 after mixing. Subsequently, the mobile phone can pack the audio data 3 into individual data packets and send the data packets to the sound boxes, and the sound boxes can play corresponding audio after analyzing the audio data 3 in each data packet.

Through sound mixing, multi-channel audio data can be converted into one-channel audio data. However, in the mixing process, due to processing such as sampling, the audio characteristics of the audio data 1 and the audio data 2 before mixing cannot be completely retained in the mixed audio data 3, for example, the loudness and frequency of the audio data 1 or the audio data 2 change in the mixed audio data 3, so that the audio data 3 is distorted. That is, a component of the audio data 1 in the audio data 3 may deviate from the audio data 1 before mixing, resulting in that the audio characteristics of the audio data 1 before mixing are different from the audio characteristics of the audio data 1 contained in the audio data 3 after mixing; similarly, a component of the audio data 2 in the audio data 3 may deviate from the audio data 2 before mixing, so that the audio characteristic of the audio data 2 before mixing is different from the audio characteristic of the audio data 2 included in the audio data 2 after mixing, so that a distortion phenomenon occurs when the sound box plays the audio data 3, and the audio use experience of the user is affected.

In this regard, in the embodiment of the present application, a mixing policy may be preset in a host device (e.g., a mobile phone), and the mixing policy may be used to indicate whether different types of audio data allow mixing. For example, as shown in the mixing strategy shown in table 1, the user generally has higher requirements on the audio quality of MUSIC type, so that it can be preset in the mixing strategy that MUSIC type MUSIC is not allowed to be mixed with other audio data, and NOTIFICATION type NOTIFICATION tone and DTMF type touch tone are allowed to be mixed with other audio data. Subsequently, when the mobile phone generates multiple channels of audio data and the multiple channels of audio data need to be played on the same audio output device, the mobile phone may determine which audio data do not need to be mixed and which audio data need to be mixed according to the mixing strategy shown in table 1 according to the type of each channel of audio data.

Watch 10

Type (B)	Whether or not to allow mixing
		NOTIFICATION (Notification)	Allow for
DTMF (Dual tone multi-frequency)	Allow for
		MUSIC (MUSIC)	Is not allowed to
……	……

It is to be understood that only the type of audio data that allows mixing may be set in the above-described mixing policy (e.g., the mixing policy shown in table 10). At this time, other types of audio data may default to audio data that does not allow mixing. Alternatively, only the type of audio data that is not permitted to be mixed may be set in the above-described mixing strategy. At this time, other types of audio data may be defined as audio data that allows mixing by default, and the embodiment of the present application does not limit this.

Still take the mobile phone as the master device for example, after the mobile phone establishes network connection with the sound box, the sound box can be used as the slave device for audio switching. For example, the mobile phone generates MUSIC type audio data 1 when running MUSIC APP, and if only audio data 1 is the same audio data, the mobile phone can send the audio data 1 to the speaker for playing. Of course, the mobile phone may also perform parsing, sampling, encoding, decoding, encapsulating, or decapsulating, and the like on the audio data 1 before sending the audio data 1, which is not limited in this embodiment of the application. Subsequently, in the process of playing the audio data 1 by the loudspeaker, if the WeChat APP of the mobile phone receives a new short message, the WeChat APP can generate the audio data 2 of the NOTIFICATION type. At this time, the mobile phone needs to send two paths of audio data from the WeChat APP and the music APP to the sound box.

Then, the mobile phone can determine, through the sound mixing strategy shown in the lookup table 10, that the audio data 1 of MUSIC type is not allowed to be mixed with other audio data, so that the audio data 1 and the audio data 2 do not need to be mixed, that is, the two paths of audio data need to be independently transmitted to the loudspeaker box. Further, as shown in fig. 48, the mobile phone may pack (may also be referred to as a packet) audio data 1 from the music APP to obtain a first packet 4801 (there may be a plurality of first packets 4801) corresponding to the audio data 1, and the mobile phone may pack (may also be referred to as a packet) audio data 2 from the WeChat APP to obtain a second packet 4802 (there may be a plurality of second packets 4802) corresponding to the audio data 2. Subsequently, the mobile phone may send the first data packet 4801 and the second data packet 4802 to the sound box, and the sound box may obtain the audio data 1 and the audio data 2 that are not subjected to audio mixing by analyzing the received first data packet 4801 and second data packet 4802. That is to say, the mobile phone can send the audio data 1 and the audio data 2 to the sound box relatively independently without mixing the two paths of audio data.

Therefore, the audio data 1 and the audio data 2 acquired by the sound box from the mobile phone are not subjected to sound mixing processing, so that the audio data 1 can more truly reflect the audio characteristics of the song in the music APP, and the audio data 2 can more truly reflect the audio characteristics of the notification sound of the short message APP. That is, the audio data 1 and the audio data 2 acquired by the sound box do not have distortion phenomenon caused by the sound mixing processing of the mobile phone. Therefore, distortion phenomena generated when the sound box plays the audio data 1 and the audio data 2 are correspondingly reduced, so that the distortion condition of multi-channel audio data during cross-equipment playing is reduced, and the audio output quality of an audio switching scene is improved.

In some embodiments, in the process of packaging the audio data 1, the mobile phone may further add feature information of the audio data 1 to the first data packet 4801, for example, the identifier of the audio data 1 is 01, the type of the audio data 1 is MUSIC type, the volume level of the audio data 1 is 5, and so on. The above feature information can reflect the actual audio features of the audio data 1. Thus, after the speaker parses the first data packet 4801, not only the unmixed audio data 1 can be obtained, but also the audio characteristics of the audio data 1 can be obtained according to the characteristic information in the first data packet 4801. Subsequently, when the sound box plays the audio data 1, the audio characteristics of the audio data 1 can be more truly restored according to the characteristic information of the audio data 1, and the distortion phenomenon in the audio switching process is reduced.

Similarly, in the process of packaging the audio data 1 by the mobile phone, the feature information of the audio data 2 may also be added to the second data packet 4802, so that the sound box may obtain the audio data 2 that is not mixed, and the audio features of the audio data 2 may also be obtained according to the feature information in the second data packet 4802, so as to reduce the distortion phenomenon in the audio switching process. The specific process of the mobile phone for packaging the multiple channels of audio data will be described in detail in the following embodiments, and therefore will not be described herein.

Of course, when the sound box plays the audio data 1 and the audio data 2, the sound box may also perform processing operations such as parsing, sampling, encoding, decoding, encapsulating or decapsulating on the audio data 1 and the audio data 2, which is not limited in this embodiment of the present application.

In this embodiment of the application, as shown in fig. 49, after the DV APP of the mobile phone creates a DMSDP Audio HAL for the slave device of the mobile phone, the DV APP may also issue a mixing policy of different types of Audio data to AudioPolicy. For example, the DV APP may issue, to AudioPolicy, a mixing policy shown in table 10 above, in which whether mixing is allowed or not for each type of audio data is recorded.

Then, when the AudioFlinger receives at least two different types of audio data, as shown in fig. 49, the AudioFlinger requests the AudioPolicy to query which audio data in the current multiple audio data need to be transmitted independently and which audio data need to be mixed and then transmitted. In response to the request of AudioFlinger, AudioPolicy may query which audio data need to be independently transmitted and which audio data need to be mixed and then transmitted according to the mixing strategy shown in table 10.

For example, if the AudioFlinger receives both the audio data B of the NOTIFICATION type of the audio data A, DTMF type and the audio data C of the MUSIC type, the AudioFlinger may send a first request to AudioPolicy requesting the AudioPolicy to determine whether the three pieces of audio data need to be mixed. Further, AudioPolicy may refer to the mixing policy shown in the first request lookup table 10, thereby determining that MUSIC type audio data C does not need to be mixed, and NOTIFICATION type audio data a and DTMF type audio data B need to be mixed. Then, AudioPolicy may output a mixing indication to AudioFlinger, in which it is indicated that audio data a and audio data B are mixed into one audio data, and audio data C is not mixed with audio data a and audio data B. The AudioFlinger can respond to the audio mixing instruction, perform audio mixing processing on the audio data a and the audio data B, and not perform audio mixing on the audio data C, so that the AudioFlinger can finally output two paths of audio data, one path is the audio data a + B after audio mixing, and the other path is the audio data C which is not subjected to audio mixing.

Subsequently, the AudioFlinger may call the DMSDP Audio HAL to output the Audio data a + B and the Audio data C to the slave device for playing, respectively. Therefore, for MUSIC (namely audio data C) with high audio requirement in the MUSIC type, the master device can independently transmit the audio data C to the slave device without mixing with other audio data, so that the audio data C received by the slave device cannot lose some audio characteristics in the audio data C due to mixing, the audio distortion phenomenon caused by mixing processing is avoided, the distortion condition of multi-channel audio data during cross-device playing is reduced, and the audio output quality of an audio switching scene is improved.

It can be seen that, based on the audio architecture shown in fig. 49, the host apparatus can set a mixing policy corresponding to various types of audio data in advance. Therefore, when the multi-channel audio data are required to be output to the slave equipment, the master equipment can select certain audio data in the multi-channel audio data not to be subjected to audio mixing processing according to the audio mixing strategy and independently transmit the audio data to the slave equipment for playing, so that the distortion phenomenon caused by the audio mixing processing on certain type of audio data is avoided, and the audio output quality of an audio switching scene is improved.

Still take a mobile phone as an example of a main device, an audio control method provided by the embodiments of the present application will be described below with reference to the accompanying drawings.

As shown in fig. 50, an audio control method provided in an embodiment of the present application includes the following steps:

s5001, the mobile phone establishes network connection with the first slave device by running the DV APP.

For example, the user can switch the audio output device when the mobile phone outputs audio in the mobile phone. For example, as shown in (a) in fig. 51, the mobile phone can display a switch button 5101 for audio switching while running a music APP. If the user is detected to click the switch button 5101, the handset can search for one or more nearby candidate devices available for audio switching by running the DV APP. As shown in fig. 51 (b), the DV APP may display candidate devices such as a television, a speaker, and a headset, which are located on the same Wi-Fi network as the mobile phone, in the menu 5102. The user can select one of the candidate devices as an audio output device for the subsequent audio output of the mobile phone, namely, the slave device of the mobile phone.

For another example, the user may open the control center of the mobile phone and switch the audio output device in the control center when the mobile phone outputs audio. In response to an operation of the user to turn on the control center, as shown in (a) in fig. 52, the cellular phone may display the control center 5201. A switching button 5202 for audio switching is provided in the control center 5201. If it is detected that the user clicks the switch button 5202, the handset can also search for one or more candidate devices nearby that are available for audio switching by running the DV APP. As shown in fig. 52 (b), the DV APP can display the searched candidate devices, such as a television, a speaker, and a headphone, which are located on the same Wi-Fi network as the mobile phone, in the control center 5201. Similarly, the user can select one of the candidate devices as an audio output device for the subsequent audio output of the mobile phone, namely, a slave device of the mobile phone.

For example, if the user selects the speaker as the slave device of the mobile phone (i.e., the first slave device in step S5001), after detecting that the user selects the candidate device, i.e., the speaker, from the candidate devices, the DV APP may establish a network connection between the speaker and the speaker using the speaker as the slave device of the mobile phone. For example, the network connection may be a P2P connection based on a TCP (transmission control protocol) or a UDP (user datagram protocol), which is not limited in this embodiment.

It should be noted that, in step S5001, the network connection established between the handset and the slave device may specifically refer to a network connection of a service channel (i.e., a service connection). For example, the handset may have established a connection with the speaker over the Wi-Fi network before the user selects the speaker as a slave device of the handset, where the connection refers to a connection of a data channel (i.e., a data connection). After the user selects the sound box as the slave device of the mobile phone, the mobile phone and the sound box can establish service connection on the basis of the established data connection.

After the network connection between the mobile phone and the sound box is established, as shown in fig. 53, the DV APP can obtain the audio capability parameter of the sound box based on the network connection, and the audio capability parameter can reflect the audio output function specifically supported by the sound box. For example, the audio capability parameters may include sampling rate supported by the speaker, number of channels, sound effect mode, play delay, and the like. Furthermore, the DV APP may create a corresponding DMSDP Audio HAL at the HAL according to the Audio capability parameters of the loudspeaker. Subsequently, the mobile phone can transmit Audio data to the sound box through the DMSDP Audio HAL, so that the Audio function in the mobile phone is switched to the sound box for realization.

In some embodiments, the user may also select multiple slave devices for outputting the handset's audio. For example, the user may select the sound box and the television from the candidate devices as slave devices of the mobile phone, and at this time, the DV APP may establish a network connection with the sound box and the television respectively and obtain corresponding audio capability parameters. Further, the DV APP may create a first DMSDP Audio HAL corresponding to the loudspeaker at the HAL according to the Audio capability parameters of the loudspeaker, and the DV APP may create a second DMSDP Audio HAL corresponding to the television at the HAL according to the Audio capability parameters of the television. Subsequently, the mobile phone may send corresponding first Audio data to the sound box through the first DMSDP Audio HAL, and send corresponding second Audio data to the television through the second DMSDP Audio HAL (the second Audio data may be the same as or different from the first Audio data).

S5002, the DV APP in the mobile phone issues the sound mixing strategy of the first slave device to the Audio policy.

In some embodiments of the present application, mixing strategies of a plurality of audio output devices may be stored in advance in the mobile phone. The mixing strategy of each audio output device is used for indicating the mixing requirements of the devices of the type on different types of audio data. For example, the mobile phone may store mixing strategies of four types of audio output devices, namely, a speaker type device, a large screen type device, a vehicle-mounted type device and a wearable device in advance.

For example, as shown in table 11, mixing strategies of a speaker device and a large screen device are stored in the mobile phone in advance. In the mixing policy, a mixing policy is set whether to allow mixing when outputting each type of audio data to different types of audio output apparatuses. It can be seen that, unlike the mixing strategy shown in table 10, the mobile phone can set a corresponding mixing strategy for different types of audio output devices.

For example, in a mixing strategy of a speaker-type device, mixing of MUSIC-type audio data is not allowed. That is, MUSIC type audio data needs to be transmitted to a sound box separately without being mixed with other audio data. And in the sound mixing strategy of the speaker equipment, the sound mixing of the audio data of the NOTIFICATION type and the DTMF type with other audio data is allowed. For another example, in the mixing strategy of a large-screen type device, mixing of audio data of MUSIC type and NOTIFICATION type is not allowed. That is, MUSIC type audio data and NOTIFICATION type audio data need to be separately transmitted to the television without being mixed with other audio data.

TABLE 11

Then, after the handset establishes a network connection with the first slave device (e.g. loudspeaker 1), the DV APP may obtain the identification of loudspeaker 1. And DV APP can be according to audio amplifier 1's sign discernment audio amplifier 1 is audio amplifier class equipment. Further, as also shown in fig. 53, the DV APP may send a pre-stored mixing strategy (i.e., mixing strategy 1) of the speaker class device to AudioPolicy. AudioPolicy may associate sound mixing policy 1 issued by DV APP with sound box 1, and at this time, sound mixing policy 1 may be bound with sound box 1 as the sound mixing policy of sound box 1.

In addition, the user can also manually modify the sound mixing strategy 1 of the sound box 1 through the DV APP. For example, as shown in fig. 54, the DV APP can display a setting interface 5401 of the audio switching function to the user. A switch button 5402 for an audio switching function is provided in the setting interface 5401. If the user is detected to turn on the switch button 5402, which indicates that the user wishes to turn on the audio switching function, the mobile phone may allow the audio in the mobile phone to be switched to other audio output devices for playing. Also, the user may further set a mixing policy of the current audio output apparatus in the setting interface 5401. For example, the user can select a specific type of audio data that does not require mixing and a specific type of audio data that requires mixing in the setting interface 5401. For another example, if the user needs to set a mixing policy for more types of audio data, the user may click on the "add other" option 5403 in the setting interface 5401. The mobile phone can respond to the operation of clicking the option 5403 by the user to display more types of audio data, and the user can set whether to mix the audio data.

The DV APP may update the sound mixing policy 1 of the sound box 1 according to the selection of the user in the setting interface 5401, and issue the updated sound mixing policy 1 to AudioPolicy. And determining whether to mix different audio data generated by the audio APP or not by the Audio policy according to the updated mixing strategy 1.

In this way, a user can set or modify the audio types of the audio data that need to be independently transmitted to each audio output device in the setting interface 5401 according to the needs of the user, that is, a corresponding audio mixing policy is set for each audio output device, so that various types of audio data can be independently transmitted according to the settings of the user or transmitted to the corresponding audio output device for playing after being mixed.

Of course, the mobile phone may also store the sound mixing strategy set by the user for the sound box 1. Subsequently, when the mobile phone establishes network connection with the sound box 1 again, the mobile phone may find the saved sound mixing policy of the sound box 1 according to the identifier of the sound box 1, and determine whether to mix different audio data generated by the audio APP by using the sound mixing policy.

In other embodiments of the present application, the handset may default to every type of audio output device other than the handset itself to allow mixing of any type of audio data. Then, after the handset establishes network connection with the first slave device (e.g. loudspeaker 1), the DV APP may send the mixing strategy shown in table 12 to AudioPolicy. In the mixing strategy shown in table 13, any type of audio data sent by the handset to the loudspeaker 1 is allowed to be mixed. Subsequently, the mobile phone may update the sound mixing policy of the sound box 1 shown in table 12 according to the selection of the user in the setting interface 1101, and issue the updated sound mixing policy of the sound box 1 to AudioPolicy. And determining whether to mix different audio data generated by the audio APP or not by the Audio policy according to the updated mixing strategy of the sound box 1. Or, the mobile phone may default to set that each type of audio output device except the mobile phone itself does not allow any type of audio data to be mixed, and a subsequent user may set a mixing policy of the current slave device in the mobile phone according to the needs of the subsequent user.

TABLE 12

Of course, the mixing policy may also be obtained by the mobile phone from the server, or the mixing policy may also be set in advance in the mobile phone by a developer, which is not limited in this embodiment of the present application.

In other embodiments, the DV APP of the mobile phone may also issue the audio mixing policy of various audio output devices to the AudioPolicy in advance. After the DV APP establishes a network connection with a first slave device (e.g., enclosure 1), the DV APP may indicate to AudioPolicy the identity of enclosure 1. Thus, AudioPolicy may find a sound mixing policy corresponding to the device type of the sound box 1 from sound mixing policies of various audio output devices according to the identifier of the sound box 1, and determine the found sound mixing policy as the sound mixing policy of the sound box 1. Or, the audio mixing strategy of each audio output device may also be stored in the AudioPolicy in advance, and there is no need for the DV APP to issue the audio mixing strategy of the audio output device to the AudioPolicy, which is not limited in this embodiment of the present application.

S5003, after the first audio application in the mobile phone generates the first audio data, the AudioFlinger sends the first audio data to the first slave device for playing.

In step S5003, as also shown in fig. 53, if audio data (which may be referred to as first audio data) is generated while the handset is running the first audio APP (e.g., music APP), the first audio APP may call an audio player (e.g., AudioTrack), and input the currently generated first audio data into an audio processor (e.g., AudioFlinger). Also, the music APP may set the type of audio data 1 in the AudioTrack. For example, the audio APP may set a mode parameter corresponding to audio data 1 to 01 in the AudioTrack to indicate that the type of audio data 1 currently input to the AudioTrack is MUSIC type audio data.

After the AudioFlinger receives audio data (e.g., audio data 1) from the first audio APP, a specific type of the audio data 1 can be identified by reading the above mode parameter in the AudioTrack. And, after the AudioFlinger receives the audio data 1, it can be determined that several audio data are received at this time, that is, the number of audio data that need to be played on the slave at the same time. If the AudioFlinger only receives the audio data, no mixing process is needed no matter what type of audio data 1 is. Therefore, as shown in fig. 53, the AudioFlinger may call the DMSDP Audio HAL corresponding to the sound box in the HAL, and send the Audio data 1 to the sound box through the DMSDP Audio HAL, so that the sound box plays the Audio data 1.

It should be noted that before sending the audio data 1 to the sound box, the AudioPolicy may also set a corresponding audio policy according to the audio capability parameter of the sound box, and output the audio policy to the AudioFlinger. For example, AudioPolicy may set whether to sample audio data 1, add sound effects, etc. according to audio capability parameters of the loudspeaker. Further, the AudioFlinger may process the audio data 1 accordingly according to the audio policy output by AudioPolicy. Then, the AudioFlinger calls DMSDP Audio HAL to send Audio data 1 to the sound box, where the Audio data is the Audio data processed by the AudioFlinger according to the Audio policy.

S5004, after the second audio application in the mobile phone generates the second audio data, the AudioPolicy determines whether to mix the first audio data and the second audio data according to the mixing policy of the first slave device.

As shown in fig. 55, taking the second audio application as an example of the short message APP, during the process of playing a song by the music APP (i.e., the first audio application), if the short message APP receives a new short message, the short message APP may generate audio data 2 (i.e., second audio data) of the NOTIFICATION type. Similar to the first audio data, the second audio APP may also invoke AudioTrack to input the currently generated second audio data into the AudioFlinger.

After receiving the audio data 2 from the short message APP, the AudioFlinger can inquire whether other audio data need to be played on the slave device at this time. For example, if the AudioFlinger receives not only audio data 1 from the music APP but also audio data 2 from the short message APP at the same time, the AudioFlinger may determine that audio data 1 and audio data 2 need to be played on the current slave (i.e., a sound box) at the same time.

Also, the AudioFlinger can recognize that the audio data 1 is different in type from the audio data 2 by reading mode parameters respectively corresponding to the audio data 1 and the audio data 2 in the AudioTrack. For example, the AudioFlinger may determine that the audio data 1 is MUSIC type audio data and the audio data 2 is NOTIFICATION type audio data. Since the audio data that needs to be played on the sound box is multiple paths of audio data of different types, the AudioFlinger may send a query request to AudioPolicy, where the query request may include specific types of audio data 1 and audio data 2 to request AudioPolicy to determine whether to mix audio data 1 and audio data 2.

If the audio data 1 and the audio data 2 do not need to be mixed, the mobile phone can continue to execute the following steps S5005-S5006; if audio data 1 and audio data 2 need to be mixed, the handset can continue to perform the following steps S5007-S5009.

It should be noted that the first audio application and the second audio application may be the same application or different applications. For example, when the WeChat APP is used as the audio APP, the WeChat APP may play NOTIFICATION sound of a NOTIFICATION type or MUSIC of a MUSIC type, and the embodiment of the present application does not limit this. In addition, if the types of the first audio data and the second audio data are the same, which means that the audio characteristics of the first audio data and the second audio data are the same, and the distortion influence of the audio mixing on the first audio data and the second audio data is small, AudioPolicy may allow, by default, whether to mix the first audio data and the second audio data.

S5005, if the first audio data and the second audio data do not need to be mixed, the AudioPolicy sends a first mixing instruction to the AudioFlinger, where the first mixing instruction is used to instruct to independently transmit the first audio data and the second audio data.

Still taking the first slave device as the sound box 1 for example, after receiving the query request sent by the AudioFlinger, the AudioPolicy may determine, according to the types of the audio data 1 and the audio data 2, that the audio data 1 of the MUSIC type is not allowed to be mixed, that is, the audio data 1 needs to be transmitted independently, with reference to the mixing policy shown in table 11. Although the NOTIFICATION type audio data 2 allows mixing with other audio data, since there is no audio data supporting mixing other than the audio data 1 and the audio data 2 at this time, AudioPolicy may determine that mixing of the audio data 1 and the audio data 2 is not necessary. That is, AudioPolicy may determine that mixing of audio data 1 and audio data 2 is not required when one of audio data 1 and audio data 2 does not allow mixing.

At this time, AudioPolicy may send a first audio mixing instruction to AudioFlinger, where the first audio mixing instruction is used to indicate that audio data 1 and audio data 2 do not need to be mixed, that is, audio data 1 and audio data 2 need to be independently transmitted to sound box 1, so as to ensure that sound box 1 can restore the audio features of audio data 1 and audio data 2 as much as possible to play according to received audio data 1 and audio data 2.

S5006, in response to the first mixing instruction, the AudioFlinger in the mobile phone sends the first audio data and the second audio data to the first slave device, respectively.

After receiving the first audio mixing instruction sent by AudioPolicy, the AudioFlinger may determine that the currently received audio data 1 and audio data 2 need to be independently transmitted to the sound box 1, and then the AudioFlinger does not need to perform audio mixing processing on the audio data 1 and the audio data 2 according to the method in the prior art. Accordingly, as shown in fig. 55, the AudioFlinger may call the DMSDP Audio HAL in the HAL, send two channels of Audio data (i.e., Audio data 1 and Audio data 2) to the DMSDP Audio HAL, and send the Audio data 1 and the Audio data 2 to the sound box 1 through the DMSDP Audio HAL. Thus, the audio data 1 and the audio data 2 received by the sound box 1 are two paths of audio data which are relatively independent, and the sound box 1 can restore the respective audio characteristics of the audio data 1 and the audio data 2 as much as possible when playing the audio data 1 and the audio data 2.

For example, after receiving the first mixing indication, the AudioFlinger may pack (may also be referred to as a packet) audio data 1 and audio data 2, respectively, for example, a mobile phone may pack each 3 frames of audio data in the audio data 1 into one packet, and a mobile phone may pack each 2 frames of audio data in the audio data 2 into one packet. As shown in fig. 56, in the process of packetizing the audio data 1, an identification of the audio data 1, for example, 01, may be added in each packet. Accordingly, in the process of packetizing the audio data 2, an identification of the audio data 2, for example 02, may be added in each packet. Still as shown in fig. 56, the AudioFlinger may store the packet obtained by packing in a buffer 5601 of the AudioFlinger in real time, and send the packet in the buffer 5601 to the DMSDP audiohal in real time. Thus, although the DMSDP Audio HAL is a packet of the Audio data 1 and the Audio data 2 obtained from the same buffer 5601, the DMSDP Audio HAL can restore which packets are the packets of the Audio data 1 and which packets are the packets of the Audio data 2 according to the identifier in the packets.

Alternatively, a plurality of buffers may be provided in the AudioFlinger, and each path of audio data to be transmitted to the slave device (for example, the sound box 1) corresponds to one buffer. For example, the AudioFlinger may pack the audio data 1 into a plurality of packets and store the packets in a first buffer, and pack the audio data 2 into a plurality of packets and store the packets in a second buffer. At this time, the AudioFlinger may or may not add an identifier indicating specific audio data to the packet. Thus, the packet obtained by the DMSDP Audio HAL from the first buffer is the packet of Audio data 1, and the packet obtained by the DMSDP Audio HAL from the second buffer is the packet of Audio data 2.

After obtaining each data packet carrying the identifier from the buffer 5601, the DMSDP Audio HAL may unpack, i.e., decapsulate, each data packet. Thus, the DMSDP Audio HAL can restore two paths of unmixed Audio data (i.e., Audio data 1 and Audio data 2) according to the identifier carried in each packet. Furthermore, DMSDP Audio HAL may send Audio data 1 and Audio data 2 to loudspeaker 1 relatively independently through the network connection established in step S5001, so that loudspeaker 1 may obtain unmixed Audio data 1 and Audio data 2.

Or, after acquiring each data packet carrying the identifier from the buffer 5601, the DMSDP Audio HAL may directly send the acquired data packet to the sound box 1, and the sound box 1 unpacks each received data packet, so as to restore two paths of unmixed Audio data (i.e., Audio data 1 and Audio data 2) according to the identifier carried in each data packet. Therefore, the processing complexity of the DMSDP Audio HAL in the mobile phone can be reduced, and the time delay generated when the mobile phone and the loudspeaker box 1 carry out Audio switching is reduced.

In some embodiments, if the audio data 1 and the audio data 2 need to be sent to two different audio output devices, for example, the audio data 1 is sent to the sound box 1 for playing, and the audio data 2 is sent to the television 2 for playing. Then, after acquiring each data packet carrying the identifier from the buffer 5601, the DMSDP Audio HAL needs to unpack each data packet first to restore two paths of Audio data, i.e., Audio data 1 and Audio data 2, and then respectively send the Audio data 1 to the sound box 1 and the Audio data 2 to the television 2.

After the audio data 1 and the audio data 2 are sent to the sound box 1, because the audio data 1 and the audio data 2 are not subjected to audio mixing processing, audio features of the audio data 1 and the audio data 2 are reserved to the greatest extent, so that the sound box 1 can restore the audio features of the audio data 1 and the audio data 2 to the greatest extent when the audio data 1 and the audio data 2 are played, and the phenomenon of playing distortion in an audio switching scene is reduced.

Illustratively, the user has set the volume of MUSIC type audio data 1 to 7 levels and the volume of NOTIFICATION type audio data 2 to 4 levels in the cellular phone. When the volume level is larger, the AudioFlinger sets the gain of the corresponding audio data to be larger, so that the loudness of the audio data when played is larger. Then, as shown in fig. 57, since the Audio data 1 and the Audio data 2 sent by the DMSDP Audio HAL in the mobile phone to the sound box 1 are two paths of Audio data which are relatively independent, the gain of the Audio data 1 received by the sound box 1 also corresponds to the 7-level volume in the mobile phone, and the gain of the Audio data 2 also corresponds to the 4-level volume in the mobile phone. Then, the sound box 1 can play the audio data 1 and the audio data 2 simultaneously according to the gain of the audio data 1 and the audio data 2 directly. Alternatively, the sound box 1 may also play the audio data 1 and the audio data 2 after amplifying or reducing the gain of the audio data 1 and the audio data 2 proportionally based on the audio data 1 and the audio data 2. In any way, because the volume of the audio data 1 received by the sound box 1 is greater than the volume of the audio data 2, the volume of the audio data 1 finally played by the sound box 1 is still greater than the volume of the audio data 2, so that the original volume characteristics of the audio data 1 and the audio data 2 on the mobile phone are restored, and the problem that a user feels that the volume of a certain path of audio data in the multi-path audio data is too large or too small is avoided.

It should be noted that, before the audio data 1 and the audio data 2 are played, the sound box 1 may further perform processing operations such as parsing, sampling, encoding, decoding, encapsulating, or decapsulating on the played audio data 1 and the played audio data 2, and then play the processed audio data 1 and the processed audio data 2, which is not limited in this embodiment of the present application.

S5007, if the first audio data and the second audio data need to be mixed, the AudioPolicy sends a second mixing instruction to the AudioFlinger, where the second mixing instruction is used to instruct to mix the first audio data and the second audio data.

Corresponding to step S5005, AudioPolicy may determine whether audio data 1 allows mixing and determine whether audio data 2 allows mixing according to the mixing policy shown in table 11 according to the types of audio data 1 and audio data 2 in step S5007. AudioPolicy may determine that audio data 1 and audio data 2 need to be mixed if both audio data 1 and audio data 2 allow mixing.

At this time, AudioPolicy may send a second mixing instruction to AudioFlinger, where the second mixing instruction is used to indicate that audio data 1 and audio data 2 need to be mixed, that is, audio data 1 and audio data 2 do not need to be independently transmitted to sound box 1.

S5008, in response to the second mixing instruction, the AudioFlinger in the mobile phone mixes the first audio data and the second audio data into third audio data.

After receiving the second mixing instruction sent by AudioPolicy, the AudioFlinger may perform mixing processing on audio data 1 and audio data 2, as shown in fig. 58. For example, when the sampling rate of the audio data 1 is greater than that of the audio data 2, the AudioFlinger may uniformly sample the audio data 1 and the audio data 2 using the sampling rate of the audio data 2 (i.e., a smaller sampling rate). For another example, when the volume of the audio data 1 is not identical to the volume of the audio data 2, the AudioFlinger may adjust the increase sizes of the audio data 1 and the audio data 2 to the same value. Then, the audio data 1 and the audio data 2 after the mixing are combined into one audio data (i.e., the third audio data).

S5009, the AudioFlinger in the mobile phone sends the third audio data to the first slave device.

In step S5009, as also shown in fig. 58, the AudioFlinger may send the mixed third Audio data to the sound box 1 through the DMSDP Audio HAL, and the sound box 1 plays the mixed third Audio data. Since the audio characteristics of the audio data 1 and the audio data 2 in the third audio data are modified during audio mixing, a certain distortion may occur when the sound box 1 plays the third audio data. However, since the user has set in the mixing strategy corresponding to the loudspeaker 1 to allow the audio data corresponding to the types of the audio data 1 and the audio data 2 to be mixed, the distortion phenomenon occurring when the third audio data is played is generally within the receiving range of the user.

In the above embodiment, it is illustrated that whether two paths of audio data, i.e., the audio data 1 and the audio data 2, are mixed, and it can be understood that when the audiomaker in the mobile phone receives three or more paths of audio data at the same time, it can still be determined according to the method described in the above embodiment which audio data need to be independently transmitted to the slave device and which audio data need to be mixed and then transmitted to the slave device.

For example, still taking the first slave device as the sound box 1 for example, after the mobile phone switches the audio function to the sound box 1, the user may also manually modify the slave device currently performing audio switching with the mobile phone on the mobile phone. For example, when the mobile phone runs a music APP, if an operation of opening the control center by the user is received, the mobile phone may display the control center 5901 in an application interface of the music APP as shown in (a) in fig. 59. The control center 5901 includes a switch button 5902 for audio switching functions. When the user wishes to modify a slave device of the current handset, the clickable toggle button 5902 queries one or more candidate devices that may currently be slave devices of the handset. As shown in (b) of fig. 59, after the handset detects that the user clicks the switching button 5902, the handset may display the detected one or more candidate devices in the control center 5902. The speakers in the control center 5902 have been marked as the selected state, i.e., the speakers are playing audio from the handset as a slave (i.e., the first slave) of the handset. If the user needs to replace the slave device of the handset, the user can re-select the slave device (i.e., the second slave device) for audio switching in the control center 5902.

For example, the second slave device in which the user selects the car machine as the mobile phone may respond to the operation of the user selecting the car machine in the control center 5902, disconnect the network connection already established with the sound box 1, and establish a network connection with the car machine. Similar to step S5001, after the network connection between the mobile phone and the car machine is established, the Audio capability parameter of the car machine can be acquired through the DV APP, and a corresponding DMSDP Audio HAL is created in the HAL according to the Audio capability parameter of the car machine. Or, the mobile phone may update the DMSDP Audio HAL created for the sound box 1 in step S5001 according to the Audio capability parameter of the car machine, so that the updated DMSDP Audio HAL matches with the Audio capability of the car machine.

Further, similar to step S5002, as shown in fig. 60, after the mobile phone determines the car machine as a current slave device (i.e., a second slave device) and establishes a network connection, the DV APP may obtain an identifier of the car machine, so as to determine that the currently connected second slave device is an on-vehicle device. Furthermore, the DV APP may issue the audio mixing policy stored for the vehicle-mounted device in advance as the audio mixing policy 2 of the vehicle-mounted device to the AudioPolicy. Subsequently, AudioPolicy may determine whether to mix currently received audio data according to the mixing policy 2 of the car machine.

For example, a mixing policy 2 of a car machine may be as shown in table 13, and different from the mixing policy shown in table 11, one or more identifiers including an application, for example, a Package Name (Package Name) of the application, may also be set in the mixing policy 2. That is, the mobile phone may further set, in the mixing strategy 2, mixing capabilities of different applications for different types of audio data respectively. For example, for the WeChat APP, the car machine does not allow mixing of audio data of MUSIC type and COMMUNICATION type, and allows mixing of audio data of NOTIFICATION type and DTMF type. For MUSIC APP, car machines do not allow mixing MUSIC type audio data and the like. The specific application included in the mixing strategy 2 may be preset by a developer. Or, when a user installs a new application in the mobile phone, the mobile phone may obtain the sound mixing policy of the application in different audio output devices from the server, so that the mobile phone may dynamically update the sound mixing policy of each audio output device according to the application installed in the mobile phone, which is not limited in this embodiment of the present application.

Watch 13

Therefore, when the mobile phone runs the WeChat APP, the audio data of the MUSIC type and the COMMUNICATION type generated by the WeChat APP can be independently transmitted to the car machine for playing, and are not mixed with other audio data received at the same time, so that the distortion rate of the two types of audio data during playing on the car machine is reduced. In addition, the mobile phone can mix or not mix different types of audio generated by one application by taking the application as granularity, so that audio data with higher requirements on tone quality of a user in a certain application is independently transmitted to the slave equipment, and the original audio characteristics of the audio data in the main equipment (namely the mobile phone) can be truly restored as far as possible when the slave equipment plays the audio data.

In other embodiments, a mixing strategy as shown in table 14 may also be set in AudioPolicy. In the mixing strategy, for different applications, device types which do not allow mixing are set for different types of audio data in each application, that is, independent transmission is required when the audio data is sent to which types of audio output devices. For example, when the wechat APP generates MUSIC type audio data, when the audio data needs to be played on several types of audio output devices such as a mobile phone, a sound box, a television or a car machine, mixing of the audio data with other audio data is not allowed. For another example, when the music APP generates audio data of NOTIFICATION, the audio data can be mixed when being played on any type of audio output device, that is, the type of device that does not allow mixing is empty. Subsequently, AudioPolicy may determine whether to mix various types of audio data generated by the running audio APP according to the mixing policy shown in table 14, in combination with the device type of the current slave device. Of course, audio output devices that allow mixing may also be set for different types of audio data in each application in the mixing strategy shown in table 14, which is not limited in this embodiment of the application.

TABLE 14

In other embodiments, the user can also manually modify the mixing strategy of the current slave device in the mobile phone. As shown in fig. 61 (a), after the mobile phone switches the music APP to the background operation, the application interface 6100 for displaying the WeChat APP by the WeChat APP can be opened. If an operation of opening the control center is detected to be input by the user in the application interface 6100, the mobile phone can display the control center 6101 in the application interface 6100. A card 6102 for audio switching is provided in the control center 6101, and a setting button 6103 for audio switching is provided in the card 6102. If it is detected that the user clicks the setting button 6103, the DV APP may display the setting interface 6104 when the car machine is a slave device, as shown in (b) in fig. 61. Similarly to fig. 54, a switch button 6105 for the audio switching function may be provided in the setting interface 6104. When the user turns on the switch button 6105, the user can set in the setting interface 6104 what types of audio data the mobile phone allows to mix when the car machine is a slave device, or further set that the mobile phone allows to mix certain types of audio data in some applications when the car machine is a slave device. Furthermore, the DV APP may update the mixing policy 2 of the car machine shown in table 13 according to the selection of the user in the setting interface 6104, so that the user may dynamically specify whether to mix audio data of various types when outputting to the slave device.

When the slave device of the mobile phone is switched from the loudspeaker box to the car machine, the audioFlinger can request to the AudioPolicy to inquire whether the currently received audio data needs to be mixed. Furthermore, AudioPolicy may determine whether to mix the current audio data according to the mixing policy 2 issued by DV APP.

For example, taking the example that the AudioFlinger receives three paths of audio data at the same time, as shown in fig. 62, at time T1, the user opens the music APP in the mobile phone to start playing songs, at time T2(T2 > T1), after the wechat APP receives a new message sent by the contact, the user starts playing alert tones, and during the time period T3-T4(T4 > T3 > T2), the user starts playing key tones when the user clicks keys with the keyboard in the wechat APP. Then, as shown in fig. 60, the AudioFlinger of the cellular phone may simultaneously receive the NOTIFICATION type audio data a and the DTMF type audio data B from the WeChat APP and the MUSIC type audio data C from the MUSIC APP during the time period T3-T4.

Still as shown in fig. 60, AudioFlinger may request AudioPolicy to query whether audio data a, audio data B, and audio data C require mixing. Furthermore, AudioPolicy may determine, according to mixing policy 2 shown in table 13, that audio data a of NOTIFICATION type and audio data B of DTMF type may be mixed, while audio data C of MUSIC type is not allowed to be mixed with other audio data, and needs to be independently transmitted to the vehicle machine, so as to ensure the sound quality of the vehicle machine when playing audio data C of MUSIC type.

AudioPolicy may then send an indication message to AudioFlinger instructing AudioFlinger to mix audio data a and audio data B, without audio data C needing to be mixed. As also shown in fig. 60, in response to the indication message, the AudioFlinger may mix the audio data a and the audio data B into one audio data (i.e., audio data a + B), and then output two audio data, i.e., the mixed audio data a + B and the audio data C that is not mixed. Furthermore, the audioFlinger can call the DMSDP Audio HAL to send the Audio data A + B after mixing and the Audio data C without mixing to the vehicle machine, and the vehicle machine plays the Audio data A + B and the Audio data C. Therefore, the MUSIC type audio data C can be transmitted to the slave equipment (namely the vehicle machine) relatively independently without being subjected to audio mixing processing, so that the audio data C received by the slave equipment can reflect the original audio characteristics more truly, the audio distortion phenomenon caused by the audio mixing processing is avoided, the distortion condition of multi-channel audio data during cross-equipment playing is reduced, and the audio output quality of an audio switching scene is improved.

In some embodiments, for example, the slave device using the car machine as a mobile phone, when the mobile phone sends each path of audio data (for example, the audio data C) to the car machine, the audio data C may be packaged into one data packet, and the data packet is sent to the car machine. For example, the AudioFlinger in the mobile phone may package the audio data C according to a JSON (JavaScript Object Notation) protocol, and add the feature information of the audio data C to the data packet. For example, the feature information may include the name of the application package to which the audio data C belongs, the playback type (e.g., game, movie, etc.), the playback format (e.g., MP3, etc.), the volume level, whether to mix, the audio type (e.g., MUSIC type, etc.), the track information (e.g., the number of tracks, the serial number of tracks, etc.). The characteristic information can reflect the actual audio characteristics of the audio data C, so that the slave device (e.g., a car machine) receiving the data packet can truly restore the audio data C and the related characteristics of the audio data C by analyzing the characteristic information in the data packet, and thus the audio data C is played according to the characteristic information of the audio data C, and the distortion phenomenon of the audio data C in the audio switching process is reduced.

For example, the AudioFlinger may add characteristic information of corresponding audio data in a header file (head) of each packet. For example, in the audio data in PCM format, packet 1 and packet 2 encapsulated by AudioFlinger according to JSON protocol may be as follows:

in some embodiments, the AudioFlinger may send the encapsulated packets (e.g., packet 1 and packet 2 above) to the DMSDP Audio HAL, and the DMSDP Audio HAL unpacks the packet 1 and packet 2 above. DMSDP Audio HAL can recognize that packet 1 is from WeChat APP and packet 2 is from music APP by reading the characteristic information (e.g., packet name of application) in packets 1 and 2, i.e., packets 1 and 2 belong to two Audio data paths, respectively. Thus, after the DMSDP Audio HAL unpacks each data packet sent by the Audio flinger, two corresponding paths of Audio data without Audio mixing can be obtained. Furthermore, the DMSDP Audio HAL can send the two paths of Audio data to the car machine for playing relatively independently.

In other embodiments, after the DMSDP Audio HAL obtains the packet that has been encapsulated by the AudioFlinger, the DMSDP Audio HAL may directly send the obtained packet to the car machine. And unpacking each received data packet by the vehicle machine, thereby restoring two paths of corresponding audio data without audio mixing according to the characteristic information carried in each data packet. And the car machine can also select a corresponding playing mode according to the corresponding characteristic information to accurately restore the playing process of the two paths of audio data. For example, the car machine may use the MP3 format to play the audio data corresponding to the data packet 1, and use the flac format to play the audio data corresponding to the data packet 2, so as to restore the audio characteristics of the two paths of audio data when played in the mobile phone as much as possible, and reduce the phenomenon of playing distortion in the audio switching scene.

In the above embodiments, it is exemplified that all audio data in the mobile phone is switched to the slave device (e.g. a sound box) for playing by the mobile phone. It will be appreciated that in some embodiments, when the AudioFlinger of the handset receives multiple audio data from different applications or the same application, the AudioFlinger may also determine the audio output device for each of the multiple audio data by interacting with AudioPolicy.

Still taking the slave device of the mobile phone as the sound box for example, after the mobile phone is connected with the sound box, as shown in fig. 63, the DV APP in the mobile phone may issue a device selection policy for the sound box to AudioPolicy, in addition to issuing the above sound mixing policy to AudioPolicy, where the device selection policy sets the type of the audio data that the sound box allows to be played and the type of the audio data that the sound box does not allow to be played. For example, the sound box allows audio data of MUSIC type, NOTIFICATION type, and RING type to be played, but the sound box does not allow audio data of SYSTEM type and DTMF type to be played.

As also shown in fig. 62, the AudioFlinger of the cellular phone can simultaneously receive the NOTIFICATION type audio data a and the DTMF type audio data B from the WeChat APP and the MUSIC type audio data C from the MUSIC APP during the time period T3-T4. Further, as also shown in fig. 63, AudioFlinger may request AudioPolicy for an audio output device that queries whether audio data a, audio data B, and audio data C are allowed to be played in the speaker, i.e., that negotiates audio data with AudioPolicy. Further, AudioPolicy may determine that NOTIFICATION type audio data a and MUSIC type audio data C may be played on the speaker, but that DTMF type audio data B may not be allowed to be played on the speaker, according to the above-described device selection policy. Furthermore, AudioPolicy may also set an audio output device for audio data B, for example, set the audio output device for audio data B as the mobile phone itself. Furthermore, AudioPolicy may instruct AudioFlinger to output audio data a and audio data C to a sound box for playing, and output audio data B to a speaker of the mobile phone for playing.

Further, the audioFlinger can determine that the audio data A and the audio data C need to be sent to the sound box for playing according to the indication of audioPolicy. At this time, in order to ensure the audio quality of the audio data a and the audio data C played on the sound box, the AudioFlinger may request AudioPolicy to inquire whether audio data a and the audio data C need to be mixed, that is, negotiate with the AudioPolicy whether audio data needs to be mixed. Furthermore, AudioPolicy may determine whether audio data a and audio data C need to be mixed according to the mixing policy issued by DV APP according to the method in the foregoing embodiment. If the audio data a and the audio data C do not need to be mixed, the mobile phone may execute the method described in steps S5005-5006 to send the two paths of audio data, i.e., the audio data a and the audio data C, to the speaker for playing independently (not shown in fig. 63); if the audio data a and the audio data C need to be mixed, as shown in fig. 63, the mobile phone may execute the method described in steps S5007-5009, and send one path of audio data a + C obtained by mixing the audio data a and the audio data C to the sound box for playing.

Or, after AudioPolicy determines that the audio output device of the audio data a and the audio data C is a sound box and the audio output device of the audio data B is a mobile phone according to the device selection policy, since the sound box needs to play multiple paths of audio data at the same time, AudioPolicy may further determine whether to mix the audio data a and the audio data C according to a mixing policy issued by the DV APP. Further, AudioPolicy may indicate to AudioFlinger that audio output devices of audio data a and audio data C are speakers and whether audio data a and audio data C need to be mixed, and indicate to AudioFlinger that audio output device of audio data B is a mobile phone.

Subsequently, the AudioFlinger may output the Audio data B to a speaker of the mobile phone through the Primary HAL for playing, and the AudioFlinger may send the unmixed Audio data a and Audio data C (or the mixed Audio data a + C) to the sound box through the DMSDP Audio HAL for playing.

Therefore, when the mobile phone needs to switch the simultaneously generated multiple audio data to the slave device for playing, the mobile phone can select corresponding audio output devices for different audio data according to the device selection strategy of the current slave device, and can determine the audio data which needs to be independently transmitted in the multiple audio data played on the same audio output device according to the sound mixing strategy of the current slave device, so that the distortion phenomenon caused by sound mixing processing on certain type of audio data is avoided, and the audio output quality of an audio switching scene is improved.

In a distributed audio scenario, the master device may be an electronic device with a split-screen display function (also referred to as a split-screen function). For example, the split screen function means that the host device can create a plurality of windows when displaying in its own display screen, and run and display a corresponding application, software or program in each window.

Still taking the mobile phone as the main device for example, the user may trigger the mobile phone to enter the split screen mode through a preset operation (e.g., pressing the screen for a long time). As shown in fig. 64, in the split-screen mode, the cellular phone can divide a display area in the display screen into a window 6401 and a window 6402. The status bar of the mobile phone may be displayed in the window 6401 or the window 6402, or may be hidden in the split screen mode, which is not limited in this embodiment. The window 6401 and the window 6402 may be used to run and display a corresponding application, respectively, or the window 6401 and the window 6402 may also run and display different tasks of the same application, respectively, for example, the window 6401 may display a chat interface with the contact Amy in the WeChat APP, and at the same time, the window 6402 may display a chat interface with the contact Mike in the WeChat APP. Of course, the mobile phone may further divide the display area into 3 or more windows, which is not limited in this embodiment.

Illustratively, a first video APP may be run in window 6401 and a second video APP may be run in window 6402. The first video APP may create one or more audio tasks at runtime, for example, audio task 1 playing video a, audio task 2 playing a notification message alert tone. Similarly, the second video APP may also create one or more audio tasks at runtime.

In embodiments of the present application, each window displayed by a host device (e.g., a cell phone) may be associated with an audio output device. That is, the audio data output by the audio task in each window may be played using the audio output device associated with that window. For example, the association relationship between the window and the audio output device may be user-set. For example, if the user setup window a is associated with a wired headset. Then, when the audio task created by the application in the window a outputs the corresponding audio data, the mobile phone can send the audio data to the wired headset connected with the mobile phone for playing. In some embodiments, the handset may also modify the association between the window and the audio output device. Still taking the example that the user sets the audio output device associated with the window a as the wired headset, if the mobile phone is not connected with the wired headset, the mobile phone may re-determine the audio output device associated with the window a, for example, the mobile phone may determine the mobile phone itself as the audio output device associated with the window a, and then the mobile phone may send the audio data output in the window a to the speaker of the mobile phone for playing. Alternatively, if the wired headset does not support playing the audio data currently in window a, for example, some wired headsets with a lower configuration may not support playing the audio data in high definition format, the handset may re-determine the audio output device associated with window a and send the audio data to the associated audio output device for playing. Of course, the mobile phone may also modify the association relationship between the window and the audio output device in response to the setting of the user, which is not limited in this embodiment of the present application.

For example, when the cellular phone enters the split-screen mode, as shown in fig. 65, the cellular phone may display a first dialog box 6501 in a window 6401, and prompt the user to select an audio output device associated with the window 6401 in the first dialog box 6501. For example, the handset may display an electronic device with audio playback capabilities that is located within the same Wi-Fi network as the handset or has established a connection with the handset in the first dialog box 6501. In this way, the user may select the audio output device (i.e., one of the slaves of the handset) associated with the window 6401 in the first dialog box 6501. For example, if the user selects a bluetooth headset in the first dialog box 6501, the mobile phone may establish an association between the window 6401 and the bluetooth headset if it is detected that the user selects the bluetooth headset in the first dialog box 6501. Subsequently, the mobile phone can send the audio data output by the audio task in the window 6401 to the bluetooth headset for playing. The first dialog box 6501 may be automatically displayed in the window 6401 after the mobile phone enters the split-screen mode, or may be displayed in the window 6401 by a user being triggered by a certain key or gesture after the mobile phone enters the split-screen mode.

Similarly, as also shown in fig. 65, after the handset enters the split-screen mode, the handset may also display a second dialog box 6502 in the window 6402, in which the user is prompted to select an audio output device associated with the window 6402. For example, the handset may display an electronic device with audio playback capabilities that is located within the same Wi-Fi network as the handset or has established a connection with the handset in the second dialog box 6502. In this way, the user may select an audio output device (i.e., another slave device of the handset) associated with the window 6402 in the second dialog box 6502. For example, if the user selects a sound box in the second dialog box 6502, if it is detected that the user selects a sound box in the second dialog box 6502, the mobile phone may establish an association relationship between the window 6402 and the sound box. Subsequently, the mobile phone can send the audio data output by the audio task in the window 6402 to the sound box for playing. Similarly, the second dialog box 6502 may be automatically displayed in the window 6402 after the mobile phone enters the split-screen mode, or may be displayed in the window 6402 by a user being triggered by a certain key or gesture after the mobile phone enters the split-screen mode.

In some embodiments, the associated audio output device set by the user for window 6401 may be the same as the associated audio output device set by the user for window 6402, e.g., the user has selected native in the first dialog box 6501 and the user has also selected native in the second dialog box 6502. At this point, as shown in fig. 66, the handset may display a prompt message 6601, the prompt message 6601 being used to prompt the user to associate window 6401 and window 6402 to the same audio output device (i.e., the handset itself). Subsequently, the mobile phone may output the audio data corresponding to the window 6401 and the audio data corresponding to the window 6402 to a speaker of the mobile phone for playing after mixing.

In some embodiments, the handset may also provide the user with access to modify the association between different windows and different audio output devices. For example, as shown in (a) of fig. 67, the cellular phone may display a first switching button 6701 of the audio output device in a window 6401, and the cellular phone may display a second switching button 6702 of the audio output device in a window 6402. If it is detected that the user clicks the first switching button 6701, the cellular phone may display one or more electronic devices having an audio playing function detected at this time in the dialog 6703 for the user to select, similarly to fig. 65. If the user selects an electronic device other than the bluetooth headset, for example, the user selects the mobile phone local in the dialog box 6703, which indicates that the user wishes to play audio data output by the audio task in the window 6401 by the mobile phone local, the mobile phone may modify the association relationship between the window 6401 and the bluetooth headset to the association relationship between the window 6401 and the mobile phone local in response to the user's operation.

Similarly, as shown in (b) of fig. 67, if it is detected that the user clicks the second switching button 6702, the cellular phone may display one or more electronic devices having an audio playing function detected at this time in the dialog 6704 for the user to select. If the user selects an electronic device other than the speaker, for example, the user selects a television in dialog 6704, indicating that the user wishes to play audio data output by the audio task in television play window 6402, the handset may modify the association between window 6402 and the speaker to the association between window 6402 and the television in response to the user's operation.

Of course, in addition to providing the switching buttons (e.g., the first switching button 6701 and the second switching button 6702 described above) in the window, a person skilled in the art may also provide other ways to modify the association between the window and the audio output device. For example, if it is detected that the user inputs a long press operation in the window 6402, the mobile phone may display the dialog 6704 so that the user modifies the audio output device corresponding to the window 6402, which is not limited in this embodiment.

In addition, the mobile phone can store the audio output devices set by the user for each window, namely, the association relationship between different windows and different audio output devices. And after the mobile phone enters the split-screen mode again, the mobile phone can respectively send the audio data corresponding to the windows to the corresponding audio output devices for playing according to the stored association relationship between the windows and the audio output devices.

Of course, besides the audio output device associated with each window in the split-screen mode being manually set by the user, the mobile phone may also obtain the association relationship between different windows and different audio output devices from the server. Or, the mobile phone can also automatically set the audio output device associated with the window according to the type of the application in the different window or the type of the audio data. For example, if the audio task created by the MUSIC APP in the window 6402 outputs MUSIC type audio data, the mobile phone may automatically send the MUSIC type audio data to the speaker for playing, which is not limited in this embodiment of the present application.

It can be seen that in the embodiment of the present application, a master device (e.g., a mobile phone) may set association relationships between different windows and different audio output devices at window granularity. Under the scene of multi-window multi-audio task concurrence, the mobile phone can send the audio data output by the related audio task in each window to the corresponding audio output equipment for playing. Therefore, the multi-channel audio data which are concurrent by the main equipment cannot interfere with each other, the multi-channel audio data can be played to different users through different audio output equipment, each user can listen to the corresponding audio data by using the audio output equipment which is associated with the related window, the influence of the audio data from other windows cannot be caused, and the audio using experience of the users is improved.

Still taking a mobile phone as an example of the main device, fig. 6 illustrates a schematic architecture diagram of an Android system in the mobile phone, where an audio architecture 601 for implementing an audio function is provided in the Android system. On the basis of the Android system shown in fig. 6, as shown in fig. 68, a display framework 6801 for realizing a display function is further provided in the Android system. In the application framework layer, the display architecture 6801 can include a windows manager (WindowManager) and a display module.

Still taking the example of a video APP, the video APP may output a series of rendering instructions of display data to be displayed into a window manager. For example, the draw instruction may be an OpenGL instruction or the like. The window manager can create a corresponding display module and draw corresponding display data in the display module according to a drawing instruction issued by the video APP. For example, the window manager may query whether the current handset is in split screen mode. If not, the window manager can create a display module for providing display data to the display screen. Furthermore, the window manager may draw the display data to be displayed of the video APP in the display module according to the received drawing instruction. Subsequently, the window manager may send the Display data in the Display module to the Display screen through the Display HAL for displaying. That is, in the non-screen mode, the mobile phone may display the display data of the video APP with the entire display area in the display screen as one window.

If the handset is in split screen mode, the window manager can create multiple display modules, each corresponding to a window in the display screen. For example, as also shown in fig. 68, the cell phone may divide the display area in the display screen into two windows by default in the split screen mode, and then if an operation of opening the split screen mode by a user input is detected, the window manager may create display module 1 corresponding to window 1 and display module 2 corresponding to window 2. The display module 1 can be understood as a virtual storage space for storing display data to be displayed in the window 1; similarly, the display module 2 can also be understood as a virtual storage space for storing display data to be displayed in the window 2. If the video APP runs in the window 1, after receiving a drawing instruction from the video APP, the window manager may draw the corresponding display data 1 in the display module 1 according to the drawing instruction. Subsequently, the window manager may send the Display data 1 in the Display module 1 to the window 1 in the Display screen through the Display HAL for displaying. Similarly, an application running in window 2 (e.g., music APP) may also transmit corresponding drawing instructions to the window manager. The window manager may draw the corresponding Display data 2 in the Display module 2 according to the drawing instruction, and send the Display data 2 in the Display module 2 to the window 2 in the Display screen for displaying through the Display HAL. In this way, the handset can display the display data of the relevant application in each window for granularity in the split screen mode.

Generally, in the split screen mode, the display data in the display module 1 and the display module 2 together constitute the display data in the display screen. Then, the Display HAL may periodically obtain 2 corresponding channels of Display data from the Display module 1 and the Display module 2, combine the 2 channels of Display data into one channel of Display data corresponding to the entire Display screen, and send the channel of Display data to the Display screen for displaying.

The audio control method in the split screen scenario will be described in detail below with reference to specific examples.

For example, a user may input a preset screen splitting operation to the mobile phone (i.e., the master device) to trigger the mobile phone to enter the screen splitting mode. For example, as shown in (a) of fig. 69, when the mobile phone runs the WeChat APP, if it is detected that the user uses the finger joint to draw a straight line on the display screen, the mobile phone may enter the split screen mode in response to the operation. As shown in (b) of fig. 69, after entering the split screen mode, the mobile phone may automatically divide the display screen into two windows, i.e., a first window 6901 and a second window 6902. The first window 6901 can be used for continuously running the WeChat APP, and the second window 6902 can be used for displaying a desktop of the mobile phone. The user can use various functions provided by the mobile phone in the first window 6901, and can also use various functions provided by the mobile phone in the second window 6902.

For example, before entering the split-screen mode, if the WeChat APP detects an operation that the user draws a straight line on the display screen by using a finger joint, the WeChat APP may send an indication message to the window manager to indicate the window manager to enter the split-screen mode. Further, as shown in fig. 70, the window manager may create a display module 1 corresponding to a first window 6901 and a display module 2 corresponding to a second window 6902 in response to the indication message, and the display module 1 and the display module 2 may be distinguished by different display IDs. The display module 1 is configured to provide corresponding display data to the first window 6901, and the display module 2 is configured to provide corresponding display data to the second window 6902.

After the window manager creates the display module 1 and the display module 2, as shown in fig. 70, the window manager may receive a drawing instruction from the wechat APP, and draw display data corresponding to the wechat APP in the display module 1 according to the drawing instruction. Also, the window manager may draw the display data 2 of the desktop in the display module 2 by default. Subsequently, the Display HAL in the HAL may synthesize the Display data 1 in the Display module 1 and the Display data 2 in the Display module 2 into the Display data 3 of the entire Display screen, and send the Display data 3 to the Display screen of the mobile phone, so that the Display screen displays the Display interface shown in (b) in fig. 69. Of course, the mobile phone can also run the WeChat APP in the second window 6902 and the desktop in the first window 6901. Or, besides the desktop, the mobile phone may also execute other applications by default in a window other than the window for running the WeChat APP, which is not limited in this embodiment of the present application.

For example, in the android system, an Audio Focus (Audio Focus) preemption mechanism is provided in the android system, and only one application can hold an Audio Focus at a time. Generally, the application that acquires the audio focus has the right to play the audio. For example, after the application 1 creates an audio task, the application 1 may call the requestAudioFocus () interface to obtain an audio focus, and the application 1 (which may be referred to as an audio focus application) that obtains the audio focus may start to execute the audio task to play corresponding audio data. When the audio task execution ends, application 1 may call abandonaudio focus () to release the above audio focus (which may also be referred to as out-of-focus), and stop playing the corresponding audio data. Or, when the application 1 holds the audio focus, if the application 2 calls the requestAudioFocus () interface to apply for obtaining the audio focus, the application 1 may also release the audio focus, so that the application 2 obtains the application 2 and becomes the current audio focus application.

In the embodiment of the present application, the mobile phone may set an Audio Focus (Audio Focus) for each window in the split-screen mode. For example, the first window 6901 may correspond to audio focus 1, and the second window 6902 may correspond to audio focus 2. In this way, the audio focus application that acquires the audio focus 1 in the first window 6901 may start to execute a related audio task, and similarly, the audio focus application that acquires the audio focus 2 in the second window 6902 may start to execute a related audio task, so that the audio tasks between the windows may not interfere with each other, and subsequently, the audio data output by the audio tasks in different windows may also be distributed to different audio output devices for playing, so that the audio data between the windows may not interfere with each other.

Illustratively, the window manager may be used to manage the acquisition and release of audio focus 1 in the first window 6901, and the acquisition and release of audio focus 62 in the second window 902. For example, after the cell phone enters the split-screen mode, the window manager can create and maintain a correspondence between each window and the audio focus application. For example, as shown in table 15, the ID of the display module 1 corresponds to the packet name of the wechat APP, that is, the audio focus application in the first window 6901 is the wechat APP, and the wechat APP currently holds the audio focus 1 of the first window 6901; the ID of the display module 2 corresponds to the packet name of the desktop, that is, the audio focus application in the second window 6902 is the desktop, and the desktop currently holds the audio focus 2 in the second window 6902. Subsequently, if the user opens another application in the desktop displayed in the second window 6902, for example, if an operation of the user to open the video APP is detected, the video APP may call the requestAudioFocus () interface to apply for obtaining the audio focus 2 of the second window 6902 to the window manager. The window manager may modify the application packet name corresponding to the ID of the display module 2 in the table 15 to a video APP packet name in response to the application of the video APP, so that the desktop in the second window 6902 releases the audio focus 2, and the video APP in the second window 6902 obtains the audio focus 2 at the same time.

Watch 15

Display ID	Package name for audio focus application
		ID of display Module 1	WeChat APP bag name
ID of display Module 2	Bag name of desktop

In this way, the window manager can determine the specific application corresponding to each current window by recording the audio focus application corresponding to each display module. In some embodiments, multiple applications may also be running in one window. For example, foreground applications and background applications may be set in each window at window granularity. For example, a WeChat APP may be run in the foreground of the first window 6901 and a music APP may be run in the background of the first window 6901. As another example, a video APP may be run in the foreground of the second window 6902 and a motion APP may be run in the background of the second window 6902. At this time, as shown in table 16, the applications corresponding to the display module 1 include a WeChat APP and a music APP; applications corresponding to the display module 2 include a video APP and a motion APP.

TABLE 16

Display ID	Package names for foreground and/or background applications
		ID of display Module 1	WeChat APP packet name, music APP packet name
ID of display Module 2	Packet name of video APP and packet name of sports APP

In this embodiment of the application, the corresponding relationship between the display module and the application shown in table 15 or table 16 may be referred to as application information of each window, and the window manager may be configured to record and maintain the application information of each window, so that the mobile phone may determine which specific applications run in each window are according to the application information.

After the mobile phone enters the split-screen mode, the window manager can also set corresponding audio output equipment for each window by taking the window as granularity. Illustratively, the handset may be connected to one or more audio output devices, i.e. there may be one or more slave devices of the handset. For example, the mobile phone can establish a bluetooth connection with a bluetooth headset, and the mobile phone can also establish a network connection with a sound box and a television through a Wi-Fi network, and meanwhile, a speaker of the mobile phone can also be used as an audio output device to play audio.

Still referring to the audio architecture shown in fig. 68, after the DV APP in the handset establishes a network connection with each slave device, the slave device may be registered in the audio policy module, for example, the identity, device name, device model or device type of the slave device may be registered. Then, the audio manager of the handset can query the audio policy module for which audio output devices are currently available.

For example, the audio focus application in the first window 6901 is a WeChat APP, and the audio focus application in the second window 6902 is a video APP, after the mobile phone enters the split-screen mode, the audio manager may query one or more audio output devices available to the current mobile phone in the audio policy module, and notify the window manager of the queried audio output devices. Further, as shown in fig. 71, the window manager may display a first dialog box 7101 in a first window 6901 running the WeChat APP, the first dialog box 7101 including one or more audio output devices currently available to the handset; also, the window manager may display a second dialog box 7102 in a second window 6902 running the video APP, the second dialog box 7102 also including one or more audio output devices currently available to the handset. In this way, the user may select an audio output device associated with the first window 6901 in the first dialog box 7101 and the user may select an audio output device associated with the second window 6902 in the second dialog box 7102.

Illustratively, if it is detected that the user selects a bluetooth headset in the first dialog box 7101, indicating that the user wants the audio data output by the audio task in the first window 6901 to be played by the bluetooth headset, the window manager may establish an association relationship between the display module 1 and the bluetooth headset, that is, the first window 6901 is associated with the bluetooth headset, as shown in table 17; if it is detected that the user selects a speaker in the second dialog box 7102, which indicates that the user wishes to play the audio data output by the audio task in the second window 6902 by the speaker, the window manager may establish an association relationship between the display module 2 and the speaker, as shown in table 17, that is, the second window 6902 is associated with the speaker. Thus, as shown in table 17, the window manager can establish and maintain the correspondence between different windows and different audio output devices in the split-screen mode.

TABLE 17

Display ID	Audio output device
		ID of display Module 1	Bluetooth earphone
ID of display Module 2	Sound box

In some embodiments, the window manager may further set the type of audio data that the corresponding audio output device allows to play, in addition to setting the corresponding audio output device for each window. For example, in the android system, the types of audio data can be divided into: ALARM, MUSIC, RING, SYSTEM, NOTIFICATION, DTMF, COMMUNICATIONs, and VOICE CALL.

For example, as shown in table 18, the window manager may set the audio output device associated with the first window 6901 to be a bluetooth headset, and the window manager may set the bluetooth headset to allow MUSIC type and NOTIFICATION type audio data to be played and not to allow RING type audio data to be played. In this way, when audio data of MUSIC type or NOTIFICATION type is generated in the first window 6901, the audio data can be output to the bluetooth headset for playing. While when a RING type audio data is generated in the first window 6901, the audio data is not outputted to the bluetooth headset to be played, thereby filtering out the RING type audio data for the user when the user uses the bluetooth headset. Of course, the window manager may also default to set that the audio output device associated with the first window 6901 allows all types of audio data to be played, which is not limited in this embodiment of the present application.

Watch 18

Alternatively, the window manager may set only the type of audio data that the audio output device allows to play in table 18, where other types of audio data are audio data that the audio output device does not allow to play; alternatively, the window manager may set only the type of audio data that the audio output device does not allow to be played in table 18, and the other type of audio data is the audio data that the audio output device allows to be played.

In some embodiments, the user may also manually modify the audio output devices associated with the respective windows, and the types of audio data that the audio output devices allow or disallow from being played. For example, as shown in (a) of fig. 72, the cellular phone may display a first setting button 7201 in a first window 6901. For example, the first setting button 7201 may be a float button. If it is detected that the user clicks the first setting button 7201, the cellular phone may display a first setting menu 7202 as shown in (b) of fig. 72. In the first setting menu 7202, the user may reset the audio output device associated with the first window 6901. Also, in the first setting menu 7202, the user may also set the types of audio data that the audio output device associated with the current window allows or does not allow to be played.

Similarly, as shown in (a) of fig. 73, the cellular phone may also display a second setting button 7301 in a second window 6902. If it is detected that the user clicks the second setting button 7301, the cellular phone may display a second setting menu 7302 as shown in (b) of fig. 73. In the second setting menu 7302, the user may reset the audio output device associated with the second window 6902. Also, in the second setting menu 7302, the user may also set the types of audio data that the audio output device associated with the current window allows or does not allow to be played.

Further, the window manager may update the correspondence between different windows in the table 18, audio output devices, and types of audio data that are allowed or not allowed to be played according to a user's selection in the first setting menu 7202 or the second setting menu 7302. Therefore, the user can set the audio output equipment of each window by taking the window as granularity according to the self requirement and specific audio data output on the audio output equipment, so that the audio data in each window can be distributed to the corresponding audio output equipment by the mobile phone to be played according to the setting of the user.

In some embodiments, the window manager may further set the volume of each type of audio data corresponding to each window during playing. Illustratively, as shown in table 19, the audio output device corresponding to the display module 1 (i.e., the first window 6901) is a bluetooth headset that allows audio data of MUSIC type and NOTIFICATION type to be played. Here, the volume level when MUSIC type audio data is played is 15 levels, and the volume level when NOTIFICATION type audio data is played is 10 levels. The audio output device corresponding to the display module 2 (i.e., the second window 6902) is a sound box that allows audio data of MUSIC type, RING type, and NOTIFICATION type to be played. The volume level when MUSIC type audio data is played is 12 levels, the volume level when RING type audio data is played is 8 levels, and the volume level when NOTIFICATION type audio data is played is 6 levels.

Watch 19

For example, the user may manually modify the volume of each of the audio data types in each window during playing. As shown in fig. 74, when the mobile phone runs the WeChat APP in the first window 6901, if a new message sent by the contact Sara is received, the WeChat APP may generate a NOTIFICATION prompt tone of NOTIFICATION type. If the user wishes to adjust the volume of the notification tone in the first window 6901, the user may trigger the mobile phone to display a volume adjustment bar 7401 in the first window 6901 through a preset gesture. In this way, the user can adjust the volume of the notification tone in the first window 6901 by adjusting the position of the slider on the volume adjustment bar 7401. At this time, the window manager may modify the volume level of the audio data of the NOTIFICATION type corresponding to the display module 1 in the table 19 according to the position of the slider on the volume bar 7401.

Similarly, when the audio data of MUSIC type is played in the first window 6901, the user may also trigger the window manager to modify the volume level of the audio data of MUSIC type corresponding to the display module 1 through the volume adjustment bar 7401 in the first window 6901. In addition, when no audio task is playing in the first window 6901, if it is detected that the user adjusts the position of the slider in the volume adjustment bar 7401, the window manager may modify the volume level of a certain type (e.g., MUSIC type) of audio data corresponding to the display module 1 by default. Alternatively, when multiple types of audio data are played in the first window 6901, the window manager may modify the volume level of a certain type of audio data corresponding to the display module 1 by default.

Similarly, as also shown in fig. 74, when the mobile phone runs the video APP in the second window 6902, the volume adjustment bar 7402 may also be displayed in the second window 6902. Since the audio data generated during the running of the video APP is MUSIC type audio data, if the mobile phone detects that the user adjusts the slider on the volume adjustment bar 7402, the window manager may modify the volume level of the MUSIC type audio data corresponding to the display module 2 in the table 19 according to the position of the slider on the volume adjustment bar 7402.

Likewise, if a RING type audio data is being played in the second window 6902, the user may also trigger the window manager to modify the volume level of the RING type audio data corresponding to the display module 2 via the volume adjustment bar 7402 in the second window 6902. If the NOTIFICATION type audio data is being played in the second window 6902, the user may also trigger the window manager to modify the volume level of the NOTIFICATION type audio data corresponding to the display module 2 through the volume adjustment bar 7402 in the second window 6902. If no audio task is playing in the second window 6902, at which time the window manager may default to modifying the volume level of a certain type of audio data (e.g., MUSIC type) corresponding to the display module 2 if it detects that the user adjusts the position of the slider in the volume adjustment bar 7402. Alternatively, when multiple types of audio data are played in the second window 6902 at the same time, the window manager may also modify the volume level of a certain type of audio data corresponding to the display module 2 by default.

Of course, the mobile phone may also set an option of modifying the volume of each type of audio data in the first setting menu 7202 or the second setting menu 7302, so that the user may manually set the volume of each type of audio data when playing on different windows and different audio output devices, which is not limited in this embodiment of the present application.

In the embodiment of the present application, the specific audio configuration contents shown in tables 17 to 19 above, which contain the association relationship between different windows and different audio output devices, may be referred to as audio configuration information of each window. For example, the audio configuration information of the first window 6901 (i.e., the display module 1) includes an associated audio output device, and may further include the types of audio data that the audio output device allows and/or does not allow to play, the volume level of the audio data that allows to play, and the like.

That is, in the split-screen mode, the window manager may establish and maintain application information (e.g., correspondence between different display modules and corresponding applications shown in table 15 or table 16) for each window, and the window manager may establish and maintain audio configuration information (e.g., audio configuration information shown in any one of table 17 to table 19) for each window. Therefore, after the corresponding audio tasks are created by the applications in different windows in the mobile phone, the mobile phone can manage the audio data corresponding to each window by taking the window as granularity according to the application information and the audio configuration information of each window, so that the audio data output by the audio tasks in each window cannot interfere with each other.

For example, as shown in fig. 75, in the split-screen mode, the window manager may send the application information and audio configuration information of the respective windows to the audio manager, for example, to an audio policy module (e.g., AudioPolicy) of the audio manager. And after the window manager detects that the application information or the audio configuration information of a certain window changes, the window manager can dynamically update the application information or the audio configuration information of the window and send the latest application information and audio configuration information of each window to the AudioPolicy.

Still taking the audio focus application in the first window 6901 as the WeChat APP, and the audio focus application in the second window 6902 as the video APP, for example, if it is detected that the user clicks the play button of the video APP in the second window 6902, the video APP can create an audio task 1 for playing audio data in the video on the one hand, and can create a display task 1 for playing display data in the video on the other hand.

Still as shown in fig. 75, for the display task 1, the video APP may issue a related drawing instruction (a dotted arrow from the video APP in fig. 75) to the window manager, and since the window manager already establishes a corresponding relationship between the display module 2 and the second window 6902 when the display module 2 is created, after receiving the drawing instruction issued by the video APP in the second window 6902, the window manager may draw the display data 1 of the video APP in the corresponding display module 2 according to the drawing instruction issued by the video APP.

For audio task 1, the video APP may create a corresponding audio player 1 (e.g., AudioTrack 1), and send audio data 1 to be played by the video APP to the AudioTrack 1. The AudioTrack 1 may send the audio data 1 from the video APP to an audio processor (e.g., AudioFlinger), and the AudioFlinger may sample the audio data 1, add sound effects, and the like.

In this embodiment of the application, as shown in fig. 75, after receiving the audio data 1, the AudioFlinger may send a query request to AudioPolicy to request the AudioPolicy to query a specific audio output device of the audio data 1. For example, when the video APP creates AudioTrack 1, AudioTrack 1 may be associated with the video APP, for example, a corresponding relationship between package names of AudioTrack 1 and the video APP is established. When the AudioFlinger receives the audio data 1, it can be determined that the received audio data 1 is video data from the video APP according to the correspondence. Further, AudioFlinger may send the package name of the video APP to AudioPolicy with the query request.

After receiving the query request sent by the audioFlinger, the audioPolicy can query which window the video APP specifically runs in according to the packet name of the video APP in the query request. For example, as shown in table 15, if the packet name of the video APP corresponds to the ID of the display module 2, it indicates that the video APP is the audio focus application currently running in the second window 6902, that is, the video APP holds the audio focus 2 in the second window 6902 and has a play right to play audio data. Further, the AudioPolicy may determine the audio output device corresponding to the display module 2 according to the audio configuration information of each window issued by the window manager.

In some embodiments, the AudioTrack 1 that the video APP is creating may also be associated with the process ID of the video APP. The video APP runtime may create multiple processes, i.e., the video APP may correspond to multiple process IDs, and the audio task created in each process may correspond to a corresponding AudioTrack. The window manager may then send the window, the audio focus application, and the correspondence between different process IDs in the audio focus application to AudioPolicy. Thus, AudioPolicy may also determine that audio data 1 is audio data from video APP based on the process ID associated with AudioTrack 1. Further, AudioPolicy may query in which window the video APP specifically runs, that is, the display module corresponding to the video APP, according to the method described above.

Taking the audio configuration information of each window shown in table 17 as an example, after AudioPolicy determines that audio data 1 from video APP corresponds to display module 2, AudioPolicy may determine that the audio output device corresponding to display module 2 is a sound box according to the audio configuration information shown in table 17. Further, AudioPolicy may send an instruction message to the AudioFlinger, where the instruction message is used to instruct the AudioFlinger to send the processed Audio data 1 to the sound box through the DMSDP Audio HAL for playing.

Taking the audio configuration information of each window shown in table 18 as an example, the query request sent by the AudioFlinger to AudioPolicy may also carry a type identifier of the audio data 1, for example, the audio data 1 is audio data of MUSIC type. Furthermore, after AudioPolicy determines that audio data 1 corresponds to display module 2, AudioPolicy may determine that the audio output device corresponding to display module 2 is a sound box according to the audio configuration information shown in table 18. Also, AudioPolicy may determine, according to the audio configuration information shown in table 18, that the speaker allows to play MUSIC type audio data, that is, the speaker allows to play the above audio data 1. Further, AudioPolicy may send an instruction message to the AudioFlinger, where the instruction message is used to instruct the AudioFlinger to send the processed Audio data 1 to the sound box through the DMSDP Audio HAL for playing.

In some embodiments, if the audio output device for audio data 1 is a sound box, but the sound box does not allow the audio data of the type of audio data 1 to be played, AudioPolicy may re-determine the audio output device for audio data 1. For example, AudioPolicy may default to playing audio data 1 using its own speaker as the audio output device. Or, AudioPolicy may use the audio output device, which is recently connected to the mobile phone except the speaker, as the audio output device of the audio data 1. Of course, AudioPolicy may also determine that the audio output device of the audio data 1 is empty, that is, the audio data 1 is played without using any audio output device, which is not limited in this embodiment of the present application.

Taking the audio configuration information of each window shown in table 19 as an example, after AudioPolicy determines that audio data 1 from the video APP corresponds to the display module 2, AudioPolicy may determine that the audio output device corresponding to the display module 2 is a sound box according to the audio configuration information shown in table 19. Also, AudioPolicy can determine that the speaker allows MUSIC type audio data to be played according to the audio configuration information shown in table 19, and the volume level when playing is 12 levels. Furthermore, AudioPolicy may send an indication message to the AudioFlinger, where the indication message may not only indicate to the AudioFlinger to send the processed Audio data 1 to the sound box for playing through the DMSDP audiohal, but also indicate that the volume level of the Audio data 1 is 12 levels. Thus, the AudioFlinger can amplify or reduce the gain of the Audio signal in the Audio data 1 at a volume level of 12 stages in response to the instruction message, and send the output Audio data 1 to the sound box through the DMSDP Audio HAL for playing.

In some embodiments, a scenario may arise in which multiple audio tasks in multiple windows are played concurrently in a split-screen mode. For example, as shown in fig. 76, while the video APP in the second window 6902 creates the above-mentioned audio task 1 to play video, if the user clicks the voice message 7601 in the WeChat APP in the first window 6901, the WeChat APP may create an audio task 2 to play the voice message 7601 on the one hand and a display task 2 to display an application interface in the WeChat APP on the other hand.

Similar to the above-described display task 1, and as also shown in fig. 75, for display task 2, the WeChat APP may issue the related drawing instructions (e.g., the dashed arrow from the WeChat APP in fig. 75) to the window manager. Since the window manager already establishes the corresponding relationship between the display module 1 and the first window 6901 when the display module 1 is created, after the window manager receives the drawing instruction issued by the wechat APP in the first window 6901, the window manager can draw the display data 2 corresponding to the application interface of the wechat APP in the corresponding display module 1 according to the drawing instruction issued by the wechat APP.

For example, as shown in fig. 75, the Display HAL in the HAL may periodically obtain the corresponding Display data from the Display module 1 and the Display module 2 according to a certain frame rate. For example, the Display HAL may acquire the Display data 2 from the Display module 1, and acquire the Display data 1 from the Display module 2, and the Display HAL may synthesize the Display data 1 and the Display data 2 into Display data 3 corresponding to a Display screen of a mobile phone. Further, the Display HAL may send the Display data 3 to the Display screen of the mobile phone for Display, so that the Display screen may Display the first window 6901 and the second window 6902 as shown in fig. 76.

For the above audio task 2, as also shown in fig. 75, similar to the above audio task 1, the wechat APP may create a corresponding audio player 2 (e.g., AudioTrack 2) and send the audio data of the voice message 7601 to be played in the wechat APP (i.e., audio data 2) into the AudioTrack 2. The AudioTrack 2 may send the audio data 2 from the WeChat APP to an audio processor (e.g., audioFlinger), and the audioFlinger samples the audio data 2, adds sound effects, and so on.

Similarly, after receiving the audio data 2, the AudioFlinger may carry the packet name of the WeChat APP in a query request to send the query request to the AudioPolicy, and request the AudioPolicy to query the specific audio output device of the audio data 2. After receiving the query request sent by the audioFlinger, the audioPolicy can query which window the WeChat APP specifically runs in according to the packet name of the WeChat APP in the query request. For example, as shown in table 16, if the packet name of the wechat APP corresponds to the ID of the display module 1, it indicates that the wechat APP is currently running in the first window 6901. Further, the AudioPolicy may determine the audio output device corresponding to the display module 1 according to the audio configuration information of each window issued by the window manager.

Taking the audio configuration information of each window shown in table 17 as an example, AudioPolicy may determine that the audio output device corresponding to the display module 1 is a bluetooth headset according to the audio configuration information shown in table 17. Further, AudioPolicy may send an indication message to the AudioFlinger, where the indication message is used to instruct the AudioFlinger to send the processed Audio data 2 to the bluetooth headset via the DMSDP Audio HAL for playing.

Further, taking the audio configuration information of each window shown in table 18 as an example, the query request sent by the AudioFlinger to AudioPolicy may also carry a type identifier of the audio data 2, for example, the audio data 2 is audio data of MUSIC type. Further, AudioPolicy may determine that the audio output device corresponding to the display module 1 is a bluetooth headset according to the audio configuration information shown in table 18. Also, AudioPolicy may determine that the bluetooth headset allows MUSIC type audio data to be played, i.e., the bluetooth headset allows the above audio data 2 to be played, according to the audio configuration information shown in table 18. Further, AudioPolicy may send an indication message to the AudioFlinger, instructing the AudioFlinger to send the processed audio data 2 to the bluetooth headset for playing through A2dp HAL.

Taking the audio configuration information of each window shown in table 19 as an example, AudioPolicy may determine that the audio output device corresponding to the display module 1 is a bluetooth headset according to the audio configuration information shown in table 19. Also, AudioPolicy can determine that the bluetooth headset allows MUSIC type audio data to be played according to the audio configuration information shown in table 19, and the volume level at the time of playing is 15 levels. Furthermore, AudioPolicy may send an indication message to AudioFlinger, where the indication message may not only indicate to AudioFlinger to send the processed audio data 2 to the bluetooth headset for playing through A2dp HAL, but also indicate that the volume level of the audio data 2 is 15 level. Thus, the AudioFlinger can amplify or reduce the gain of the audio signal in the audio data 2 at a volume level of 15 steps in response to the indication message, and transmit the output audio data 2 to the bluetooth headset through the A2dp HAL for playing.

That is to say, when the audio task in the first window 6901 and the audio task in the second window 6902 are played concurrently, the AudioPolicy may send, according to the application information and the audio configuration information of each window issued by the window manager, the audio data output by the audio task in the first window 6901 to the first audio output device (for example, the bluetooth headset) for playing with the window as a granularity, and send the audio data output by the audio task in the second window 6902 to the second audio output device (for example, the sound box) for playing. Thus, the user a can listen to the audio data from the first window 6901 by using the first audio output device, and the user B can listen to the audio data from the second window 6902 by using the second audio output device, so that the audio data output between the windows will not interfere with each other, thereby improving the audio using experience of the user.

In addition, in the scenario where the audio task 1 and the audio task 2 are concurrently played, if the AudioPolicy receives audio configuration information of each window newly issued by the window manager, the AudioPolicy may re-determine information, such as audio output devices, volume sizes, and the like of audio data in different windows according to the latest audio configuration information of each window according to the method, so as to instruct the AudioFlinger to output the audio data corresponding to each window to the latest audio output device for playing, which is not described in detail herein.

In still other embodiments, still exemplified by audio task 1 and audio task 2 above, as shown in fig. 77, after audio task 1 is created by video APP in second window 6902, AudioTrack 1 corresponding to audio task 1 may be created in the application framework layer. Also, after audio task 2 is created by the WeChat APP in the first window 6901, an Audio track 2 corresponding to audio task 2 can be created in the application framework layer. Unlike the above-described embodiment, the window manager may transmit the application information and the audio configuration information of each window to the AudioTrack 1 and AudioTrack 2 without transmitting the application information and the audio configuration information of each window to AudioPolicy.

Thus, as shown in fig. 77, the AudioTrack 1 can determine that the audio data received by itself corresponds to the display module 1 according to the application information. Furthermore, the AudioTrack 1 may determine, according to the audio configuration information, that the audio output device associated with the display module 1 is a sound box, that is, the audio data received by the AudioTrack 1 needs to be output to the sound box for playing. Then, after receiving the Audio data 1 from the video APP, the subsequent AudioTrack 1 may send the Audio data 1 to the AudioFlinger for processing such as sampling, and at the same time, the AudioTrack 1 may instruct the AudioFlinger to send the processed Audio data 1 to the sound box for playing through the DMSDP Audio HAL.

Similarly, as shown in fig. 77, the AudioTrack 2 may determine that the audio data received by itself corresponds to the display module 2 according to the application information. Furthermore, the AudioTrack 2 may determine, according to the audio configuration information, that the audio output device associated with the display module 2 is a bluetooth headset, that is, the audio data received by the AudioTrack 2 needs to be output to the bluetooth headset for playing. Then, after receiving the audio data 2 from the WeChat APP, the subsequent AudioTrack 2 may send the audio data 2 to the AudioFlinger for sampling and other processing, and meanwhile, the AudioTrack 1 may instruct the AudioFlinger to send the processed audio data 2 to the Bluetooth headset through the A2dp HAL for playing.

Similar to the above embodiments, if the user modifies the audio focus application in a window, the window manager may send the latest application information to the current plurality of audiotracks in response to the user's modification. Alternatively, if the user modifies the audio output device associated with a window, the volume level of the audio data, etc., the window manager may send the most recent audio configuration information to the current plurality of audiotracks in response to the user's modification. In this way, each AudioTrack may re-determine information such as audio output devices, volume sizes, and the like of audio data in different windows according to the latest application information and audio configuration information by the above method, so as to instruct the audiomaker to output the audio data corresponding to each window to the latest audio output device for playing, which is not described in detail herein.

In other embodiments, after the mobile phone enters the split-screen mode, each window in the display screen of the mobile phone can be projected to other electronic devices for display through the screen projection function.

Illustratively, as shown in fig. 78, after the mobile phone enters the split screen mode, a WeChat APP is run in a first window 6901, and a video APP is run in a second window 6902. If the user needs to use the screen projection function, the user can open the control center 7801 of the mobile phone. A card 7802 may be provided in the control center 7801, and the card 7802 is used to show one or more nearby candidate devices that can be projected for screen detection by the mobile phone. If it is detected that the user selects a certain electronic device (e.g., a television) in the card 7802, indicating that the user wishes to start the screen projection function, the display data in the mobile phone is projected to the television for display, that is, the display output device of the mobile phone at this time is the television.

In one possible implementation, as shown in fig. 79, in response to a user selecting a television operation in a card 7802, the window manager may instruct the Display HAL to combine Display data 1 in Display module 1 and Display data 2 in Display module 2 into corresponding Display data 3 in real time. Furthermore, the Display HAL can send the Display data 3 to the tv for displaying in real time, so that the tv can Display the first window 6901 and the second window 6902 of the mobile phone in the split screen mode. Of course, the Display HAL may also send the Display data 3 to the Display screen of the mobile phone in real time for Display, and at this time, the mobile phone and the tv may synchronously Display the first window 6901 and the second window 6902.

In addition, if the user selects the tv from the card 1702 and the network connection between the mobile phone and the tv is not established, the mobile phone may establish the network connection with the tv first, and then send the Display data 3 to the tv in real time through the Display HAL.

At this time, the audio data generated by the WeChat APP in the first window 6901 and the audio data generated by the video APP in the second window 6902 can still be sent to different audio output devices by the Audio Flinger according to the instruction of the Audio policy for playing. For example, Audio data 1 generated by video APP can be sent to a sound box for playing through DMSDP Audio HAL, and Audio data 2 generated by WeChat APP can be sent to a bluetooth headset for playing through A2dp HAL. That is to say, in the screen-shot scene of the split-screen mode, the distribution process of the display data and the audio data can be respectively controlled by the main device, and the audio data in different windows are distributed to the corresponding audio output devices by the main device for playing.

In another possible implementation, as shown in fig. 80, in response to the user selecting a tv in the card 7802, the mobile phone may not only send the Display data 3 containing the first window 6901 and the second window 6902 to the tv for displaying in real time through the Display HAL, but also send the Audio data 1 from the first window 6901 and the Audio data 2 from the second window 6902 to the tv in real time through the DMSDP Audio HAL. Also, as shown in fig. 80, the AudioPolicy in the mobile phone may also send the application information and the audio configuration information of each window sent by the window manager to the television.

For example, as shown in fig. 80, an agent APP may be installed in an application layer of a television, where the agent APP is configured to receive display data and audio data sent by a mobile phone, for example, the display data may be the display data 3, and the audio data may be the audio data 1 and the audio data 2. Also, similar to the display architecture and audio architecture of a mobile phone, the application framework layer of a tv may also include a window manager, an audio policy module (e.g., AudioPolicy), an audio processor (e.g., AudioFlinger), and the like. After receiving the Display data 3, the agent APP may call the window manager to send the Display data 3 to the Display screen of the television for displaying through the Display HAL of the television.

As shown in fig. 80, the agent APP of the television is further configured to receive application information and audio configuration information of each window sent by the mobile phone, and then the agent APP may send the application information and the audio configuration information to the AudioPolicy of the television. And after the agent APP of the television receives the audio data 1 and the audio data 2 sent by the mobile phone, an AudioTrack 1 can be created for playing the audio data 1, and an AudioTrack 2 can be created for playing the audio data 2. Further, the AudioTrack 1 may transmit the received audio data 1 to the AudioFlinger of the television for processing such as sampling, and the AudioTrack 2 may similarly transmit the received audio data 2 to the AudioFlinger of the television for processing such as sampling.

Similar to the interactive process between the AudioFlinger and the AudioPolicy in the mobile phone, after receiving the audio data 1 and the audio data 2, the AudioFlinger of the television may also send a query request to the AudioPolicy of the television to request the AudioPolicy to query specific audio output devices of the audio data 1 and the audio data 2. AudioPolicy may respond to the query request, and determine that the audio output device of audio data 1 is a sound box and the audio output device of audio data 2 is a bluetooth headset according to the application information and audio configuration information of each window. Further, the AudioPolicy of the television may instruct its AudioFlinger to send Audio data 1 to the sound box for playing through DMSDP audiohal, and instruct the AudioFlinger to send Audio data 2 to the bluetooth headset for playing through A2dp HAL. The interactive process between AudioPolicy and AudioFlinger in the television is similar to that between AudioPolicy and AudioFlinger in the mobile phone, and thus is not described herein again.

It can be seen that, in the screen projection scene of the split screen mode, the mobile phone (i.e., the master device) can transmit both the display data and the audio data generated in each window to the television (i.e., the slave device). And the mobile phone can send the application information and the audio configuration information of each window to the television together, so that the television can distribute the audio data of different windows in the mobile phone to corresponding audio output equipment according to the application information and the audio configuration information for playing, and the isolation effect that the audio data among multiple windows are not interfered with each other is realized.

In some embodiments, the user may also change the audio output device associated with each window when the cell phone is projected to the television in the split screen mode. For example, as shown in fig. 81, during the process that the mobile phone projects the first window 6901 and the second window 6902 to the television for display, if a preset gesture (for example, long pressing the first window 6901) performed by the user on the first window 6901 is detected, the mobile phone may display a first device list 8101, and the user may select an audio output device associated with the first window 6901 in the first device list 8101. Similarly, if the user is detected to perform a preset gesture on the second window 6902 (e.g., long-pressing the second window 6902), the handset may display a second device list 8102, in which the user may select an audio output device associated with the second window 6902.

For example, the television devices in the first device list 8101 (or the second device list 8102) may be one or more television devices with audio output capabilities connected to a cell phone. Alternatively, the television devices in the first device list 8101 (or the second device list 8102) may be one or more television devices with an audio output function, which are connected to a television, and the embodiment of the present application does not limit this.

The window manager in the mobile phone may update the audio configuration information of each window according to the selection of the user in the first device list 8101 (or the second device list 8102), and send the updated audio configuration information to the AudioPolicy of the mobile phone. Furthermore, the AudioPolicy of the mobile phone can send the updated audio configuration information to the television, and the agent APP of the television sends the received updated audio configuration information to the AudioPolicy of the television, so that the AudioPolicy of the television can acquire the latest audio configuration information in real time. In this way, the television can distribute the audio data from different windows to corresponding audio output devices for playing according to the latest audio configuration information.

In addition, if the television detects that the audio output device associated with a certain window in the audio configuration information does not establish network connection with the television, the television can establish network connection with the audio output device first and then send the audio data from the window to the audio output device through the established network connection.

In the above embodiments, the application scenario in which multiple windows are displayed in the mobile phone for performing multi-audio task concurrent playing is illustrated, and in some embodiments, the mobile phone may also hide one or more windows displayed in the display screen. For example, a mobile phone may apply a granularity for screen projection, as shown in fig. 82, when the mobile phone projects a certain application (e.g., a video APP) to a television, the window manager may create a display module 1, and draw display data 1 generated by the video APP in the display module 1, that is, the display module 1 corresponds to the television. The Display HAL of the mobile phone can acquire the Display data 1 from the Display module 1 in real time and send the Display data 1 to the television for displaying. The difference is that the Display HAL does not need to send the Display data 1 to the Display screen of the mobile phone for Display, i.e. the mobile phone can hide the window corresponding to the Display module 1 in the Display screen.

Meanwhile, the user can run other applications in the mobile phone and display the application interface of the application. For example, after a user opens a wechat APP in a mobile phone, the wechat APP may create the display module 2 through a window manager. The window manager can draw the display data 2 generated by the WeChat APP in the display module 2, namely the display module 2 corresponds to the display screen of the receiving mobile phone. Furthermore, the Display HAL of the mobile phone can acquire the Display data 2 from the Display module 2 in real time and send the Display data 2 to the Display screen of the mobile phone for displaying. At the moment, the mobile phone projects the running video APP into the television for displaying, and the mobile phone keeps the running WeChat APP in the display screen of the mobile phone for displaying. That is, the mobile phone can also work normally when projecting a certain application to the television, and the user can also run other applications on the mobile phone.

In this scenario, the audio output device associated with display module 1 is the audio output device of a slave device (i.e., a television), and the audio output device associated with display module 2 is the audio output device of a master device (i.e., a cell phone). Similar to the above embodiment, still as shown in fig. 82, the video APP can send the generated audio data 1 to the AudioFlinger by creating AudioTrack 1, and at the same time, the WeChat APP can send the generated audio data 2 to the AudioFlinger by creating AudioTrack 2. AudioPolicy stores the latest application information and audio configuration information of each window. The AudioFlinger can determine that the audio output device of the audio data 1 is a sound box and the audio output device of the audio data 2 is a bluetooth headset through interaction with AudioPolicy. Further, the AudioFlinger can send Audio data 1 to the sound box for playing through the DMSDP Audio HAL, and the AudioFlinger can send Audio data 2 to the bluetooth headset for playing through the A2dp HAL.

That is to say, after the mobile phone performs the split screen mode, the display data can be provided to different display output devices (including the mobile phone itself) through different display modules, and the different display output devices can be associated with different audio output devices, and the mobile phone can send the audio data matched with the corresponding display output device to the associated audio output device for playing. Therefore, the display data in each display output device can not interfere with each other, and the audio data in each display output device can not interfere with each other, so that the audio use experience of a user is improved.

In a distributed audio scenario, the master device may have a screen recording function; the slave device of the master device may also have a screen recording function. The host device (e.g., the mobile phone) may record first screen data of itself, and the first screen data may include display data and audio data played by the host device. And, the master device may obtain second screen data recorded by the slave device from a slave device (e.g., a television) having a screen recording function, where the second screen data may include display data and audio data played by the slave device.

After a user starts a screen recording function in the master device, the master device can acquire first screen data through the screen recording function of the master device, and meanwhile, the master device can also send a screen recording instruction to the slave device. After receiving the screen recording instruction, the slave device may record second screen data of the slave device in response to the screen recording instruction if the slave device has a screen recording function. And, the slave device may transmit the recorded second screen data to the master device in real time. Therefore, the main device can generate a screen recording file recorded by the screen according to the first screen data of the main device and the second screen data of the slave device, the screen recording file can restore the content played when the main device records the screen and can also restore the content played when the slave device records the screen, so that the service scenes realized on the main device and the slave device are more comprehensively restored, and the use experience of a user in a distributed system is improved.

For example, the mobile phone may display a screen recording button for starting a screen recording function at a control center, a negative one-screen menu, a pull-down menu, or the like. For example, as shown in fig. 83, after detecting an operation of opening a pull-down menu in the mobile phone by the user, the mobile phone may display a pull-down menu 8301, and a screen recording button 8302 is provided in the pull-down menu 8301. If the fact that the user clicks the screen recording button 8302 is detected, that the user currently has a requirement for using the screen recording function is indicated, at this time, the mobile phone can search one or more electronic devices which are currently connected to the same communication network with the mobile phone. For example, a handset may search for one or more electronic devices that are located in the same Wi-Fi network as the handset. For another example, the cell phone may query the server for one or more electronic devices that are logged into the same account as the cell phone (e.g., huayao account).

Further, as shown in fig. 84, the cellular phone may display the searched one or more electronic devices in a device list 8401. Alternatively, the mobile phone may select an electronic device having a display function among the searched one or more electronic devices, and display the electronic device having the display function in the device list 8401. Or, when the mobile phone searches one or more electronic devices currently accessing the same communication network as the mobile phone, the mobile phone can inquire whether the searched electronic devices have a screen recording function. If the mobile phone has a screen recording function, the mobile phone can display the electronic device in the device list 8401. Of course, the handset itself (i.e., the master device) may also be included in the device list 8401. The user can select which device or devices of screen data to record for this screen recording operation from the device list 8401. For example, the mobile phone may default that the user needs to record the screen data of the mobile phone itself, and at this time, the user only needs to select one or more electronic devices in the device list 8401 that the user wants to record the screen simultaneously with the mobile phone.

Illustratively, if it is detected that the user selects the television 8402 in the device list 8401, the user is required to record screen data played in the mobile phone and the television 8402 at the same time. Then, as shown in fig. 85, on one hand, the mobile phone may perform a screen recording operation to start recording the display data and the audio data played in the mobile phone, so as to obtain the first screen data. On the other hand, the mobile phone can transmit a screen recording instruction to the television 8402 by using the television 8402 as a slave device. The television 8402 can also execute a screen recording operation in response to a screen recording instruction sent by the mobile phone, and start recording display data and audio data played in the television 8402 to obtain second screen data. And the television 8402 may send the second screen data recorded in real time to the mobile phone. The mobile phone can generate the screen recording file according to the first screen data and the second screen data.

The specific process of the master device (e.g., the mobile phone) completing the screen recording operation by interacting with the slave device (e.g., the television 8402) will be described in detail in the following embodiments, and thus, details are not repeated herein.

It can be seen that, in the embodiment of the present application, a user may select to record screen data in the master device and the slave device at the same time in a screen recording scene. The master device as a master device can send a screen recording instruction to the slave device selected by the user to trigger the slave device to record the screen data of the slave device, and the master device can record the screen data of the master device at the same time. Subsequently, the main device can generate the screen recording file according to the screen data recorded by the two devices, and the screen recording file can restore the content played by the main device and the content played by the slave device, so that the service scenes realized on the main device and the slave device can be more comprehensively recorded and restored, and the use experience of a user in a distributed system is improved.

Based on the Android system architecture shown in fig. 6, as shown in fig. 86, a screen recording application with a screen recording function may be included in the application layer. The application framework layer may include MediaRecord, display module, AudioRecord, and AudioFlinger. The display module is used for storing display data to be played and outputting the display data to be played to the display screen in real time for playing. The AudioFlinger is used to process audio data from an application by mixing, resampling, sound effect setting, etc., and output the processed audio data to a corresponding audio output device (e.g., speaker, headphone, etc.) through the HAL. The AudioRecord is used for obtaining corresponding audio data from the AudioFlinger for audio recording. After the screen recording application detects that the screen recording function is opened by the user, the screen recording application can call the MediaRecord to start the screen recording process.

Illustratively, as also shown in fig. 86, the screen recording application may send a screen recording request to the MediaRecord, for example, the screen recording request may indicate that the display data played on the display screen and the audio data played on the speaker need to be recorded. Furthermore, the MediaRecord can respond to the screen recording request and acquire display data played by the display screen from the display module in real time. Also, the MediaRecord may call the AudioRecord, acquire audio data currently played by a speaker from the AudioFlinger in real time by the AudioRecord, and transmit the acquired audio data to the MediaRecord. Subsequently, the MediaRecord can synthesize the display data and the audio data acquired in real time into first screen data, and the first screen data can restore the picture and the audio which are played by the mobile phone when the screen is recorded. Further, the MediaRecord may transmit the first screen data to the screen recording application. The screen recording application can store the obtained first screen data in a screen recording file form locally in the mobile phone. Or, the screen recording application can also send the obtained first screen data to a server or other electronic equipment in real time.

Also as shown in fig. 86, HALs corresponding to different hardware modules of the handset are provided in the HAL, e.g., Audio HAL, Camera HAL, Wi-Fi HAL, etc. By way of example, the Audio HAL may be further divided into a Primary HAL, a Remote perfect HAL, A2dp HAL, etc. according to the Audio output device of the mobile phone. The Primary HAL may correspond to a microphone of a mobile phone, the A2dp HAL may correspond to a Bluetooth headset connected to the mobile phone, and the Remote Submix HAL may correspond to system tones played in the mobile phone. For example, the AudioFlinger may call the Primary HAL to obtain audio data collected by a microphone of a handset. For another example, the AudioFlinger may call A2dp HAL to output audio data to a bluetooth headset connected to a cell phone for playback. As another example, the AudioFlinger may call the Remote Submix HAL to obtain system audio for speaker playback.

In a screen recording scene, if a screen recording request sent by the screen recording application to the MediaRecord indicates to record the system audio (i.e., the in-recording scene) of the mobile phone, the AudioFlinger may obtain the audio data being played by the mobile phone from the Remote Submix HAL in real time, and send the obtained audio data to the MediaRecord through the AudioRecord. Or, if the screen recording request sent by the screen recording application to the MediaRecord indicates to record the sound collected by the microphone of the mobile phone (i.e., the external recording scene), the AudioFlinger may obtain the audio data recorded by the microphone of the mobile phone from the Primary HAL in real time and send the obtained audio data to the MediaRecord through the AudioRecord. Of course, the screen recording application can also instruct the MediaRecord to record the audio data being played by the mobile phone and the audio data collected by the microphone of the mobile phone at the same time. The AudioFlinger may mix audio data acquired from the Primary HAL and audio data acquired from the Remote Submix HAL and then transmit the mixed data to the MediaRecord through the AudioRecord.

In some embodiments, as still shown in fig. 86, the application framework layer is further provided with a Camera module, and the Camera module can acquire the display data acquired by the current Camera through the Camera HAL. For example, the screen recording application sends a screen recording request to the MediaRecord to indicate that the display data required to be recorded at this time is the display data acquired by the camera. Furthermore, the MediaRecord can respond to the screen recording request, acquire corresponding display data from the Camera module in real time, report the display data to the MediaRecord, and synthesize the display data and the audio data acquired in real time into first screen data by the MediaRecord.

That is to say, the first screen data recorded by the mobile phone during screen recording may include at least one of display data played in a display screen of the mobile phone, display data acquired by a camera of the mobile phone, audio data played by the mobile phone itself, or audio data acquired by a microphone of the mobile phone.

In this embodiment, after the network connection between the mobile phone and the slave device is established, the mobile phone may create a DMSDP HAL corresponding to the slave device in the HAL, where the DMSDP HAL corresponds to the slave device to which the mobile phone is currently connected. The mobile phone can be used as a master control device to transmit and receive data with the slave device through the DMSDP HAL, and the slave device is used as a virtual device of the mobile phone to cooperate with the slave device to complete services such as audio or display in a distributed scene.

For example, when a user starts a screen recording function provided by a screen recording application in a mobile phone, the screen recording application may display a currently searched electronic device that can record a screen synchronously with the mobile phone to the user. For example, the screen recording application may present to the user an electronic device that is currently logged into the same account as a cell phone. For example, when the user selects the electronic device 1, the mobile phone may establish a connection with the electronic device 1 selected by the user as a slave device. For example, the handset may establish a WiFi P2P (peer-to-peer) connection or a bluetooth connection with the electronic device 1. As shown in fig. 87, the mobile phone may create a DMSDP HAL corresponding to the slave device in the HAL, that is, the DMSDP HAL corresponds to the electronic device 1.

In the embodiment of the present application, as also shown in fig. 87, a screen recorder may be disposed in the application framework layer, and the screen recorder may not only record the first screen data of the mobile phone itself in accordance with the MediaRecord function, but also instruct the slave device of the mobile phone to record the second screen data of the slave device. Furthermore, the screen recorder can generate a corresponding screen recording file according to the first screen data and the second screen data, and the function of synchronously recording the screen of the mobile phone and the slave equipment is achieved.

For example, the screen recorder may obtain the screen recording capability parameter of the electronic device 1 through the DMSDP HAL. The screen recording capability parameter may include a display capability parameter and an audio capability parameter supported when the slave device records a screen. For example, the display capability parameters may specifically include capability parameters for recording display data, such as resolution of the slave device, DPI (Dots Per Inch), and the like; the audio capability parameters may specifically include a sampling rate of the slave device for audio data, whether the slave device has the capability of recording system sound or whether the slave device has the capability of collecting audio using a microphone, and other capability parameters for recording audio data.

In addition, the screen recorder can acquire the recording capability parameters of the mobile phone, namely the display capability parameters and the audio capability parameters of the mobile phone. Furthermore, the screen recorder can determine the screen recording parameters used by the mobile phone and the electronic device 1 during the screen recording according to the screen recording capability parameters of the electronic device 1 and the screen recording capability parameters of the mobile phone. Corresponding to the screen recording capability parameter, the screen recording parameter may include a display recording parameter such as resolution, DPI, and an audio recording parameter such as a sampling rate of audio data, whether the system audio is recorded or not, or whether the system audio is recorded or not, and the audio acquisition capability of a microphone is provided. That is to say, the master device may determine, according to the screen recording capability of the slave device, the screen recording parameter used when the mobile phone and the slave device record the screen synchronously.

Further, as also shown in fig. 87, the screen recorder may transmit the screen recording parameters determined for the electronic device 1 to the electronic device 1 (i.e., the slave device). For example, the screen recorder may send the identifier of the electronic device 1 and the screen recording parameter to the DMSDP HAL, and the DMSDP HAL may send a screen recording instruction to the electronic device 1 according to the identifier of the electronic device 1, where the screen recording instruction includes the screen recording parameter. Therefore, after the electronic device 1 receives the screen recording instruction sent by the mobile phone, the screen data of the electronic device 1 can be recorded according to the screen recording parameters in the screen recording instruction. For example, the screen data of the electronic device 1 may include one or more of display data being played by a display screen of the electronic device 1, display data being collected by a camera of the electronic device 1, audio data being played by a speaker of the electronic device 1, or audio data being collected by a microphone of the electronic device 1. Before the screen recording is finished, the electronic device 1 may send the recorded screen data (which may be referred to as second screen data) to the screen recorder in real time.

As shown in fig. 87, for the mobile phone, after the screen recorder determines the corresponding screen recording parameter for the mobile phone, the screen recorder may start to record the screen data (which may be referred to as the first screen data subsequently) of the mobile phone according to the screen recording parameter. For example, the first screen data may include one or more of display data being played by a display screen of the cell phone, display data being collected by a camera of the cell phone, audio data being played by a speaker of the cell phone, or audio data being collected by a microphone of the cell phone.

Therefore, the screen recorder in the mobile phone can acquire the first screen data recorded by the mobile phone and the second screen data recorded by the electronic device 1 in real time in the screen recording process. Subsequently, the screen recorder may report the first screen data and the second screen data to the screen recording application. The screen recording application may encapsulate the received first screen data and the second screen data into one screen recording file, or the screen recording application may encapsulate the received first screen data and the received second screen data into two screen recording files, respectively. Of course, the screen recorder may also package the first screen data and the second screen data into one or more screen recording files and report the screen recording files to the screen recording application.

In addition, the screen recorder or other functional modules in the mobile phone may further perform operations such as parsing, encoding and decoding, mixing or adding sound effects to the first screen data and/or the second screen data, which is not limited in this application.

It can be seen that, in the screen recording scene, the mobile phone (i.e., the master device) may synchronously acquire the first screen data recorded by the master device and the second screen data recorded by the electronic device 1 (i.e., the slave device). Then, the screen recording file generated by the mobile phone according to the first screen data and the second screen data can restore the content played by the master device and can also restore the content played by the slave device, so that the service scenes realized on the master device and the slave device can be more comprehensively recorded and restored, the user can record the screen data in a plurality of electronic devices simultaneously, and the screen recording use experience of the user in a distributed system is improved.

Still take a mobile phone as an example of a main device, a screen recording method provided by the embodiment of the present application will be specifically described below with reference to the accompanying drawings.

As shown in fig. 88 (a), the mobile phone is exemplified by providing a screen recording button 8802 in the drop-down menu 8801. If the user is detected to click the screen recording button 8802, the mobile phone can be triggered to acquire one or more electronic devices currently located in the same communication network as the mobile phone. For example, a cell phone may search for electronic devices that access the same Wi-Fi network or log into the same account as the cell phone. Furthermore, the mobile phone can inquire the electronic equipment with the screen recording function in the searched electronic equipment. For example, the mobile phone may send a request message to the searched electronic device, and the electronic device requesting to receive the request message reports whether the electronic device has a screen recording function.

For example, in the case where the television 1 and the tablet computer 1 have a screen recording function, as shown in (b) of fig. 88, the mobile phone may display the television 1 and the tablet computer 1 in a dialog box 8803. The user may select one or more slave devices in dialog box 8803 that are currently being recorded in the synchronized screen with the handset. That is, after the user turns on the screen recording function in the mobile phone (i.e., the master device), the user can select one or more slave devices to perform screen recording synchronously with the mobile phone.

In some embodiments, the cell phone may also display the cell phone itself as an option in the dialog box 8803. At this time, the user may select to record the screen data in the mobile phone or may select not to record the screen data in the mobile phone. For example, the user may select to record screen data in the television 1 and the tablet pc 1 synchronously, but not record screen data in a mobile phone. Of course, in addition to clicking the screen recording button 8802, the user may also open the function of synchronously recording the screen of the mobile phone and other devices by inputting preset gestures or voice commands to the mobile phone, which is not limited in this embodiment of the present application.

For example, when the user selects the television 1 in the dialog box 8803, and after detecting that the user selects the television 1, as shown in fig. 89, the mobile phone may use the television 1 as a slave device of the mobile phone, and trigger the screen recording application to acquire the screen recording capability parameter (which may be referred to as a second screen recording capability parameter) of the television 1. The second screen recording capability parameter may include a display capability parameter and an audio capability parameter supported by the television 1 during screen recording. For example, the display capability parameter may include parameters such as resolution or DPI of the television 1; the audio capability parameters may include parameters such as the sampling rate of the television 1, the ability to record system sounds, or the ability to capture audio using a microphone. It is understood that the parameters that may be used when the television 1 records the screen can be used as the second screen recording capability parameter. For example, the second screen recording capability parameter may further include parameters such as an audio or image codec format supported by the television 1, and a focal length and a resolution of a camera in the television 1, which are not limited in this embodiment of the present application.

Illustratively, the second screen recording capability parameter of the television 1 is used to indicate that the resolution of the television 1 is 1080 × 720, the DPI of the television 1 is 200, the sampling rate of the television 1 is 36KHz, and the television 1 supports recording system audio and does not support recording audio by using a microphone. The television 1 may send the second screen recording capability parameter to the DMSDP HAL of the mobile phone, and the DMSDP HAL sends the second screen recording capability parameter to the screen recording application.

As also shown in fig. 89, the screen recording application may acquire the screen recording capability parameter of the mobile phone itself (may be referred to as a first screen recording capability parameter) in addition to the second screen recording capability parameter of the television 1. Similar to the second screen recording capability parameter, the first screen recording capability parameter may include a display capability parameter and an audio capability parameter supported when the mobile phone records a screen. For example, the first screen recording capability parameter of the mobile phone is used to indicate that the resolution of the mobile phone is 1920 × 1080, the DPI of the mobile phone is 300, the sampling rate of the mobile phone is 48KHz, and the mobile phone supports recording system audio and also supports recording audio by using a microphone.

Further, as also shown in fig. 89, the screen recording application may send a screen recording request 1 to the screen recorder 1 in the application framework layer, where the screen recording request 1 may include the first screen recording capability parameter and the second screen recording capability parameter. In response to the screen recording request 1, the screen recorder 1 may determine, according to the first screen recording capability parameter and the second screen recording capability parameter, a screen recording parameter used in the screen recording. The screen recording parameters may include a display recording parameter corresponding to the display capability parameter and an audio recording parameter corresponding to the audio capability parameter. Therein, the screen recorder 1 may provide a screen recording interface, for example, the name HwMediaProject, to an application (e.g., a screen recording application) in the application framework layer. The screen recording application may interact with the screen recorder 1 by calling a screen recording interface HwMediaProject.

For example, when the resolution 1920 × 1080 of the mobile phone is greater than the resolution 1080 × 720 of the television 1, the screen recorder 1 may set the resolution at this screen recording time to 1080 × 720 (i.e., the resolution of the television 1), so as to ensure that both the mobile phone and the television 1 can complete the screen recording operation.

For another example, when the DPI 300 of the mobile phone is larger than the DPI 200 of the television 1, the screen recorder 1 may set the DPI of the current screen recording to 200 (i.e., the DPI of the television 1), so as to ensure that both the television 1 and the mobile phone can complete the current screen recording operation.

For another example, when the sampling rate 1 of the mobile phone to the audio data is greater than the sampling rate 2 of the television 1 to the audio data, the screen recorder 1 may set the sampling rate to the audio data during this screen recording to be the sampling rate 2 (i.e., the sampling rate of the television 1), so as to ensure that both the television 1 and the mobile phone can complete this screen recording operation.

For another example, the television 1 supports recording system audio, and does not support recording system audio by using a microphone, and the mobile phone supports recording system audio, and also supports recording system audio by using a microphone, so the screen recorder 1 can set the television 1 to record system audio in the current screen recording operation, and the mobile phone records system audio and microphone audio simultaneously in the current screen recording operation.

Thus, as shown in table 20, the screen recorder 1 can determine the screen recording parameters used by the mobile phone and the television 1 in the screen recording operation according to the first screen recording capability parameter and the second screen recording capability parameter, where the screen recording parameters include a display recording parameter and an audio recording parameter. The screen recording parameters simultaneously meet the screen recording capabilities of the mobile phone (namely, the master device) and the television 1 (namely, the slave device), so that the mobile phone and the television 1 can complete the screen recording operation according to the determined screen recording parameters.

Watch 20

In some embodiments, the screen recorder 1 may also determine the above-mentioned screen recording parameters in conjunction with the current network status. For example, when the network bandwidth between the mobile phone and the television 1 is smaller than the preset value, which indicates that the network state between the mobile phone and the television 1 is not good, at this time, the screen recorder 1 may set the resolution used by the mobile phone and the television 1 in the screen recording parameters as a default value, where the default value may be smaller than the current resolution of the mobile phone and the television 1. Therefore, the data volume of the screen data acquired by the television 1 during screen recording according to the resolution is small, and the condition that the television 1 cannot timely send the recorded screen data to the mobile phone due to insufficient bandwidth during synchronous screen recording of the television 1 and the mobile phone can be avoided.

In some embodiments, the screen recorder 1 may also determine the above-mentioned screen recording parameters in connection with the current service scenario. For example, when the screen recording application is a live broadcast type application, since the live broadcast service has a high requirement on the delay of audio and picture, the screen recorder 1 may set the resolution used by the mobile phone and the television 1 in the screen recording parameters to the default value. Therefore, the data volume of the screen data acquired by the television 1 during screen recording according to the resolution is small, and the condition that the screen data recorded by the mobile phone and the television 1 are not synchronous due to overhigh data volume during the subsequent synchronous screen recording of the television 1 and the mobile phone can be avoided.

In some embodiments, the display recording parameter of the screen parameters may further include a source of the display data. The source of the display data may include a display screen and/or a camera. For example, when the screen recording application is a live-type application, the screen recorder 1 may set the source of the display data recorded by the mobile phone as the camera, and the source of the display data recorded by the television as the display screen. That is to say, in the screen recording operation, the screen recording application can record the display data acquired by the camera on the mobile phone and the display data played by the display screen in the television 1 at the same time.

In other embodiments, the user may also manually set or modify the screen parameters in the screen recording process. For example, the screen recorder 1 may determine one or more optional screen parameters according to the first screen recording capability parameter and the second screen recording capability parameter. The screen recorder 1 may display selectable screen parameters to the user through the screen recording application, for example, whether to start a microphone recording, whether to start a camera recording, and the like, which are manually set by the user.

As shown in fig. 90, the recording screen application may display a recording control column 9001 of the mobile phone and a recording control column 9002 of the television 1 during the recording process. Information such as the current recording time length may be displayed in recording control field 9001 and recording control field 9002.

And when the first screen recording capability parameter of the mobile phone indicates that the mobile phone supports the microphone to record audio data and supports the camera to collect display data, the screen recording application may display a microphone button 9003 and a camera button 9004 in the recording control box 9001. For example, the user may turn on or off the microphone of the mobile phone by clicking the microphone button 9003, and the screen recorder 1 may modify the audio recording parameters used by the mobile phone among the screen parameters according to the user's operation of the microphone button 9003. For another example, the user may open or close a camera of a mobile phone by clicking the camera button 9004, and the screen recorder 1 may modify display recording parameters used by the mobile phone among screen parameters according to the user's operation of the camera button 9004.

Similarly, when the second screen recording capability parameter of the television 1 indicates that the television 1 does not support microphone recording of audio data and a camera is enabled to capture display data, the screen recording application may display a camera button 9005 in the recording control box 9001. The user can turn on or off the camera of the television 1 by clicking the camera button 9005, and the screen recorder 1 can modify the display recording parameters used by the television 1 among the screen parameters according to the user's operation of the camera button 9005. Of course, during the recording process, the mobile phone may also instruct the television 1 to display the recording control column 9001 and/or the recording control column 9002, which is not limited in this embodiment.

After detecting that the user modifies the screen recording parameters used by the mobile phone or the television 1, the screen recording application can send the latest screen recording parameters to the screen recorder 1. Subsequently, the screen recorder 1 can control the mobile phone and the television 1 to perform synchronous screen recording operation according to the latest screen recording parameters.

In addition, in the foregoing embodiment, the screen recorder 1 determines the screen recording parameters used by the mobile phone and the television 1 in the screen recording operation according to the first screen recording capability parameter and the second screen recording capability parameter, which is exemplified, it can be understood that the screen recording application may also determine the screen recording parameters used by the mobile phone and the television 1 according to the method in the foregoing embodiment, and then the screen recording application may carry the determined screen recording parameters in the screen recording request 1 and send the screen recording parameters to the screen recorder 1, which is not limited in this embodiment.

As shown in fig. 89, after the screen recorder 1 determines the screen recording parameters used by the mobile phone and the television 1, on one hand, the screen recorder 1 may trigger the mobile phone to perform the screen recording operation according to the screen recording parameters corresponding to the mobile phone. On the other hand, the screen recorder 1 can send the screen recording parameters used by the television 1 to the television 1, and the television 1 executes the screen recording operation according to the screen recording parameters corresponding to the television 1. For example, the screen recorder 1 may transmit the identifier of the television 1 and the screen recording parameters corresponding to the television 1 in the table 20 to the DMSDP HAL through the AudioFlinger, and the DMSDP HAL sends a screen recording instruction to the television 1 according to the identifier of the television 1, where the screen recording instruction may include the screen recording parameters used by the television 1.

And for the mobile phone itself, the screen recorder 1 may call the display module, and obtain the display data 1 being played in the display screen of the mobile phone in real time from the display module according to the display recording parameter corresponding to the mobile phone in the recording screen parameter. Or, if the display recording parameters indicate that the display data acquired by the mobile phone Camera needs to be recorded, the screen recorder 1 may call the Camera module, and acquire the display data acquired by the mobile phone Camera in real time from the Camera module according to the corresponding display recording parameters. Of course, the screen recorder 1 may also obtain the display data being played in the display screen of the mobile phone from the display module, and obtain the display data collected by the Camera of the mobile phone from the Camera module.

Meanwhile, the screen recorder 1 may call the AudioRecord, and obtain the audio data 1 of the mobile phone in real time from the AudioFlinger according to the audio recording parameter corresponding to the mobile phone in the screen recording parameters. For example, if the mobile phone records system audio as indicated in the audio recording parameter, the AudioFlinger may obtain the audio data being played by the mobile phone from the Remote Submix HAL. Thus, the audio data 1 acquired by the screen recorder 1 from the AudioFlinger is the system audio of the mobile phone. For another example, if the handset is instructed in the audio recording parameter to record microphone audio, the AudioFlinger may acquire audio data recorded by the handset's microphone from the Primary HAL. Thus, the audio data 1 acquired by the screen recorder 1 from the AudioFlinger is the audio collected by the microphone of the mobile phone. Still alternatively, if the mobile phone recording system audio and the microphone audio are indicated in the audio recording parameter, the AudioFlinger may mix audio data acquired from the Remote sub mix HAL and audio data acquired from the Primary HAL. In this way, the audio data 1 acquired by the screen recorder 1 from the AudioFlinger is the audio collected by the microphone of the mobile phone and the system audio of the mobile phone.

As shown in fig. 89, the screen recorder 1 can acquire the display data 1 and the audio data 1 of the mobile phone in real time, that is, the first screen data recorded by the mobile phone, and the first screen data can record and restore the audio and the picture being played (or collected) in the mobile phone. Meanwhile, for the television 1, after receiving a screen recording instruction sent by the mobile phone, the television 1 can execute screen recording operation according to the screen recording parameters corresponding to the television 1 in the screen recording instruction by utilizing the screen recording capability of the television 1.

As shown in fig. 91, similar to the software architecture of the operating system in the mobile phone, an agent application may be set in the application layer of the television 1, a screen recorder 2, a display module, a Camera module, an AudioRecord, an AudioFlinger and other functional modules are set in the application framework layer of the television 1, and a Camera HAL, a Remote sub HAL, a Primary HAL and the like are set in the HAL of the television 1.

Wherein the proxy application of the television 1 may be used for interacting with the mobile phone. For example, after the television 1 is connected to the mobile phone, the agent application of the television 1 may call the screen recorder 2 to obtain the screen recording capability parameter of the television 1 (i.e., the second screen recording capability parameter), and send the second screen recording capability parameter to the mobile phone (e.g., the screen recording application of the mobile phone or the screen recording recorder 1 of the mobile phone). For another example, the agent application may receive a screen recording instruction sent by a mobile phone (e.g., a screen recording application of the mobile phone or the screen recording recorder 1 of the mobile phone), where the screen recording instruction includes screen recording parameters corresponding to the television 1. In response to the screen recording instruction, the agent application may call the screen recorder 2 to execute the screen recording operation according to the screen recording parameter.

Similar to the process of executing the screen recording operation by the mobile phone, the screen recorder 2 may call the display module of the television 1, and obtain the display data 2 being played in the display screen of the television 1 in real time from the display module according to the display recording parameter corresponding to the television 1 in the screen recording parameters. Or, if the display recording parameters indicate that the display data acquired by the Camera of the television 1 needs to be recorded, the screen recorder 2 may call the Camera module of the television 1, and acquire the display data acquired by the Camera of the television 1 in real time from the Camera module according to the corresponding display recording parameters. Of course, the screen recorder 2 may also obtain the display data being played in the display screen of the television 1 from the display module, and obtain the display data collected by the Camera of the television 1 from the Camera module, which is not limited in this embodiment of the present application.

Meanwhile, the screen recorder 2 can call the AudioRecord of the television 1, and the audio data 2 of the television 1 is obtained in real time from the audioFlinger according to the audio recording parameter corresponding to the television 1 in the screen recording parameters. For example, the AudioFlinger may obtain audio data being played by the television 1 from the Remote Submix HAL. As another example, the AudioFlinger may acquire audio data collected by a microphone of the television 1 from the Primary HAL.

In this way, the screen recorder 2 in the television 1 can acquire the display data 2 and the audio data 2 of the television 1 in real time, that is, the second screen data recorded by the television 1, and the second screen data can record and restore the audio and the picture being played (or collected) in the television 1. Further, the screen recorder 2 may send the acquired second screen data to the mobile phone. Alternatively, the screen recorder 2 may report the acquired second screen data to the agent application, and the agent application sends the second screen data to the mobile phone. The data interaction between the television 1 and the mobile phone can be realized through the DMSDP HAL in the mobile phone.

As also shown in fig. 89, the screen recorder 1 in the mobile phone can receive the second screen data sent from the screen recorder 2 in the television 1 (or the proxy application of the television 1) through the DMSDP HAL. In this way, the screen recorder 1 can acquire the first screen data recorded by the mobile phone and the second screen data recorded by the television 1 in real time. The first screen data comprises display data 1 and audio data 1 recorded by a mobile phone, and the second screen data comprises display data 2 and audio data 2 recorded by a television 1. Subsequently, the screen recorder 1 may report the first screen data and the second screen data to the screen recording application, and the screen recording application may call the corresponding interface to make the first screen data and the second screen data into the screen recording file. Of course, the screen recorder 1 may also make the screen recording file according to the first screen data and the second screen data, and report the screen recording file to the screen recording application, which is not limited in this embodiment of the application.

In some embodiments, for example, the screen recording application may create the screen recording file according to the first screen data and the second screen data, and the screen recording application may create the first screen data and the second screen data as one screen recording file. For example, as shown in fig. 92, since the first screen data and the second screen data are recorded according to the unified recording parameters shown in table 20, and therefore, the display recording parameters such as the resolution of the display data 1 in the first screen data and the display data 2 in the second screen data are unified, the recording application may re-encode the display data 1 and the display data 2, and combine the display data 1 and the display data 2 into the display data 3 corresponding to one display screen. The screen recording application may set a display track in the screen recording file, the display track corresponding to the display data 3. In the embodiment of the present application, no limitation is imposed on the position, shape, and ratio of the display data 1 and the display data 2 in the combined display screen.

As also shown in fig. 92, the screen recording application may set two audio tracks in the screen recording file, where audio track 1 corresponds to audio data 1 in the first screen data and audio track 2 corresponds to audio data 2 in the second screen data. Of course, the screen recording application may also perform operations such as encoding on the audio data 1 and the audio data 2, which is not limited in this embodiment of the application. Finally, the screen recording file produced by the screen recording application includes the display data 3 and the audio data on the audio track 1 and the audio track 2.

Alternatively, the screen recording application may synthesize the audio data 1 and the audio data 2 into the audio data 3 by audio mixing. At this time, the screen recording application may set an audio track and audio data 3 in the screen recording file, and set a display track corresponding to the display data 3, which is not limited in this embodiment of the application.

In the process of synchronously recording the screen of the mobile phone and the television 1, a certain time consumption is required for the television 1 to transmit the recorded second screen data to the mobile phone through a communication network, and the phenomenon that the first screen data recorded by the mobile phone and the second screen data recorded by the television 1 are not synchronized in time may be caused. In some embodiments, the screen recording application may further obtain a transmission delay T in the screen recording process. For example, the screen recording application may periodically query the current transmission delay T of the Wi-Fi module through the DMSDP HAL. When the transmission delay T is larger than the threshold value, the screen recording application can synchronize the first screen data and the second screen data according to the current transmission delay T, and then the screen recording application can make the synchronized first screen data and the synchronized second screen data into a screen recording file according to the method, so that the phenomenon that the screen data are not synchronized when multiple devices record screens synchronously is avoided.

In some embodiments, the screen recording application may make the first screen data and the second screen data as corresponding two screen recording files, respectively. For example, as shown in fig. 93 (a), the screen recording application may perform operations such as encoding and decoding or packaging on the display data 1 and the audio data 1 in the first screen data, so as to obtain a screen recording file 1. The display track in the screen recording file 1 corresponds to the display data 1, and the audio track in the screen recording file 1 corresponds to the audio data 1. Further, as shown in (b) of fig. 93, the screen recording application may perform operations such as encoding and decoding or packaging on the display data 2 and the audio data 2 in the second screen data, so as to obtain the screen recording file 2. The display track in the screen recording file 2 corresponds to the display data 2, and the audio track in the screen recording file 2 corresponds to the audio data 2.

It should be noted that the screen recording application may store the one or more screen recording files obtained by the creation in the local of the mobile phone, or may send the one or more screen recording files obtained by the creation to the television 1, and the television 1 stores the screen recording files in the local of the television 1. Or, the screen recording application may also send the generated screen recording file to a server or other electronic devices in real time, which is not limited in this embodiment.

In addition, if the user also selects to use other slave devices to record the screen synchronously with the mobile phone and the television 1 when the screen recording function is turned on, the mobile phone can also instruct the other slave devices to record the screen data of the mobile phone according to the screen recording parameters determined by the mobile phone, similar to the process of synchronously recording the screen by the mobile phone and the television 1 in the above embodiment. At this time, the screen recording application may also receive screen data sent from other devices, and the screen recording application may still make the acquired screen data recorded by the multiple devices into corresponding screen recording files according to the above method.

In some embodiments, in addition to the handset interacting with a slave device (e.g., television 1) using a DMSDP HAL by creating the DMSDP HAL, the handset may also interact with the slave device by the proxy application of the handset by setting up the proxy application at the application layer. Illustratively, as shown in fig. 94, the proxy application is provided in each of the mobile phone and the television 1. The proxy application of the mobile phone can interact with the proxy application of the television 1, and data receiving and sending between the mobile phone and the television 1 are achieved. For example, in the process of synchronously recording the screen of the mobile phone and the television 1, the mobile phone may send the screen recording parameters of the television 1 to the proxy application of the television 1 through the proxy application, and the proxy application of the television 1 calls the screen recorder 2 to record the second screen data according to the method described in fig. 91. And, the proxy application of the television 1 may transmit the second screen data to the proxy application of the mobile phone, and the proxy application of the mobile phone may transmit the second screen data to the screen recording application. Similarly, the screen recording application may acquire not only the first screen data reported by the screen recorder 1 but also the second screen data sent by the agent application. Subsequently, the screen recording application can make a corresponding screen recording file according to the first screen data and the second screen data according to the method. In this implementation, the mobile phone does not need to create a DMSDP HAL corresponding to the slave device in the HAL, and can also implement a synchronous screen recording operation between the mobile phone and other devices.

It can be seen that, in the embodiment of the present application, a mobile phone (i.e., a master device) can synchronously acquire first screen data recorded by the master device and second screen data recorded by a slave device in a screen recording scene. Furthermore, the mobile phone can generate a screen recording file according to the first screen data and the second screen data, so that audio and pictures played (or collected) in the master device and the slave device are synchronously restored, service scenes realized on the master device and the slave device are more comprehensively recorded and restored, a user can record the screen data in a plurality of electronic devices simultaneously, and screen recording use experience of the user in a distributed system is improved.

In a distributed audio scene, for example, a mobile phone is taken as a master device, and the master device may specifically include the following various scenes when projecting display data and/or audio data to a slave device.

Scene one

As shown in fig. 95, the video played by the mobile phone when running the video APP generally includes audio data and display data. The mobile phone can display data generated by the video APP by using a display screen of the mobile phone, and meanwhile, the mobile phone can send audio data generated by the video APP to the slave device (such as a sound box) for playing. That is, in scenario one, the mobile phone may project the audio data to the slave device for playing, and the mobile phone itself may play the corresponding display data.

In such a scenario, the speed of processing the display data by the mobile phone is high, and human eyes have a persistence phenomenon on the image, so that the display time delay S spent by the mobile phone in processing and displaying the display data can be ignored. The display delay S may specifically refer to a time taken for one data packet of the display data to be displayed from generation to final display. The data packet of the display data may be generated in response to a play instruction input by a user, for example, in response to a play instruction that the user clicks a play button in the video APP, the video APP may start generating the data packet of the display data to be played. Or, the data packet of the display data may also be automatically generated by the video APP, for example, the video APP may automatically generate the data packet of the display data to be played when displaying the advertisement, which is not limited in this embodiment of the present application.

And the mobile phone starts to process the audio data generated by the video APP, and sends the audio data to the sound box, and finally the audio time delay L spent by the sound box for playing the audio data is generally longer. As also shown in fig. 95, the audio delay L of one packet of audio data may include: the time L1 taken for the packet to be processed in the handset (i.e., the master device) (hereinafter referred to as the master audio delay), the time L2 taken for the packet to be transmitted between the handset and the speaker (i.e., the slave device) (hereinafter referred to as the audio transmission delay), and the time L3 taken for the speaker to start processing the packet until the audio data in the packet is finally played after the packet is received by the speaker (hereinafter referred to as the slave audio delay). It is understood that the starting time and the ending time of the foregoing processing procedure and transmission procedure can be implemented in various ways, and those skilled in the art can select the starting time and the ending time according to the needs when implementing the product.

For example, the starting time of the master audio delay L1 may be the time when the video APP generates a packet of audio data to be played, and the ending time of the master audio delay L1 may be the time when the packet is sent out from the mobile phone. The video APP can respond to a play instruction input by a user to generate a data packet of audio data to be played, or the video APP can automatically generate a data packet of audio data to be played.

For example, the start time of the audio propagation delay L2 may be the time when the packet is sent out from the handset (i.e. the end time of the audio propagation delay L1 of the host device), and the end time of the audio propagation delay L2 may be the time when the packet is transmitted to the speaker, i.e. the time when the packet is received by the speaker.

For example, the starting time of the slave device audio delay L3 may be the time when the sound box receives the data packet (i.e., the ending time of the audio transmission delay L2), and the ending time of the slave device audio delay L3 may be the time when the sound box plays the audio data in the data packet.

For example, the starting time of the main device audio delay L1 may specifically be a time when the video APP in the main device receives a user click on a play button in the video APP, or the starting time of the main device audio delay L1 may be a time when the video APP generates a play instruction in response to the user click on the play button, or the starting time of the main device audio delay L1 may be a time when the audio player in the main device receives the play instruction, and the like.

Similarly, the ending time of the audio delay L1 of the master device (i.e. the starting time of the audio transmission delay L2) may be the time when the HAL of the master device sends out the packet of audio data, or the time when the hardware module such as Wi-Fi of the master device sends out the packet of audio data.

Similarly, the ending time of the audio transmission delay L2 (i.e., the starting time of the audio delay L3) may be the time when the packet is received from the relevant application in the device, the time when the packet is received from the relevant hardware module in the device, or the like.

It can be understood that, those skilled in the art may adjust the specific time nodes of the start time and the end time according to the requirement of the detection accuracy, the actual application scenario, or the actual experience, and at this time, although the specific values of the master audio delay L1, the audio transmission delay L2, or the slave audio delay L3 slightly float, if the error caused by the float is within the preset value range, the calculation and the update process of the audio delay L will not be significantly affected.

That is, the audio delay L is the master audio delay L1+ audio transfer delay L2+ slave audio delay L3. Of course, when the audio delay L is calculated, the master device may also multiply the master device audio delay L1, the audio transmission delay L2, and the slave device audio delay L3 by a certain scaling factor and then add the result to obtain the audio delay L, that is, the audio delay L is k1, the master device audio delay L1+ k2, the audio transmission delay L2+ k3, and the slave device audio delay L3, and k1+ k2+ k3 is 1.

In addition, the master audio delay L1 may also be referred to as a first delay, the audio transmission delay L2 may also be referred to as a second delay, and the slave audio delay L3 may also be referred to as a third delay, which is not limited in this embodiment of the present application.

Then the synchronization delay T between the audio data and the display data is the audio delay L-display delay S. When the synchronous time delay T is larger than 0, the sound production of the audio data is later than the display of the display data; when the synchronous time delay T is less than 0, the sound production of the audio data is earlier than the display of the display data; when the synchronization delay T is equal to 0, it is indicated that the sound production of the audio data corresponds to the display of the display data. Because the display time delay S of the display data in the scene one can be ignored, the synchronization time delay T is approximately equal to the audio time delay L. I.e. the audio delay L may reflect the time difference between the audio data and the display data at the same moment of play.

For example, the audio delay L is 500ms, and the display delay S can be ignored, which indicates that the audio data playing at the same time is 500ms slower than the display data playing at the same time. Then, after the mobile phone obtains the audio time delay L, the audio time delay L may be used as a synchronization time delay T, and a synchronization policy for performing sound and picture synchronization processing currently is set according to the audio time delay L. For example, after the video APP receives an operation of clicking a play button in the video by a user, the current and latest audio delay L (the audio delay L is 500ms) may be obtained. Furthermore, video APP can set up in the synchronization strategy when video APP produces 1 st second display data after, the cell-phone waits for 500ms after uses the display screen to show this display data, through 500 ms's audio delay L this moment, 1 st second's audio data corresponding with 1 st second's display data has been handled and sent to and plays in the audio amplifier, thereby make the audio frequency of broadcast in the audio amplifier and the image of broadcast in the cell-phone can reach the synchronous effect of sound picture, improve user's use and experience.

Scene two

As shown in fig. 96, still by using the mobile phone to run the video APP to play the video, in a second scenario, the mobile phone may send both the audio data and the display data generated by the video APP to the slave device (e.g., a television) to play. That is, in scene two, the mobile phone can project the audio and the image in the video to the same slave device for playing.

In this scenario, the mobile phone starts processing the display data generated by the video APP, the mobile phone sends the display data to the television, and finally, the time taken for the television to play the display data is the display delay S. As also shown in fig. 96, the display delay S of one packet of display data may include: the time taken for the packet to be processed in the handset (i.e., the master device) S1 (hereinafter referred to as master display latency), the time taken for the packet to be transmitted between the handset and the television (i.e., the slave device) S2 (hereinafter referred to as display transmission latency), and the time taken for the television to start processing the packet until the display data in the packet is finally played after the packet is received by the television S3 (hereinafter referred to as slave display latency).

Similar to the host audio delay L1, the start time of the host display delay S1 may be the time when the video APP generates a packet of display data to be played, and the end time of the host display delay S1 may be the time when the packet is output from the mobile phone. The start time of the display transmission delay S2 may be the time when the packet is output from the mobile phone (i.e., the end time of the display delay S1 of the host device), and the end time of the display transmission delay S2 may be the time when the packet is transmitted to the television, i.e., the time when the packet is received by the television. The start time of the slave device display delay S3 may be the time when the tv receives the above-mentioned packet (i.e., the end time of the display transmission delay S2), and the end time of the slave device display delay S3 may be the time when the tv displays the display data in the packet.

That is, the display delay S is the master display delay S1+ the display transmission delay S2+ the slave display delay S3. Similar to the audio time delay L, when the display time delay S is calculated, the master device may also multiply the master device display time delay S1, the display transmission time delay S2, and the slave device display time delay S3 by a certain proportionality coefficient and then add the products to obtain the display time delay S.

In the case of the master device or the slave device, since the processing process of the display data in the electronic device is generally simple, that is, the processing speed is fast, the master device display time delay S1 and the slave device display time delay S3 of the display data can be ignored. At this time, the above-described display delay S ≈ display propagation delay S2.

For the audio data generated when the mobile phone runs the video APP, similar to the scenario one described above, the audio delay L of the audio data is the master device audio delay L1+ the audio transmission delay L2+ the slave device audio delay L3.

Then the synchronization delay T between the audio data and the display data is audio delay L-display delay S ≈ audio delay L-display transmission delay S2. The synchronization delay T may reflect the time difference between playing the audio data and the display data at the same time.

For example, the audio delay L is 500ms, and the display transmission delay S2 is 200ms, at this time, the synchronization delay T between the audio data and the display data is 500ms-200 ms-300 ms, that is, the audio data playing at the same time of the television is 300ms slower than the display data playing at the same time. Then, after the mobile phone obtains the synchronization delay T, a synchronization policy for performing sound and picture synchronization processing currently may be set according to the synchronization delay T. For example, the video APP may obtain the latest synchronization delay T (synchronization delay T is 300ms) when playing the nth frame of display data, and then the video APP may set, in the synchronization policy, that the television waits for 300ms to play the display data after receiving the (N + 1) th frame of display data. Because it still takes 200ms (that is, the display transmission delay S2 is 200ms) to transmit the display data of the (N + 1) th frame from the mobile phone to the television, the audio data corresponding to the display data of the (N + 1) th frame can be synchronously played in the television after being processed and transmitted for 500ms, so that the audio and the image played in the television can achieve the effect of synchronizing sound and picture, and the use experience of the user is improved.

In some embodiments, the network connection between the handset and the tv is generally fixed when the handset sends the audio data and the display data to the tv, for example, the handset may transmit the audio data and the display data based on a Wi-Fi P2P connection between the handset and the tv, then the transmission rate of the audio data and the display data over the Wi-Fi P2P connection is generally the same, and the size of the data packet in the audio data and the size of the data packet in the display data are mostly in the same order, so the time consumption of the handset to transmit one data packet in the audio data over the Wi-Fi P2P connection is substantially the same as the time consumption of transmitting one data packet in the display data, i.e., the display transmission delay S2 of the display data is approximately equal to the audio transmission delay L2 of the audio data, at which the synchronization delay T is approximately equal to (the master audio delay L1+ the audio transmission delay L2+ the slave audio delay L3) -the display transmission delay S2 is approximately equal to the master audio delay S2 L1+ slave audio delay L3. In this way, the handset can calculate the current synchronization delay T by obtaining the master audio delay L1 and the slave audio delay L3.

Scene three

As shown in fig. 97, still by using the mobile phone to run the video APP to play the video, in scene three, the mobile phone may send the display data generated by the video APP to the first slave device (for example, a television) to be played, and send the audio data generated by the video APP to the second slave device (for example, a sound box) to be played. That is, in scene three, the mobile phone may project the audio and the image in the video to different slave devices for playing.

In this scenario, the display latency S of the display data is the master display latency S1+ display transmission latency S2+ slave display latency S3. Similar to scenario two, the master display latency S1 and the slave display latency S3 of the display data are negligible. Namely: the display delay S ≈ display propagation delay S2. Unlike scenario two, the display transmission delay S2 refers to the delay in transmitting display data between the mobile phone and the television (i.e., the first slave device).

Similarly, the audio delay L of the audio data is the master audio delay L1+ audio transmission delay L2+ slave audio delay L3. Unlike scenario two, the audio transmission delay L2 refers to the delay of audio data transmission between the handset and the speaker (i.e., the second slave device).

Since the first network connection between the mobile phone and the television and the second network connection between the mobile phone and the speaker box may be different, for example, a Wi-Fi connection is established between the mobile phone and the television, and a bluetooth connection is established between the mobile phone and the speaker box, in scenario three, the display transmission delay S2 and the audio transmission delay L2 may be different.

Then, the synchronization delay T between the audio data and the display data is audio delay L — display delay S (master audio delay L1+ audio transmission delay L2+ slave audio delay L3) — display transmission delay S2.

Similarly, the mobile phone can set a synchronization strategy for currently performing sound and picture synchronization processing according to the synchronization time delay T by acquiring the synchronization time delay T, so that audio data and video data are respectively projected to the sound box and the television according to the synchronization strategy, and the audio played in the sound box and the image played in the television can achieve the sound and picture synchronization effect.

It can be seen that, in any scenario, when the mobile phone projects display data and/or audio data to the slave device, in order to achieve the effect of sound-picture synchronization, the mobile phone needs to acquire the synchronization delay T between the audio delay L and the display delay S, and then sets a synchronization policy for performing sound-picture synchronization processing according to the synchronization delay T.

In the embodiment of the present application, the mobile phone may dynamically detect and update the audio time delay L, so that the mobile phone may more accurately calculate the synchronization time delay T between the audio time delay L and the display time delay S according to the latest audio time delay L, that is, the synchronization time delay T is also dynamically updated along with the audio time delay L. Subsequently, the mobile phone can carry out a synchronization strategy of sound and picture synchronization processing according to the latest synchronization time delay T, so that the audio and the image projected by the mobile phone can more accurately achieve the sound and picture synchronization effect, and the use experience of a user is improved.

Based on the Android system architecture shown in fig. 6, as shown in fig. 98, in a process of playing an audio, audio data needs to be processed layer by an audio player, an audio processor, an audio manager, and a HAL and finally output to hardware modules such as a speaker for playing. That is, the audio delay L of the audio data starts from the generation of the audio data by the video APP, and the audio data is processed layer by the audio player, the audio processor, the audio manager, and the HAL until the audio data is played from the speaker. Obviously, the audio delay L of the audio data is typically relatively long.

When the mobile phone displays an image, the mobile phone still takes the video APP as an example, as shown in fig. 98, display data generated by the video APP can be directly analyzed through the surface module, and the analyzed display data is sent to the display screen for displaying through the display driver by the surface module. It can be seen that, when the image is displayed on the mobile phone, the processing process of the display data in one image display process is simple, and the display time delay S of the display data can be ignored at this time. In this scenario, the audio delay L of the audio data may reflect the time difference between the audio data and the display data at the same time when the mobile phone plays, that is, the audio delay L ≈ the synchronization delay T.

In order to ensure that images and audio played in the video APP are synchronous (i.e. audio and video synchronization), when the mobile phone runs the video APP, the video APP may periodically call a getLatency () interface to obtain an audio delay L of audio data. Furthermore, the video APP can set a synchronization strategy for performing sound and picture synchronization processing according to the obtained audio time delay L. For example, the video APP may periodically obtain the latest audio delay L. The period is 3 seconds for example, if the current audio time delay L is 500ms when the video APP generates 3 rd second audio data to be played and display data, then the video APP can set the 3 rd second audio data to be input into the audio player 500ms in advance relative to the display data in the synchronization strategy, and thus the 3 rd second audio data can be played synchronously with the 3 rd second display data after the 500ms processing process, and the effect of sound and picture synchronization is achieved.

For example, when an APP (e.g., a video APP) running on a mobile phone needs to perform a sound-picture synchronization process, the video APP may query an audio service (AudioService) for a synchronization delay T (audio delay L — display delay S) between audio data and display data that is calculated last time by using a getLatency () interface. Alternatively, the video APP may call a player (e.g., Nuplayer, Mediaplayer), and call the getLatency () interface by the player to query the audio service (AudioService) for the most recently calculated synchronization delay T between the audio data and the display data. Subsequently, after the video APP or the player obtains the synchronization delay T, a synchronization policy for audio and video synchronization processing may be set according to the synchronization delay T, so that audio data and/or display data subsequently generated by the video APP is output to the slave device according to the synchronization policy.

In the embodiment of the application, in order to improve the accuracy of audio and image synchronization during audio and video synchronization, the synchronization delay T stored in the AudioService may be dynamically updated, so that the synchronization delay T obtained by the video APP or the player may more accurately indicate the time difference between the audio data and the display data currently playing at the same time. In the following, a specific method for dynamically updating the synchronization delay T will be described in detail with reference to specific embodiments.

Scene one

In a first scenario, when the mobile phone runs a video APP, audio data generated by the video APP can be projected to the slave device to be played, and display data generated by the APP is still played in the mobile phone.

Illustratively, as shown in fig. 99 (a), the handset may display a switch button 9901 for audio switching while running the video APP. If the fact that the user clicks the switching button 9901 is detected, that the user wants to switch the audio data generated by the video APP to other audio output devices for playing, the mobile phone searches for one or more candidate devices which are nearby and can be used for audio switching by running the DV APP. For example, as shown in fig. 99 (b), the DV APP may display candidate devices such as a television, a sound box, and a headphone, which are located on the same Wi-Fi network as the cellular phone, in the menu 9902. The user can select one of the candidate devices as an audio output device for the subsequent audio output of the mobile phone, namely, the slave device of the mobile phone.

For another example, the user may open the control center of the mobile phone and switch the audio output device in the control center when the mobile phone outputs audio. In response to an operation of the user to turn on the control center, as shown in (a) of fig. 100, the cellular phone may display the control center 10001. The control center 10001 is provided with a switch button 10002 for audio switching. If the user is detected to click the switch button 10002, the handset can also run the DV APP to search for one or more candidate devices available for audio switching nearby. As shown in (b) of fig. 100, the DV APP may display the searched candidate devices, such as a television, a sound box, and a headset, which are located in the same Wi-Fi network as the mobile phone, in the control center 10001. Similarly, the user can select one of the candidate devices as an audio output device for the subsequent audio output of the mobile phone, namely, a slave device of the mobile phone.

For example, a user selects a sound box as a slave device of the mobile phone, and after detecting that the user selects the sound box as the candidate device from the candidate devices, the DV APP can establish network connection between the sound box and the sound box by using the sound box as the slave device of the mobile phone. For example, the network connection may be a P2P connection based on a TCP (transmission control protocol) or a UDP (user datagram protocol), which is not limited in this embodiment.

After the network connection between the mobile phone and the loudspeaker box is established, the DV APP can obtain the audio capability parameters of the loudspeaker box based on the network connection, and the audio capability parameters can reflect the audio output function specifically supported by the loudspeaker box. For example, the audio capability parameters may include sampling rate supported by the speaker, number of channels, sound effect modes, and the like. Furthermore, the DV APP may create a corresponding DMSDP Audio HAL at the HAL according to the Audio capability parameters of the loudspeaker. Subsequently, the mobile phone can transmit Audio data to the sound box through the DMSDP Audio HAL, so that the Audio function in the mobile phone is switched to the sound box for realization.

In some embodiments, after the mobile phone establishes the network connection with the speaker, the mobile phone may further query whether the current slave device (e.g., the speaker) has a display function. For example, the mobile phone may send a query request to the speaker requesting the speaker to report whether the speaker has the display capability. For another example, the mobile phone may obtain information such as the device model of the sound box, and then determine whether the sound box has the display function according to the information such as the device model of the sound box. If the sound box has a display function, as shown in fig. 101, the mobile phone may prompt the user whether the display data in the mobile phone needs to be projected into the sound box for display, that is, the display data and the audio data are projected to the slave device at the same time.

If the user selects to project the display data to the sound box for display, the mobile phone can project the display data and the audio data in the mobile phone to the slave device for playing according to a specific method in the following scene two.

If the user selects to keep the display data in the mobile phone for display, or if the mobile phone determines that the sound box does not have the display function, the mobile phone can determine that only the audio data in the mobile phone needs to be projected into the sound box for playing, and the display data in the mobile phone is still displayed by the display screen of the mobile phone.

In the process of projecting the audio data to the sound box for playing by the mobile phone, as shown in fig. 95, the audio delay L generated by each data packet of the audio data is the master device audio delay L1+ the audio transmission delay L2+ the slave device audio delay L3. And the display time delay S of the mobile phone playing the display data by using the display screen of the mobile phone is approximately equal to 0. Then the synchronization delay T between the audio data and the display data is audio delay L-display delay S ≈ audio delay L.

For example, after the DV APP creates a corresponding DMSDP Audio HAL for the sound box in the HAL, the DMSDP Audio HAL may periodically acquire a synchronization delay T between the Audio data and the display data, and in a scene one, the synchronization delay T is an Audio delay L when the mobile phone projects the Audio data to the sound box. For example, the DMSDP Audio HAL may periodically obtain the current master Audio delay L1, the current Audio transmission delay L2, and the current slave Audio delay L3 with 1 second(s) as a period, and further calculate the sum of the obtained master Audio delay L1, the obtained Audio transmission delay L2, and the obtained slave Audio delay L3 to obtain the latest Audio delay L. Of course, those skilled in the art may set the specific periods of the DMSDP Audio HAL acquiring the master device Audio delay L1, the Audio transmission delay L2 and the slave device Audio delay L3 according to practical experience or practical application scenarios.

The size of the audio delay L1 of the master device mainly depends on the specific processing procedure of the audio data by the handset (i.e., the master device). For example, when the handset needs to perform sound effect processing on the audio data, the corresponding audio delay L1 of the host device is increased. For another example, when the mobile phone uses different encoding algorithms to encode audio data, the corresponding master audio delay L1 will also be different. As another example, when the handset needs to resample the audio data, the corresponding master audio delay L1 may increase.

In this embodiment of the application, after the network connection between the mobile phone and the sound box is established, the AudioPolicy in the mobile phone may determine a corresponding audio policy according to the audio capability parameter of the current slave device (e.g., the sound box), where the audio policy specifically includes an audio processing instruction such as whether to add a sound effect, whether to perform resampling, how to perform encoding, and the like. Then DMSDP Audio HAL may request an Audio policy query for the current Audio policy and determine the master Audio delay L1 at that time based on the current Audio policy.

For example, as shown in fig. 102, if it is set in the Audio policy that an Audio effect needs to be added to the Audio data (i.e., Audio effect processing is performed), the DMSDP Audio HAL may determine that the Audio effect processing process takes 20 ms. Accordingly, if no sound effect processing is set to be performed on the Audio data in the Audio policy, the DMSDP Audio HAL may determine that the sound effect processing process takes 0 ms.

For another example, as shown in fig. 102, if the Audio policy sets that Audio data is encoded using the AAC (Advanced Audio Coding) format, the DMSDP Audio HAL may determine that the encoding process takes 15 ms; if the Audio policy sets the encoding of Audio data using the H.264 format, the DMSDP Audio HAL may determine that the encoding process takes 10 ms.

For another example, as shown in fig. 102, if the Audio policy sets resampling of Audio data, DMSDP Audio HAL may determine that the resampling process takes 5 ms; the DMSDP Audio HAL may determine that the resampling process takes 0ms if no resampling of the Audio data is set in the Audio policy.

Further, as shown in fig. 102, the DMSDP Audio HAL may add the time consumption of the sound effect processing procedure, the time consumption of the encoding procedure, and the time consumption of the resampling procedure to obtain the current master Audio delay L1. It should be noted that the specific time consumption of the sound effect processing process, the specific time consumption of the encoding process, and the specific time consumption of the resampling process may be experience values preset by a developer, or may be obtained from a server by a mobile phone. The handset can update the specific time-consuming of these audio processes. For example, when the time consumption of the sound effect processing process in the new version of the operating system is updated from 20ms to 15ms, the mobile phone updates the new version of the operating system and then updates the time consumption of the sound effect processing process from 20ms to 15 ms.

Of course, if other Audio processing procedures are set in the Audio policy, the DMSDP Audio HAL may also determine time consumption of the other Audio processing procedures, and use the time consumption as a part of the Audio delay L1 of the master device, which is not limited in this embodiment of the present application.

In the embodiment of the present application, since the audio capability parameter obtained by the DV APP from the speaker is dynamically updated, the audio policy determined by the AudioPolicy according to the audio capability parameter is also dynamically updated. Then the master Audio delay L1 determined by DMSDP Audio HAL according to the Audio policy above is also dynamically updated.

For example, a first flag bit for storing the master Audio delay L1 may be set in DMSDP Audio HAL. DMSDP Audio HAL may periodically (e.g., at 1 second intervals) calculate the latest master Audio delay L1 in the manner described above, and store each calculated master Audio delay L1 update in the first flag bit.

For another example, each time AudioPolicy updates the Audio policy according to the latest Audio capability parameters, an update message may be sent to the DMSDP Audio HAL, that is, the DMSDP Audio HAL is notified that the Audio policy has been updated. The DMSDP Audio HAL may obtain the latest Audio policy from the Audio policy in response to the update message, and calculate the latest master Audio delay L1 according to the latest Audio policy, and further, the DMSDP Audio HAL may update the calculated master Audio delay L1 to be stored in the first flag.

In some embodiments, the size of the audio propagation delay L2 is mainly determined by the network condition of the network connection between the handset (i.e., the master device) and the speaker (i.e., the slave device). For example, when the network quality of the network connection between the mobile phone and the sound box is good, the corresponding audio transmission delay L2 is reduced; when the network quality of the network connection between the mobile phone and the sound box is poor, the corresponding audio transmission delay L2 is increased. For another example, when the distance between the mobile phone and the sound box is short, the corresponding audio transmission delay L2 may decrease; when the distance between the mobile phone and the sound box is long, the corresponding audio transmission delay L2 is increased.

For example, the network connection between the mobile phone and the sound box is Wi-Fi P2P connection, and the Wi-Fi HAL in the mobile phone HAL may periodically detect the audio transmission delay L2 between the mobile phone and the sound box. For example, as shown in fig. 103, the Wi-Fi HAL may send a test packet to the sound box through the Wi-Fi P2P connection between the mobile phone and the sound box at a period of 500ms, and the sound box may send a response packet to the Wi-Fi HAL of the mobile phone after receiving the test packet. Then, the Wi-Fi HAL may calculate the audio transmission delay L2 between the current handset and the speaker by detecting the time interval between sending the test packet and receiving the response packet.

For another example, after the mobile phone establishes Wi-Fi P2P connection with the speaker, the mobile phone may periodically receive a heartbeat packet or a heartbeat response from the speaker through a heartbeat mechanism. When the mobile phone does not receive the heartbeat packet or heartbeat response sent by the sound box within the preset time, which indicates that the network condition between the mobile phone and the sound box may change at the moment, the Wi-Fi HAL of the mobile phone can be triggered to send a test data packet to the sound box, and a response data packet sent by the sound box in response to the test data packet is received. Thus, the Wi-Fi HAL may calculate the audio transmission delay L2 between the current handset and the speaker by detecting the time interval between sending the test packet and receiving the response packet.

For example, if the time interval between the time when the mobile phone sends the test packet to the sound box and the time when the mobile phone receives the response packet from the sound box is N, the Wi-Fi HAL may calculate the audio transmission delay L2 between the mobile phone and the sound box as N/2, that is, the time consumed by the mobile phone to transmit one packet of audio data to the sound box.

In the embodiment of the present application, a second flag bit for storing the Audio transmission delay L2 may be set in the DMSDP Audio HAL. As also shown in fig. 103, the DMSDP Audio HAL may periodically read the Audio transmission delay L2 stored in the Wi-Fi HAL, and update the Audio transmission delay L2 that is read each time to be saved in the second flag. The period of the Audio transmission delay L2 acquired by the DMSDP Audio HAL may be the same as or different from the period of the Audio transmission delay L1 acquired by the DMSDP Audio HAL in the above embodiment.

For another example, each time the Wi-Fi HAL calculates the latest Audio transmission delay L2, an update message may be sent to the DMSDP Audio HAL, that is, the DMSDP Audio HAL is notified that the Audio transmission delay L2 has been updated. The DMSDP Audio HAL may read the Audio transmission delay L2 stored in the Wi-Fi HAL in response to the update message, and update the read Audio transmission delay L2 to be stored in the second flag bit.

In some embodiments, the size of the slave device audio delay L3 depends mainly on the specific processing and playing process of the audio data by the speaker (i.e., the slave device). For example, when the sound box needs to perform sound effect processing on the audio data, the audio delay L3 of the corresponding slave device is increased. For another example, when a speaker needs to process audio data using a DSP, the audio delay L3 of the corresponding slave device may also increase. For another example, when the speakers are used to play audio data with speakers of different models or different manufacturers, the corresponding slave device audio delay L3 may also be different.

In this embodiment of the application, the audio capability parameter obtained by the DV APP of the mobile phone from the sound box may include a slave device audio time delay L3 when the sound box is used as a slave device. That is, the speaker can calculate its own audio delay, and report the audio delay to the handset (i.e., the master device) as the slave device audio delay L3 carried in the audio capability parameter.

Illustratively, the process of the loudspeaker as a slave device obtaining the slave device audio delay L3 is similar to the process of the mobile phone as a master device obtaining the master device audio delay L1. For example, if the sound box needs to perform sound effect processing on the audio data, the sound box may determine that the sound effect processing process takes 15 ms. Correspondingly, if the sound box does not need to perform sound effect processing on the audio data, the sound box can determine that the sound effect processing process takes 0 ms. For another example, if the loudspeaker needs to process audio data using a DSP, the loudspeaker may determine that the processing process of the DSP takes 10 ms. Accordingly, if the loudspeaker does not need to process the audio data using the DSP, the loudspeaker may determine that the DSP processing takes 0 ms.

The loudspeaker box can sum the time consumption of each processing process of the audio data to obtain the slave device audio time delay L3. Furthermore, the sound box can carry the current slave device audio delay L3 in the audio capability parameter to report to the DV APP of the mobile phone, and the DV APP of the mobile phone registers the audio capability parameter in AudioPolicy. As shown in fig. 104, a third flag bit for storing slave Audio delay L3 may be set in DMSDP Audio HAL. DMSDP Audio HAL may save the read slave Audio delay L3 in the third flag bit by reading the slave Audio delay L3 in the Audio capability parameter in AudioPolicy.

In the embodiment of the application, the audio capability parameters acquired by the DV APP from the loudspeaker box are dynamically updated. For example, after the user sets the sound effect mode of the cabinet from the dolby sound effect mode to the subwoofer sound effect mode, the cabinet may update the sound effect processing time consumption from 15ms to 20ms in response to the user's setting. At this time, the speaker may recalculate and update the slave audio delay L3, and carry the updated slave audio delay L3 with the DV APP that is reported to the handset in the audio capability parameters. The DV APP may issue the latest audio capability parameters to AudioPolicy. The AudioPolicy may notify the DMSDP Audio HAL after obtaining the latest Audio capability parameter, and the DMSDP Audio HAL may obtain the latest slave Audio delay L3 from the AudioPolicy at this time, and update the latest slave Audio delay L3 in the third flag bit.

Thus, the DMSDP Audio HAL has the latest master Audio delay L1 stored in the first flag, the latest Audio transmission delay L2 stored in the second flag, and the latest slave Audio delay L3 stored in the third flag. Then, the DMSDP Audio HAL may periodically obtain the latest master Audio delay L1, Audio transmission delay L2, and slave Audio delay L3 from the first flag, the second flag, and the third flag. Furthermore, the DMSDP Audio HAL may calculate a corresponding Audio delay L, which is the master device Audio delay L1+ the Audio transmission delay L2+ the slave device Audio delay L3. Thus, the DMSDP Audio HAL can dynamically update the Audio delay L of the current Audio data.

Alternatively, the DMSDP Audio HAL may be configured to trigger the DMSDP Audio HAL to obtain the master Audio delay L1 in the first flag, the Audio transmission delay L2 in the second flag, and the slave Audio delay L3 in the third flag when the content of one of the first flag, the second flag, and the third flag is updated, and calculate the latest Audio delay L — the master Audio delay L1+ the Audio transmission delay L2+ the slave Audio delay L3. Thus, the DMSDP Audio HAL may also dynamically update the Audio delay L of the current Audio data.

For example, the DMSDP Audio HAL in the handset may dynamically calculate the Audio delay L of the current Audio data according to the above method. Furthermore, the DMSDP Audio HAL may store the last calculated Audio delay L in a preset flag. If the difference value between the Audio time delay L calculated by the DMSDP Audio HAL this time and the Audio time delay L calculated last time is smaller than the threshold value, which indicates that the current Audio time delay L has no large fluctuation, the DMSDP Audio HAL does not need to modify the Audio time delay L stored in the preset zone bit.

As shown in fig. 105, after the DMSDP Audio HAL in the mobile phone can calculate the Audio time delay L of the current Audio data according to the above method, if the difference between the Audio time delay L calculated this time and the Audio time delay L calculated last time is greater than the threshold, which indicates that the current Audio time delay L has large fluctuation, the DMSDP Audio HAL can update the content in the preset flag bit to the Audio time delay L calculated this time. Furthermore, the DMSDP Audio HAL may send a notification message to the AudioFlinger, where the notification message carries the Audio delay L in the preset flag bit, that is, informs the AudioFlinger of the latest Audio delay L. Furthermore, the AudioFlinger may send the latest audio delay L in the notification message to the AudioService, and the AudioService stores the latest audio delay L.

Subsequently, the video APP or the player may call a getLatency () interface to query the AudioService for the synchronization delay T between the current audio data and the display data during the operation process. Since the synchronization delay T is approximately equal to the audio delay L in the scene one, the AudioService can feed back the stored latest audio delay L as the synchronization delay T to the video APP or the player. In this way, the video APP or player can set the synchronization strategy of the sound and picture synchronization processing according to the latest audio delay L.

In other embodiments, as shown in fig. 106, the DMSDP Audio HAL in the mobile phone may calculate the Audio delay L of the current Audio data according to the above method, and update the Audio delay L in the preset flag bit. Furthermore, DMSDP Audio HAL may send a notification message to the Audio flinger indicating that the current Audio delay L has been updated, at which point the notification message does not include the updated Audio delay L. Then, the AudioFlinger, upon receiving the notification message, may read the updated Audio delay L (i.e., the most recent Audio delay L) from the preset flag of the DMSDP Audio HAL. Furthermore, the AudioFlinger may send the latest audio delay L to the AudioService, and the AudioService stores the latest audio delay L.

Subsequently, similar to fig. 105, the video APP or player may call the getLatency () interface to query the AudioService about the synchronization delay T between the current audio data and the display data during the operation process. The AudioService can feed back the stored latest audio time delay L as the synchronization time delay T to the video APP or the player, so that the video APP or the player can set a synchronization strategy for audio and video synchronization processing according to the latest audio time delay L.

Because the AudioService can dynamically acquire the newly calculated Audio delay L by the DMSDP Audio HAL, the Audio delay L acquired by the video APP or the player from the AudioService each time is also the newly calculated Audio delay L, and the Audio delay L can more accurately reflect the time difference between the current Audio data and the display data. Therefore, when the video APP or the player performs sound and picture synchronization processing according to the audio time delay L, the audio played in the sound box and the image played in the mobile phone can better achieve the sound and picture synchronization effect, and the sound and picture synchronization experience of a user is improved.

For example, the video APP or player may periodically call the getLatency () interface to obtain the latest audio delay L during operation. After the latest audio time delay L is obtained each time, the video APP or the player may set a corresponding synchronization policy for subsequently output audio data and display data according to the latest audio time delay L to perform sound and picture synchronization. For example, the video APP may obtain the latest audio delay L with 5 frames of audio data as one period. After outputting the nth frame of audio data, the video APP may call a getLatency () interface to obtain the latest audio delay L, and as an example, the audio delay L at this time is 400ms, and the video APP may set display data to play 400ms later than the audio data in a synchronization policy. Therefore, the video APP can play the corresponding audio data and video data according to the synchronization strategy in the process of playing the (N + 1) th frame audio data to the (N + 5) th frame audio data so as to realize the effect of sound and picture synchronization.

Furthermore, after the video APP outputs the N +5 th frame of audio data, the video APP may call the getLatency () interface again to obtain the latest audio delay L, and the video APP may update the synchronization policy according to the latest audio delay L, for example, the audio delay L at this time is 500 ms. For example, the video APP may set the display data to play 500ms later than the audio data in the synchronization policy. Then, the video APP may play the corresponding audio data and video data according to the above synchronization policy in the process of playing the audio data of the (N + 6) th frame to the audio data of the (N + 10) th frame. Therefore, the video APP can continuously update the corresponding synchronization strategy according to the latest audio time delay L in the running process, so that a user can obtain a better sound and picture synchronization effect when playing the audio.

Or the AudioService may actively report the latest audio delay L to the video APP in the running process of the video APP. For example, after the AudioService acquires the latest audio delay L reported by the AudioFlinger, the AudioService may actively send the latest audio delay L to the video APP, and trigger the video APP to set a corresponding synchronization policy according to the latest audio delay L. That is to say, once the mobile phone detects that the current audio delay L changes, the mobile phone can notify the video APP of the latest audio delay L, so that the video APP can update the corresponding synchronization policy according to the latest audio delay L, and a user can obtain a better sound and picture synchronization effect during audio playing.

In some embodiments, in the process of projecting the audio data to the sound box for playing, the user may also modify the slave device currently performing audio switching with the mobile phone on the mobile phone. For example, when the mobile phone runs the video APP, if an operation of opening the control center by the user is received, the mobile phone may display the control center 10701 in an application interface of the video APP as shown in (a) in fig. 107. The control center 10701 includes a switching button 10702 for an audio switching function. When the user wishes to modify the slave device of the current handset, the clickable toggle button 10702 queries one or more candidate devices that may currently be slave devices of the handset. As shown in (b) of fig. 107, upon detecting that the user clicks the switching button 10702, the handset may display the detected one or more candidate devices in the control center 10703. At this point, the speaker in the control center 10703 has been marked as the selected state, i.e., the speaker is playing audio from the handset as a slave to the handset. If the user needs to replace the slave device of the handset, the user can reselect the slave device for audio switching in the control center 10703.

For example, if the user selects the car machine as a new slave device of the mobile phone, the mobile phone may respond to the operation of the user selecting the car machine in the control center 10703, disconnect the network connection already established with the sound box, and establish a network connection with the car machine. Furthermore, the mobile phone can obtain the Audio capability parameters of the car machine through the DV APP, and update the DMSDP Audio HAL in the HAL according to the Audio capability parameters of the car machine, so that the updated DMSDP Audio HAL is matched with the Audio capability of the car machine.

Furthermore, the updated DMSDP Audio HAL may use the car machine as a slave device, and continue to periodically obtain the master device Audio delay L1, the Audio transmission delay L2, and the slave device Audio delay L3 according to the above method, so as to periodically calculate the latest Audio delay L. Subsequently, similar to the process in fig. 105 or 106, the DMSDP Audio HAL may store the latest Audio delay L as the synchronization delay T in the AudioService through the AudioFlinger each time, so that the video APP or player may obtain the latest Audio delay L in the AudioService, thereby setting a synchronization policy for Audio and video synchronization processing according to the latest Audio delay L.

Scene two

In a second scenario, when the mobile phone runs the video APP, the audio data and the display data generated by the video APP can be projected to the same slave device for playing, that is, the audio and the image of the mobile phone are simultaneously switched to the slave device for playing by using the screen projection function in the mobile phone.

For example, as shown in fig. 108, when the mobile phone runs the video APP, if the user needs to use the screen projection function, the user may open a control center 10801 of the mobile phone in an application interface of the video APP. A card 10802 may be disposed in the control center 10801, where the card 10802 is used to display one or more nearby candidate devices that may be projected for detection by the mobile phone. If the user selects a certain electronic device (such as a television) in the card 10802, it indicates that the user wishes to project audio data and display data generated by the video APP to the television for playing through the screen projection function of the mobile phone.

Similar to the first scenario, when the mobile phone detects that the user selects the television as the slave device of the mobile phone to perform screen projection, the DV APP can establish network connection with the television, further acquire the Audio capability parameters of the television through the network connection, and create a corresponding DMSDP Audio HAL in the HAL according to the Audio capability parameters of the television. Subsequently, the handset can transmit Audio data generated by the video APP to the television via the DMSDP Audio HAL. And in a screen projection scene, the mobile phone can send display data generated by the video APP to the television for display according to the existing screen projection mode. Or, the mobile phone may also create a DMSDP Display HAL corresponding to the Display function in the HAL, and then transmit the Display data generated by the video APP to the television through the DMSDP Display HAL, which is not limited in this embodiment of the present application.

In the second scenario, as shown in fig. 96, for the audio data generated when the mobile phone runs the video APP, similarly to the first scenario, the audio delay L of the audio data is the master audio delay L1+ the audio transmission delay L2+ the slave audio delay L3.

In the second scenario, as shown in fig. 96, for the display data generated when the mobile phone runs the video APP, the display delay S of the display data is the master display delay S1+ the display transmission delay S2+ the slave display delay S3. Because the processing process of the display data in the electronic device is generally simple, that is, the processing speed is fast, the master device display time delay S1 and the slave device display time delay S3 of the display data can be ignored. At this time, the above-described display delay S ≈ display propagation delay S2.

In the second scenario, the display data and the audio data generated by the video APP are transmitted through the network connection between the mobile phone and the television, and at this time, the time for transmitting one data packet in the display data from the mobile phone to the television is substantially the same as the time for transmitting one data packet in the video data from the mobile phone to the television, that is, the display transmission delay S2 is approximately equal to the audio transmission delay L2.

Then the synchronization delay T between the audio data and the display data ≈ audio delay L-display delay S ≈ (master audio delay L1+ audio transmission delay L2+ slave audio delay L3) -display transmission delay S2 ≈ master audio delay L1+ slave audio delay L3.

In the embodiment of the present application, the DMSDP Audio HAL may dynamically obtain the current master Audio delay L1 and the slave Audio delay L3, so as to dynamically obtain the latest synchronization delay T (the master Audio delay L1+ the slave Audio delay L3). The specific method for the DMSDP Audio HAL to dynamically acquire the master device Audio delay L1 and the slave device Audio delay L3 may refer to the related description in scenario one, and therefore, details are not described here again. Because the synchronous time delay T in the screen projection process of the mobile phone and the television is dynamically updated, the synchronous time delay T can accurately indicate the time difference between the audio data and the display data of the television at the same playing time.

Similar to the process in fig. 105 or 106, the DMSDP Audio HAL in the mobile phone can store the updated synchronization delay T (i.e., the latest synchronization delay T) in the AudioService through the AudioFlinger each time, so that the video APP or player can obtain the latest synchronization delay T in the AudioService, thereby setting the synchronization policy for Audio and video synchronization processing according to the latest synchronization delay T. Because the APP or the player can accurately indicate the time difference between the audio data and the display data of the television playing at the same moment when acquiring the latest synchronous time delay T, when the video APP or the player performs sound and picture synchronous processing according to the synchronous time delay T, the effect of sound and picture synchronization of the audio and the image played in the television can be better achieved, and the sound and picture synchronous experience of a user is improved.

Scene three

In a third scenario, when the mobile phone runs the video APP, the mobile phone may send display data generated by the video APP to the first slave device for playing, and the mobile phone may send audio data generated by the video APP to the second slave device for playing. That is, in scene three, the mobile phone may project the audio and the image in the video to different slave devices for playing.

For example, as shown in (a) in fig. 109, when the mobile phone runs a video APP, the user may open a control center 10901 of the mobile phone in an application interface of the video APP. A first button 10902 for audio switching and a second button 10903 for display switching may be provided in the control center 10901.

If it is detected that the user clicks the first button 10902, which indicates that the user wishes to switch the audio data generated by the video APP to other audio output devices for playing, the mobile phone searches for one or more candidate devices nearby by running the DV APP, which can be used for audio switching. As shown in fig. 109 (b), the DV APP can display candidate devices for audio switching with the cellular phone in the menu 10904. The user can select one of the candidate devices as an audio output device for the subsequent output of audio data by the mobile phone.

If it is detected that the user clicks the second button 10903, which indicates that the user wishes to switch the display data generated by the video APP to other display devices for playing, at this time, as shown in (c) of fig. 109, the mobile phone may display the currently detected candidate device that can be used for receiving and displaying the display data of the mobile phone in the menu 10905. The user can select one of the candidate devices as the display output device for the subsequent output of the display data by the mobile phone.

For example, if the user selects a speaker as the audio output device for audio data in menu 10904 and selects a tv as the display output device for display data in menu 10905, the mobile phone may establish a first network connection with the speaker and a second network connection with the tv. The mobile phone can send audio data generated by the video APP to the sound box for playing through the first network connection, and can send display data generated by the video APP to the television for displaying through the second network connection.

In the embodiment of the application, after the first network connection between the mobile phone and the sound box is established, the DV APP in the mobile phone can obtain the Audio capability parameter of the sound box through the first network connection, and create a DMSDP Audio HAL corresponding to the Audio capability parameter of the sound box in the HAL. The DMSDP Audio HAL may dynamically acquire the synchronization delay T between the current Audio data and the display data.

In scenario three, as shown in fig. 97, similarly to scenario one and scenario two described above, the audio delay L of the audio data is the master device audio delay L1+ the audio transmission delay L2+ the slave device audio delay L3.

In scenario three, as shown in fig. 97, similar to scenario two described above, the display latency S of the display data is the master display latency S1+ the display transmission latency S2+ the slave display latency S3 is approximately equal to the display transmission latency S2.

Then, the synchronization delay T between the audio data and the display data ≈ audio delay L-display delay S ≈ (master audio delay L1+ audio transmission delay L2+ slave audio delay L3) -display transmission delay S2.

By taking the network connection between the mobile phone and the sound box as a first Wi-Fi connection and the network connection between the mobile phone and the television as a second Wi-Fi connection, for example, the Wi-Fi HAL in the mobile phone HAL can send a first test data packet to the sound box through the first Wi-Fi connection and receive a first response data packet sent by the sound box in response to the first test data packet, so that the audio transmission delay L2 between the current mobile phone and the sound box is calculated according to the time interval between the sending of the first test data packet and the receiving of the first response data packet. Similarly, the Wi-Fi HAL may send a second test packet to the tv via the second Wi-Fi connection, and receive a second response packet sent by the tv in response to the second test packet, so as to calculate the current display transmission delay between the mobile phone and the tv according to the time interval between sending the second test packet and receiving the second response packet S2.

That is, in scenario three, the Wi-Fi HAL of the handset may dynamically detect and update the audio transmission delay L2 and the display transmission delay S2. Thus, the DMSDP Audio HAL may read the latest Audio transmission delay L2 and display transmission delay S2 in the Wi-Fi HAL.

Moreover, the DMSDP Audio HAL may obtain the latest master Audio delay L1 and slave Audio delay L3 according to the method in scenario one. Furthermore, DMSDP Audio HAL can calculate the latest Audio delay L from the latest master Audio delay L1, Audio transfer delay L2, and slave Audio delay L3. Furthermore, the DMSDP Audio HAL may calculate the latest synchronization delay T from the latest Audio delay L and the display transmission delay S2 (synchronization delay T ═ Audio delay L — display transmission delay S2).

Subsequently, similar to the process in fig. 105 or 106, after the DMSDP Audio HAL in the mobile phone calculates the latest synchronization delay T each time, the latest synchronization delay T may be stored in the AudioService through the AudioFlinger, so that the video APP or player may obtain the latest Audio delay L in the AudioService, thereby setting a synchronization policy for Audio and video synchronization processing according to the latest Audio delay L.

Or, the DMSDP Audio HAL in the mobile phone may report the latest Audio delay L and display transmission delay S2 to the AudioService through the AudioFlinger. And calculating the latest synchronization time delay T by the Audio service according to the latest audio time delay L and the display transmission time delay S2, thereby setting a synchronization strategy for sound and picture synchronization processing according to the latest audio time delay L.

Similarly, the APP or the player obtains the time difference when the latest synchronization delay T can accurately indicate that the audio data and the display data at the same time are respectively played on two slave devices (namely, a sound box and a television), so that when the video APP or the player performs sound and picture synchronization processing according to the synchronization delay T, images played in the television and audio played in the sound box can better achieve the sound and picture synchronization effect, and the sound and picture synchronization experience of a user is improved.

In other embodiments, the user may also connect other audio output devices to the audio output device of the handset. For example, after the user selects the television as the audio output device of the mobile phone, the mobile phone may send the audio data generated by the video APP to the television for playing. As shown in fig. 110, when the television is an audio output device of the mobile phone, the user may further connect the television with the sound box, and at this time, the television may send the audio data sent by the mobile phone to the sound box again for playing, that is, the sound box is an audio output device of the television at this time.

As also shown in fig. 110, in the scenario where the multiple audio output devices are cascaded, the audio delay L of the audio data is the master device audio delay L1+ the audio transmission delay L2+ the slave device audio delay L3. Differently, the slave device audio delay L3 specifically includes: the audio data processing method comprises a first time delay Y1 spent on processing a data packet of the audio data in a television, a second time delay Y2 spent on transmitting the data packet between the television and a sound box, and a third time delay Y3 spent on the sound box starting to process the data packet to finally play the audio data in the data packet after the sound box receives the data packet, namely L3-Y1 + Y2+ Y3.

Then, similar to the method for acquiring the master device audio delay L1, the audio transmission delay L2, and the slave device audio delay L3 by the cell phone in the scene one, the current first delay Y1, the current second delay Y2, and the current third delay Y3 may also be acquired by the master device of the television as a sound box according to the above method, so as to calculate the latest slave device audio delay L3. Furthermore, the television can carry the latest slave audio delay L3 in the audio capability parameter and report the audio delay L to the mobile phone, and the mobile phone updates the audio delay L of the current audio data according to the method described in the scenario one.

Therefore, the mobile phone can dynamically update the audio time delay L of the current audio data in the scene of cascading multiple audio output devices, so that the video APP or the player can determine the corresponding synchronous time delay T according to the latest audio time delay L to perform sound and picture synchronous processing, and the sound and picture synchronous effect in the scene of cascading multiple audio output devices is improved.

In the above embodiment, a scene in which the mobile phone runs the video APP to perform audio and video synchronization is exemplified, and it can be understood that the mobile phone may also need to call the getLatency () interface to obtain the audio delay L of the audio data from the AudioService when running other applications. The method provided by the embodiment of the application can dynamically update the audio time delay L stored in the AudioService, so that the audio time delay L can more accurately reflect the time spent on playing the current audio data, and the accuracy of other processing performed by other applications according to the audio time delay L can be correspondingly improved.

For example, as shown in (a) of fig. 111, the handset may display a first button 20101 corresponding to an audio output capability and a second button 20102 corresponding to an audio input capability in the control center when running the karaoke application. The user can control the switching of the audio output and input functions of the mobile phone on the slave device through the first button 20101 and the second button 20102 respectively. For example, if the user is detected to click the first button 20101, as shown in (b) of fig. 111, the handset may display a device list 20103 of candidate devices having an audio output function, where the current audio output device of the handset is native to the handset. The user may switch the audio output device of the handset in the device list 20103. Similarly, if the user is detected to click the second button 20102, as shown in (c) of fig. 111, the handset may display a list 20104 of candidate devices with audio input capabilities, where the handset's current audio input device is a speaker. The user may switch the audio input device of the handset in the device list 20104. Therefore, the user can respectively switch the audio output and input capabilities of the mobile phone to the corresponding devices according to the own requirements, and separately control the audio output and input functions of the mobile phone.

For example, the user selects a sound box in the device list 20104 as an Audio input device, and similarly to the case that the mobile phone outputs Audio data to the Audio output device through the DMSDP Audio HAL, the mobile phone may receive recording data collected by the sound box through the DMSDP Audio HAL and report the recording data to the karaoke application, so that the Audio input function (also referred to as an Audio recording or recording function) of the mobile phone is distributed to the slave devices to be implemented.

Similar to the audio delay L for playing the audio data in the above embodiment, the audio delay of the recorded data may be referred to as recording delay R, and the recording delay R of one packet of the recorded data is the recording delay R1 of the master device + the recording transmission delay R2+ the recording delay R3 of the slave device.

For example, the start time of the recording delay R3 from the device may be the time when a packet of user-input recording data is collected from the device (e.g., speaker), and the end time of the recording delay R3 from the device may be the time when the packet is output from the speaker. The start time of the recording transmission delay R2 may be the time when the packet is output from the speaker (i.e., the end time of the recording delay R3 of the slave device), and the end time of the recording transmission delay R2 may be the time when the host device (e.g., a mobile phone) receives the packet. The start time of the recording delay R1 of the master device may be the time when the data packet is received by the mobile phone (i.e., the end time of the recording transmission delay R2), and the end time of the recording delay R1 of the master device may be the time when the corresponding application (e.g., the karaoke application) in the mobile phone acquires the data packet.

Similar to the DMSDP Audio HAL dynamically acquiring the latest master device Audio time delay L1, Audio transmission time delay L2, and slave device Audio time delay L3 according to the method in the foregoing embodiment, in the above recording scenario, the DMSDP Audio HAL may also dynamically acquire the latest master device recording time delay R1, recording transmission time delay R2, and slave device recording time delay R3, thereby calculating the recording time delay R of the latest recording data, and updating the recording time delay R in the AudioService. Subsequently, the karaoke application may also call the getLatency () interface to obtain the recording delay R of the recording data from the Audio service. Furthermore, the K song application can mix the recording data and the currently played accompaniment music according to the recording time delay R to obtain a recorded song file.

For example, as shown in fig. 112, when the audio delay L of the recording data is 400ms, it indicates that the time when the K song application receives the recording data is 400ms later than the time when the user actually utters the sound, and also indicates that the current recording data is 400ms later than the accompaniment music, then the K song application can mix the currently received recording data with the accompaniment music before 400ms, so as to obtain a song file more accurately synchronized with the accompaniment music.

Still alternatively, besides the above-mentioned karaoke application, the mobile phone may also project audio data (e.g. songs) from the music APP to the slave device (e.g. a sound box) for playing when running the music APP. Then, the music APP may also obtain the latest audio delay L from the AudioService according to the method in the above embodiment, and further, the music APP may synchronize the played lyric data and the audio data according to the audio delay L, so that the lyrics seen by the user correspond to the songs heard. That is to say, after the audio delay L is dynamically updated by the master device according to the method in the foregoing embodiment, a person skilled in the art may set a relevant application according to actual experience or an actual application scenario to obtain a latest audio delay L, so as to perform synchronization processing related to audio according to the latest audio delay L, so that a result of the synchronization processing is more accurate.

At present, in the process of screen projection between a master device (such as the above-mentioned mobile phone) and a slave device, the mobile phone needs to decode audio data and display data first and then encode the audio data and the display data according to a preset format, so that the data processing process during screen projection is complex, and the delay of the screen projection process is large.

For example, when a user uses a video APP in a mobile phone to project a screen to a television, a video file played during the running of the video APP is usually acquired in the form of a data packet. For example, the video APP may obtain one or more data packets of a video file from a server on the network side. For another example, the video APP may retrieve one or more packets of a video file from memory local to the cell phone. That is, one video file may include one or more data packets, and of course, the video file may also include other information such as format, duration, name, etc. of the video file, and these information and the data packets of the video file together form the video file.

As shown in fig. 113, the data packet of the video file generally includes a header (head) and a data (data) portion.

Wherein the data portion includes specific display data and audio data. For example, the data (data) portion of the data packet 1 includes 3 frames of audio data and 2 frames of display data. These display data and audio data are generally data encoded by a producer of a video file according to a certain encoding format. The amount of the encoded display data and audio data is usually small, so that the encoded display data and audio data can be carried in a data packet for faster transmission.

The header (head) includes audio parameters and display parameters corresponding to the data portion of the data packet. For example, a header (head) may include display parameters such as encoding format, resolution, and the like of the display data. The display parameter may specifically be a parameter used by a producer of the video file when encoding the display data. For another example, the header (head) may include audio parameters such as the encoding format, sampling rate, number of channels, and the like of the audio data. The audio parameters may specifically be parameters used by the producer of the video file when encoding the audio data.

In addition, the header (head) may further include some information of the video file, for example, a format of the video file (for example, MP4 format), a name of the video file (for example, a movie), a duration of the video file (for example, 120 minutes), and the like, which is not limited in this embodiment.

When the video APP in the mobile phone plays the video file, as shown in fig. 114, the mobile phone may parse a header (head) of a data packet in the video file. By analyzing the packet header, the mobile phone can extract the corresponding audio parameters and display parameters in the packet header. And, the mobile phone can decapsulate the data (data) part in the data packet. By de-encapsulation, the handset can extract corresponding X (X > 0) frame audio data and Y (Y > 0) frame display data from the data portion. The X frame audio data and the Y frame display data are both encoded data. For example, the audio data obtained after the decapsulation may be referred to as audio data 1 in fig. 114, and similarly, the display data obtained after the decapsulation may be referred to as display data 1 in fig. 114.

Further, as shown in fig. 114, the mobile phone may decode the X frames of audio data (i.e. audio data 1) according to the analyzed audio parameters. For example, if the audio data indicating the data portion in the audio parameters are encoded according to the ACC format, the mobile phone may decode the extracted X frames of audio data 1 according to the ACC format to obtain decoded X frames of audio data, i.e. audio data 2 in fig. 114. For example, the decoded audio data 2 may be audio data in a PCM format.

Similarly, as shown in fig. 114, the mobile phone may also decode the above-mentioned Y frame display data (i.e. display data 1) according to the display parameters obtained by parsing. For example, if the display data indicating the data portion in the display parameters is encoded according to the h.265 format, the mobile phone may decode the extracted Y frame display data 1 according to the h.265 format to obtain decoded Y frame display data, i.e., the display data 2 in fig. 4. For example, the decoded display data 2 may be display data in YUV format (Y denotes brightness, and U and V denote chroma).

Then, for each data packet of the video file, the mobile phone can perform operations such as parsing, decapsulating, decoding, and the like according to the method, and finally the mobile phone plays the display data and the audio data obtained after decoding, thereby realizing the playing function of the corresponding video file in the video APP in the mobile phone.

At present, when a user projects a video in a mobile phone to a slave device (e.g., a television) for playing, as shown in fig. 114, the mobile phone may encode the decoded display data 2 and audio data 2 again according to a preset format according to a relevant screen projection protocol. For example, if the screen-casting protocol specifies that display data interacted between the master device and the slave device is unified into an h.264 format and audio data is unified into an ACC format, the mobile phone needs to encode the decoded display data according to the h.264 format again and encode the decoded audio data according to the ACC format again. Further, the cellular phone can transmit the re-encoded display data (i.e., display data 3 in fig. 114) and audio data (i.e., audio data 3 in fig. 114) to the television. Accordingly, the television may decode the received display data 3 and audio data 3 according to the relevant format as specified by the screen-casting protocol to obtain the decoded display data 2 and audio data 2. Furthermore, the television can play the decoded display data 2 and the decoded audio data 2, and the screen projection function of the corresponding video file in the video APP on the television is realized.

It can be seen that, when a host device (e.g., a mobile phone) performs screen projection, a series of processing such as parsing, decapsulating, decoding, and re-encoding on a data packet of a video file is required, so that the time consumed from the time when a user triggers the host device to perform screen projection to the time when a slave device (e.g., a television) starts playing display data and audio data is long, that is, the time delay of the screen projection process is long, and the user experience is poor.

In this regard, in the embodiment of the present application, as shown in fig. 115, when a master device (e.g., a mobile phone) is on screen, the master device may directly transmit encoded audio data 1 and display data 1 extracted from a data packet by the master device to a television via the decoding capability of a slave device (e.g., a television). Furthermore, the television can decode the received audio data 1 and display data 1 by using its own decoding capability, and obtain decoded display data 2 and audio data 2. Furthermore, the television can play the decoded audio data 2 and the decoded display data 2, and the screen projection process of the video file in the video APP is completed.

Therefore, when the screen is projected, the main device does not need to decode and re-encode the encoded audio data and the encoded display data in the data packet, but directly sends the encoded audio data and the encoded display data in the video file to the slave device, and the slave device decodes and plays the decoded audio data and the decoded display data, so that the time delay of the whole screen projection process is shortened, and the use experience of a user in the screen projection scene is improved.

Moreover, because the audio data and the display data sent by the master device to the slave device are data which are not decoded and re-encoded in the video file, the audio data and the display data received by the slave device are not distorted due to decoding and re-encoding of the master device, so that lossless transmission of the audio data and the display data between the master device and the slave device is realized, and the fidelity of the audio and the image in a screen projection scene is improved.

Taking an operating system in a mobile phone as an example of an Android system, at present, as shown in fig. 116, when an APP in an application layer needs to play audio or video, a corresponding audio/video player, such as a nulayer, may be created in an application framework layer. For example, the video APP can issue the acquired video file to the Nuplayer in real time in the form of a data packet when the video APP runs. Nulayer can invoke a multimedia extractor (MediaExtractor) to parse and decapsulate the received packet. Through analysis operation, the Nulayer can acquire audio parameters and display parameters carried in the packet header of the data packet; by the decapsulation operation, the nulayer can extract the corresponding display data 1 and audio data 1 from the data portion of the data packet. At this time, the extracted display data 1 and audio data 1 after decapsulation are display data and audio data that have been encoded in the video file.

Further, the nulayer may call an audio decoder (audio decoder) to decode the decapsulated audio data 1 according to the parsed audio parameters, and obtain decoded audio data 2, such as PCM data. Then, the nulayer-ready image decoder (video decoder) decodes the decapsulated display data 1 according to the display parameters obtained by the analysis, and obtains decoded display data 2, for example, YUV data.

Further, the Nuplayer may input the decoded audio data 2 to the audio manager. As also shown in fig. 116, an audio player (AudioTrack), an audio processor (AudioFlinger), and an audio policy module (AudioPolicy) may be included in the audio manager. Specifically, the Nuplayer may input the decoded audio data 2 to the AudioTrack, and the AudioTrack calls the audioflanger to perform processing such as mixing, resampling, and sound effect setting on the decoded audio data 2. When the AudioFlinger processes the audio data 2, the AudioPolicy may set a corresponding audio policy (e.g., a mixing policy, a sampling policy, etc.) for the audio data currently processed by the AudioFlinger, and the AudioPolicy may send the set audio policy to the AudioFlinger, so that the AudioFlinger may process the decoded audio data 2 according to the audio policy.

Also, as shown in fig. 116, the nulayer may input the decoded display data 2 to a surface module, for example, a surface flunger, and render the decoded display data 2 by the surface flunger.

Subsequently, as also shown in fig. 116, the AudioFlinger may send the output audio data (which is still the decoded audio data 2 at this time) to the corresponding one of the HALs. Among them, HALs are provided which correspond to different hardware modules of the mobile phone, for example, an Audio HAL, a Camera HAL, a Wi-Fi HAL, a display HAL, and the like. The AudioFlinger may call the Audio HAL, send the processed Audio data to the Audio HAL, and send the Audio data 2 to a corresponding Audio output device (e.g., a speaker, an earphone, etc.) for playing.

Illustratively, the Audio HAL may further include a Primary HAL, A2dp HAL, etc., depending on the Audio output device. For example, when the audio output device of the mobile phone is the mobile phone itself, the AudioFlinger may call the Primary HAL to output the audio data 2 to the speaker (speaker) of the mobile phone for playing. For another example, when the audio output device of the mobile phone is a bluetooth headset, the AudioFlinger may call A2dp HAL to output the audio data 2 to the bluetooth headset connected to the mobile phone for playing.

Similarly, as also shown in fig. 116, the surfefinger can send the drawn display data 2 to the display output device of the mobile phone through the display HAL for display. For example, the surface flicker may output the display data 2 to the display screen of the mobile phone through the display HAL for display.

Through the processing process, each data packet of the video file to be played when the video APP operates can be analyzed, unpacked and decoded according to the method and then played, so that the mobile phone can play the relevant video file to the user through operating the video APP.

In this embodiment, for example, still taking the mobile phone as the main device, as shown in fig. 117, when the mobile phone runs the video APP to play the video file a, a nulayer for playing the video file a may be created in the application framework layer. Furthermore, the video APP can issue the acquired video file a to the Nuplayer in a data packet form in real time. After the DV APP in the mobile phone establishes network connection with the television 203, the DV APP may send a first broadcast message to the video APP to notify the video APP to project the video file a being played to the television 203 for playing. After receiving the first broadcast message, the video APP may continue to send a data packet to be played in the video file a, for example, data packet 1, to the nulayer.

Taking the example that the nulayer receives the data packet 1 from the video APP, after receiving the data packet 1, the nulayer may call the mediaextra to analyze and decapsulate the data packet 1.

By decapsulating the data packet 1, the nulayer can extract the display data and audio data carried by the data portion of the data packet 1. The display data and the audio data are data encoded in the video file a according to a certain encoding format. The embodiment of the application does not limit the encoding format of the display data and the audio data in the data packet of the video file A.

By analyzing the data packet 1, the nulayer can obtain the audio parameter 1 and the display parameter 1 carried in the packet header of the data packet 1. The audio parameter 1 is a parameter used when encoding the audio data in the data packet 1.

For example, the audio parameters 1 may include an encoding format (e.g., AAC format, etc.) used when encoding audio data. For another example, each coding format may include various different coding specifications or versions, for example, the AAC format further includes 9 different coding specifications, such as a Low Delay specification (LD), a variable Sample Rate (SSR), a High Efficiency specification (HE), and a Low Complexity specification (LC). The audio parameters 1 may then also comprise a specific coding specification or version, e.g. AAC-LC, in the coding format used when encoding the audio data in the data packet 1. For another example, the audio parameter 1 may further include parameters such as a sampling rate and a number of channels used when encoding the audio data, which is not limited in this embodiment of the present application.

The display parameter 1 is a parameter used when encoding the display data in the packet 1. For example, display parameter 1 may include an encoding format (e.g., h.265 format, h.264 format, etc.) used when encoding the display data in packet 1. Similarly, each encoding format may include various encoding specifications or versions, and the display parameter 1 may also include a specific encoding specification or version of the encoding format used for displaying the data in the encoded data packet 1, for example, a 3.1 version of h.265. For another example, the display parameter 2 may further include a resolution of the display data in the encoded data packet 1, for example, 1080 × 720.

Certainly, the header of the data packet 1 may also carry parameters such as the name, duration, or file format of the currently played video file a, which is not limited in this embodiment of the present application.

In some embodiments, after the video APP receives the first broadcast message sent by the DV APP, the video APP may further send a first notification message (also referred to as a screen projection indication) to the nulayer, and after receiving the first notification message, the nulayer may start to project the display data and the audio data in the received data packet to the television 203 according to a screen projection mode for playing. For example, if the nulayer receives packet 1 after receiving the screen projection indication, it indicates that the nulayer needs to project packet 1 to the slave device (i.e., the tv 203) for playing.

Still as shown in fig. 117, in the screen projection mode, the nulayer may send the parsed audio parameter 1 and display parameter 1 to the television 203 through the DV APP. Alternatively, the DV APP may also run in the application framework layer in the form of a system service (proxy service in fig. 117). At this time, the nulayer may transmit the audio parameter 1 and the display parameter 1 obtained by the analysis to the television 203 through the proxy service.

After receiving the audio parameter 1 and the display parameter 1, the television 203 may query whether itself has the corresponding audio decoding capability and image decoding capability according to the audio parameter 1 and the display parameter 1. For example, if the audio parameter 1 indicates that the encoding format used when encoding the audio data is AAC format, the encoding specification is AAC-LC, the sampling rate is 44100Hz, and the number of channels is 2, the television 203 may inquire whether the corresponding audio data can be decoded according to the audio parameter 1. If the corresponding audio data can be decoded, it indicates that the television 203 has the audio decoding capability for the audio data in the data packet 1. Similarly, if the display parameter 1 indicates that the encoding format used in encoding the display data is h.265 format, the encoding version is 3.1 version, and the resolution is 1080 × 720, the television 203 may query whether it is possible to decode the corresponding display data according to the display parameter 1. If the corresponding display data can be decoded, it indicates that the television 203 has the image decoding capability for the display data in the data packet 1.

Further, as also shown in fig. 117, if tv 203 has corresponding image decoding capability and audio decoding capability, tv 203 may send a first response message to the DV APP (or proxy service) of the handset indicating that tv 203 is capable of decoding the audio data and display data in packet 1. The DV APP (or proxy service) may send the first response message to the nulayer, triggering the nulayer to send the display data and audio data decapsulated in the data packet 1 to the television 203.

For example, the nulayer may send the display data and the audio data decapsulated in the data packet 1 to the television 203 through the DV APP (or proxy service) of the mobile phone. Or, the Nuplayer may send the display data and the Audio data decapsulated in the data packet 1 to the television 203 through a Wi-Fi module of the mobile phone, or the Nuplayer may send the display data and the Audio data decapsulated in the data packet 1 to the television 203 through the created DMSDP Audio HAL, which is not limited in this embodiment of the present application.

The display data and the audio data received by the television 203 from the mobile phone are decapsulated and not decoded data, the television 203 can decode the received display data by using its own image decoding capability, and similarly, the television 203 can decode the received audio data by using its own audio decoding capability. Furthermore, the television 203 can play the decoded display data and audio data, so that the video file a played by the video APP in the mobile phone is projected to the television 203 for playing.

In addition, in the screen projection process, the mobile phone (namely, the main device) only needs to analyze and decapsulate the data packet of the video file, does not need to decode the audio data and the display data in the data packet, and does not need to recode the decoded audio data and display data according to a uniform format like the prior art, so that the processing process of the main device on the audio data and the display data during screen projection is simplified. Therefore, the time consumption of the screen projection process of the master device to the television 203 (i.e. the slave device) is greatly reduced, so that the time delay of the screen projection is reduced, and the use experience of the user in the screen projection is improved.

Additionally, if tv 203 does not have audio decoding capability for the audio data in packet 1 and/or does not have image decoding capability for the display data in packet 1, tv 203 may send a second response message to the handset's DV APP (or proxy service) indicating that tv 203 is unable to decode the audio data and/or display data in packet 1. At this time, the DV APP (or the agent service) may send a screen-casting failure message to the video APP, and the video APP may prompt the user that this screen casting failure occurs after receiving the message.

Or, after receiving the screen projection failure message, the video APP may also instruct the Nuplayer to decode the audio data and the display data in the data packet 1 through the decoder according to the existing screen projection mode. For example, the Nuplayer may decode audio data in the packet 1 using an audio decoder and decode display data in the packet 1 using a video decoder. Furthermore, the Nuplayer may re-encode the decoded audio data and display data according to a preset unified format by using an associated encoder, and transmit the re-encoded audio data and display data to the television 203. The television 203 may decode and play the received audio data and display data according to a predetermined unified format.

In some embodiments, still taking the slave device when the mobile phone is projected as the television 203 for example, the software architecture of the operating system in the television 203 may be similar to that of the operating system of the mobile phone. As shown in fig. 118, the television 203 may include an application layer, an application framework layer, and a HAL. System-level system applications, such as agent APP, may be installed in the application layer of television 203. The agent APP interacts with the DV APP in the mobile phone to realize the interaction process between the mobile phone and the television 203. For example, the agent APP may interact with the DV APP to complete the process of establishing a network connection between the mobile phone and the television 203. The agent APP in tv 203 may also run in the application framework layer in the form of a system service (agent service in tv 203 in fig. 118).

For another example, as shown in fig. 118, during the screen projection process, the nulayer of the mobile phone may send the audio parameter 1 and the display parameter 1 in the parsed data packet 1 to the agent APP (or agent service) of the television 203. The agent APP (or agent service) of the tv 203 may query whether itself has a corresponding audio decoding capability according to the audio parameter 1, and similarly, the agent APP (or agent service) may query whether itself has a corresponding image decoding capability according to the display parameter 1. If tv 203 has corresponding audio decoding capability and image decoding capability, then the agent APP (or agent service) may send the first response message to the DV APP (or agent service) of the handset, indicating that tv 203 is capable of decoding the audio data and display data in data packet 1. Correspondingly, if the tv 203 does not have the corresponding audio decoding capability and/or image decoding capability, the agent APP (or agent service) of the tv 203 may send the second response message to the DV APP (or agent service) of the handset, so as to indicate that the tv 203 cannot decode the audio data and/or display data in the data packet 1.

If the agent APP (or agent service) of the tv 203 sends the first response message to the DV APP (or agent service) of the mobile phone, the nulayer of the mobile phone may be triggered to send the display data and the audio data decapsulated in the data packet 1 to the agent APP (or agent service) of the tv 203. For example, the display data decapsulated in the data packet 1 may be referred to as display data 1, the audio data decapsulated in the data packet 1 may be referred to as audio data 1, and both the display data 1 and the audio data 1 may be encoded in the video file a.

As also shown in fig. 118, a nulayer is also included in the application framework layer of the television 203. After the agent APP (or agent service) of the tv 203 receives the display data 1 and the audio data 1 sent by the mobile phone, the agent APP (or agent service) may send the display data 1 and the audio data 1 to the nulayer of the tv 203. Since the display data 1 and the audio data 1 in the data packet 1 sent by the mobile phone are encoded data in the video file a, the Nuplayer can create a corresponding audio decoder and a video decoder. Furthermore, the Nuplayer may send the received audio data 1 to an audio decoder for decoding, so as to obtain decoded audio data 2. And, the nulayer may send the received display data 1 to a video decoder for decoding, resulting in decoded display data 2.

In some embodiments, when the agent APP (or agent service) in tv 203 determines that it is capable of decoding audio data 1 and display data 1 in packet 1, nulayer may be instructed to create a corresponding audio decoder according to received audio parameter 1, and nulayer may be instructed to create a corresponding video decoder according to received display parameter 1. Thus, when the Nuplayer receives the display data 1 and the audio data 1 in the data packet 1 sent by the mobile phone, the display data 1 can be directly input to the video decoder for image decoding, and the audio data 1 can be input to the audio decoder for audio decoding. Thus, the time taken for the television 203 to process the audio data 1 and the display data 1 at the time of screen projection can be further shortened, and the delay time at the time of screen projection can be further reduced.

Subsequently, as also shown in fig. 118, the audio decoder may input the decoded audio data 2 to the audio flag through the audio track, and the audio flag performs audio processing such as sound effect setting on the decoded audio data 2. The video decoder may input the decoded display data 2 to the surfefinger, and render the decoded display data 2 by the surfefinger. Furthermore, the AudioFlinger may send the output Audio data 2 to an Audio output device (e.g., a speaker of the tv 203) for playing through the Audio HAL of the HAL layer; the surfefinger can send the output display data 2 to a display output device (for example, a display screen of the television 203) through a display HAL of the HAL layer for displaying, so that the video file a played by the video APP in the mobile phone is projected to the television 203 for playing.

It can be seen that the audio data and the display data in the video file a sent to the television 203 when the mobile phone is on screen are not decoded, that is, the audio data and the display data received by the television 203 are both source data in the video file a that are not decoded, and therefore, distortion caused by operations such as mobile phone decoding or re-encoding does not occur in the audio data and the display data. Furthermore, the television 203 can directly decode and play the source data which is not decoded in the video file a, so that the fidelity of the played video is higher.

The above embodiment exemplifies the case where the audio data and the display data in the data packet 1 are projected to the television 203 when the mobile phone is turned on. It can be understood that when the video APP plays the video file a in the mobile phone, the video APP further includes other data packets after the data packet 1, and the processing procedure of the other data packets when the video APP is projected on the screen may also refer to the processing procedure of the data packet 1 in the above embodiment.

For example, after the video APP issues the data packet 1 of the video file a to the nulayer, the video APP may continue to issue the data packet 2 of the video file a to the nulayer. Nulayer may call mediaextra to parse and decapsulate packet 2. Different from the foregoing embodiment, the audio parameters of the audio data and the display parameters of the display data in each data packet in the video file a are generally uniform, for example, the audio data in the data packet 1, the data packet 2 and other data packets in the audio file are uniformly encoded according to the encoding specification ACC-LC, and the audio parameters such as the sampling rate and the number of sound channels during encoding are uniform. For example, the display data in the packet 1, the packet 2, and the other packets in the audio file are all encoded in a unified manner in accordance with the version 3.1 of h.265, and the display parameters such as the resolution at the time of encoding are all the same. Then, if the television 203 has corresponding decoding capabilities for the audio data and display data in packet 1, the television 203 has corresponding decoding capabilities for both the audio data and display data within packet 2 and the other packets in the audio file.

Then, after the nulayer extracts the audio data and the display data in the data packet 2 through the MediaExtractor, the nulayer can directly send the extracted audio data and display data in the data packet 2 to the television 203 without sending corresponding audio parameters and display parameters to the television 203. At this time, the audio data and the display data are still the data in the video file a that is not decoded. Subsequently, the television 203 can decode the received audio data and display data, and play the decoded audio data and display data, so as to project the video file a played by the video APP in the mobile phone to the television 203 for playing.

If the video APP plays a new video file (e.g., video file B) when the mobile phone is on screen, the video APP can create a nulayer (i.e., a new nulayer) playing the video file B, and input a data packet of the video file B into the new nulayer. And the video APP may send a second notification message to the new nulayer, and the new nulayer may start to project the display data and the audio data in the received data packet to the television 203 to play after receiving the second notification message. For example, when the new nulayer receives a data packet (e.g., data packet 3) of the video file B, according to the processing procedure of data packet 1 in the above embodiment, the audio parameter and the display parameter in the parsed data packet 3 are sent to the television 203, and the television 203 is triggered to query whether the decoding capability for decoding the audio data and the display data in the data packet 3 is available. If the television 203 has corresponding decoding capabilities, the television 203 may send a first response message to the handset indicating that the television 203 is capable of decoding the audio data and the display data in the data packet 3. Furthermore, the new nulayer in the mobile phone can send the display data and the audio data extracted from the data packet 3 and the subsequent other data packets to the television 203, and the television 203 decodes the received display data and audio data and plays the decoded data.

In some embodiments, the handset may also replace the slave device when it is projected. For example, in the process that the mobile phone runs the video APP and projects the video file a to the television 203 for playing, if it is detected that the user input modifies the slave device from the television 203 to the notebook computer, the DV APP in the mobile phone may disconnect the network connection with the television 203, and establish the network connection with the notebook computer as a new slave device.

As shown in fig. 119, after the DV APP of the mobile phone establishes network connection with the notebook computer, the DV APP may send a second broadcast message to the video APP that is playing the video file a, so as to notify the video APP to project the video file a that is playing into the notebook computer for playing. After receiving the second broadcast message, the video APP may send a third notification message (which may also be referred to as a change message) to the Nuplayer, in addition to continuing to send a data packet (for example, data packet 4) to be played in the video file a to the Nuplayer, where the third notification message may include an identifier of a new slave device (i.e., a notebook computer). The nulayer can determine that the slave equipment of the mobile phone is changed from the television 203 to the notebook computer after receiving the third notification message.

As shown in fig. 119, after receiving the third notification message, the nulayer may start to project the display data and the audio data in the received data packet to the notebook computer for playing. For example, the nulayer may call mediaextra to parse and decapsulate the data packet 4, and obtain the display data and the audio data in the data packet 4, and the display parameter 2 of the display data and the audio parameter 2 of the audio data. After receiving the change message, the Nuplayer needs to acquire the decoding capability of whether the notebook computer can decode the display data and the audio data in the video file a. Still as shown in fig. 119, the nulayer may send the parsed display parameter 2 and audio parameter 2 to the notebook computer, and trigger the notebook computer to query whether the notebook computer has the decoding capability of decoding the audio data and the display data in the data packet 4.

If the notebook computer has the corresponding decoding capability, the notebook computer may send a first response message to the mobile phone for indicating that the notebook computer can decode the audio data and the display data in the data packet 4. Further, as shown in fig. 119, the nulayer in the mobile phone can send the display data and the audio data extracted from the data packet 4 to the notebook computer, and the notebook computer decodes the received display data and audio data and plays them.

And before the nulayer receives the change message of the slave device again, the nulayer can send the display data and the audio data in other data packets in the video file a after the data packet 4 to the notebook computer, and the notebook computer decodes and plays the received display data and audio data. Therefore, when the slave equipment is changed when the mobile phone is projected to the screen, the mobile phone can project the video file A played in the video APP to the television 203 for playing. The processing procedure of the notebook computer on the received display data and audio data is similar to the processing procedure of the television 203 on the received display data and audio data in the above embodiment, and therefore is not described herein again.

Similarly, after switching the slave device when the mobile phone (i.e. the master device) is turned on, if the slave device has the capability of decoding the currently played video file, the mobile phone only needs to parse and decapsulate the data packet of the video file. The mobile phone does not need to decode the audio data and the display data in the data packet, and does not need to recode the decoded audio data and the decoded display data according to a uniform format like the prior art. Therefore, the processing process of the audio data and the display data by the main equipment during screen projection is simplified, and the time consumed for the main equipment to project the screen to the notebook computer (namely the slave equipment) is shortened.

Moreover, the audio data and the display data received by the notebook computer are both source data which are not decoded in the video file A, so that the audio data and the display data can not be distorted due to operations such as mobile phone decoding or recoding. Subsequently, the notebook computer can directly decode and play the source data which is not decoded in the video file A, so that the fidelity of the played video is higher.

In some embodiments, in the process of the mobile phone running the video APP to project a screen to the slave device, as shown in fig. 120, after receiving a data packet of a video file, the nulayer 1 in the mobile phone may directly send the undecoded audio data and display data extracted from the data packet to the slave device, and the slave device decodes and plays the received audio data and display data.

That is, the nulayer 1 in the mobile phone does not need to call the relevant decoder to decode the audio data and the display data in the data packet, so the nulayer 1 does not need to input the decoded audio data to the AudioFlinger for audio processing, and similarly, the nulayer 1 does not need to input the decoded display data to the SurfaceFlinger for rendering. That is to say, in the process of projecting a screen to the slave device, functional modules used for playing audio and video data, such as AudioTrack, AudioFlinger and SurfaceFlinger, in the mobile phone are not occupied by the video APP and are in an idle state.

Thus, as shown in fig. 120, in the process of running the video APP to project a screen, the mobile phone may also run other applications, and play display data and/or audio data generated by the other applications by using the functional modules such as the AudioTrack, the AudioFlinger, and the surface flinger. For example, the user may switch a video APP running in the handset to the background, at which point the nulayer 1 in the handset may continue to send the undecoded audio data and display data from the video APP to the slave device.

The video APP is switched to the background, if the user opens the WeChat APP in the mobile phone, as shown in fig. 120, the WeChat APP can create a corresponding Nulayer 2, and the Nulayer 2 performs operations such as parsing and decapsulation on the display data 1 and/or the audio data 1 from the WeChat APP. Further, Nuplayer2 may input audio data 1 to an audio decoder for decoding, resulting in decoded audio data 2. And/or nulayer 2 may input display data 1 to a video decoder for decoding, resulting in decoded display data 2. Further, the audio decoder may input the decoded audio data 2 to the AudioTrack, and the AudioTrack calls the AudioFlinger to perform processing such as mixing, resampling, and sound effect setting on the decoded audio data 2. And/or, the video decoder may input the decoded display data 2 to the surface flunger, and render the decoded display data 2 by the surface flunger.

Subsequently, the audioFlinger in the mobile phone can call the audioHAL, the decoded Audio data 2 output by the audioFlinger is sent to the audioHAL, and the audioHAL sends the Audio data 2 to the Audio output equipment such as a loudspeaker or an earphone of the mobile phone for playing. Similarly, the SurfaceFlinger in the handset may send the decoded display data 2 to the display of the handset via the display HAL for display. In this way, while the mobile phone projects the video file in the video APP to the slave device, other applications (for example, the above WeChat APP) can be run, and display data and/or audio data in the other applications are played, so that a user can use the application functions provided by the other applications in the mobile phone while using the mobile phone to project a screen.

Certainly, in the process of running the video APP to perform screen projection on the mobile phone, the nulayer 1 may also decode the audio data and the display data in the received video file according to the above method, and then use the functional modules such as the AudioTrack, the AudioFlinger, and the surfafinger to play the audio data and/or the display data in the video APP, so that the mobile phone can play the audio data and/or the display data in the relevant video file synchronously with the slave device when performing screen projection, which is not limited in this embodiment of the present application.

In the above embodiments, it is exemplified that the mobile phone projects the video file in the video APP to the slave device to be played, and in other embodiments of the present application, the mobile phone may also project the audio file in the application to the slave device to be played. Similarly, an audio file may also include one or more data packets, and of course, the audio file may also include other information such as format, duration, name, etc. of the audio file, and these information and the data packets of the audio file together form the video file.

As shown in fig. 121, similar to running the video APP, when the mobile phone runs the music APP to play the audio file a, a nulayer playing the audio file a may be created in the application framework layer. And the music APP can issue the audio file A to the Nulayer in a data packet mode in real time.

When the mobile phone runs the music APP, if the user sets the sound box as the audio output device of the mobile phone, as shown in fig. 121, the DV APP in the mobile phone may establish a network connection with the sound box as a slave device. In turn, the DV APP may send a first broadcast message to the music APP to inform the music APP to project the audio file a being played into the television 203 for playing. After receiving the first broadcast message, the music APP may continue to send a data packet to be played in the audio file a, for example, data packet 5, to the nulayer. Unlike the above-described video file a (or video file B), the data packet of the audio file a includes corresponding audio data therein, but does not include display data.

As shown in fig. 121, after the nulayer receives the data packet 5 sent from the music APP, the mediaextra may be called to parse and decapsulate the data packet 5, so as to obtain the audio data 1 in the data packet 5 and the audio parameter a of the audio data 1. Similarly, the audio parameter a is a parameter used when the audio data 1 in the data packet 5 is encoded, such as an encoding format, an encoding specification or version, a sampling rate, a number of channels, and the like, which is not limited in this embodiment.

As shown in fig. 121, the nulayer may send the audio parameter a to an agent APP (or agent service) of the sound box through the DV APP. The agent APP (or agent service) of the loudspeaker can inquire whether the agent APP (or agent service) has corresponding audio decoding capability according to the audio parameter a. If the sound box has the corresponding decoding capability, the agent APP (or agent service) of the sound box can send a first response message to the DV APP (or agent service) of the mobile phone, so as to indicate that the sound box can decode the audio data 1 in the data packet 5. In turn, the DV APP (or proxy service) of the handset sends a first response message to the nulayer of the handset to indicate that the loudspeaker is capable of decoding the audio data 1 in the data packet 5. And after determining that the agent APP (or agent service) of the sound box has the audio decoding capability for the audio data 1 in the data packet 5, the agent APP (or agent service) of the sound box can also instruct the nulayer of the sound box to create a corresponding audio decoder according to the audio parameter a.

Subsequently, as shown in fig. 121, the nulayer in the mobile phone may send the undecoded audio data 1 extracted from the data packet 5 to the proxy APP (or proxy service) of the speaker, and the proxy APP (or proxy service) of the speaker may send the audio data 1 to the nulayer of the television 203. The nulayer may send the received audio data 1 to an audio decoder for decoding, resulting in decoded audio data 2. Further, the audio decoder may input the decoded audio data 2 to the AudioFlinger through the AudioTrack, and the AudioFlinger may perform audio processing such as sound effect setting on the decoded audio data 2. Finally, the AudioFlinger may send the output decoded Audio data 2 to the speakers of the sound box for playing through the Audio HAL of the HAL layer.

If the audio output device of the mobile phone is not changed, the Nuplayer can send the audio data in other data packets in the audio file A after the data packet 5 to the sound box according to the method, the sound box decodes the received audio data by utilizing the decoding capability of the sound box and plays the decoded audio data, and therefore the audio file A played by the music APP in the mobile phone is projected to the sound box to be played.

Similar to the projection process of the video file, when the mobile phone projects the audio file to the slave device, if the slave device has the decoding capability for the currently played audio file, the mobile phone only needs to parse and decapsulate the data packet of the audio file, and directly send the audio data that is not decoded in the audio file to the slave device (e.g., a sound box). The mobile phone does not need to decode the audio data in the data packet, and does not need to recode the decoded audio data according to a uniform format like the prior art. Therefore, the processing process of the audio data by the master device during audio switching is simplified, and the time consumed for the screen projection of the master device to the slave device is shortened.

Moreover, the audio data received by the sound box is source data which is not decoded in the audio file A, so that distortion caused by operations such as mobile phone decoding or recoding and the like cannot occur in the audio data. Subsequently, the sound box can directly decode and play the source data which is not decoded in the audio file A, so that the fidelity of the played audio is higher.

It should be noted that, similar to the process of projecting the video file, when the mobile phone plays a new audio file, or the audio output device (i.e., the slave device) of the mobile phone is changed, the mobile phone may retrieve the decoding capability of the slave device for the currently played audio file. If the slave device can decode the currently played audio file, the mobile phone can send the audio data which is not decoded in the audio file to the slave device according to the method, and the slave device decodes and plays the received audio data, so as to reduce the time delay when the master device projects the audio to the slave device.

In a distributed audio scene, the master device can switch one or more services related to audio functions (for example, an audio playing service, a recording service, a video playing service or a camera service) of the master device to the slave device for execution, so that the distributed scene of the cooperative work of the multiple devices is presented.

Wherein a certain service may correspond to one or more functions in an application. For example, a video APP may provide a video playing service for implementing a screen projection function. For another example, the audio APP may provide an audio playing service for implementing an audio playing function, and may also provide a karaoke service for implementing an audio recording function. In a distributed scenario, the master device may switch a service to the slave device, and at this time, the master device may perform the service in cooperation with the slave device.

For example, the master device may send the audio data being played to the slave device, and the slave device plays the audio data, thereby switching the audio playing service of the master device to be executed on the slave device. For another example, the master device may send display data in a video to one slave device for playing, and send audio data in the video to another slave device for playing, so as to switch a video playing service of the master device to be executed on multiple slave devices. One service may include one or more subtasks. For example, the video playing service may include a display subtask related to a display function and an audio subtask related to an audio function.

For another example, the master device may collect audio data using a microphone of the slave device, and send the collected audio data to the master device for saving or playing, so as to switch the recording service of the master device to the slave device for execution. For another example, the master device may capture a captured image using a camera of the slave device, and send the captured image to the master device for display, so as to switch the camera service of the master device to the slave device for execution.

However, in the distributed audio scenario, when a master device switches a service to a slave device, service quality problems such as poor audio quality, poor display quality, asynchronous voice and image, and long delay may occur due to differences in device performance.

In this regard, in this embodiment of the present application, the master device may predict the service quality when the current service is switched to different slave devices for execution, and further, the master device may recommend one or more electronic devices executing the service to the user according to the prediction result. Therefore, the user can select proper equipment to cooperate with the master equipment to execute corresponding services according to the recommendation of the master equipment, and the service quality of the cooperative work of the master equipment and the slave equipment in a distributed scene is improved.

Illustratively, as shown in fig. 122, a master device may establish a network connection with one or more slave devices. Further, when the master device is ready to switch service 1 to one or more slave devices for execution, the master device may obtain device parameters associated with service 1 from each slave device.

For example, when the service 1 is a camera service, the device parameters associated with the service 1 may include an image processing algorithm supported by the slave device, the number of cameras, a resolution of the cameras, or a FOV of the cameras, and the like.

For another example, when the service 1 is an audio playing service, the device parameters associated with the service 1 may include sound effect algorithms supported by the slave device, the number of channels, a sampling rate, a mixing strategy, a model number of a speaker, and the like.

For another example, when the service 1 is a recording service, the device parameters associated with the service 1 may include a voice processing algorithm supported by the slave device, a codec algorithm of audio data, the number of microphones, and the like.

For another example, when the service 1 is a display service, the device parameters associated with the service 1 may include resolution, frame rate, and codec algorithm of display data supported by the slave device.

As also shown in fig. 122, after the master device acquires the device parameters associated with the service 1 from each slave device, the service quality of the service 1 executed by each slave device may be predicted according to the acquired device parameters. Furthermore, the master device may determine a corresponding recommendation device according to the service quality of each slave device executing the service 1. For example, the master device may determine the slave device performing service 1 with the highest quality of service as the recommendation device. Or, after the service 1 is switched to a certain slave device for execution, that is, in the process of executing a certain service cooperatively by the master device and the slave device, the master device may obtain the device parameter associated with the service 1 from each slave device, and determine the corresponding recommendation device according to the method.

After the main device determines the corresponding recommendation device, the determined recommendation device can be recommended to the user. For example, the main device may display a prompt message, and the prompt message may carry an identifier of the recommendation device. If the fact that the user clicks the identification of the recommendation device is detected, the main device can be triggered to switch the service 1 to the corresponding recommendation device again for execution.

In this way, the user can timely know the specific device suitable for executing the service 1 in the current distributed scene through the prompt message. And the user can quickly switch and execute the specific equipment of the service 1 through the identifier in the prompt message, so that the service quality when multiple pieces of equipment cooperatively work in a distributed scene is improved, and the use experience of the user is improved.

Still taking the mobile phone as the example of the above-mentioned master device, based on the software architecture of the Android system shown in fig. 6, as shown in fig. 123, a prediction sensing module 21301 may be set in an application framework layer of the mobile phone. After the network connection is established between the mobile phone and one or more slave devices, the prediction sensing module 21301 may detect the currently running service in real time. Such as an audio play service, a video call service, a live broadcast service, or a recording service. Furthermore, the prediction sensing module 21301 may obtain, according to the currently running service, device parameters corresponding to the service from each slave device. For example, the device parameters may include one or more of audio parameters, display parameters, or camera parameters.

The audio parameters may include one or more items of sound effect algorithms, the number of channels, sampling rates, mixing strategies, coding and decoding algorithms of audio data, the number and model of microphones, the number and model of loudspeakers, and the like, which are supported by the slave device. The audio parameters may reflect audio capabilities (e.g., playback capabilities or recording capabilities) of the slave device.

The display parameters may include one or more of resolution, frame rate, and codec algorithm of the display data supported by the slave device. The display parameter may reflect the display capabilities of the slave device.

The camera parameters may include one or more of an image processing algorithm supported by the slave device, the number of cameras, a resolution of the cameras, or a field of view (FOV) of the cameras. The camera parameters may reflect the shooting capabilities of the slave device.

For example, after the prediction sensing module 21301 of the mobile phone determines the device parameters corresponding to the current service, the device parameters may be obtained from the corresponding slave device through the corresponding DMSDP HAL. For example, as shown in fig. 123, the prediction sensing module 21301 may send an acquisition request to the slave device 1 through the DMSDP HAL 1, the slave device 1 may send its own device parameters to the DMSDP HAL 1 of the mobile phone in response to the acquisition request, and then the DMSDP HAL 1 sends the device parameters of the slave device 1 to the prediction sensing module 21301. Thus, according to the above method, the prediction sensing module 21301 may obtain device parameters corresponding to the current service from each slave device.

Subsequently, the prediction sensing module 21301 may predict, according to the obtained device parameters, the service quality when each slave device executes the service. For example, the prediction awareness module 21301 may score the service quality of each slave device when the slave device performs the service according to the device parameter of the slave device. Further, the predictive awareness module 21301 may recommend one or more slave devices with higher scores to the user as recommendation devices. For example, the prediction sensing module 21301 may report the determined recommendation device to an APP running the current service, and the APP carries the recommendation device in a prompt message and displays the prompt message in a display interface.

In this way, a user can know the recommendation device suitable for executing the service in the current distributed scene, so that the current service is switched to the recommendation device determined by the prediction sensing module 21301 to be executed, the service quality when multiple devices cooperatively work in the distributed scene is improved, and the use experience of the user is improved.

The following describes a method for recommending devices when a mobile phone recommends different services to a user in a distributed scenario in detail with reference to specific scenarios and examples.

For example, a user may select one or more electronic devices in a cell phone as slave devices of the cell phone. For example, after the mobile phone detects that the user opens the control center, it may detect the electronic device currently accessing the same communication network as the mobile phone. For example, a handset may search for one or more electronic devices that are located in the same Wi-Fi network as the handset. For another example, the cell phone may query the server for one or more electronic devices that are logged into the same account as the cell phone (e.g., huayao account). Further, as shown in fig. 124, the cellular phone may display the detected one or more electronic devices in the control center 21401. At this time, the user may select one or more electronic devices in the control center 21401 as slave devices of the cellular phone.

As also shown in fig. 124, the control center 21401 displays a tv 21402, a speaker 21403, a watch 21404, and a tv 21405, which are detected by the mobile phone. If the user is detected to click on television 21402, the DV APP of the handset may establish a network connection with television 21402 as a slave to television 21402. Further, as shown in fig. 125, the DV APP of the mobile phone may obtain capability parameters of the television 21402, such as shooting capability parameters, audio capability parameters, display capability parameters, and the like of the television 21402, from the television 21402 based on the network connection. Subsequently, the DV APP of the mobile phone may create a corresponding DMSDP HAL 1 in the HAL according to the capability parameters of the television 21402, so that the mobile phone may interact with the television 21402 through the DMSDP HAL 1.

Similarly, if it is further detected that the user clicks the speaker 21403, as shown in fig. 125, the mobile phone may obtain the capability parameters of the speaker 21403 according to the above method, and create a corresponding DMSDP HAL2 in the HAL according to the capability parameters of the speaker 21403. If it is detected that the user clicks the television 21405, the mobile phone may obtain the capability parameter of the television 21405 according to the above method, and create a corresponding DMSDP HAL 3 in the HAL according to the capability parameter of the television 21405. In this way, the mobile phone can be used as a master device to interact with three slave devices, namely, the television 21402, the loudspeaker 21403 and the television 21405, through the corresponding DMSDP HAL, so as to implement various services in a distributed scenario.

The services performed by a handset or slave device typically involve one or more of an audio function, a display function or a camera function. For example, the services related to the audio function may include services such as audio playing, recording, karaoke, game, voice call, or video call. For example, the services related to the display function may include services of video playing, video calling, games, and the like. Also for example, the services related to the camera function may include services of taking a picture, recording a video, live broadcasting, scanning a code, or video calling.

In this embodiment, after the mobile phone establishes a network connection with the slave devices (e.g., the television 21402, the speaker 21403, and the television 21405), the prediction sensing module 21301 in the mobile phone may detect, in real time, a specific service (which may be referred to as a target service in the following embodiments) being executed by the mobile phone in a distributed scenario.

For example, the prediction awareness module 21301 may obtain an identifier of an application (e.g., a package name of the application) that the mobile phone is running, and then determine the target service being executed according to the identifier of the application. For example, if the running application is a video APP, the prediction awareness module 21301 may determine that the target service is a video playback service.

Or, the prediction and perception module 21301 may further obtain the use states of the devices in the mobile phone, and determine the target service being executed according to the use states of the devices. For example, when the mobile phone is running the WeChat APP, if the front camera of the mobile phone is in an operating state, the prediction sensing module 21301 may determine that the target service is a video call service. Correspondingly, if the rear camera of the mobile phone is in a working state, the prediction sensing module 21301 may determine that the target service is a code scanning service.

The device parameters affecting the quality of service may be different for different services at run time. For example, for audio playing services, the number of speakers of the device, the sound effect algorithm and the mixing strategy have a large impact on the quality of the services. For another example, for a live broadcast service, the display resolution of the device, the model of the camera, the image processing algorithm, and the time delay have a large influence on the service quality.

Then, as shown in table 21, the corresponding relationship between different services and associated device parameters may be stored in the handset in advance. After the current target service is determined by the prediction sensing module 21301 of the mobile phone, the specific device parameters associated with the target service may be determined according to the corresponding relationship shown in table 21. Further, as also shown in fig. 125, the prediction aware module 21301 of the handset may obtain device parameters associated with the target service from each slave device through the corresponding DMSDP HAL.

TABLE 21

Business	Parameters of the equipment
		Service A	Device parameter A and device parameter B
Service B	Device parameter A, device parameter B, and device parameter C
		……	……

Taking the target service as an example of the audio playing service, as shown in fig. 126, when the mobile phone runs the music APP to play song a, the user can switch the song a to the television 21402 for playing through the audio switching function. After recognizing that the current target service is an audio playing service, the prediction sensing module 21301 of the mobile phone can determine, according to the correspondence shown in table 21, the device parameters related to the audio playing service, including three audio parameters, namely, the number of speakers, the sound effect algorithm, and the sound mixing policy. Furthermore, the prediction sensing module 21301 of the mobile phone can obtain the device parameter 1 of the television 21402 from the television 21402 by calling DMSDP HAL 1, where the device parameter 1 includes the number of speakers of the television 21402, a sound effect algorithm, and a sound mixing strategy. Moreover, the prediction sensing module 21301 of the mobile phone may obtain the device parameters 2 of the sound box 21403 from the sound box 21403 by calling DMSDP HAL 2, where the device parameters 2 include the number of speakers of the sound box 21403, a sound effect algorithm, and a sound mixing policy. Similarly, the prediction sensing module 21301 of the mobile phone can obtain the device parameters 3 of the television 21405 from the television 21405 by calling the DMSDP HAL 3, where the device parameters 3 include the number of speakers of the television 21405, a sound effect algorithm, and a sound mixing policy.

Furthermore, the prediction sensing module 21301 of the mobile phone may predict, according to the obtained device parameter 1, device parameter 2, and device parameter 3, the service quality when each slave device executes the audio playing service. For example, the mobile phone may store in advance quality parameters corresponding to the device parameters in different services. For example, as shown in table 22, for different services, quality parameters of respective device parameters associated with the services are preset, and each quality parameter may reflect the requirement of the corresponding service on the device parameter. For example, in the audio playing service, the quality parameters corresponding to the number of speakers may be set to be 2 or more than 2, which indicates that 2 or more than 2 speakers can meet the service requirements of the audio playing service. For another example, in the live broadcast service, the quality parameter corresponding to the delay may be set to be within 500ms, which indicates that the delay within 500ms may satisfy the service requirement of the live broadcast service.

TABLE 22

Then, after the prediction sensing module 21301 of the mobile phone obtains the device parameter 1 from the television 21402, the service quality score 1 when the television 21402 executes the audio playing service using the device parameter 1 may be predicted by combining various quality parameters corresponding to the audio playing service in the table 22.

For example, the prediction sensing module 21301 may score each device parameter in the device parameters 1 by combining quality parameters corresponding to the audio playing service in the table 22, that is, determine a service quality score of each device parameter. For example, if the number of horns in the equipment parameter 1 is 0, the fraction F1 of the equipment parameter, the number of horns, is 0; if the number of the horns in the equipment parameter 1 is 2, the fraction F1 of the equipment parameter, namely the number of the horns, is 60; if the number of horns in the plant parameter 1 is 4, the fraction F1 of the plant parameter, the number of horns, is 80.

According to the scoring method, the prediction sensing module 21301 may obtain a score F1 scoring the number of loudspeakers in the device parameter 1, a score F2 scoring the sound effect algorithm in the device parameter 1, and a score F3 scoring the mixing strategy in the device parameter 1. Further, the prediction perception module 21301 may calculate a quality of service score 1 when the television 21402 performs an audio playing service using the device parameter 1 according to the score F1, the score F2, and the score F3. For example, the prediction sensing module 21301 may calculate a weighted average of the score F1, the score F2, and the score F3 according to a preset weight, and take the calculated weighted average as the quality of service score 1.

Similarly, after the prediction sensing module 21301 obtains the device parameter 2 from the sound box 21403, the service quality score 2 when the sound box 21403 executes the audio playing service using the device parameter 2 can be predicted according to the method described above. Moreover, after the prediction sensing module 21301 obtains the device parameter 3 from the television 21405, the service quality score 3 when the television 21405 executes the audio playing service by using the device parameter 3 can be predicted according to the method described above.

In other embodiments, the mobile phone may also pre-store a quality parameter scoring curve or scoring rule corresponding to each device parameter in different services. Still taking the audio playing service as an example, as shown in (a) of fig. 127, in the audio playing service, the corresponding relationship between the number of different speakers and the quality parameter score is curve 1. The predictive sensing module 21301 may determine a fraction F1 of the plant parameter corresponding to the number of horns in plant parameter 1, based on curve 1. As shown in (b) of fig. 127, a scoring rule 1 between different sound effect algorithms and quality parameter scoring in the audio playing service is shown. The predictive perception module 21301 may determine a score F2 for the device parameter, which is the audio effect algorithm in device parameter 1, according to scoring rule 1. Similarly, as shown in (c) of fig. 127, a scoring rule 2 between different mixing strategies and quality parameter scoring in the audio playing service is provided. The prediction perception module 21301 may determine a score F3 of the device parameter corresponding to the mixing strategy in the device parameter 1 according to the scoring rule 2.

Further, the prediction perception module 21301 may calculate a quality of service score 1 when the television 21402 performs an audio playing service using the device parameter 1 according to the score F1, the score F2, and the score F3. Of course, the prediction sensing module 21301 may also obtain, according to the above method, the service quality score 2 when the sound box 21403 executes the audio playing service by using the device parameter 2, and the service quality score 3 when the television 21405 executes the audio playing service by using the device parameter 3.

After the prediction sensing module 21301 of the mobile phone obtains the service quality score 1, the service quality score 2, and the service quality score 3, the recommendation device corresponding to the current audio playing service can be determined according to the service quality scores. For example, if the quality of service score 2 > the quality of service score 1 > the quality of service score 3 indicates that the quality of service for audio playback service performed by speaker 21403 is higher than the quality of service for audio playback service performed by current television 21402, then prediction awareness module 21301 may determine speaker 21403 as the recommendation device. At this time, as also shown in fig. 125, the prediction sensing module 21301 may report the determined identifier of the recommendation device (i.e., speaker 21403) to the running APP (i.e., music APP).

Further, as shown in fig. 128, the music APP may display a prompt message 21801 in the display interface 21801 according to the recommendation device reported by the prediction sensing module 21301. The prompt message 21801 is used to prompt the user to use the speaker 21403 with higher quality of service to execute the current audio playing service. For example, the prompting message 21801 may include an identification 21802 of the speaker 21403. In this way, the user can perceive the quality of service when other slave devices perform a certain service in a distributed scenario through prompt message 21801. Subsequently, if it is detected that the user clicks the indication 21802 of the sound box 21403, the mobile phone may switch the current audio playing service to the sound box 21403 for continuous execution, and simultaneously stop continuously executing the audio playing service on the television 21402, thereby improving the service quality of the audio playing service.

In some embodiments, the prediction sensing module 21301 of the mobile phone may also determine a corresponding recommendation device according to a quality of service score of a certain device parameter. For example, if the score of the device parameter of the mixing strategy of the speaker 21403 is higher than the scores of the device parameters of the mixing strategies of other slave devices, but the score of the device parameter of the sound effect algorithm of the television 21402 is higher than the scores of the device parameters of the sound effect algorithms of other slave devices, it indicates that the mixing capability of the speaker 21403 is higher, and the sound effect processing capability of the television 21402 on audio data is higher. At this time, prediction sensing module 21301 may use speaker 21403 as a recommendation device and notify music APP that the mixing capability of speaker 21403 is higher than that of other slave devices. Further, as shown in fig. 129, the music APP may prompt the user in a prompt message 21901 that the speaker 21403 has a stronger mixing capability in the audio playing service. Of course, the music APP may also prompt the user in the prompt message 21901 that the sound effect processing capability of the television 21402 currently playing audio data is stronger.

In this way, in a distributed scenario, a user may perceive different service capabilities when different slave devices execute a certain service, so as to selectively select a corresponding slave device to execute the service.

Alternatively, the prediction sensing module 21301 may notify the determined recommendation device (e.g., the speaker 21403) to other APPs, for example, notify a system APP such as a central APP. For example, after receiving the identifier of the recommendation device sent by the prediction sensing module 21301, similar to fig. 128 or fig. 129, the notification center APP may prompt the user to use the sound box 21403 with higher service quality to execute the current audio playing service in the form of a notification message, which is not limited in this embodiment of the present application.

Still alternatively, the prediction sensing module 21301 may further notify the determined recommendation device to the television 21402 that is executing the audio playing service through the corresponding DMSDP HAL 1. Further, as shown in fig. 130, after acquiring the identifier of the recommendation device sent by the mobile phone, the television 21402 may display a prompt message 22001 to prompt the user to execute the current audio playing service by using the loudspeaker 21403 with higher service quality. Subsequently, if the television 21402 detects that the user selects the identifier of the speaker 21403 in the prompt message 22001, the television 21402 may stop executing the audio playing service. Meanwhile, the television 21402 may send a service switching message to the mobile phone, so that the mobile phone may switch the current audio playing service to the speaker 21403 for continued execution in response to the service switching message, thereby improving the service quality of the audio playing service.

Of course, if the prediction sensing module 21301 determines that the television 21402 currently performing the audio playing service is the device with the highest quality of service, the prediction sensing module 21301 may notify the music APP that the television 21402 currently performing the audio playing service is the optimal device. At this time, the mobile phone or the television 21402 may no longer recommend the other device to perform the audio playing service to the user.

In other embodiments, the prediction sensing module 21301 of the mobile phone may predict, in addition to the service quality scores of the slave devices (e.g., the tv 21402, the speaker 21403, and the tv 21405) executing the audio playing service, the service quality scores of the mobile phone itself executing the audio playing service.

For example, the prediction sensing module 21301 of the mobile phone may further obtain device parameters 4 associated with the current audio playing service, that is, the number of speakers in the mobile phone, the sound effect algorithm, and the sound mixing strategy. Furthermore, the prediction sensing module 21301 may predict the service quality score 4 when the mobile phone uses the device parameter 4 to execute the audio playing service according to the method described above. Subsequently, the prediction sensing module 21301 may determine the current recommendation device according to the service quality score 4 and the service quality scores corresponding to the slave devices. That is, the main device (i.e. the mobile phone) may also be a candidate device for the recommendation device, and when the service quality of the target service (e.g. the audio playing service) executed by the main device is high, the user may also be prompted to execute the target service by using the main device.

In the foregoing embodiment, an audio playing service is taken as an example for explanation, and it can be understood that, when a mobile phone executes other services related to an audio function in a distributed scene, a recommendation device with higher service quality when other services are executed may also be recommended to a user according to the foregoing method.

For example, as shown in fig. 131, when the mobile phone runs the video APP, the mobile phone may project the video B played in the video APP to the television 21402 to be played in response to the screen-projecting operation of the user. In the screen projection process, the mobile phone can send the display data of the video B to the television 21402 through the DMSDP HAL 1, and meanwhile, the mobile phone can send the audio data of the video B to the television 21402 through the DMSDP HAL 1.

In addition, in the screen projection process, the prediction sensing module 21301 of the mobile phone can identify that the current target service is a video playing service. Furthermore, the prediction sensing module 21301 may determine the device parameters associated with the video playing service according to the corresponding relationship shown in table 21. For example, the device parameters associated with the video playing service may include display parameters related to a display function, such as resolution of a display screen, and may also include audio parameters related to an audio function, such as the number of microphones and an audio effect algorithm.

Still taking the examples that the slave devices of the mobile phone include the television 21402, the speaker 21403, and the television 21405, for example, after the prediction sensing module 21301 of the mobile phone determines the device parameters associated with the video playing service, the device parameters associated with the video playing service may be acquired from each slave device through the corresponding DMSDP HAL. Furthermore, the prediction sensing module 21301 may predict, according to the method in the foregoing embodiment, a service quality score of each slave device when executing the video playing service according to the obtained device parameter, and determine a corresponding recommendation device according to the service quality score of each slave device when executing the video playing service.

For example, if the quality of service score when the television 21405 executes the video playing service is higher than the quality of service score when the television 21402 executes the video playing service, and the quality of service score when the television 21405 executes the video playing service is higher than the quality of service score when the sound box 21403 executes the video playing service, the prediction and perception module 21301 may determine the television 21405 as a recommendation device for executing the video playing service this time.

Further, as shown in (a) of fig. 132, the prediction sensing module 21301 may notify the video APP that the current recommendation device is the television 21405, so that the video APP may display a prompt message 22201 in the display interface to prompt the user to perform the current video playing service using the television 21405 with higher service quality. The prompt message 22201 may include an identifier of the television 21405. If the user clicks the identifier of the television 21405, the mobile phone can switch the current video playing service to the television 21405 for continuous execution, and simultaneously stop executing the video playing service continuously on the television 21402, so that the service quality of the video playing service is improved.

Alternatively, as shown in (b) of fig. 132, the prediction sensing module 21301 may notify the television 21402 that is performing a video playing service of the determined recommendation device through the corresponding DMSDP HAL 1. After the television 21402 determines that the recommendation device for executing the video playing service this time is the television 21405, a prompt message 22202 may be displayed to prompt the user to execute the current video playing service using the television 21405 with higher service quality. Similarly, if the television 21402 detects that the user selected the identity of the television 21405 in the prompt message 22202, the television 21402 may stop performing the video playback service described above. Meanwhile, the television 21402 may send a service switching message to the mobile phone, so that the mobile phone may switch the current video playing service to the television 21405 for continued execution in response to the service switching message, thereby improving the service quality of the audio playing service.

Of course, if the prediction sensing module 21301 determines that the television 21402 currently performing the video playing service is the device with the highest service quality, the prediction sensing module 21301 may notify the video APP that the television 21402 currently performing the video playing service is the optimal device. At this time, the mobile phone or the television 21402 may no longer recommend the other device to perform the video playing service to the user.

In other embodiments, the prediction sensing module 21301 of the mobile phone may also determine that multiple slave devices execute the current video playing service (i.e., the target service) according to the service quality scores of different device parameters in different slave devices. In this case, there may be a plurality of recommendation devices determined by the prediction perception module 21301.

For example, the prediction aware module 21301 may obtain device parameter 1 associated with a video playback service from the television 21402. The device parameters 1 include audio parameters and display parameters of the television 21402. Likewise, the prediction awareness module 21301 may obtain the device parameter 2 associated with the video playing service from the speaker 21403. The device parameters 2 include audio parameters and display parameters of the speaker 21403. If the service quality score of the display parameter in the device parameter 1 is higher than the service quality score of the display parameter in the device parameter 2, the display capability of the television 21402 is higher than the display capability of the speaker 21403. If the service quality score of the audio parameter in the device parameter 2 is higher than the service quality score of the audio parameter in the device parameter 1, the prediction sensing module 21301 indicates that the audio capability of the speaker 21403 is higher than the display capability of the television 21402.

At this time, the prediction sensing module 21301 may determine both of the speaker 21403 and the television 21402 as recommendation devices, where the television 21402 may be configured to execute a first subtask related to a display function in the video playing service, and the speaker 21403 may be configured to execute a second subtask related to an audio function in the video playing service. Furthermore, the prediction and perception module 21301 may notify the video APP of the mobile phone that the television 21402 is a recommendation device for executing the first sub-task, and the speaker 21403 is a recommendation device for executing the second sub-task. Further, as shown in fig. 133, the video APP may display a prompt message 22301 according to the recommendation device notified by the prediction perception module 21301. The prompt message 22301 may prompt the user to cooperatively execute the video playing service using the device group consisting of the television 21402 and the speaker 21403, so that the user may enjoy the optimal video playing service quality.

Subsequently, if it is detected that the user selects the confirmation button 22302 in the prompt message 22301, the mobile phone may switch the display data in the video B being played to the television 21402 for playing (i.e., switch the first sub task in the video playing service to the television 21402), and simultaneously switch the audio data in the video B to the sound box 21403 for playing (i.e., switch the second sub task in the video playing service to the sound box 21403), so as to improve the service quality of the video playing service in a multi-device cooperative working manner.

That is to say, the master device (for example, a mobile phone) may divide the current target service into a plurality of subtasks, and then determine the best recommendation device for each subtask according to the device parameter of each slave device, so as to distribute the target service to the plurality of recommendation devices for execution, and improve the service quality of the target service by means of cooperative work of the plurality of devices.

In addition to the above audio playing service and video playing service, the mobile phone may recommend a corresponding recommendation device according to the device parameter of the slave device, and when the mobile phone executes a service related to the camera function, the mobile phone may also recommend a recommendation device with higher quality of service to the user according to the above method.

For example, when the mobile phone runs the WeChat APP, the user can open the video call function of the WeChat APP to perform video call with a contact (e.g., the contact Sam). As shown in fig. 134, during a video call, the mobile phone may use a camera of the television 21402 to acquire image data, and the mobile phone may use a display screen of the television 21402 to display a video call interface. At this time, the mobile phone may obtain image data acquired by a camera of the television 21402 through the DMSDP HAL 1, and the mobile phone may send display data of the video call interface to the television 21402 through the DMSDP HAL 1 for display.

In the process of video call, the prediction sensing module 21301 of the mobile phone can identify that the current target service is a video call service. Furthermore, the prediction sensing module 21301 may determine the device parameters associated with the video call service according to the correspondence shown in table 21. For example, the device parameters associated with the video call service may include display parameters related to a display function, such as resolution of a display screen, audio parameters related to an audio function, such as the number of microphones and an audio effect algorithm, camera parameters related to a camera function, such as the number of cameras, and parameters such as time delay.

Still taking the examples that the slave devices of the mobile phone include the television 21402, the speaker 21403, and the television 21405, after the prediction sensing module 21301 of the mobile phone determines the device parameters associated with the video call service, the device parameters associated with the video call service may be acquired from each slave device through the corresponding DMSDP HAL. Furthermore, the prediction sensing module 21301 may predict, according to the method in the foregoing embodiment, a quality of service score when each slave device executes the video call service, and determine a recommendation device according to the quality of service score when each slave device executes the video call service.

For example, if the quality of service score when the television 21405 performs the video call service is higher than the quality of service score when the television 21402 performs the video call service, and the quality of service score when the television 21405 performs the video call service is higher than the quality of service score when the speaker 21403 performs the video call service, the prediction sensing module 21301 may determine the television 21405 as a recommendation device for performing the video call service this time.

Further, as shown in (a) of fig. 135, the prediction sensing module 21301 may notify the WeChat APP that the current recommendation device is the television 21405, so that the WeChat APP may display a prompt message 22501 in a display interface to prompt the user to perform the current video call service using the television 21405 with higher quality of service. Alternatively, as shown in (b) of fig. 135, the prediction sensing module 21301 may notify the television 21402 that is performing the video call service of the determined recommendation device through the corresponding DMSDP HAL 1, and the television 21402 displays a prompt message 22502 to prompt the user to perform the current video call service using the television 21405 with higher quality of service.

In addition, for services with high real-time requirements, such as a video call service, a live broadcast service or a karaoke service, the prediction sensing module 21301 of the mobile phone can also predict the time delay required for executing the corresponding service, and recommend corresponding recommendation equipment based on the predicted time delay. That is, for a service with a high real-time requirement, the time delay required for each device to execute the service can be used as a criterion for determining the quality of the service. For example, when calculating the service quality score of each device executing the service, the mobile phone may set the weight value of the device parameter of time delay to 1, and at this time, the service quality of the current service completely depends on the device parameter of time delay.

Still taking the video call service as an example, after the prediction sensing module 21301 of the mobile phone identifies that the current target service is the video call service, the network delay between the mobile phone and each slave device and the processing delay of each slave device for executing the video call service can be obtained. In this way, the delay of each slave device executing the video call service in the distributed scenario is the sum of the network delay and the processing delay.

Taking the network connection between the television 21402 and the mobile phone as an example of Wi-Fi connection, the prediction sensing module 21301 of the mobile phone may call Wi-Fi HAL to detect the network delay L11 between the mobile phone and the television 21402. Moreover, the prediction sensing module 21301 may obtain the processing delay L12 for the television 21402 to execute the video call service from the television 21402 through DMSDP HAL 1. Further, the predictive sensing module 21301 may calculate the time delay L1-L11 + L12 of the tv 21402 executing the video call service. Similarly, the prediction and sensing module 21301 may also calculate the time delay L2 for the speaker 21403 to execute the video call service and the time delay L3 for the television 21405 to execute the video call service according to the above method.

Illustratively, if the quality parameter of the delay in the video call service is set to be within 500ms in table 22, it is indicated that the delay within 500ms can meet the service quality requirement of the video call service. Then, if the prediction sensing module 21301 of the mobile phone calculates that the delay L1 of the television 21402 for performing the video call service is greater than 500ms, and the delay L3 of the television 21405 for performing the video call service is less than 500ms, the prediction sensing module 21301 may determine the television 21405 as a recommendation device for the video call service. At this time, as shown in fig. 136, the prediction sensing module 21301 may notify the WeChat APP that the current recommendation device is the television 21405, so that the WeChat APP displays a prompt message 22601 in the display interface according to the recommendation device, and prompts the user to execute the current video call service using the television 21405 with lower time delay. Of course, the prompt message may also be displayed in the television 21402 that is performing the video call service, which is not limited in this embodiment.

In some embodiments, when the delay L1 of the tv 21402 for performing the video call service is greater than 500ms, the prediction and perception module 21301 of the handset may further analyze the specific reason why the delay L1 exceeds 500 ms. For example, if the network delay L11 between the television 21402 and the mobile phone is greater than the threshold 1, which results in the delay L1 exceeding 500ms, which indicates that the current network state is poor and the delay when interacting with each slave device is long, the prediction sensing module 21301 may determine the mobile phone itself as a recommendation device for executing the video call service. At this time, as shown in (a) of fig. 137, the prediction awareness module 21301 may notify the WeChat APP (or the television 21402) in the mobile phone to display a prompt message 22701, and prompt the user in the prompt message 22701 that the current network status is poor, so as to suggest that the user performs the current video call service using the mobile phone.

Alternatively, if the delay L1 exceeds 500ms because the processing delay L12 of the television 21402 for performing the video call service is greater than the threshold 2, the prediction awareness module 21301 may simplify the processing of the television 21402 for performing the video call service. For example, the prediction awareness module 21301 may determine video call strategy 1, where the television 21402 does not need to sound effect processing on the audio data. For another example, the prediction awareness module 21301 may determine a video call policy 2, where the tv 21402 may perform codec processing on the display data using a specified codec algorithm. Furthermore, as shown in (b) of fig. 137, the prediction sensing module 21301 may notify the WeChat APP (or the television 21402) in the mobile phone to display a prompt message 22702 according to the determined video call policy, and prompt the user in the prompt message 22702 to simplify the processing procedure of the television 21402 for executing the video call service, so as to reduce the time delay for executing the video call service. Subsequently, if it is detected that the user selects the confirmation button in the prompt message 22702, the prediction sensing module 21301 may send the video call policy to the television 21402 through the corresponding DMSDP HAL 1, so that the television 21402 may simplify the processing procedure for executing the video call service according to the video call policy, thereby reducing the time delay of the entire video call service and improving the service quality of the video call service.

Of course, the prediction sensing module 21301 of the mobile phone may also directly send the video call policy to the television 21402 after determining the video call policy for simplifying the video call service processing procedure, so that the television 21402 may simplify the processing procedure for executing the video call service according to the video call policy. At this time, the mobile phone (or the television 21402) may display a prompt message to prompt the user that the processing procedure of the video call service has been simplified, so as to reduce the time delay of the entire video call service.

Still alternatively, if the network delay L11 between the tv 21402 and the mobile phone is greater than the threshold 1, and the processing delay L12 of the tv 21402 executing the video call service is greater than the threshold 2, the prediction awareness module 21301 may select a slave device with a delay of less than 500ms as the recommendation device among other slave devices when executing the video call service. Further, similar to fig. 136, the prediction sensing module 21301 may notify the WeChat APP (or the television 21402) in the mobile phone to display a corresponding prompt message, prompting the user to execute the current video call service using the electronic device with a lower time.

In the above embodiments, it is described by way of example that the mobile phone serves as the master device and recommends the recommendation device more suitable for executing a certain service to the user in the process of executing the service in cooperation with the slave device.

Still taking the example in which the slave devices of the mobile phone include the television 21402, the speaker 21403, and the television 21405, as shown in (a) in fig. 138, a screen projection button 22802 may be provided in the pull-down menu 22801 of the mobile phone. If the user clicks the screen projection button 22802, it indicates that the mobile phone has opened the screen projection function to perform the screen projection service (e.g., video playing service). At this time, the prediction sensing module 21301 of the mobile phone may obtain device parameters associated with the screen projection service from each slave device according to the method in the foregoing embodiment, and predict a service quality score when each slave device executes the screen projection service according to the obtained device parameters, so as to recommend a corresponding recommendation device to the user according to the predicted service quality score.

For example, when the prediction and sensing module 21301 predicts that the service quality score 1 when the television 21405 performs the screen projection service is higher than the service quality score 2 when the television 21402 performs the screen projection service, and that the service quality score 2 when the television 21402 performs the screen projection service is higher than the service quality score 3 when the speaker 21403 performs the screen projection service, it indicates that the service quality of the screen projection services performed by the three slave devices, i.e., the television 21405, the television 21402, and the speaker 21403, is sequentially reduced. Then, as shown in (b) of fig. 138, the cellular phone may display the device list 222803 in response to the user clicking on the screen-projection button 22802. In the device list 222803, the handset may arrange the slave devices in order of highest quality of service score for performing the screen projection service. The user can select which slave device the content in the mobile phone is projected to play in the device list 222803, and the above arrangement sequence can recommend the user to execute the screen-casting service this time by using the device with higher service quality score, so as to improve the service quality of the subsequent screen-casting service.

Or, the prediction sensing module 21301 of the mobile phone may predict, according to the obtained device parameters, the specific service quality of different slave devices when the screen projection service is executed. For example, the prediction awareness module 21301 may predict that the television 21402 may have a phenomenon of picture and sound asynchronization when performing a screen projection service according to the device parameters of the television 21402. For another example, the prediction sensing module 21301 may predict, according to the device parameters of the speaker 21403, that the speaker 21403 cannot display data when performing a screen projection service. Further, as shown in fig. 139, the cellular phone may display the device list 222803 in response to the user clicking on the screen-projection button 22802. In the device list 222803, the mobile phone may prompt the user for a specific quality of service for each slave device to execute the screen-casting service, that is, notify the user of a specific reason why a certain device is recommended or not recommended to execute the screen-casting service.

That is to say, in a distributed scenario, a mobile phone as a master device may predict, before executing a certain service, a specific service quality when the service is switched to each slave device for execution, so as to prompt a user of a slave device with a better service quality when the user selects the slave device for executing the service. Therefore, the mobile phone can recommend the recommendation equipment suitable for executing the service to the user before the mobile phone and the multiple equipment cooperatively work, so that the user can conveniently switch the service to the recommendation equipment for execution, the service quality in the distributed scene during the cooperative work of the multiple equipment is improved, and the use experience of the user is improved.

Currently, as shown in fig. 140, when performing ultrasonic positioning indoors, a plurality of ultrasonic receivers 23001 are installed, and the ultrasonic receivers 23001 may be connected to a controller (not shown). The electronic device 23002 transmits an ultrasonic signal after the positioning function is turned on. After the ultrasonic receiver 23001 at a Different location receives the ultrasonic signal transmitted by the electronic device 23002, the controller may locate the electronic device 23002 based on a Time Of Arrival (TOA) or a Time Difference Of Arrival (TDOA) Of the ultrasonic signal at the Different ultrasonic receiver 23001. In such a positioning scenario, a user needs to purchase and install devices such as the ultrasonic receiver 23001 to position the electronic device 23002, which is complex in operation process and high in implementation cost.

In the embodiment of the present application, a plurality of ultrasonic receivers may be provided on the main device. For example, the plurality of ultrasonic receivers may be a plurality of microphones in a microphone array, each microphone being operable to acquire an ultrasonic signal. An ultrasonic generator may be provided on the slave device. For example, the ultrasonic generator may be a speaker, which may be used to emit ultrasonic signals. The master device may be used to locate the slave device via the ultrasonic signal.

Also, taking a mobile phone as a main device, for example, a plurality of ultrasonic receivers of the mobile phone may be provided inside the mobile phone, or a plurality of ultrasonic receivers may be incorporated outside the mobile phone. In some scenarios, a user may input an instruction to the handset to query the location of a device in the vicinity of the handset. For example, as shown in fig. 141, a button 23102 for inquiring nearby devices is provided in a control center 23101 of the mobile phone, and if it is detected that the user clicks the button 23102, the mobile phone can acquire an electronic device currently accessing the same communication network as the mobile phone. For example, a handset may search for one or more electronic devices that are located in the same Wi-Fi network as the handset. For another example, the cell phone may query the server for one or more electronic devices that are logged into the same account as the cell phone (e.g., huayao account).

For example, when the mobile phone and the television, the sound box 1 and the sound box 2 are located in the same communication network, after the mobile phone searches for the television, the sound box 1 and the sound box 2, the television, the sound box 1 and the sound box 2 can be used as slave devices to locate the slave devices, so as to determine the positions of the television, the sound box 1 and the sound box 2 (i.e., the slave devices) near the mobile phone.

In some embodiments, the handset may send positioning instructions to the television, speaker 1, and speaker 2, respectively. After receiving the positioning instruction, the television can respond to the positioning instruction and transmit an ultrasonic signal 1 through the ultrasonic generator. The multiple microphones in the microphone array in the mobile phone can receive the ultrasonic signal 1, and then the mobile phone can calculate the positioning result 1, namely the position of the television, through the TDOA algorithm based on the time of each microphone in the microphone array receiving the ultrasonic signal 1. The ultrasonic generator can be a loudspeaker, so that auxiliary equipment such as a television does not need to be additionally provided with an ultrasonic generator to adapt to the ultrasonic positioning method provided by the embodiment of the application. Similarly, the ultrasonic receiver used for receiving the ultrasonic signal in the main device such as the mobile phone can be a microphone of the mobile phone, so that the mobile phone does not need to additionally add an ultrasonic receiver to adapt to the ultrasonic positioning method provided by the embodiment of the application, and the implementation cost and difficulty of indoor positioning are reduced.

Similarly, after receiving the positioning command, the loudspeaker 1 may emit an ultrasonic signal 2 via an ultrasonic generator (e.g., a speaker) in response to the positioning command. The ultrasonic signals 2 can be received by a plurality of microphones in a microphone array in the mobile phone. Furthermore, the mobile phone can calculate the positioning result 2, that is, the position of the sound box 1, by using the TDOA algorithm based on the time when each microphone in the microphone array receives the ultrasonic signal 2.

Similarly, the loudspeaker 2, after receiving the above-mentioned positioning command, may emit an ultrasonic signal 3 via an ultrasonic generator (e.g. a loudspeaker) in response to the positioning command. The ultrasonic signals 3 can be received by a plurality of microphones in a microphone array in the mobile phone. Furthermore, the mobile phone can calculate the positioning result 3, that is, the position of the sound box 2, by using the TDOA algorithm based on the time when each microphone in the microphone array receives the ultrasonic signal 3.

Subsequently, as shown in fig. 142, the mobile phone may display a positioning interface 23201 according to the calculated positioning result 1, positioning result 2, and positioning result 3, where the positioning interface 23201 shows the positional relationship between the mobile phone and the television, and the position relationship between the sound box 1 and the sound box 2, so that the user can visually see the positional relationship between the mobile phone and each nearby device.

In other embodiments, the mobile phone is still used as the master device, and the television, the speaker 1 and the speaker 2 are used as the slave devices, and the master device performs positioning examples on the slave devices. Firstly, the mobile phone can select one slave device from the television, the sound box 1 and the sound box 2 as a cooperative device, and the cooperative device can cooperate with the mobile phone to complete the positioning process of other slave devices. For example, the mobile phone may use the device with stronger positioning capability among the television, the sound box 1 and the sound box 2 as the cooperative device. Taking a television as an example of the cooperative device, the mobile phone can first locate the television to obtain a relative position relationship between the mobile phone and the television, and then obtain a first locating result. For example, the handset may send a positioning instruction to the television, which may emit an ultrasonic signal 1 through an ultrasonic generator such as a speaker in response to the positioning instruction. The ultrasonic signals 1 can be received by a plurality of microphones in the microphone array in the mobile phone, and then the mobile phone can calculate the position relationship between the mobile phone and the television through the TDOA algorithm based on the time when each microphone in the microphone array receives the ultrasonic signals 1.

Furthermore, the mobile phone can send a positioning instruction to other slave devices, and the mobile phone can instruct the television to turn on the positioning function to position the other slave devices in cooperation with the mobile phone. For example, the mobile phone may send a positioning instruction to the sound box 1 and send a co-positioning instruction to the television. As shown in fig. 143, the sound box 1 can transmit an ultrasonic signal 2 in response to a positioning instruction sent by the mobile phone. The television can respond to the co-location instruction sent by the mobile phone to open the microphone array of the television, and the ultrasonic wave signals 2 are received by a plurality of microphones in the microphone array of the television. Also, the handset can receive the ultrasonic signal 2 through a plurality of microphones in its microphone array.

In one implementation, the television may send its own detected ultrasound data (e.g., the ultrasound data may include the waveform of the ultrasound signal 2 received by each microphone, the arrival time of the ultrasound signal 2, etc.) to the handset. In this way, the mobile phone can acquire not only the ultrasonic data (i.e., the first ultrasonic data) detected by the microphone array of the mobile phone, but also the ultrasonic data (i.e., the second ultrasonic data) detected by the microphone array of the television. After the mobile phone has determined the position relationship with the television, the mobile phone can use the microphone array in the television as a part of the own microphone array. Then, the mobile phone can use each microphone in the mobile phone and the television as a complete microphone array, and calculate the position of the sound box 1 according to the first ultrasonic data and the second ultrasonic data.

In another implementation, after each microphone of the microphone array in the television detects the ultrasonic signal 2, the television may calculate an ultrasonic positioning result (i.e., a first ultrasonic positioning result) for the sound box 1 according to the time when the ultrasonic signal 2 reaches each microphone. And the television can send the first ultrasonic positioning and positioning result to the mobile phone. After each microphone of the microphone array in the mobile phone detects the ultrasonic signal 2, a positioning result (second ultrasonic positioning result) of the sound box 1 can also be calculated according to the time of the ultrasonic signal 2 reaching each microphone. After the mobile phone receives the first ultrasonic positioning result sent by the television, the second ultrasonic positioning result calculated by the mobile phone is corrected according to the first ultrasonic positioning result, and finally the position of the sound box 1 is determined.

In the scene that the sound box 1 is positioned by the mobile phone and the television working together, the ultrasonic signals 2 transmitted by the sound box 1 can be received by the microphones in the mobile phone and the television 1, so that the number of the microphones for detecting the ultrasonic signals 2 is larger, and the accuracy of the positioning result calculated by the mobile phone according to the time of the ultrasonic signals 2 reaching each microphone is higher. In addition, the microphones for detecting the ultrasonic signals 2 come from different positions in the mobile phone and the television, so that the mobile phone can position the sound box 1 according to the ultrasonic signals 2 detected in multiple directions, and the positioning precision of the sound box 1 can also be improved.

In the above embodiment, the sound box 1 is located by the cooperation of the mobile phone and the television, and it can be understood that the mobile phone and the television can also locate the sound box 2 by the above method to obtain the location of the sound box 2. Of course, if the slave device of the mobile phone further includes other devices, the mobile phone may also position the other devices according to the above method, which is not limited in this embodiment of the present application.

Finally, similar to the positioning interface 401 shown in fig. 142, the mobile phone (i.e., the master device) may present the mobile phone and the position relationship between the slave devices to the user according to the positioning result of each slave device, so that the user can visually see the position relationship between the mobile phone and each device nearby.

It can be seen that, in the embodiment of the present application, the master device may utilize the existing ultrasonic positioning capability (for example, the capability of a speaker to transmit an ultrasonic signal or the capability of a microphone to receive an ultrasonic signal) in the electronic device, and simultaneously turn on the microphones of the multiple devices to receive the ultrasonic signal sent by the slave device in a multi-device cooperation manner, so as to position the slave device, thereby improving the positioning accuracy of the slave device, and simultaneously simplifying the operation process and implementation cost of indoor positioning.

Still taking the mobile phone as the main device for example, based on the Android system architecture shown in fig. 6, as shown in fig. 144, one or more audio function modules such as an AudioRecord, an AudioTrack, an AudioFlinger, and an ultrasonic processing module are arranged in an application framework layer of the mobile phone.

Illustratively, a handset may have the capability to receive ultrasonic signals. For example, the AudioRecord may obtain an ultrasonic signal recorded by a microphone of a mobile phone from the AudioFlinger. When the mobile phone is provided with a plurality of microphones (i.e. a microphone array), the AudioFlinger can receive a plurality of ultrasonic signals reported by the plurality of microphones. At this time, the ultrasonic processing module may combine the multiple ultrasonic signals in the AudioFlinger and send the combined signals to the AudioRecord, and the AudioRecord may report the recorded ultrasonic signals to a location application or other applications.

Illustratively, a handset may have the capability to transmit ultrasonic signals. For example, the AudioTrack may call the AudioFlinger in response to an instruction from an application such as a positioning application, and control a device such as a speaker of a mobile phone to transmit an ultrasonic signal through the AudioFlinger.

Also shown in fig. 144, HALs corresponding to different hardware modules of the handset are provided in the HALs of the handset, e.g., Audio HAL, Camera HAL, Wi-Fi HAL, etc.

Among them, the Audio HAL can correspond to an ultrasonic receiver such as a microphone by ultrasonic driving of the core layer. When the mobile phone is provided with a plurality of microphones, the plurality of microphones respectively correspond to the plurality of ultrasonic drives of the inner core layer. Each microphone may report the received ultrasonic signal to the Audio HAL via the corresponding ultrasonic drive. The Audio HAL may report the received multiple ultrasonic signals to the Audio flinger, which sends the received ultrasonic signals to the ultrasonic processing module.

For example, after the ultrasonic processing module receives N ultrasonic signals, the N ultrasonic signals may be combined into a set of ultrasonic data. The ultrasonic data may include information such as a waveform of each ultrasonic signal, and a time when each ultrasonic signal reaches a corresponding microphone. Further, the ultrasonic processing module may determine the location result of the device (e.g., slave device 1) transmitting the ultrasonic signal using the TOA algorithm or TDOA algorithm based on the ultrasonic data. Subsequently, the ultrasonic processing module can report the positioning result to the positioning application through the AudioRecord.

Or after the ultrasonic processing module combines the N ultrasonic signals into a group of ultrasonic data, the ultrasonic data can be reported to a positioning application through the AudioRecord. Furthermore, the positioning result of the device (e.g. slave device 1) emitting the ultrasonic signal may be determined by the positioning application according to the ultrasonic data, which is not limited in any way by the embodiment of the present application.

In the embodiment of the present application, as shown in fig. 145, the handset may create a DMSDP HAL corresponding to a slave device (e.g., slave device 1) in the HAL. DMSDP HAL corresponds to the slave device 1 to which the handset is currently connected. The mobile phone can be used as a master device to transmit and receive data with the slave device 1 through the DMSDP HAL, and the slave device 1 is used as a virtual device of the mobile phone to cooperate with the slave device 1 to complete various services in a distributed scenario.

For example, when the positional relationship of the mobile phone and the slave device 1 is fixed, if the mobile phone needs to locate another slave device (for example, the slave device 2), the mobile phone may transmit a location instruction to the slave device 2 so that the slave device 2 transmits an ultrasonic wave signal using an ultrasonic wave generator such as a speaker in response to the location instruction. Meanwhile, the mobile phone may send a co-location instruction to the slave device 1, so that the slave device 1 opens its own microphone array to receive the ultrasonic signal in response to the co-location instruction.

In one aspect, as also shown in fig. 145, N microphones in the handset may be used to receive the ultrasonic signals transmitted from the device 2, i.e., N ultrasonic signals (i.e., the first ultrasonic data) may be acquired by the handset. The N microphones can report the acquired ultrasonic signals to the Audio HAL respectively, so that the Audio HAL can acquire N paths of ultrasonic signals. The Audio HAL may report the acquired N ultrasonic signals to the ultrasonic processing module through the Audio flinger.

On the other hand, as also shown in fig. 145, after the slave device 1 opens its own microphone array, M microphones in the microphone array may also be used to receive the ultrasonic signals transmitted from the slave device 2, that is, M ultrasonic signals (i.e., the second ultrasonic data) may be acquired from the slave device 1. Then, the slave device 1 may send the acquired M ultrasonic signals to the AudioFlinger of the mobile phone through the DMSDP HAL in the mobile phone, and the AudioFlinger reports the M ultrasonic signals to the ultrasonic processing module.

In this way, the ultrasonic processing module of the mobile phone can acquire N ultrasonic signals (i.e. first ultrasonic data) acquired by the microphone array in the mobile phone, and also can acquire M ultrasonic signals (i.e. second ultrasonic data) acquired by the microphone array in the device 1. Subsequently, the ultrasonic processing module may combine the N ultrasonic signals and the M ultrasonic signals into a set of multi-channel ultrasonic data. The multi-channel ultrasound data may include: the waveform of each ultrasonic signal in the N ultrasonic signals and the time of each ultrasonic signal reaching the corresponding microphone, and the waveform of each ultrasonic signal in the M ultrasonic signals and the time of each ultrasonic signal reaching the corresponding microphone. Furthermore, the ultrasonic processing module may report the multi-channel ultrasonic data to a positioning application, which calculates the position of the slave device 2 using a preset positioning algorithm (e.g., TOA algorithm or TDOA algorithm) according to the multi-channel ultrasonic data.

Or, the ultrasonic processing module may directly calculate the position of the slave device 2 according to the multi-channel ultrasonic data by using a preset positioning algorithm, and report the position of the slave device 2 to the positioning application, which is not limited in this embodiment of the present application.

It can be seen that the handset can take advantage of the location capability of the slave device 1 when locating the slave device 2 by simultaneously turning on the handset and receiving the ultrasonic signal emitted from the device 2 from the microphone of the slave device 1 in a coordinated manner with the slave device 1. In this way, in a positioning scene, the number of microphones for acquiring ultrasonic signals is increased, and the microphones for acquiring ultrasonic signals are located at different positions of the mobile phone and the slave device 1, so that the mobile phone can position the slave device 2 according to more ultrasonic signals and more directions, and the positioning accuracy of the device is improved.

Meanwhile, the positioning process utilizes the existing ultrasonic positioning capability (for example, the capability of a loudspeaker for transmitting ultrasonic signals or the capability of a microphone for receiving ultrasonic signals) in the mobile phone and the slave equipment (for example, the slave equipment 1 and the slave equipment 2), and the hardware structures in the mobile phone and the slave equipment do not need to be changed, so that the operation process of indoor positioning of a user can be simplified, and the realization cost of the indoor positioning can be reduced.

Still take a mobile phone as an example of a main device, an ultrasonic positioning method provided by the embodiments of the present application will be specifically described below with reference to the accompanying drawings.

In some embodiments, as shown in fig. 146, the cell phone may have a screen shot button 23602 located at a location such as a drop down menu 23601. When the user desires to use the screen projection function of the cellular phone, the screen projection button 23602 may be clicked. If the user is detected to click the screen projection button 23602, the mobile phone needs to present the nearby electronic devices to the user, so that the user can select a specific screen projection device to trigger the mobile phone to project the content in the mobile phone to the screen projection device for playing. In this embodiment of the present application, after it is detected that the user clicks the screen-projecting button 23602, the mobile phone may use the ultrasonic positioning method provided in this embodiment of the present application to position the electronic device near the mobile phone, and present the electronic device near the mobile phone to the user according to the positioning result, so that the user may select a corresponding screen-projecting device.

Illustratively, upon detecting that the user has clicked the screen-shot button 23602, the handset may search for one or more electronic devices that are currently accessing the same communication network as the handset. For example, the cell phone may query one or more electronic devices that are logged into the same account as the cell phone through the server. For example, the electronic devices currently logged in to the same account as the mobile phone include the television 1, the speaker 2, the speaker 3, and the air purifier 4. Further, the mobile phone can acquire the positioning capabilities of the television 1, the speaker 2, the speaker 3, and the air cleaner 4. For example, the server may store therein the positioning capabilities of the television 1, the loudspeaker 2, the loudspeaker 3 and the air purifier 4. The mobile phone can acquire that the television 1, the sound box 2 and the sound box 3 have positioning capability from the server, and the air purifier 4 does not have positioning capability. Wherein, the positioning capability of the television 1 is higher than that of the sound box 2 and the sound box 3. Then, the mobile phone can use the television 1, the sound box 2 and the sound box 3 with the positioning capability as slave devices of the mobile phone in the positioning process to position the television 1, the sound box 2 and the sound box 3.

For example, when the number of microphones provided on a certain electronic device is larger, the positioning capability of the electronic device can be set to be higher. Alternatively, when the distance between microphones provided on a certain electronic device is larger, the positioning capability of the electronic device may be set to be higher. Or, when the performance of a hardware parameter such as a processor of a certain electronic device is higher, the positioning capability of the electronic device may be set to be higher, which is not limited in this embodiment of the present application.

In some embodiments, the handset may select the tv 1 with the highest location capability among the tv 1, the speaker 2, and the speaker 3 as the cooperating device. After determining that the television 1 is a cooperative device, the mobile phone may first locate the television 1, and then the mobile phone may cooperate with the television 1 to complete a process of locating other slave devices (e.g., the speaker 2 and the speaker 3).

Illustratively, as shown in fig. 147, the architecture of the operating system in the television 1 is similar to that in the cellular phone. The application layer of the television 1 may have a proxy application installed therein, and the proxy application is used for data transmission and reception with other devices (e.g., a mobile phone). The application framework layer of the television 1 is provided with audio function modules such as AudioTrack, AudioFlinger, and AudioRecord (not shown in the figure). An Audio HAL is provided in the HAL of the television 1, and the Audio HAL corresponds to ultrasonic drive of the core layer of the television 1. The ultrasonic drive of the television 1 can drive an ultrasonic generator such as a speaker in the hardware layer to emit an ultrasonic signal. Alternatively, the ultrasonic drive of the television 1 may also drive a microphone array (not shown in fig. 147) of the television 1 to acquire an ultrasonic signal.

Still as shown in fig. 147, after the mobile phone determines the television 1 as a cooperative device, the mobile phone may send a positioning instruction 1 to the proxy application in the television 1 through the positioning application, where the positioning instruction 1 is used to trigger the television 1 to transmit an ultrasonic signal. After receiving the location instruction 1, the agent application of the television 1 may invoke AudioTrack in response to the location instruction 1, and send an ultrasonic transmission instruction to the audiohal through the AudioFlinger by the AudioTrack. The Audio HAL may send a drive signal to the ultrasonic driver in response to the ultrasonic emission instruction so that the ultrasonic driver may drive the speaker of the television 1 to emit an ultrasonic signal in response to the drive signal.

As also shown in fig. 147, the microphone array on the handset can capture the ultrasonic signals emitted by the speakers of the television 1. For example, a microphone array of the mobile phone includes N microphones, and each of the N microphones can drive the acquired ultrasonic signal to report to the Audio HAL of the mobile phone through a corresponding ultrasonic wave. Then, the Audio HAL of the mobile phone can receive N ultrasonic signals, such as ultrasonic signal 1, ultrasonic signal 2, etc. The Audio HAL of the mobile phone can report the N ultrasonic signals to the ultrasonic processing module of the mobile phone through the Audio flinger. As shown in fig. 148, since the N microphones of the mobile phone are disposed at different positions of the mobile phone, the waveforms of the ultrasonic signals collected by the different microphones may be different, and the times at which the ultrasonic signals are collected by the different microphones may also be different. The ultrasonic processing module can combine the received N paths of ultrasonic signals into multi-channel ultrasonic data, wherein N sound channels in the multi-channel ultrasonic data correspond to the N paths of ultrasonic signals one by one, and the acquisition time of each ultrasonic signal, namely the time of the ultrasonic signal reaching the corresponding microphone, is recorded in the multi-channel ultrasonic data. Furthermore, the ultrasonic processing module of the mobile phone can position the television 1 by using a preset positioning algorithm according to the multi-channel ultrasonic data to obtain a positioning result 1 of the television 1.

For example, the ultrasonic processing module of the mobile phone may use the TOA algorithm or the TDOA algorithm to locate the television 1 according to the time when the ultrasonic signal reaches the corresponding microphone. Of course, a person skilled in the art may also set the ultrasonic processing module to use other positioning algorithms to position the television 1, and the embodiment of the present application does not limit this.

For example, the positioning result 1 calculated by the ultrasonic processing module for the television 1 may be an absolute position coordinate of the television 1 in a preset coordinate system, or may be a relative position relationship between the television 1 and a mobile phone, which is not limited in this embodiment of the present application.

For example, with the mobile phone as the coordinate origin O (0, 0), the above positioning result 1 may include the position coordinate a1(X1, Y1) of the tv 1 in the coordinate system. At this time, the position coordinates a1(X1, Y1) may indicate the relative position between the television 1 and the cellular phone. The ultrasonic processing module of the mobile phone can report the positioning result 1 to the positioning application through the AudioRecord, so that the positioning application can acquire the position coordinate of the television 1. Or, the ultrasonic processing module of the mobile phone may also directly report the multi-channel ultrasonic data to the positioning application through the AudioRecord, and the positioning application calculates the positioning result 1 of the television 1 according to the multi-channel ultrasonic data, which is not limited in this embodiment of the present application.

At this time, as shown in fig. 149, after the positioning application of the mobile phone obtains the positioning result 1 of the television 1, the position relationship between the television 1 and the mobile phone may be displayed in the form of characters, animation, icons, or the like in the interactive interface 23901 according to the positioning result 1. For example, the television 1 is located 5 meters directly in front of the handset. Since the mobile phone has not obtained the positioning results of other slave devices such as the sound box 2 and the sound box 3, the positioning application may display the positioning prompt information 23902 in the interactive interface 23901, and prompt the user to wait for the mobile phone to position other slave devices.

For example, after the positioning application of the mobile phone obtains the positioning result 1 of the television 1, the mobile phone may complete the positioning process of the sound box 2, the sound box 3, and other slave devices by cooperating with the television 1 by using the positioning capability of the television 1.

For example, by using the mobile phone and the television 1 to cooperate to locate the sound box 2, as shown in fig. 150, the architecture of the operating system in the sound box 2 is similar to that of the operating system in the mobile phone (or the television 1). The application layer of the sound box 2 may have a proxy application installed therein, and the proxy application is used for data transceiving with other devices (e.g., the mobile phone and the television 1). Audio functional modules such as AudioRecord (not shown in the figure), AudioTrack, and AudioFlinger are disposed in the application framework layer of the sound box 2. An Audio HAL is provided in the HAL of the cabinet 2, and corresponds to ultrasonic driving of the core layer of the cabinet 2. The ultrasonic drive can drive an ultrasonic generator such as a loudspeaker in the hardware layer to emit an ultrasonic signal.

For example, as shown in fig. 150, after the positioning application of the mobile phone obtains the positioning result 1 of the television 1, the positioning application of the mobile phone may send a positioning instruction 2 to the proxy application of the sound box 2, where the positioning instruction 2 is used to trigger the sound box 2 to transmit an ultrasonic signal. Meanwhile, the positioning application of the mobile phone can send a co-positioning instruction to the agent application of the television 1, and the co-positioning instruction is used for triggering the television 1 to detect the ultrasonic signals transmitted by the sound box 2 by using the M ultrasonic receivers of the television 1.

Similar to the process of the television 1 responding to the positioning instruction 1, after receiving the positioning instruction 2, the agent application of the sound box 2 can respond to the positioning instruction 2 to call the AudioTrack, and the AudioTrack sends an ultrasonic wave transmitting instruction to the audiohal through the AudioFlinger. The Audio HAL may send a driving signal to the ultrasonic driver in response to the ultrasonic emission instruction so that the ultrasonic driver may drive the speaker of the cabinet 2 to emit an ultrasonic signal in response to the driving signal.

On the other hand, as shown in fig. 150, after the agent application of the television 1 receives the co-location instruction sent by the mobile phone, the agent application of the television 1 may open its own microphone array to receive the ultrasonic signal in response to the co-location instruction. The ultrasonic signal received by the television 1 is the ultrasonic signal emitted by the speaker of the speaker box 2. For example, the microphone array of the television 1 includes M microphones, and each of the M microphones can drive the acquired ultrasonic signal to report to the Audio HAL of the television 1 through a corresponding ultrasonic wave. Then, the Audio HAL of the television 1 can receive M ultrasonic signals, such as ultrasonic signal 1, ultrasonic signal 2, and so on. The Audio HAL of the television 1 can report the M ultrasonic signals to the Audio flinger of the television 1, and then the Audio record of the television 1 records the M ultrasonic signals from the Audio flinger, and reports the M ultrasonic signals to the agent application of the television 1. The proxy application of the television 1 can send the M ultrasonic signals acquired by the proxy application to the DMSDP HAL of the mobile phone, and the DMSDP HAL of the mobile phone reports the M ultrasonic signals acquired by the television 1 to the ultrasonic processing module of the mobile phone through the AudioFlinger. That is, when the sound box 2 is located by using ultrasonic waves, the ultrasonic processing module of the mobile phone may acquire the ultrasonic signals transmitted by the sound box 2 and collected by the cooperating device (e.g., the television 1).

Meanwhile, as also shown in fig. 150, the microphone array on the mobile phone can also collect the ultrasonic signals emitted by the speakers of the sound box 2. For example, after the sound box 2 emits the ultrasonic signals, the N microphones on the mobile phone may drive the collected ultrasonic signals to the Audio HAL of the mobile phone through the corresponding ultrasonic waves. Then, the Audio HAL of the mobile phone can receive N ultrasonic signals, such as ultrasonic signal 1, ultrasonic signal 2, etc. The Audio HAL of the mobile phone can report the N ultrasonic signals to the ultrasonic processing module of the mobile phone through the Audio flinger.

Thus, when the mobile phone performs ultrasonic positioning on the sound box 2, the ultrasonic processing module of the mobile phone can acquire N ultrasonic signals (i.e., first ultrasonic data) acquired by the microphone array in the mobile phone, and can also acquire M ultrasonic signals (i.e., second ultrasonic data) acquired by the microphone array in the television 1. Furthermore, the ultrasonic processing module of the mobile phone can perform ultrasonic positioning on the sound box 2 according to the first ultrasonic data and the second ultrasonic data. For example, the ultrasonic processing module of the mobile phone may combine the N ultrasonic signals and the M ultrasonic signals into a set of multi-channel ultrasonic data. The multi-channel ultrasound data may include: the waveform of each ultrasonic signal in the N ultrasonic signals and the time of each ultrasonic signal reaching the corresponding microphone, and the waveform of each ultrasonic signal in the M ultrasonic signals and the time of each ultrasonic signal reaching the corresponding microphone. Furthermore, the ultrasonic processing module of the mobile phone can perform ultrasonic positioning on the sound box 2 based on the multi-channel ultrasonic data.

In some embodiments, the sampling rate 1 when the microphone array in the mobile phone acquires the N ultrasonic signals may be the same as the sampling rate when the microphone array in the television 1 acquires the M ultrasonic signals. For example, when the mobile phone sends the co-location instruction to the television 1, the co-location instruction may carry a preset sampling rate a. In this way, the television 1 can acquire the M ultrasonic signals by using its own microphone array according to the sampling rate a carried in the co-location instruction, and meanwhile, the mobile phone can also acquire the N ultrasonic signals by using its own microphone array according to the sampling rate a. Therefore, the ultrasonic processing module of the mobile phone can combine the M paths of ultrasonic signals and the N paths of ultrasonic signals with the same sampling rate to obtain corresponding multi-channel ultrasonic data.

Since the mobile phone has acquired the positioning result 1 of the television 1, that is, the positional relationship between the mobile phone and the television 1 is determined, then, the M microphones on the television 1 can also be used as the microphones of the mobile phone, which is equivalent to that a part of the microphones (for example, the N microphones) of the mobile phone are disposed on the local mobile phone, and another part of the microphones (for example, the M microphones) of the mobile phone are disposed on the television 1. Furthermore, the ultrasonic processing module of the mobile phone can position the sound box 2 emitting the ultrasonic signal according to the positioning result 1 of the television 1 (i.e. the positions of the M microphones) and the time when the ultrasonic signal in the multi-channel ultrasonic data respectively reaches the M microphones and the N microphones, so as to obtain the positioning result 2 of the sound box 2.

Similarly, the positioning result 2 calculated by the ultrasonic processing module for the sound box 2 may be an absolute position coordinate of the sound box 2 in a preset coordinate system, or a relative position relationship between the sound box 2 and the mobile phone, which is not limited in this embodiment of the present application.

For example, still using the mobile phone as the coordinate origin O (0, 0), the positioning result 2 may include the position coordinate a2(X2, Y2) of the sound box 2 in the coordinate system. At this time, the position coordinates a2(X2, Y2) may indicate the relative position between the sound box 2 and the cellular phone. The ultrasonic processing module of the mobile phone can report the positioning result 2 to the positioning application through the AudioRecord, so that the positioning application can acquire the position coordinate of the sound box 2. Or, the ultrasonic processing module of the mobile phone may also directly report the multi-channel ultrasonic data to the positioning application through the AudioRecord, and the positioning application calculates the positioning result 2 of the sound box 2 according to the multi-channel ultrasonic data, which is not limited in this embodiment of the present application.

At this time, as shown in fig. 151, after the positioning application of the mobile phone obtains the positioning result 2 of the sound box 2, the positioning application may display the position relationship between the sound box 2 and the nearby device such as the mobile phone in the form of characters, animation, or icons on the interactive interface 24101 according to the positioning result 2. For example, the speaker 2 is located on the left side of the television 1 and on the front left side of the handset.

It can be seen that, when the mobile phone locates the sound box 2, the mobile phone can utilize the locating capability of the located television 1 to simultaneously turn on the mobile phone and the microphone of the television 1 in a manner of cooperating with the television 1 to receive the ultrasonic signal sent by the sound box 2. Therefore, in a positioning scene, the number of microphones for acquiring ultrasonic signals is increased, the orientation angles of the microphones for acquiring the ultrasonic signals are more, and the mobile phone can position the sound box 2 according to the ultrasonic signals in more numbers and directions, so that the positioning accuracy of the equipment is improved.

Similarly, the mobile phone may cooperate with the television 1 to locate the sound box 3 according to the above method, so as to obtain the location result 3 of the sound box 3. The positioning result 3 may include position coordinates a3(X3, Y3) of the sound box 3 in the above coordinate system. Further, as shown in fig. 152, after the positioning application of the mobile phone obtains the positioning result 3 of the sound box 3, the position relationship between the sound box 3 and the nearby device such as the mobile phone may be displayed in the form of characters, animation, or icons on the interactive interface 24201 according to the positioning result 3. For example, the speaker 3 is located on the right side of the television and on the front right of the mobile phone. If the slave device of the mobile phone does not have any other slave device needing positioning besides the television 1, the sound box 2 and the sound box 3, the positioning application of the mobile phone can prompt the user that the searching and positioning of the nearby devices are finished in the interactive interface 24201, and the user can select the corresponding device to screen.

Of course, the positioning application of the mobile phone may also display the position relationship between the mobile phone and each slave device in the interface after acquiring the positioning results of all the slave devices, such as the television 1, the sound box 2, the sound box 3, and the like, which is not limited in this embodiment of the present application.

In some embodiments, the handset may simultaneously locate speakers 2 and 3 in conjunction with television 1. For example, the mobile phone may send a positioning instruction to the sound box 2 and the sound box 3 at the same time, and trigger the sound box 2 and the sound box 3 to start transmitting the ultrasonic signal. Different from the above, the sound box 2 and the sound box 3 may carry their own identifiers in the transmitted ultrasonic signal, for example, the ultrasonic signal 1 sent by the sound box 2 includes the identifier of the sound box 2, and the ultrasonic signal 2 sent by the sound box 3 includes the identifier of the sound box 3. Thus, the mobile phone can identify the ultrasonic signal 1 from the sound box 2 and the ultrasonic signal 2 from the sound box 3 according to the identification carried in the acquired ultrasonic signal.

In other embodiments, the mobile phone may cooperate with the tv 1 to locate the speaker 2, and cooperate with the tv 1 to locate the speaker 3. Or, the mobile phone may cooperate with the television 1 to locate the sound box 3, and then cooperate with the television 1 to locate the sound box 2 until the current location results of all the slave devices are obtained.

In other embodiments, the handset may coordinate multiple slave devices to locate other slave devices that are not located. For example, the mobile phone can obtain the positioning result 1 of the television 1 after positioning the television 1 according to the method, and can obtain the positioning result 2 of the sound box 2 after the mobile phone cooperates with the television 1 to position the sound box 2 according to the method. Furthermore, the mobile phone can use both the television 1 and the sound box 2 as cooperative devices to continue to locate the sound box 3.

For example, while the mobile phone sends the positioning instruction 3 to the sound box 3, it may also send a cooperative positioning instruction to the television 1 and the sound box 2 (i.e., two cooperative devices). Furthermore, similar to the method in the above embodiment in which the television 1 is used as a cooperative device to cooperate with a mobile phone to locate other slave devices, the sound box 3 may transmit an ultrasonic signal using a speaker in response to the location instruction 3, and the television 1 and the sound box 2 may open their own microphone arrays in response to the cooperative location instruction to collect the ultrasonic signal and send the collected multiple paths of ultrasonic signals to the mobile phone. Meanwhile, the mobile phone can open a microphone array of the mobile phone to acquire ultrasonic signals.

For example, the mobile phone may acquire N ultrasonic signals (i.e., first ultrasonic data) from the mobile phone itself, M ultrasonic signals (i.e., second ultrasonic data) from the television 1, and Z ultrasonic signals (i.e., third ultrasonic data) from the sound box 2 when positioning the sound box 3, where the microphone array of the mobile phone includes N microphones, the microphone array of the television 1 includes M microphones, and the microphone array of the sound box 2 includes Z microphones. Furthermore, the mobile phone can combine the N-channel ultrasonic signal, the M-channel ultrasonic signal, and the Z-channel ultrasonic signal into corresponding multi-channel ultrasonic data, so that the ultrasonic processing module (or positioning application) of the mobile phone can calculate the positioning result 3 of the sound box 3 according to the multi-channel ultrasonic data.

Therefore, when the mobile phone locates the sound box 3, the mobile phone can simultaneously utilize the locating capabilities of the located television 1 and sound box 2, and simultaneously turn on the microphones of the mobile phone, the television 1 and the sound box 2 to receive the ultrasonic signal sent by the sound box 3 in a multi-device cooperation mode. Therefore, in a positioning scene, the number of microphones for acquiring the ultrasonic signals is larger, the orientation angles of the microphones for acquiring the ultrasonic signals are also larger, and the mobile phone can position the sound box 3 according to the ultrasonic signals in more numbers and directions, so that the positioning accuracy of the equipment is improved.

Of course, if the slave device of the mobile phone further includes a device that is not located by another device, for example, the sound box 4, the mobile phone may add a new cooperative device according to the above method to continue to locate the other device that is not located, which is not limited in this embodiment of the present application.

In addition, after the mobile phone acquires the N paths of ultrasonic signals, audio processing operations such as sound effect setting, sound mixing and the like do not need to be performed on the N paths of ultrasonic signals, so that the N paths of ultrasonic signals acquired by the mobile phone are ensured to be relatively original ultrasonic signals transmitted by slave equipment, and the positioning accuracy of a subsequent positioning process is improved. Similarly, after the M-channel ultrasonic signals are acquired by the cooperative device of the mobile phone (for example, the television 1), audio processing operations such as sound effect setting and sound mixing are not required to be performed on the M-channel ultrasonic signals, so that the M-channel ultrasonic signals acquired by the mobile phone are relatively original ultrasonic signals transmitted from the device, and the positioning accuracy of the subsequent positioning process is improved.

In other embodiments, for example, the sound box 2 is still located by the cooperation of the mobile phone and the television 1, as shown in fig. 153, a location application may also be installed in the application program layer of the television 1, and an ultrasound processing module may also be included in the application program framework layer of the television 1.

When the sound box 2 is positioned, the N microphones in the mobile phone can report the collected ultrasonic signals to the Audio HAL of the mobile phone through the corresponding ultrasonic drive, so that the Audio HAL of the mobile phone can receive N ultrasonic signals such as ultrasonic signal 1 and ultrasonic signal 2. The Audio HAL of the handset may report the N ultrasonic signals to the Audio flinger of the handset. At this time, the ultrasonic processing module of the mobile phone may obtain the N ultrasonic signals from the AudioFlinger, and further, the ultrasonic processing module of the mobile phone may combine the N ultrasonic signals into corresponding multi-channel ultrasonic data 1, where the multi-channel ultrasonic data 1 may include: the waveform of each ultrasonic signal in the N ultrasonic signals and the time for each ultrasonic signal to reach the corresponding microphone. Furthermore, the ultrasonic processing module or the positioning application of the mobile phone can position the sound box 2 according to the multi-channel ultrasonic data 1 to obtain a positioning result 2 of the sound box 2.

Similarly, when the sound box 2 is located, M microphones in the television 1 may drive and report the acquired ultrasonic signals to the Audio HAL of the television 1 through corresponding ultrasonic waves, so that the Audio HAL of the television 1 may receive M ultrasonic signals such as the ultrasonic signal 1 and the ultrasonic signal 2. The Audio HAL of television 1 may report the M ultrasonic signals to the Audio flinger of television 1. At this time, the ultrasonic processing module of the television 1 may acquire the M ultrasonic signals from the AudioFlinger, and further, the ultrasonic processing module of the television 1 may combine the M ultrasonic signals into corresponding multi-channel ultrasonic data 2, where the multi-channel ultrasonic data 2 may include: the waveform of each ultrasonic signal in the M ultrasonic signals and the time for each ultrasonic signal to reach the corresponding microphone. Then, the ultrasonic processing module or the positioning application of the tv 1 can position the sound box 2 according to the above-mentioned multi-channel ultrasonic data 2, and at this time, the tv 1 can also obtain the positioning result 2' of the sound box 2.

Furthermore, the positioning application in the television 1 may send the calculated positioning result 2 'of the sound box 2 to the DMSDP HAL of the mobile phone, and the DMSDP HAL of the mobile phone reports the positioning result 2' to the ultrasonic processing module of the mobile phone. In this way, the ultrasonic processing module of the mobile phone can obtain the positioning result 2 calculated when the mobile phone positions the sound box 2, and can also obtain the positioning result 2' calculated when the television 1 positions the sound box 2. Furthermore, the ultrasonic processing module of the mobile phone can correct the self-calculated positioning result 2 according to the positioning result 2' to obtain the corrected positioning result 2 of the sound box 2, so as to improve the positioning accuracy of the sound box 2.

For example, the positioning result 2 calculated when the mobile phone positions the sound box 2 includes the position coordinate a21 of the sound box 2, and the positioning result 2' calculated when the television 1 positions the sound box 2 includes the position coordinate a22 of the sound box 2. Then, the ultrasonic processing module of the mobile phone can calculate the position coordinate a2 of the sound box 2, i.e. the final positioning result of the sound box 2, by using a preset correction algorithm according to the position coordinate a21 and the position coordinate a 22. For example, the ultrasonic processing module of the mobile phone may perform weighted average on the position coordinate a21 and the position coordinate a22 to obtain the position coordinate a2 of the sound box 2, which is not limited in this embodiment.

Of course, the mobile phone may further continue to locate other slave devices (e.g., the sound box 3) of the mobile phone according to the above method, which is not limited in this embodiment of the present application.

Still taking the slave devices of the mobile phone including the television 1, the sound box 2, and the sound box 3 as an example, as shown in fig. 154, after the mobile phone displays the positional relationship between the mobile phone and the television 1, the sound box 2, and the sound box 3 in the interface 24201, the user can conveniently and quickly complete the content projection operation across the devices according to the positional relationship of each slave device displayed in the interface 24201.

For example, if it is detected that the identifier 24401 of the mobile phone in the user drag interface 24201 moves to the located position of the television 1, which indicates that the user needs to project the content played in the mobile phone into the television 1 for playing, the mobile phone may start to perform a screen projection operation on the television 1. For example, the mobile phone may start to perform screen projection operation on the television 1 through Miracast protocol or DLNA (DIGITAL LIVING NETWORK ALLIANCE ) protocol, which is not limited in this embodiment of the present application.

For another example, if it is detected that the user drags the identifier 24401 of the mobile phone in the interface 24201 to move to the located position of the sound box 3, which indicates that the user needs to project the audio content in the mobile phone to the sound box 3 for playing, the mobile phone may project the audio content played by the mobile phone to the sound box 3 for playing through the bluetooth protocol.

In the above embodiment, the example that the mobile phone is triggered by the user in a scene of screen projection or music projection to position the slave device is illustrated, it can be understood that a person skilled in the art may also set that the ultrasonic positioning method is used in other application scenes, and the embodiment of the present application does not limit this.

For example, after the user projects the video played in the mobile phone to the television 1 for playing, the mobile phone may automatically start to locate the searched slave device according to the ultrasonic locating method provided in the above embodiment. As shown in fig. 155, if the mobile phone detects that the left and right sides of the television 1 are respectively provided with speakers, such as speaker 2 and speaker 3, the mobile phone may prompt the user to set a binaural stereo effect in an interface 245801. For example, if it is detected that the user clicks the determination button 24502 in the interface 245801, the mobile phone may send the left channel audio data projected to the tv 1 to the sound box 2 for playing, and send the right channel audio data projected to the tv 1 to the sound box 3 for playing, so as to present the audio playing effect of two-channel stereo to the user during the screen projection process.

In other embodiments, the positioning method described in the above embodiments can be applied to other sound wave signals, such as 20Hz to 20000Hz sound signals that can be recognized by human ears, in addition to positioning the electronic device by ultrasonic signals according to the positioning method described in the above embodiments.

For example, in a meeting scene, an electronic device having the above positioning function may be provided at the position of each participating user. As shown in fig. 156, when a participant user speaks, each electronic device may open its own microphone array to collect a sound signal emitted by the participant user when speaking, and further determine a position relationship between the speaking participant user and other participant users according to the above positioning method. At this time, the electronic device may also display the position relationship between the participating user speaking at this time and other participating users in the interface, so as to prompt the participating users of the position of the user speaking in the current conference.

It should be noted that, in the embodiment, a mobile phone is taken as an example of the main device in the positioning scene, and it is understood that the main device in the positioning scene may also be an electronic device with the positioning function, such as a tablet computer and a television, and the embodiment of the present application does not limit this.

It should be noted that, in the above embodiment, a mobile phone is taken as an example of the main device in the distributed audio scene, and it may be understood that the main device in the distributed audio scene may also be an electronic device with an audio function, such as a tablet computer and a television, and the embodiment of the present application does not limit this.

In addition, in the embodiment, a specific method for implementing audio control among the functional modules is described by taking an Android system as an example, and it can be understood that corresponding functional modules may also be set in other operating systems to implement the audio control method. As long as the functions implemented by the respective devices and functional modules are similar to the embodiments of the present application, they are within the scope of the claims of the present application and their equivalents.

As shown in fig. 157, an embodiment of the present application discloses an electronic device, which may be the above-mentioned main device (e.g., a mobile phone). The electronic device may specifically include: a touch screen 24701, the touch screen 24701 including a touch sensor 24706 and a display screen 24707; one or more processors 24702; a memory 24703; a communication module 24708; one or more application programs (not shown); and one or more computer programs 24704, which can be connected by one or more communication buses 24705. Wherein the one or more computer programs 24704 are stored in the memory 24703 and configured to be executed by the one or more processors 24702, the one or more computer programs 24704 comprising instructions that can be used to perform the steps associated with the master device in the embodiments described above.

As shown in fig. 158, an embodiment of the present application discloses an electronic device, which may be the slave device (e.g., a sound box, a television, etc.) described above. The electronic device may specifically include: one or more processors 24802; a memory 24803; a communication module 24806; one or more application programs (not shown); and one or more computer programs 24804, which can be connected by one or more communication buses 24805. Of course, a device such as a touch screen may also be provided in the slave device, which is not limited in this embodiment. Wherein the one or more computer programs 24804 are stored in the memory 24803 and configured to be executed by the one or more processors 24802, the one or more computer programs 24804 include instructions that can be used to perform the steps associated with performing a slave device in the embodiments described above.

Through the above description of the embodiments, it is clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions. For the specific working processes of the system, the apparatus and the unit described above, reference may be made to the corresponding processes in the foregoing method embodiments, and details are not described here again.

Each functional unit in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially implemented or make a contribution to the prior art, or all or part of the technical solutions may be implemented in the form of a software product stored in a storage medium and including several instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: flash memory, removable hard drive, read only memory, random access memory, magnetic or optical disk, and the like.

The above description is only a specific implementation of the embodiments of the present application, but the scope of the embodiments of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the embodiments of the present application should be covered by the scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims (179)

1. An audio control method, comprising:

responding to a first operation input by a user, displaying a candidate device list by a first device, wherein the candidate device list comprises at least one candidate device with an audio output function;

in response to an operation that a user selects a second device in the candidate device list, the first device acquires an audio capability parameter of the second device, wherein the audio capability parameter is used for indicating hardware capability of the second device for playing audio and software capability of the second device for playing audio; the second equipment is a sound box, a television or a mobile phone;

the first equipment determines a first audio playing strategy according to the audio capacity parameter of the second equipment;

the first equipment carries out first processing on first audio data from a first audio application according to the first audio playing strategy to obtain second audio data;

And the first equipment sends the second audio data to the second equipment so that the second equipment processes the second audio data and then plays the second audio data, or so that the second equipment plays the second audio data.

2. The method of claim 1, wherein the audio capability parameter comprises an audio effect parameter indicating whether the second device is in audio effect mode;

wherein, the first device determines a first audio playing strategy according to the audio capability parameter of the second device, and the method comprises the following steps:

if the second device starts a sound effect mode, the first device sets the first audio playing strategy to carry out no sound effect processing on the first audio data;

and if the second equipment does not start a sound effect mode, the first equipment sets sound effect processing on the first audio data in the first audio playing strategy.

3. The method of claim 1, wherein the audio capability parameter comprises an audio effect parameter indicating an audio effect mode supported by the second device; the first audio application sets a sound effect mode of audio playing to be a first sound effect mode;

Wherein, the first device determines a first audio playing strategy according to the audio capability parameter of the second device, and the method comprises the following steps:

if the first sound effect mode is the sound effect mode supported by the second equipment, the first equipment sets that sound effect processing is not carried out on the first audio data in the first audio playing strategy;

and if the first sound effect mode is not the sound effect mode supported by the second equipment, the first equipment sets the first audio data in the first audio playing strategy to carry out sound effect processing according to the first sound effect mode.

4. The method of claim 1, wherein the audio capability parameter comprises a target sampling rate, the target sampling rate being a sampling rate supported by the second device;

wherein, the first device determines a first audio playing strategy according to the audio capability parameter of the second device, and the method comprises the following steps:

and the first device sets the first audio data to be sampled according to the target sampling rate in the first audio playing strategy.

5. The method of claim 1, wherein the audio capability parameter comprises a target number of channels, the target number of channels being a number of channels supported by the second device;

Wherein, the first device determines a first audio playing strategy according to the audio capability parameter of the second device, and the method comprises the following steps:

and the first equipment sets the first audio data to be mixed according to the target channel number in the first audio playing strategy.

6. The method of claim 1, wherein the audio capability parameter comprises a latency parameter indicating whether the second device supports low latency audio processing capabilities;

wherein, the first device determines a first audio playing strategy according to the audio capability parameter of the second device, and the method comprises the following steps:

if the second device supports the audio processing capability with low time delay, the first device sets the first audio data to be processed according to a low time delay mode in the first audio playing strategy;

and if the second equipment does not support the audio processing capacity with low time delay, the first equipment sets the first audio data to be processed according to a non-low time delay mode in the first audio playing strategy.

7. The method of claim 1, wherein the audio capability parameter comprises a latency parameter indicating audio playback latency of the second device;

Wherein, the first device determines a first audio playing strategy according to the audio capability parameter of the second device, and the method comprises the following steps:

if the audio playing time delay of the second device is smaller than a preset value, the first device sets the first audio data to be processed according to a low time delay mode in the first audio playing strategy;

and if the audio playing time delay of the second equipment is greater than or equal to a preset value, the first equipment sets the first audio data to be processed according to a non-low time delay mode in the first audio playing strategy.

8. The method according to any one of claims 1-7, wherein an application layer of the first device is installed with a first preset application; wherein the obtaining, by the first device, the audio capability parameter of the second device includes:

and the first equipment acquires the audio capability parameter of the second equipment from the second equipment by running the first preset application.

9. The method of claim 8, after the first device obtains the audio capability parameters of the second device, further comprising:

the first preset application creates a corresponding hardware abstraction module in a hardware abstraction layer HAL according to the audio capability parameter of the second device;

Wherein the first device sending the second audio data to the second device comprises:

and the first equipment calls the hardware abstraction module and sends the second audio data to the second equipment.

10. The method of claim 9, wherein the application framework layer of the first device comprises an audio policy module; wherein, the first device determines a first audio playing strategy according to the audio capability parameter of the second device, and the method comprises the following steps:

the first preset application sends the audio capacity parameter of the second device to the audio policy module, so that the audio policy module determines the first audio playing policy according to the audio capacity parameter of the second device.

11. The method of claim 10, wherein the application framework layer of the first device comprises an audio processor; the first device performs first processing on first audio data from a first audio application according to the first audio playing policy to obtain second audio data, and the method includes:

the audio processor receiving first audio data from the first audio application;

And the audio processor outputs second audio data after performing first processing on the first audio data according to a first audio playing strategy output by the audio strategy module.

12. The method of claim 11, further comprising:

when the audio capability parameter of the second device is updated, the first preset application acquires the updated audio capability parameter of the second device;

the first preset application sends the updated audio capacity parameters to the audio strategy module, so that the audio strategy module updates the first audio playing strategy into a second audio playing strategy according to the updated audio capacity parameters;

and the audio strategy module outputs the second audio playing strategy to the audio processor, so that the audio processor performs second processing on the received audio data according to the second audio playing strategy.

13. The method of claim 12, after the first preset application obtains the updated audio capability parameters of the second device, further comprising:

and the first preset application updates the hardware abstraction module according to the updated audio capability parameter, so that the hardware abstraction module supports sending the audio data subjected to the second processing to the second device.

14. The method of claim 11, after the first device transmits the second audio data to the second device, further comprising:

the first equipment receives a second operation of setting the volume by the user;

the audio strategy module responds to the second operation to set a volume level in the first audio playing strategy to obtain a third audio playing strategy;

and the audio strategy module outputs the third audio playing strategy to the audio processor, so that the audio processor modifies the gain of the received audio data according to the volume level set in the third audio playing strategy.

15. The method of claim 11, wherein the application layer of the first device has a second audio application installed; after the first device transmits the second audio data to the second device, the method further includes:

the audio processor receiving audio data from the second audio application;

when the types of the audio data of the second audio application and the audio data of the first audio application are different, the audio policy module updates the first audio playing policy to a fourth audio playing policy;

The audio policy module outputs the fourth audio playing policy to the audio processor, so that the audio processor performs third processing on the received audio data according to the fourth audio playing policy;

and the audio processor sends the audio data subjected to the third processing to the second device through the hardware abstraction module.

16. The method of claim 11, after the first device transmits the second audio data to the second device, further comprising:

the first device receives a third operation that a user selects a third device to play audio, wherein the third device is different from the second device;

responding to the third operation, the first preset application acquires the audio capacity parameter of the third device from the third device;

the first preset application sends the audio capability parameter of the third device to the audio policy module, so that the audio policy module updates the first audio playing policy to a fifth audio playing policy according to the audio capability parameter of the third device;

and the audio policy module outputs the fifth audio playing policy to the audio processor, so that the audio processor performs fourth processing on the received audio data according to the fifth audio playing policy.

17. The method according to claim 16, further comprising, after the first preset application obtains the audio capability parameter of the third device from the third device:

and the first preset application updates the hardware abstraction module according to the audio capability parameter of the third device, so that the updated hardware abstraction module supports sending the audio data subjected to the fourth processing to the third device.

18. The method according to any one of claims 1 to 7, wherein the application layer of the first device is installed with a second preset application, and the second preset application is the same as or different from the first preset application; wherein the first device sending the second audio data to the second device comprises:

and the first equipment sends the second audio data to the second preset application, and the second preset application sends the second audio data to the second equipment.

19. The method of any of claims 1-18, further comprising, prior to the first device displaying the list of candidate devices:

the first equipment displays an application interface of the first audio application, and an audio switching button is arranged in the application interface;

The first operation is the operation that a user clicks the audio switching button.

20. The method of any of claims 1-18, further comprising, prior to the first device displaying the list of candidate devices:

the first equipment displays a control center, and the control center is provided with an audio switching button;

the first operation is the operation that a user clicks the audio switching button.

21. The method of any of claims 1-20, further comprising, after the first device determines a first audio playback policy based on the audio capability parameters of the second device:

the first device determines whether to perform the first processing on the first audio data according to the first audio playing strategy;

the first device performs first processing on first audio data from a first audio application according to the first audio playing policy to obtain second audio data, and the method includes:

and if the first equipment determines to perform the first processing on the first audio data, the first equipment performs the first processing on the first audio data from the first audio application according to the first audio playing strategy to obtain second audio data.

22. The method of claim 21, further comprising, after the first device determines whether to perform the first processing on the first audio data according to the first audio playback policy:

and if the first equipment determines that the first audio data is not subjected to the first processing, the first equipment sends the first audio data to the second equipment for playing.

23. The method of any of claims 1-7, wherein the audio capability parameter of the second device comprises a recording parameter indicating an audio input capability of the second device; the method further comprises the following steps:

the first equipment determines an audio recording strategy according to the recording parameters;

after the first device receives the first recording data recorded by the second device, processing the first recording data according to the audio recording strategy to obtain second recording data;

and the first equipment reports the second sound recording data to the first audio application.

24. An audio control method, comprising:

responding to a first operation input by a user, displaying a candidate device list by a first device, wherein the candidate device list comprises at least one candidate device with an audio output function;

Responding to the operation that a user selects a second device and a third device from the candidate device list, wherein the first device respectively obtains an audio capability parameter of the second device and an audio capability parameter of the third device, the audio capability parameter of the second device is used for indicating the hardware capability of the second device for playing audio and the software capability of the second device for playing audio, and the audio capability parameter of the third device is used for indicating the hardware capability of the third device for playing audio and the software capability of the third device for playing audio;

the first equipment determines a multi-equipment audio playing strategy according to the audio capability parameter of the second equipment and the audio capability parameter of the third equipment;

the first device performs first processing on first audio data from a first audio application according to the multi-device audio playing strategy to obtain second audio data corresponding to the second device and third audio data corresponding to the third device;

and the first equipment sends the second audio data to the second equipment for playing and sends the third audio data to the third equipment for playing.

25. The method of claim 24, wherein the audio capability parameters of the second device and the audio capability parameters of the third device are different, and wherein the multi-device audio playback policy comprises a first policy and a second policy;

wherein, the first device performs a first process on first audio data from a first audio application according to the multi-device audio playing policy to obtain second audio data corresponding to the second device and third audio data corresponding to the third device, including:

the first device copies the first audio data to obtain first data and second data, wherein the first data is the same as the first audio data, and the second data is the same as the first audio data;

the first equipment processes the first data according to the first strategy to obtain second audio data;

and the first equipment processes the second data according to the second strategy to obtain third audio data.

26. The method of claim 25, wherein the application layer of the first device is installed with a first preset application; after the first device obtains the audio capability parameters of the second device and the audio capability parameters of the third device, respectively, the method further includes:

The first preset application creates a first hardware abstraction module in a hardware abstraction layer HAL according to the audio capability parameter of the second device, and the first preset application creates a second hardware abstraction module in the HAL according to the audio capability parameter of the third device;

wherein the first device sending the second audio data to the second device and sending the third audio data to the third device includes:

the first device sends the second audio data to the second device through the first hardware abstraction module, and the first device sends the third audio data to the third device through the second hardware abstraction module.

27. The method of claim 24, wherein the audio capability parameter of the second device and the audio capability parameter of the third device are the same;

wherein, the first device performs a first process on first audio data from a first audio application according to the multi-device audio playing policy to obtain second audio data corresponding to the second device and third audio data corresponding to the third device, including:

The first device performs first processing on the first audio data according to the multi-device audio playing strategy to obtain second audio data;

and the first equipment copies the second audio data to obtain the third audio data.

28. The method of claim 27, wherein the application layer of the first device is installed with a first preset application; after the first device obtains the audio capability parameters of the second device and the audio capability parameters of the third device, respectively, the method further includes:

the first preset application creates a hardware abstraction module in the HAL according to the audio capability parameter of the second device or the audio capability parameter of the third device;

wherein the first device sending the second audio data to the second device and sending the third audio data to the third device includes:

the first device sends the second audio data to the second device through the hardware abstraction module, and the first device sends the third audio data to the third device through the hardware abstraction module.

29. An audio control method, comprising:

Responding to a first operation input by a user, displaying a candidate device list by a first device, wherein the candidate device list comprises at least one candidate device with an audio input function;

responding to the operation that a user selects a second device in the candidate device list, wherein the first device acquires an audio capability parameter of the second device, and the audio capability parameter is used for indicating the hardware capability of the second device for recording audio and the software capability of the second device for recording audio; the second equipment is a sound box, a television or a mobile phone;

the first equipment determines a first audio recording strategy according to the audio capacity parameter of the second equipment;

and when the first equipment receives first recording data sent by the second equipment, the first equipment carries out first processing on the first recording data according to the first audio recording strategy to obtain second recording data.

30. The method of claim 29, after the first device obtains the audio capability parameters of the second device, further comprising:

when the audio capability parameter of the second device is updated, the first device acquires the updated audio capability parameter of the second device;

The first equipment updates the first audio recording strategy into a second audio recording strategy according to the updated audio capacity parameters;

and when the first equipment receives third recording data sent by the second equipment, the first equipment carries out second processing on the third recording data according to the second audio recording strategy to obtain fourth recording data.

31. The method of claim 29, after the first device obtains the audio capability parameters of the second device, further comprising:

the first equipment receives a second operation that a user selects third equipment for audio recording, wherein the third equipment is different from the second equipment;

in response to the second operation, the first device acquires an audio capability parameter of the third device from the third device;

the first equipment determines a third audio recording strategy according to the audio capacity parameter of the third equipment;

and when the first equipment receives fifth recording data sent by the third equipment, the first equipment carries out third processing on the fifth recording data according to the third audio recording strategy to obtain sixth recording data.

32. The method of any of claims 29-31, further comprising, after the first device obtains audio capability parameters of the second device:

the first equipment creates a corresponding hardware abstraction module in a hardware abstraction layer HAL according to the audio capability parameter of the second equipment;

wherein, the first device receives the first sound recording data sent by the second device, and the method comprises the following steps:

and the first device receives the first recording data sent by the second device through the hardware abstraction module.

33. An audio control method, comprising:

the method comprises the steps that a second device sends a first audio capability parameter to a first device, wherein the first audio capability parameter is used for indicating the hardware capability of the second device for playing audio and the software capability of the second device for playing audio, and the second device is a sound box, a television or a mobile phone;

the second equipment receives first audio data sent by the first equipment, wherein the first audio data are obtained after the first equipment is processed according to the first audio capacity parameter;

and the second equipment plays the first audio data, or the second equipment processes the first audio data and then plays the first audio data.

34. The method of claim 33, wherein the application layer in the second device is installed with a preset application;

wherein the second device sending the first audio capability parameter to the first device comprises:

the second device sends the first audio capability parameter to the first device by running the preset application;

the second device receives first audio data sent by the first device, and the method comprises the following steps:

and the second equipment receives the first audio data sent by the first equipment by running the preset application.

35. The method according to claim 33 or 34, wherein after the second device plays the first audio data or the second device processes the first audio data for playing, further comprising:

the second device sets a third device as an audio output device, the third device being different from the second device, the third device being different from the first device;

the second device sends a second audio capability parameter to the first device, wherein the second audio capability parameter is used for indicating the hardware capability of the third device for playing audio and the software capability of the third device for playing audio;

The second device receives second audio data sent by the first device, wherein the second audio data is obtained after the first device is processed according to the second audio capability parameter;

and the second audio data is played by the second equipment, or the second audio data is played after being processed by the second equipment.

36. The method according to claim 33 or 34, wherein after the second device plays the first audio data or the second device processes the first audio data for playing, further comprising:

when the audio capability of the second device changes, the second device updates the first audio capability parameter to a third audio capability parameter;

the second device sending the third audio capability parameter to the first device;

the second device receives third audio data sent by the first device, wherein the third audio data is obtained after the first device is processed according to the third audio capability parameter;

and the second equipment plays the third audio data, or the second equipment processes the third audio data and then plays the third audio data.

37. The method of any of claims 33-36, wherein the first audio capability parameter is further used to indicate a hardware capability of the second device to record audio and a software capability of the second device to record audio;

after the second device sends the first audio capability parameter to the first device, the method further includes:

the second equipment starts to record audio to obtain recording data;

and the second equipment sends the recording data to the first equipment, so that the first equipment processes the recording data according to the first audio capability parameter.

38. The method according to any one of claims 33-37, further comprising:

the second device receives an audio policy sent by the first device, wherein the audio policy is determined by the first device according to the first audio capability parameter, and the audio policy comprises an audio playing policy and/or an audio recording policy;

the playing of the first audio data after being processed by the second device includes:

and the second equipment processes the first audio data according to the audio playing strategy and plays the processed first audio data.

39. An audio control method, comprising:

after a first device establishes network connection with a second device, the first device acquires a device selection strategy corresponding to the second device;

the first equipment acquires first audio data to be played, wherein the type of the first audio data is a first type;

the first device determining whether the first audio data is allowed to be played in the second device according to the first type and the device selection policy;

and if the first audio data is allowed to be played in the second equipment, the first equipment sends the first audio data to the second equipment for playing.

40. The method of claim 39, wherein the device selection policy comprises: at least one of a type of audio data that the second device allows to be played and a type of audio data that the second device does not allow to be played;

wherein the determining, by the first device, whether the first audio data is allowed to be played in the second device according to the first type and the device selection policy comprises:

the first device inquiring in the device selection policy whether the second device allows the first type of audio data to be played;

If the second device allows the first type of audio data to be played, the first device determines that the first audio data is allowed to be played in the second device; and if the second equipment does not allow the audio data of the first type to be played, the first equipment determines that the first audio data is not allowed to be played in the second equipment.

41. The method of claim 39, wherein the device selection policy comprises: the type of the application from which the audio data allowed to be played by the second device and the type of the audio data allowed to be played by the second device, and/or the type of the application from which the audio data not allowed to be played by the second device and the type of the audio data not allowed to be played by the second device;

wherein the determining, by the first device, whether the first audio data is allowed to be played in the second device according to the first type and the device selection policy comprises:

the first device determining that the first audio data is from a first application;

the first device querying the device selection policy whether the second device allows playing of the first type of audio data from the first application;

If the second device allows the first type of audio data from the first application to be played, the first device determines that the first audio data is allowed to be played in the second device; if the second device does not allow the first type of audio data from the first application to be played, the first device determines that the first audio data is not allowed to be played in the second device.

42. The method of claim 40 or 41, wherein the device selection policy further comprises audio output devices of different types of audio data; the method further comprises the following steps:

if the first audio data is not allowed to be played in the second device, the first device queries an audio output device of the first audio data in the device selection strategy as a third device according to the first type, wherein the third device is different from the second device;

and the first equipment sends the first audio data to the third equipment for playing.

43. The method of claim 42, wherein the third device is an audio output device that has recently accessed the first device in addition to the second device.

44. The method of claim 39, wherein the device selection policy comprises: correspondence between different types of audio data and different audio output devices;

wherein the determining, by the first device, whether the first audio data is allowed to be played in the second device according to the first type and the device selection policy comprises:

the first device querying the device selection policy whether an audio output device corresponding to the first type of audio data includes the second device;

if the second device is included, the first device determines that the first audio data is allowed to be played in the second device; if the second device is not included, the first device determines that the first audio data is not allowed to be played in the second device.

45. The method of claim 39, wherein the device selection policy comprises: correspondence between different applications, different types of audio data, and different audio output devices;

wherein the determining, by the first device, whether the first audio data is allowed to be played in the second device according to the first type and the device selection policy comprises:

The first device determining that the first audio data is from a first application;

the first device querying the device selection policy whether audio output devices corresponding to the first type and the first application include the second device;

if the second device is included, the first device determines that the first audio data is allowed to be played in the second device; if the second device is not included, the first device determines that the first audio data is not allowed to be played in the second device.

46. The method of any one of claims 39-45,

when the second device is a loudspeaker box type electronic device, the second device allows playing of audio data of a music type in the device selection strategy, and the second device does not allow playing of audio data of a notification type;

when the second device is an electronic device of a vehicle-mounted device type, the second device allows the audio data of a navigation type to be played in the device selection strategy, and the second device does not allow the audio data of a notification type to be played;

when the second device is an electronic device of a large-screen device type, the second device allows the audio data from the first application to be played in the device selection policy, and the second device does not allow the audio data from the second application to be played.

47. The method of any one of claims 39-46, wherein the application framework layer of the first device is installed with a preset application, and wherein the application framework layer of the first device comprises an audio policy module;

the obtaining, by the first device, a device selection policy corresponding to the second device includes:

the preset application determines the device type of the second device;

the preset application acquires a corresponding equipment selection strategy according to the equipment type of the second equipment;

and the preset application issues the equipment selection strategy to the audio strategy module.

48. The method of claim 47, wherein the application framework layer of the first device further comprises an audio processor;

wherein the determining, by the first device, whether the first audio data is allowed to be played in the second device according to the first type and the device selection policy comprises:

after the audio processor receives the first audio data, sending a query request to the audio policy module, wherein the query request comprises the identifier of the first type;

in response to the query request, the audio policy module determines whether the first audio data of the first type is allowed to be played in the second device according to the device selection policy.

49. The method of claim 48, further comprising, after the first device establishes the network connection with the second device:

the first equipment creates a first hardware abstraction module corresponding to the second equipment at a hardware abstraction layer HAL;

if the first audio data is allowed to be played in the second device, the first device sends the first audio data to the second device for playing, including:

the audio processor receives a first indication sent by the audio policy module, wherein the first indication is used for indicating that an audio output device of the first audio data is the second device;

in response to the first indication, the audio processor invokes the first hardware abstraction module to send the first audio data to the second device.

50. The method of claim 49, wherein the HAL comprises a second hardware abstraction module corresponding to a third device; the method further comprises the following steps:

the audio processor receives a second indication sent by the audio policy module, wherein the second indication is used for indicating that the audio output device of the first audio data is the third device;

In response to the second indication, the audio processor invokes the second hardware abstraction module to send the first audio data to the third device.

51. The method of any one of claims 39-50, after the first device obtains the device selection policy corresponding to the second device, further comprising:

the first equipment displays a setting interface of the audio output equipment;

the first equipment receives an equipment selection strategy which is set by a user and corresponds to the second equipment in the setting interface;

in response to a user setting, the first device updates the device selection policy.

52. The method of any one of claims 39-51, further comprising:

when the second device plays the first audio data, the first device acquires second audio data to be played, wherein the type of the second audio data is a second type;

the first device determines whether the second audio data is allowed to be played in the second device according to the second type and the device selection policy;

if the second audio data is allowed to be played in the second equipment, the first equipment sends the first audio data and the second audio data to the second equipment for playing after mixing; or the first device sends the unmixed first audio data and second audio data to the second device for playing.

53. The method of any one of claims 39-51, further comprising:

after the first device stops sending the first audio data to the second device, the first device acquires second audio data to be played, wherein the type of the second audio data is a second type;

the first device determines whether the second audio data is allowed to be played in the second device according to the second type and the device selection policy;

and if the second audio data is allowed to be played in the second equipment, the first equipment sends the second audio data to the second equipment for playing.

54. An audio control method, comprising:

after a first device establishes network connection with a second device, the first device acquires a sound mixing strategy corresponding to the second device;

the method comprises the steps that first audio data and second audio data to be played are obtained by first equipment, wherein the type of the first audio data is a first type, and the type of the second audio data is a second type;

the first device determines whether the first audio data and the second audio data need to be mixed according to the first type, the second type and the mixing strategy;

If the first audio data and the second audio data do not need to be mixed, the first equipment sends the first audio data and the second audio data to the second equipment for playing;

if the first audio data and the second audio data need to be mixed, the first device mixes the first audio data and the second audio data into third audio data, and audio features of the first audio data and/or the second audio data in the third audio data are changed; and the first equipment sends the third audio data to the second equipment for playing.

55. The method of claim 54, wherein the mixing strategy comprises: a type of audio data that requires mixing and/or a type of audio data that does not require mixing;

wherein the determining, by the first device, whether the first audio data and the second audio data need to be mixed according to the first type, the second type, and the mixing policy includes:

the first equipment inquires whether the audio data of the first type needs sound mixing or not and whether the audio data of the second type needs sound mixing or not in the sound mixing strategy;

If at least one of the first type of audio data and the second type of audio data does not require audio mixing, the first device determines that audio mixing of the first audio data and the second audio data is not required; if the audio data of the first type and the audio data of the second type both require audio mixing, the first device determines that the audio mixing of the first audio data and the second audio data is required.

56. The method of claim 54, wherein the mixing strategy comprises: the type of the audio data to be mixed and the application from which the audio data to be mixed comes, and/or the type of the audio data not to be mixed and the application from which the audio data not to be mixed comes;

wherein the determining, by the first device, whether the first audio data and the second audio data need to be mixed according to the first type, the second type, and the mixing policy includes:

the first device determining that the first audio data is from a first application and the second audio data is from a second application;

the first device inquires whether the audio data of the first type from the first application needs mixing and whether the audio data of the second type from the second application needs mixing in the mixing strategy;

If at least one of the first type of audio data from the first application and the second type of audio data from the second application does not require mixing, the first device determines that mixing of the first audio data and the second audio data is not required; if the audio data of the first type from the first application and the audio data of the second type from the second application both require audio mixing, the first device determines that the audio mixing of the first audio data and the second audio data is required.

57. The method of claim 54, wherein the mixing strategy comprises: correspondence between different types of audio data and audio output devices that do not allow mixing;

wherein the determining, by the first device, whether the first audio data and the second audio data need to be mixed according to the first type, the second type, and the mixing policy includes:

the first equipment inquires whether first audio output equipment corresponding to the first type of audio data contains the second equipment or not in the mixing strategy;

the first equipment inquires whether second audio output equipment corresponding to the second type of audio data comprises the second equipment or not in the mixing strategy;

If the first audio output device comprises the second device and/or the second audio output device comprises the second device, the first device determines that mixing of the first audio data and the second audio data is not required; if the first audio output device does not include the second device and the second audio output device does not include the second device, the first device determines that audio mixing of the first audio data and the second audio data is required.

58. The method of claim 54, wherein the mixing strategy comprises: correspondence between different types of audio data, different applications, and audio output devices that do not allow mixing;

wherein the determining, by the first device, whether the first audio data and the second audio data need to be mixed according to the first type, the second type, and the mixing policy includes:

the first device determining that the first audio data is from a first application and the second audio data is from a second application;

the first device inquires whether a first audio output device corresponding to the first application and the first type contains the second device in the sound mixing strategy;

The first device inquires whether a second audio output device corresponding to the second application and the second type comprises the second device or not in the mixing strategy;

if the first audio output device comprises the second device and/or the second audio output device comprises the second device, the first device determines that mixing of the first audio data and the second audio data is not required; if the first audio output device does not include the second device and the second audio output device does not include the second device, the first device determines that audio mixing of the first audio data and the second audio data is required.

59. The method of any one of claims 54-58,

when the second device is a loudspeaker box type electronic device, no sound mixing is needed when sending music type audio data to the second device in the sound mixing strategy;

when the second device is an electronic device of a vehicle-mounted device type, no sound mixing is required when audio data of a navigation type is sent to the second device in the sound mixing strategy;

and when the second equipment is large-screen equipment type electronic equipment, mixing is required when notification type audio data is sent to the second equipment in the mixing strategy.

60. The method of any of claims 54-59, wherein the first device sending the first audio data and the second audio data to the second device comprises:

the first equipment packs the first audio data to obtain at least one first data packet;

the first equipment packs the second audio data to obtain at least one second data packet;

the first device sends the first data packet and the second data packet to the second device.

61. The method of claim 60, wherein the first data packet includes a first identifier therein, the first identifier indicating the first audio data; the second data packet comprises a second identifier, and the second identifier is used for indicating second audio data.

62. The method of claim 60, wherein the first data packet includes first characteristic information therein, the first characteristic information indicating audio characteristics of the first audio data; second characteristic information is included in the second data packet, and the second characteristic information is used for indicating audio characteristics of the second audio data.

63. The method of claim 62,

the first feature information includes at least one of an application to which the first audio data belongs, a type of the first audio data, a volume level of the first audio data, a playback format of the first audio data, and track information of the first audio data;

the second feature information includes at least one of an application to which the second audio data belongs, a type of the second audio data, a volume level of the second audio data, a playback format of the second audio data, and track information of the second audio data.

64. The method according to any of claims 54-63, further comprising, after the first device establishes a network connection with the second device:

the first equipment acquires an equipment selection strategy corresponding to the second equipment;

after the first device acquires the first audio data and the second audio data to be played, the method further includes:

the first device determines that the audio output device of the first audio data and the second audio data is the second device according to the first type, the second type and the device selection policy.

65. The method of any one of claims 54-64, wherein the application framework layer of the first device is installed with a preset application, and wherein the application framework layer of the first device comprises an audio policy module;

the obtaining, by the first device, a mixing policy corresponding to the second device includes:

the preset application determines the device type of the second device;

the preset application acquires a corresponding sound mixing strategy according to the equipment type of the second equipment;

and the preset application issues the sound mixing strategy to the audio strategy module.

66. The method of claim 65, wherein the application framework layer of the first device further comprises an audio processor;

the method for acquiring the first audio data and the second audio data to be played by the first device includes:

the audio processor acquires first audio data and second audio data from an application program layer;

wherein the determining, by the first device, whether the first audio data and the second audio data need to be mixed according to the first type, the second type, and the mixing policy includes:

The audio processor sends a query request to the audio policy module, wherein the query request comprises a first type identifier and a second type identifier;

in response to the query request, the audio policy module determines whether mixing of the first audio data of the first type and the second audio data of the second type is required according to the mixing policy.

67. The method of claim 66, further comprising, after the first device establishes the network connection with the second device:

the first equipment creates a hardware abstraction module corresponding to the second equipment at a hardware abstraction layer HAL;

wherein if the first audio data and the second audio data do not need to be mixed, the method further comprises:

the audio processor receives a first indication sent by the audio policy module, wherein the first indication is used for indicating that the first audio data and the second audio data are not mixed;

in response to the first indication, the audio processor encapsulating the first audio data into at least one first data packet and storing the first data packet in a cache of the audio processor;

In response to the first indication, the audio processor encapsulating the second audio data into at least one second data packet and storing the second data packet in a cache of the audio processor;

and the audio processor sends the first data packet and the second data packet in the cache of the audio processor to the hardware abstraction module.

68. The method of claim 67, wherein the first device sending the first audio data and the second audio data to the second device comprises:

the hardware abstraction module sends a first data packet and a second data packet in a cache of the audio processor to the second device, so that the second device restores the first audio data and the second audio data by analyzing the first data packet and the second data packet; or;

the hardware abstraction module restores the first audio data and the second audio data by analyzing the first data packet and the second data packet, and sends the first audio data and the second audio data to the second device.

69. The method according to any one of claims 54-68, further comprising, after the first device obtains the mixing strategy corresponding to the second device:

The first equipment displays a setting interface of the audio output equipment;

the first equipment receives a sound mixing strategy which is set by a user and corresponds to the second equipment in the setting interface;

and in response to the setting of the user, the first equipment updates the mixing strategy.

70. The method of any one of claims 54-69, further comprising:

when the first device acquires the first audio data and the second audio data, the first device also acquires third audio data to be played, wherein the type of the third audio data is a third type;

the first device determines that the first audio data does not need to be mixed and the second audio data and the third audio data need to be mixed according to the first type, the second type, the third type and the mixing strategy;

the first device sends the first audio data to the second device, and the first device sends the second audio data and the third audio data to the second device after mixing.

71. An audio control method, comprising:

the second equipment establishes network connection with the first equipment;

The second equipment acquires unmixed first audio data and second audio data from the first equipment, wherein the type of the first audio data is a first type, and the type of the second audio data is a second type;

the second device plays the first audio data and the second audio data.

72. The method of claim 71, wherein the second device obtaining unmixed first and second audio data from the first device comprises:

the second device acquires a first data packet corresponding to the first audio data and a second data packet corresponding to the second audio data from the first device;

and the second equipment restores the first audio data and the second audio data by analyzing the first data packet and the second data packet.

73. The method of claim 72, wherein the first data packet includes a first identifier therein, the first identifier indicating the first audio data; the second data packet comprises a second identifier which is used for indicating second audio data;

wherein the second device restores the first audio data and the second audio data by parsing the first data packet and the second data packet, including:

And the second device restores the audio data in the data packet carrying the first identifier into the first audio data and restores the audio data in the data packet carrying the second identifier into the second audio data by analyzing the first data packet and the second data packet.

74. The method of claim 72, wherein the first data packet includes first characteristic information therein, the first characteristic information indicating audio characteristics of the first audio data; the second data packet comprises second characteristic information, and the second characteristic information is used for indicating audio characteristics of the second audio data;

wherein the playing of the first audio data and the second audio data by the second device comprises:

and the second equipment plays the first audio data according to the first characteristic information, and simultaneously plays the second audio data according to the second characteristic information.

75. A method for playing a multi-audio task, comprising:

the method comprises the steps that the electronic equipment displays a first window and a second window in a display interface, wherein a first application runs in the first window, and a second application runs in the second window;

The electronic equipment receives a first operation input by a user;

in response to the first operation, the electronic device establishes an association relationship between the first window and a first audio output device;

the electronic equipment receives a second operation input by a user;

in response to the second operation, the electronic device establishes an association relationship between the second window and a second audio output device;

when the electronic equipment acquires first audio data from the first application, the electronic equipment sends the first audio data to the first audio output equipment for playing according to the incidence relation between the first window and the first audio output equipment;

when the electronic equipment acquires second audio data from the second application, the electronic equipment sends the second audio data to the second audio output equipment for playing according to the incidence relation between the second window and the second audio output equipment.

76. The method of claim 75, after the electronic device displays the first window and the second window in the display interface, further comprising:

the electronic equipment displays a first dialog box in the first window, wherein the first dialog box comprises at least one first candidate equipment with an audio playing function; wherein the first operation is an operation of a user selecting the first audio output device among the at least one first candidate device;

The electronic equipment displays a second dialog box in the second window, wherein the second dialog box comprises at least one second candidate equipment with an audio playing function; wherein the second operation is an operation of the user selecting the second audio output device among the at least one second candidate device.

77. The method of claim 75 or 76, wherein the electronic device establishes an association of the first window with a first audio output device, comprising:

the electronic equipment records first audio configuration information of the first window, wherein the first audio configuration information comprises an incidence relation between the first window and the first audio output equipment;

the electronic device establishes an association relationship between the second window and a second audio output device, including:

and the electronic equipment records second audio configuration information of the second window, wherein the second audio configuration information comprises the incidence relation between the second window and the second audio output equipment.

78. The method of claim 77, further comprising:

when the first application runs in the first window, the electronic equipment records a first corresponding relation between the first window and the first application;

When the second application runs in the second window, the electronic device records a second corresponding relationship between the second window and the second application.

79. The method of claim 78, wherein the electronic device records a first correspondence between the first window and the first application, comprising:

the first application acquires a first audio focus of the first window;

when the first application acquires the first audio focus, the electronic equipment records a first corresponding relation between the first window and the first application;

the electronic device records a second corresponding relationship between the second window and the second application, and the method includes:

the second application applies for obtaining a second audio focus of the second window;

when the second application acquires the second audio focus, the electronic device records a second corresponding relation between the second window and the second application.

80. The method of claim 78, further comprising, before the electronic device sends the first audio data to the first audio output device for playback:

The electronic device determining that the first audio data is from the first application;

the electronic equipment determines that the first audio data corresponds to the first window according to the first corresponding relation;

the electronic equipment determines that the audio output equipment of the first audio data is the first audio output equipment associated with the first window according to the first audio configuration information;

before the electronic device sends the second audio data to the second audio output device for playing, the method further includes:

the electronic device determining that the second audio data is from the second application;

the electronic equipment determines that the second audio data corresponds to the second window according to the second corresponding relation;

and the electronic equipment determines that the audio output equipment of the second audio data is the second audio output equipment associated with the second window according to the second audio configuration information.

81. The method of any one of claims 77-80,

the first audio configuration information further comprises the type of audio data allowed to be played in the first audio output device and/or the volume of the audio data allowed to be played in the first audio output device;

The second audio configuration information further includes a type of audio data allowed to be played in the second audio output device and/or a volume of the audio data allowed to be played in the second audio output device.

82. The method of claim 81, wherein the first audio configuration information further comprises a volume level of audio data allowed to be played in the first audio output device;

after the electronic device displays the first window and the second window in the display interface, the method further comprises:

the electronic equipment displays a first volume adjusting bar in the first window;

if it is detected that a user inputs a first volume adjustment operation on the first volume adjustment bar, the electronic device modifies the volume of first type audio data in the first audio configuration information, where the first type is the type of audio data being played in the first window or a preset type of audio data.

83. The method of claim 81, wherein the second audio configuration information further comprises a volume level of audio data allowed to be played in the second audio output device;

After the electronic device displays the first window and the second window in the display interface, the method further comprises:

the electronic equipment displays a second volume adjusting bar in the second window;

and if the fact that the user inputs a second volume adjusting operation on the second volume adjusting bar is detected, the electronic equipment modifies the volume of second type audio data in the second audio configuration information, wherein the second type is the type of audio data being played in the second window or the type of preset audio data.

84. The method of any of claims 77-83, further comprising, after the electronic device establishes the association of the first window with the first audio output device:

responding to a third operation input by a user, the electronic equipment updates the first audio configuration information, and the updated first audio configuration information comprises an incidence relation between the first window and a third audio output device.

85. The method of any of claims 77-83, further comprising, after the electronic device establishes an association of the second window with a second audio output device:

And responding to a fourth operation input by the user, updating the second audio configuration information by the electronic equipment, wherein the updated second audio configuration information comprises the association relationship between the second window and a fourth audio output device.

86. The method of any one of claims 75-85, wherein an application framework layer of the electronic device includes an audio policy module and an audio processor, the method further comprising:

when the audio processor receives the first audio data, the audio processor sends a first query request to the audio policy module; in response to the first query request, the audio policy module determining that an audio output device of the first audio data is the first audio output device;

when the audio processor receives the second audio data, the audio processor sends a second query request to the audio policy module; in response to the second query request, the audio policy module determines that the audio output device for the second audio data is the second audio output device.

87. The method of claim 86, wherein the application framework layer of the electronic device comprises a window manager configured to send a first correspondence of the first window to the first application, a second correspondence of the second window to the second application, first audio configuration information of the first window, and second audio configuration information of the second window to the audio policy module.

88. The method as recited in claim 86, wherein a first audio abstraction module corresponding to the first audio output device and a second audio abstraction module corresponding to the second audio output device are included in a Hardware Abstraction Layer (HAL) of the electronic device;

wherein the electronic device sending the first audio data to the first audio output device comprises:

the audio processor sends the first audio data to the first audio output device through the first audio abstraction module;

wherein the electronic device sending the second audio data to the second audio output device comprises:

and the audio processor sends the second audio data to the second audio output device through the second audio abstraction module.

89. The method according to any of claims 75-88, wherein a Hardware Abstraction Layer (HAL) of the electronic device comprises a display abstraction module; the method further comprises the following steps:

the display abstraction module acquires first display data corresponding to the first window and second display data corresponding to the second window;

and the display abstraction module synthesizes the first display data and the second display data into third display data corresponding to the display interface.

90. The method according to claim 89, wherein the display output device of the electronic device is a display screen of the electronic device, and after the display abstraction module synthesizes the first display data and the second display data into third display data, the method further comprises:

and the display abstraction module sends the third display data to a display screen of the electronic equipment for display.

91. The method according to claim 89, wherein the display output device of the electronic device is a first display device, and after the display abstraction module synthesizes the first display data and the second display data into third display data to be displayed, the method further comprises:

and the display abstraction module sends the third display data to the first display device for display.

92. The method of claim 91, wherein the electronic device sends the first audio data to the first audio output device for playing according to the association relationship between the first window and the first audio output device, and the method comprises:

the electronic equipment sends the first audio data and the incidence relation between the first window and first audio output equipment to the first display equipment, so that the first display equipment sends the first audio data to the first audio output equipment for playing according to the incidence relation between the first window and the first audio output equipment;

The sending, by the electronic device, the second audio data to the second audio output device for playing according to the association relationship between the second window and the second audio output device includes:

and the electronic equipment sends the second audio data and the incidence relation between the second window and the second audio output equipment to the first display equipment, so that the first display equipment sends the second audio data to the second audio output equipment for playing according to the incidence relation between the second window and the second audio output equipment.

93. The method according to any of claims 75-88, wherein a Hardware Abstraction Layer (HAL) of the electronic device comprises a display abstraction module; the display output device corresponding to the first window is a first display device, and the display output device corresponding to the second window is a second display device; the method further comprises the following steps:

the display abstraction module acquires first display data corresponding to the first window and sends the first display data to the first display device for display;

and the display abstraction module acquires second display data corresponding to the second window and sends the second display data to the second display equipment for display.

94. A screen recording method is characterized by comprising the following steps:

responding to an operation of opening a screen recording function in first equipment by a user, wherein the first equipment displays a screen recording equipment list, and the screen recording equipment list comprises one or more electronic equipment which is accessed to the same communication network with the first equipment;

in response to an operation that a user selects a second device in the screen recording device list, the first device sends a screen recording instruction to the second device, so that the second device executes a screen recording operation in response to the screen recording instruction;

the first equipment executes screen recording operation to obtain first screen data of the first equipment;

the first equipment acquires second screen data obtained by the second equipment executing screen recording operation;

and the first equipment generates a screen recording file according to the first screen data and the second screen data.

95. The method of claim 94, wherein the first screen data comprises first display data and first audio data;

the first display data includes: the display data played by the first equipment and/or the display data collected by the first equipment; the first audio data includes: the audio data played by the first device and/or the audio data collected by the first device.

96. The method of claim 95, before the first device performs a screen recording operation to obtain the first screen data of the first device, further comprising:

the first equipment acquires a first screen recording parameter used by the first equipment for executing screen recording operation, wherein the first screen recording parameter comprises a display recording parameter and an audio recording parameter;

the method for acquiring the first screen data of the first device by the first device executing screen recording operation includes:

the first equipment records according to the display recording parameters to obtain the first display data;

and the first equipment records the audio recording parameters to obtain the first audio data.

97. The method of claim 96, prior to the first device sending a screen recording instruction to the second device, further comprising:

the first equipment acquires second screen recording parameters used by the second equipment for executing screen recording operation, wherein the second screen recording parameters comprise display recording parameters and audio recording parameters;

wherein, the first equipment sends the screen recording instruction to the second equipment, including:

and the first equipment carries the second screen recording parameter in a screen recording instruction and sends the screen recording instruction to the second equipment.

98. The method of claim 97, wherein the obtaining, by the first device, the first screen recording parameter and the second screen recording parameter comprises:

the first equipment acquires a first screen recording capability parameter of the first equipment, wherein the first screen recording capability parameter is used for reflecting the screen recording capability of the first equipment;

the first equipment acquires a second screen recording capability parameter of the second equipment, wherein the second screen recording capability parameter is used for reflecting the screen recording capability of the second equipment;

and the first equipment determines a first screen recording parameter corresponding to the first equipment and a second screen recording parameter corresponding to the second equipment according to the first screen recording capability parameter and the second screen recording capability parameter.

99. The method of claim 98,

when the resolution of the first device in the first screen recording capability parameter is greater than the resolution of the second device in the second screen recording capability parameter, the resolution of the first screen recording parameter and the resolution of the second screen recording parameter are the resolution of the second device; alternatively, the first and second electrodes may be,

when the number of Dots Per Inch (DPI) of the first equipment in the first screen recording capability parameter is larger than the DPI of the second equipment in the second screen recording capability parameter, the DPI in the first screen recording parameter and the second screen recording parameter is the DPI of the second equipment; alternatively, the first and second electrodes may be,

And when the sampling rate of the first equipment in the first screen recording capacity parameter is greater than the sampling rate of the second equipment in the second screen recording capacity parameter, the sampling rate of the first screen recording parameter and the sampling rate of the second screen recording parameter are the sampling rate of the second equipment.

100. The method of any of claims 95-99, wherein the second screen data comprises second display data and second audio data;

the second display data includes: the display data played by the second equipment and/or the display data collected by the second equipment; the second audio data includes: and audio data played by the second device and/or audio data collected by the second device.

101. The method of claim 100, wherein the first device generates a screen recording file according to the first screen data and the second screen data, and wherein the generating comprises:

the first equipment synthesizes the first display data and the second display data into third display data;

the first device makes the third display data, the first audio data, and the second audio data as a screen recording file.

102. The method of claim 100, wherein the screen-recording file comprises a first file and a second file;

the generating, by the first device, a screen recording file according to the first screen data and the second screen data includes:

the first device makes the first display data and the first audio data into the first file;

the first device creates the second display data and the second audio data as the second file.

103. The method of any one of claims 94-102, further comprising, prior to the first device displaying a list of screen recording devices:

the first equipment detects N pieces of electronic equipment which are accessed to the same communication network with the first equipment, wherein N is an integer larger than 0;

the first equipment determines the electronic equipment with the screen recording function in the N pieces of electronic equipment;

wherein, the first equipment shows and records screen equipment list, includes:

and the first equipment displays the determined electronic equipment with the screen recording function in the screen recording equipment list.

104. The method as claimed in any one of claims 94 to 103, wherein the application layer of the first device comprises a screen recording application, the application framework layer of the first device comprises a screen recorder, and the hardware abstraction layer HAL of the first device comprises a preset hardware abstraction module;

The method for acquiring the first screen data of the first device by the first device executing screen recording operation includes:

the screen recording application calls the screen recorder to execute screen recording operation to obtain first screen data of the first device;

the acquiring, by the first device, second screen data obtained by performing a screen recording operation by the second device includes:

and the screen recorder acquires second screen data obtained by the second equipment executing screen recording operation through the hardware abstraction module.

105. A screen recording method is characterized by comprising the following steps:

the method comprises the steps that a second device obtains a screen recording instruction sent by a first device, wherein the screen recording instruction comprises screen recording parameters used by the second device for executing screen recording operation, and the screen recording parameters comprise display recording parameters and audio recording parameters;

responding to the screen recording instruction, and executing screen recording operation by the second equipment according to the screen recording parameters to obtain screen data of the second equipment;

and the second equipment sends the screen data of the second equipment to the first equipment.

106. The method of claim 105, wherein the screen data of the second device comprises display data and audio data;

The display data includes: the display data played by the second equipment and/or the display data collected by the second equipment; the audio data includes: and audio data played by the second device and/or audio data collected by the second device.

107. The method of claim 106, wherein the second device performs a screen recording operation according to the screen recording parameters to obtain the screen data of the second device, comprising:

the second equipment records the display data according to the display recording parameters;

and the second equipment records the audio data according to the audio recording parameters.

108. The method as recited in any one of claims 105-107, wherein the application layer of the second device comprises a preset application, and the application framework layer of the second device comprises a screen recorder;

the preset application is used for: receiving the screen recording instruction from the first equipment; calling the screen recorder to execute screen recording operation according to the screen recording parameters to obtain screen data of the second equipment; and sending the screen data of the second equipment to the first equipment.

109. An audio delay updating method, comprising:

the first equipment and the second equipment establish network connection;

if the second device is set as an audio output device of the first device, the first device acquires a first time delay, a second time delay and a third time delay, wherein the first time delay is a time delay generated when audio data is processed in the first device, the second time delay is a time delay generated when the audio data is transmitted between the first device and the second device, and the third time delay is a time delay generated when the audio data is played in the second device;

and the first equipment determines the current audio time delay, wherein the audio time delay is calculated based on the first time delay, the second time delay and the third time delay.

110. The method of claim 109, further comprising, after the first device establishes the network connection with the second device:

the first device acquires an audio capability parameter of the second device, wherein the audio capability parameter is used for indicating the hardware capability of the second device for playing audio and the software capability of the second device for playing audio;

Wherein the obtaining, by the first device, a third time delay includes:

the first device obtains the third delay from the audio capability parameter.

111. The method of claim 110, after the first device obtains audio capability parameters of the second device, further comprising:

the first equipment determines an audio strategy according to the audio capacity parameter;

wherein the first device obtaining a first time delay comprises:

and the first equipment determines the first time delay according to the audio strategy.

112. The method of claim 111, wherein the audio policy includes N processes on the audio data, N being an integer greater than 0;

wherein the determining, by the first device, the first time delay according to the audio policy includes:

the first device determines a time consumed by each of the N processes;

and the first equipment calculates the first time delay based on the consumed time of each processing process.

113. The method of claim 111 or 112, wherein if the second device is configured as an audio output device of the first device, the method further comprises:

The first device acquires the updated audio capability parameter of the second device;

the first equipment updates the audio strategy according to the updated audio capacity parameter;

the first equipment updates the third time delay according to the updated audio capacity parameter;

and the first equipment updates the first time delay according to the updated audio strategy.

114. The method as claimed in any one of claims 109-113, wherein the hardware abstraction layer HAL of the first device comprises a first hardware abstraction module for transmitting audio data;

after the first device establishes a network connection with the second device, the method further includes:

the first device creating a second hardware abstraction module in the HAL corresponding to the second device;

wherein the obtaining of the second time delay by the first device includes:

the second hardware abstraction module obtains the second latency from the first hardware abstraction module.

115. The method of claim 114, further comprising:

the first hardware abstraction module sends a test data packet to the second device;

the first hardware abstraction module receives a response data packet sent by the second device in response to the test data packet;

The first hardware abstraction module calculates and stores the second time delay according to a time interval between sending the test data packet and receiving the response data packet.

116. The method as claimed in any one of claims 109 and 115 wherein the determining of the current audio delay by the first device comprises:

when at least one of the first delay, the second delay, or the third delay is updated, the first device updates the current audio delay.

117. The method as recited in any one of claims 109-116, wherein the display data output by the first device is displayed on a display screen of the first device;

after the first device determines the current audio delay, the method further includes:

a first application running in the first device acquires the audio time delay;

when the first application is a video application, the first application synchronizes audio data and display data of the first application according to the audio time delay; when the first application is a music application, the first application synchronizes the audio data and the lyric data of the first application according to the audio time delay; and when the first application is a karaoke application, the first application synchronizes the audio data and the accompaniment music of the first application according to the audio time delay.

118. The method as recited in any one of claims 109-116, wherein the display data output by the first device is displayed on a display screen of a third device; the method further comprises the following steps:

the first device determining a current display delay, the display delay comprising a resulting delay in display data transmission between the first device and the third device;

the first device determines a current synchronization delay based on the audio delay and the display delay;

and the second application running in the first equipment synchronizes the audio data and the display data of the second application according to the current synchronization delay.

119. The method of claim 118, wherein the synchronization delay is a difference between the audio delay and the display delay.

120. The method as recited in any one of claims 109-119 wherein the audio delay is a sum of the first delay, the second delay, and the third delay.

121. The method as claimed in any one of claims 109-120, wherein if a fourth device is provided as the audio output device of the first device, the fourth device being connected to the second device, the method further comprises:

The first device obtains the first time delay and the second time delay;

the first device obtains a fourth time delay, which includes: a time delay generated when the audio data is processed in the second device, a time delay generated when the audio data is transmitted between the second device and the fourth device, and a time delay generated when the audio data is played in the fourth device;

and the first equipment determines the current audio time delay, wherein the audio time delay is calculated based on the first time delay, the second time delay and the fourth time delay.

122. The method of claim 121, wherein the audio delay is a sum of the first delay, the second delay, and the fourth delay.

123. An audio delay updating method, comprising:

the first equipment and the second equipment establish network connection;

if the second device is set as the audio output device and the display output device of the first device, the first device obtains a first time delay and a second time delay, wherein the first time delay is a time delay generated when audio data is processed in the first device, and the second time delay is a time delay generated when the audio data is played in the second device;

The first device determines a current synchronization delay based on the first delay and the second delay.

124. The method of claim 123, further comprising, after the first device determines a current synchronization delay based on the first delay and the second delay:

a first application running in the first device acquires the synchronization delay;

and the first application synchronizes the audio data and the display data of the first application according to the synchronization delay.

125. The method of claim 123 or 124, wherein said synchronization delay is a sum of said first delay and said second delay.

126. A method for playing an audio/video file is characterized by comprising the following steps:

the method comprises the steps that first equipment receives first operation input by a user, the first operation is used for projecting a first file played in the first equipment to second equipment to be played, and the first file is an audio file or a video file;

the first device acquires a first data packet of the first file, wherein the first file comprises N data packets, the first data packet is one of the N data packets, and N is greater than 0;

The first device extracts a first audio parameter and first audio data in the first data packet, wherein the first audio data is encoded audio data, and the first audio parameter is an encoding parameter used when the first audio data is encoded;

the first device sends the first audio parameters to the second device so that the second device can determine whether the second device has the capability of decoding the first audio data according to the first audio parameters;

and if the second equipment has the capability of decoding the first audio data, the first equipment sends the first audio data to the second equipment so that the second equipment can decode the first audio data and then play the first audio data.

127. The method of claim 126, wherein the first packet comprises a header and a data portion;

wherein the extracting, by the first device, the first audio parameter and the first audio data in the first data packet includes:

the first device extracting the first audio parameter from a packet header of the first data packet;

the first device extracts the first audio data from a data portion of the first data packet.

128. The method of claim 126 or 127, wherein the first audio parameter comprises: at least one of an encoding format, an encoding specification, a version, a sampling rate, or a number of channels used in encoding the first audio data.

129. The method of claim 126, wherein the first file is a video file; after the first device acquires the first data packet of the first file, the method further includes:

the first device extracts a first display parameter and first display data in the first data packet, wherein the first display data is coded display data, and the first display parameter is a coding parameter used when the first display data is coded;

the first device sends the first display parameters to the second device so that the second device can determine whether the second device has the capability of decoding the first display data according to the first display parameters;

and if the second equipment has the capability of decoding the first display data, the first equipment sends the first display data to the second equipment so that the second equipment can play the first display data after decoding the first display data.

130. The method of claim 129 wherein the first packet comprises a header and a data portion;

wherein the extracting, by the first device, the first display parameter and the first display data in the first data packet includes:

the first device extracting the first display parameter from the header of the first data packet;

the first device extracts the first display data from a data portion of the first data packet.

131. The method of claim 129 or 130, wherein the first display parameter comprises: at least one of an encoding format, an encoding specification, a version, or a resolution used in encoding the first display data.

132. The method as claimed in any one of claims 126-131, wherein the first device comprises a first application, a second application and a first audio/video player, the first application is configured to communicate with the second device, and the second application is configured to send the data packet of the first file to be played to the first audio/video player;

after the first device receives the first operation of the user input, the method further comprises the following steps:

in response to the first operation, the first application establishes a network connection with the second device;

The first application sends a first broadcast message to the second application, wherein the first broadcast message is used for informing the second application to project the first file to the second device for playing;

and responding to the first broadcast message, the second application sends a first notification message to the first audio and video player so that the first audio and video player starts projecting audio and video files to the second device.

133. The method of claim 132, wherein the first device further comprises a first multimedia extractor;

wherein the extracting, by the first device, the first audio parameter and the first audio data in the first data packet includes:

responding to the first notification message, the first multimedia extractor calls the multimedia extractor to analyze the first data packet to obtain the first audio parameter;

and responding to the first notification message, and calling the multimedia extractor by the first multimedia extractor to decapsulate the first data packet to obtain the first audio data.

134. The method of claim 132, further comprising, after the first device transmits the first audio data to the second device:

The first audio and video player acquires a second data packet of the first file from the second application;

the first audio and video player extracts second audio data in the second data packet;

and the first audio and video player sends the second audio data to the second equipment so that the second equipment decodes the second audio data and plays the second audio data.

135. The method of claim 132, further comprising, after the first device transmits the first audio data to the second device:

responding to a second operation input by a user, and establishing network connection between the first application and a third device;

the first application sends a second broadcast message to the second application, wherein the second broadcast message is used for informing the second application to project the first file to the third equipment for playing;

and responding to the second broadcast message, the second application sends a second notification message to the first audio and video player so that the first audio and video player starts to project audio and video files to the third device.

136. The method of claim 135, further comprising, after the second application sends a second notification message to the first audiovisual player:

The first audio and video player acquires a third data packet of the first file from the second application;

the first audio and video player extracts a second audio parameter and third audio data in the third data packet, wherein the third audio data is encoded audio data, and the second audio parameter is an encoding parameter used when the third audio data is encoded;

the first audio and video player sends the second audio parameter to the third equipment, so that the third equipment determines whether the third equipment has the capacity of decoding the third audio data according to the second audio parameter;

and if the third equipment has the capability of decoding the third audio data, the first audio and video player sends the third audio data to the third equipment so that the third equipment can play the third audio data after decoding the third audio data.

137. The method of claim 132, further comprising, after the first device transmits the first audio data to the second device:

responding to a third operation input by a user, switching the currently played first file into a second file by the second application, wherein the second file is an audio file or a video file;

The second application creates a second audio and video player, and the second audio and video player is used for receiving the data packet from the second file;

and the second application sends a third notification message to the second audio and video player so that the second audio and video player starts to project audio and video files to the second device.

138. The method of claim 137, further comprising, after the second application sends a third notification message to the second audiovisual player:

the second audio and video player acquires a fourth data packet of the second file from the second application;

the second audio and video player extracts a third audio parameter and fourth audio data in the fourth data packet, wherein the fourth audio data is encoded audio data, and the third audio parameter is an encoding parameter used when the fourth audio data is encoded;

the second audio and video player sends the third audio parameter to the second device, so that the second device determines whether the fourth audio data has the capability of decoding according to the third audio parameter;

and if the second equipment has the capability of decoding the fourth audio data, the second audio and video player sends the fourth audio data to the second equipment so that the second equipment can decode the fourth audio data and then play the fourth audio data.

139. The method as recited in any one of claims 126-138, further comprising, after the first device transmits the first audio data to the second device:

the first device running a third application;

the first device plays an audio file or a video file from the third application.

140. A method for playing an audio/video file is characterized by comprising the following steps:

the method comprises the steps that a second device receives a first audio parameter sent by a first device, wherein the first audio parameter is an encoding parameter used when first audio data in a first data packet is encoded, and the first data packet is a data packet of a first file;

the second device determining whether the second device has the capability of decoding the first audio data according to the first audio parameters;

if the first audio data can be decoded, the second device sends a first response message to the first device, wherein the first response message is used for indicating the first device to send the first audio data to the second device;

after receiving the first audio data sent by the first equipment, the second equipment decodes the first audio data to obtain decoded second audio data;

And the second equipment plays the second audio data.

141. The method of claim 140, wherein the second device decodes the first audio data to obtain decoded second audio data, comprising:

and the second equipment decodes the first audio data by using a first decoding parameter to obtain decoded second audio data, wherein the first decoding parameter corresponds to the first audio parameter.

142. The method of claim 140 or 141, wherein the first data packet further comprises first display data; the method further comprises the following steps:

the second equipment receives a first display parameter sent by the first equipment, wherein the first display parameter is a coding parameter used when the first display data is coded;

the second device determining whether the second device has the capability of decoding the first display data according to the first display parameter; if the first display data is capable of being decoded, the first response message is further used for instructing the first device to send the first display data to the second device;

after receiving the first display data sent by the first equipment, the second equipment decodes the first display data to obtain decoded second display data;

And the second equipment plays the second display data.

143. The method of claim 142, wherein the second device decodes the first display data to obtain decoded second display data, comprising:

and the second equipment decodes the first display data by using a second decoding parameter to obtain decoded second display data, wherein the second decoding parameter corresponds to the first display parameter.

144. The method as claimed in any one of claims 140-143, further comprising, after the second device receives the first audio data transmitted by the first device:

if the second device receives third audio data in the first file, the second device decodes the third audio data and plays the third audio data; and/or the presence of a gas in the gas,

and if the second equipment receives third display data in the first file, the second equipment decodes the third display data and plays the third display data.

145. The method as claimed in any of claims 142-144, further comprising, after the second device determining whether the second device has the capability to decode the first audio data based on the first audio parameters:

If the first audio data can be decoded, the second device creates an audio decoder according to the first audio parameters, and the audio decoder is used for decoding the audio data in the first file;

after the second device determines whether the second device has the capability of decoding the first display data according to the first display parameter, the method further comprises the following steps:

and if the first display data has the capability of decoding the first display data, the second device creates an image decoder according to the first display parameters, and the image decoder is used for decoding the display data in the first file.

146. An apparatus recommendation method, comprising:

the method comprises the steps that first equipment receives first operation of opening a first service by a user;

responding to the first operation, the first device respectively acquires device parameters associated with the first service from N electronic devices, wherein the N electronic devices and the first device are located in the same communication network, and N is an integer greater than 0;

the first equipment predicts the service quality of the first service executed by each electronic equipment in the N pieces of electronic equipment according to the equipment parameters;

and the first device displays a device list according to the service quality of the first service executed by each of the N electronic devices, wherein the device list includes a device identifier of at least one of the N electronic devices.

147. The method of claim 146, wherein a second device is included in the N electronic devices; the device parameters acquired by the first device from the second device comprise M parameters, wherein M is an integer greater than 0;

the predicting, by the first device, the quality of service of the first service executed by the second device according to the device parameter includes:

the first equipment determines M scores corresponding to the M parameters one by one;

the first equipment calculates the service quality score of the second equipment for executing the first service according to the M scores; the higher the quality of service score, the higher the quality of service of the first service.

148. The method of claim 147, wherein a third device is further included in the N electronic devices;

when the service quality score of the second device executing the first service is higher than the service quality score of the third device executing the first service, the second device is arranged in the device list in the front of the third device.

149. The method of claim 147,

when the service quality score of the second device executing the first service is higher than the service quality scores of other electronic devices executing the first service in the N electronic devices, the second device is marked as a recommended device in the device list.

150. The method of claim 147, wherein said M parameters include a first parameter corresponding to a first function;

when the score of the first parameter of the second device is higher than the scores of the first parameters of other electronic devices in the N pieces of electronic devices, the device list comprises first prompt information, and the first prompt information is used for recommending a user to use the second device to realize the first function.

151. The method of claim 147, wherein said M parameters include a first parameter corresponding to a first function;

when the score of the first parameter of the second device is lower than the scores of the first parameters of other electronic devices in the N pieces of electronic devices, the device list includes second prompt information, and the second prompt information includes a reason why the user is not recommended to use the second device to realize the first function.

152. The method as recited in any one of claims 146-151, wherein the N electronic devices comprise a second device;

after the first device displays the device list of the service quality of the first service according to each electronic device in the N electronic devices, the method further includes:

The first device receives a second operation that a user selects the second device in the device list;

and responding to the second operation, and switching the first service to the second device by the first device for execution.

153. The method of claim 152, wherein after the handover of the first service by the first device to the second device is performed, further comprising:

the first device respectively acquires device parameters associated with the first service from the N pieces of electronic equipment;

the first equipment predicts the service quality of the first service executed by each electronic equipment in the N pieces of electronic equipment according to the equipment parameters;

when the second device is not the electronic device with the highest service quality for executing the first service in the N electronic devices, the first device displays a first prompt message, where the first prompt message is used to prompt a user to execute the first service by using a recommendation device, and the recommendation device is one or more of the N electronic devices.

154. The method of claim 153, wherein the first device predicting the quality of service for each of the N electronic devices to perform the first service based on the device parameters comprises:

And the first equipment calculates the service quality score of each electronic equipment for executing the first service according to the equipment parameters acquired from each electronic equipment.

155. The method of claim 154, wherein a third device is included in the N electronic devices;

when the service quality score of the first service executed by the third device is higher than that of other electronic devices in the N electronic devices, the recommending device is the third device; alternatively, the first and second electrodes may be,

and when the service quality score of the first subtask executed by the third device in the first service is higher than that of other electronic devices in the N electronic devices, the recommendation device is the third device.

156. The method of claim 154, wherein the first transaction comprises a first subtask and a second subtask, and wherein the N electronic devices comprise a third device and a fourth device;

and when the service quality score of the third device executing the first subtask is higher than that of other electronic devices in the N electronic devices, and the service quality score of the fourth device executing the second subtask is higher than that of other electronic devices in the N electronic devices, the recommendation device is the third device and the fourth device.

157. The method as recited in any one of claims 154-156, further comprising:

the first device predicts a quality of service score for the first device to perform the first service;

when the service quality score of the first service executed by the first device is higher than that of the N electronic devices, the recommending device is the first device.

158. The method as claimed in any one of claims 153-157, wherein the first prompting message includes a device identifier of the recommending device;

after the first device displays the first prompt message, the method further comprises the following steps:

the first device receives a third operation that the user selects the recommendation device in the first prompt message;

and responding to the third operation, and the first equipment switches the first service to the recommendation equipment for execution.

159. The method as claimed in any of claims 153-158, wherein when the second device is not the highest quality electronic device of the N electronic devices for executing the first service, the method further comprises:

and the first equipment informs the second equipment of the recommendation equipment so as to enable the second equipment to display a second prompt message, wherein the second prompt message is used for prompting the user to execute the first service by using the recommendation equipment.

160. The method as recited in any one of claims 146-159, wherein the device parameters comprise one or more of audio parameters, display parameters, camera parameters, or time delays.

161. An ultrasonic positioning method, comprising:

responding to the operation of opening a screen projection function by a user, detecting K electronic devices which are accessed to the same communication network with the first device by the first device, wherein K is an integer larger than 1;

the first equipment carries out ultrasonic positioning on second equipment to obtain a first positioning result of the second equipment, wherein the second equipment is electronic equipment in the K pieces of electronic equipment;

the first equipment carries out ultrasonic positioning on third equipment to obtain a second positioning result of the third equipment, wherein the third equipment is the electronic equipment except the second equipment in the K pieces of electronic equipment;

and the first device displays the identifier of the second device and the identifier of the third device in an interactive interface, wherein the position of the identifier of the second device in the interactive interface is related to the first positioning result, and the position of the identifier of the third device in the interactive interface is related to the second positioning result.

162. The method of claim 161, wherein the first device ultrasonically locates the second device, comprising:

the first equipment sends a first positioning instruction to the second equipment, and the first positioning instruction is used for triggering the second equipment to transmit a first ultrasonic signal;

and the first equipment positions the second equipment according to the time of the first ultrasonic signal reaching N ultrasonic receivers in the first equipment respectively, wherein N is an integer greater than 1.

163. The method of claim 161 or 162, wherein the first device ultrasonically locates a third device, comprising:

the first equipment sends a second positioning instruction to the third equipment, and the second positioning instruction is used for triggering the third equipment to transmit a second ultrasonic signal;

the first equipment acquires first ultrasonic data, wherein the first ultrasonic data comprise the time for the second ultrasonic signals to respectively reach N ultrasonic receivers in the first equipment, and N is an integer greater than 1;

the first equipment acquires second ultrasonic data, wherein the second ultrasonic data comprise the time for the second ultrasonic signal to respectively reach M ultrasonic receivers in the second equipment, and M is an integer greater than 1;

The first device locates the third device according to the first ultrasonic data and the second ultrasonic data.

164. The method of claim 163, further comprising, prior to the first device sending a second positioning instruction to the third device:

the first device determines the second device as a cooperative device among the K electronic devices, and the cooperative device is used for performing ultrasonic positioning in cooperation with the first device;

the first device sends a first co-location instruction to the second device, where the first co-location instruction is used to trigger the second device to detect the second ultrasonic signal using the M ultrasonic receivers.

165. The method of claim 164, wherein the first device determines the second device as a cooperating device among the K electronic devices, comprising:

and when the second device is the electronic device with the highest positioning capability in the K electronic devices, the first device determines the second device as the cooperative device.

166. The method of any one of claims 163-165 wherein the first device locating the third device from the first ultrasonic data and the second ultrasonic data comprises:

And the first equipment positions the third equipment according to the first positioning result of the second equipment, the first ultrasonic data and the second ultrasonic data.

167. The method as claimed in any one of claims 163-165 wherein the N ultrasonic receivers are N microphones in a microphone array of the first device; and/or the M ultrasonic receivers are M microphones in a microphone array of the second device;

wherein the first device acquires first ultrasound data, comprising:

the first equipment detects the second ultrasonic signals by using the N microphones to obtain N paths of second ultrasonic signals;

wherein the first device acquires second ultrasound data, comprising:

the first device acquires M paths of second ultrasonic signals detected by the second device by using the M microphones.

168. The method of claim 167, wherein the first device locates the third device from the first ultrasound data and the second ultrasound data, comprising:

the first equipment combines the N paths of second ultrasonic signals and the M paths of second ultrasonic signals into multi-channel ultrasonic data;

And the first equipment uses a preset positioning algorithm to position the third equipment according to the multi-channel ultrasonic data.

169. The method of claim 161 or 162, wherein the first device ultrasonically locates a third device, comprising:

the first equipment sends a second positioning instruction to the third equipment, and the second positioning instruction is used for triggering the third equipment to transmit a second ultrasonic signal;

the first equipment acquires a first ultrasonic positioning result, wherein the first ultrasonic positioning result is determined by the first equipment according to the time for the second ultrasonic signal to reach N ultrasonic receivers in the first equipment, and N is an integer greater than 1;

the first equipment acquires a second ultrasonic positioning result, wherein the second ultrasonic positioning result is determined by the second equipment according to the time for the second ultrasonic signal to reach M ultrasonic receivers in the second equipment, and M is an integer greater than 1;

and the first equipment positions the third equipment according to the first ultrasonic positioning result and the second ultrasonic positioning result.

170. The method of claim 169, wherein the N ultrasonic receivers are N microphones in a microphone array of the first device;

Wherein the first device obtains a first ultrasound localization result, comprising:

the first device detecting the second ultrasonic signal using the N microphones;

and the first equipment positions the third equipment according to the time when each microphone in the N microphones detects the second ultrasonic signal, so as to obtain the first ultrasonic positioning result.

171. The method as recited in any one of claims 161-170, wherein the K electronic devices further comprise a fourth device; the method further comprises the following steps:

the first equipment carries out ultrasonic positioning on the fourth equipment to obtain a third positioning result of the fourth equipment;

and the first device displays the identifier of the second device, the identifier of the third device and the identifier of the fourth device in the interactive interface, wherein the position of the identifier of the second device in the interactive interface is related to the first positioning result, the position of the identifier of the third device in the interactive interface is related to the second positioning result, and the position of the identifier of the fourth device in the interactive interface is related to the third positioning result.

172. The method of claim 171, wherein the first device ultrasonically locates the fourth device, comprising:

the first equipment sends a third positioning instruction to the fourth equipment, and the third positioning instruction is used for triggering the fourth equipment to transmit a third ultrasonic signal;

the first equipment acquires third ultrasonic data, wherein the third ultrasonic data comprises the time for the third ultrasonic signal to respectively reach N ultrasonic receivers in the first equipment;

the first equipment acquires fourth ultrasonic data, wherein the fourth ultrasonic data comprises the time of the third ultrasonic signal reaching M ultrasonic receivers in the second equipment respectively;

the first device locates the fourth device according to the third ultrasonic data and the fourth ultrasonic data.

173. The method of claim 172, further comprising, after said first device sends a third positioning instruction to said fourth device:

the first equipment acquires fifth ultrasonic data, wherein the fifth ultrasonic data comprise time for the third ultrasonic signal to respectively reach Z ultrasonic receivers in the third equipment, and Z is an integer greater than 1;

Wherein the first device locates the fourth device according to the third ultrasonic data and the fourth ultrasonic data, including:

the first device locates the fourth device according to the third ultrasonic data, the fourth ultrasonic data, and the fifth ultrasonic data.

174. The method of claim 173, further comprising, prior to the first device sending a third positioning instruction to the fourth device:

the first device determines the second device and the third device as cooperative devices;

when the first device sends a third positioning instruction to the fourth device, the method further includes:

the first device sends a second co-location instruction to the second device and the third device, the second co-location instruction being configured to trigger the second device to detect the third ultrasonic signal using the M ultrasonic receivers, and the second co-location instruction being configured to trigger the third device to detect the third ultrasonic signal using the Z ultrasonic receivers.

175. The method as recited in any one of claims 161-174, further comprising, after the first device displays the identifier of the second device and the identifier of the third device in an interactive interface, the steps of:

In response to an operation that a user selects an identifier of the second device, the first device projects content played by the first device to the second device for playing; alternatively, the first and second electrodes may be,

in response to an operation that the user selects the identifier of the third device, the first device projects the content played by the first device to the third device for playing.

176. An electronic device, wherein the electronic device is a first device, the first device comprising:

a display screen;

one or more processors;

a memory;

a communication module;

wherein the memory has stored therein one or more computer programs, the one or more computer programs comprising instructions, which when executed by the first device, cause the first device to perform the method of any of claims 1-175.

177. An electronic device, wherein the electronic device is a second device, the second device comprising:

one or more processors;

a memory;

a communication module;

wherein the memory has stored therein one or more computer programs, the one or more computer programs comprising instructions, which when executed by the second device, cause the second device to perform the method of any of claims 1-175.

178. A computer-readable storage medium having instructions stored therein, which when executed on an electronic device, cause the electronic device to perform the method of any of claims 1-175.

179. A computer program product comprising instructions for causing an electronic device to perform the method of any one of claims 1-175 when the computer program product is run on the electronic device.

Images