WO2022233237A1

WO2022233237A1 - Audio playing method and apparatus, and device

Info

Publication number: WO2022233237A1
Application number: PCT/CN2022/088205
Authority: WO
Inventors: 陈全全
Original assignee: 华为技术有限公司
Priority date: 2021-05-07
Filing date: 2022-04-21
Publication date: 2022-11-10
Also published as: CN115314584A

Abstract

Disclosed in the present application are an audio playing method, apparatus and system. The method comprises: when a terminal device starts a screen mirroring service, a first management module receiving a first request sent by a screen mirroring service module, wherein the first request is used for requesting the creation of a virtual audio device, and the first request comprises relevant information of a first device to which a user expects to perform screen mirroring; the first management module sending the first request to a hardware abstraction layer (HAL); receiving a device identifier, which is sent by the HAL, of a first virtual audio device; and the first management module sending the device identifier of the first virtual audio device to the screen mirroring service module, such that the screen mirroring service module establishes a binding relationship between a virtual display and the first virtual audio device. By means of the method, the audio input/output capability in an existing screen mirroring scenario is enhanced, such that when a virtual display runs an application, an audio stream is input and output following an audio device at a screen mirroring end, and in a one-to-may screen mirroring scenario, screen mirroring devices display audio in a manner independent of each other, and do not interfere with each other.

Description

An audio playback method, device and device

This application claims the priority of the Chinese patent application with the application number 202110497072.1 and the title of the invention "An audio playback method, device and equipment", which was submitted to the State Intellectual Property Office of China on May 7, 2021, the entire contents of which are incorporated by reference in this application.

technical field

The present application relates to the field of media playback, and in particular, to an audio playback method, apparatus and device.

Background technique

At present, Android (Android) mobile phone terminals can realize the screen projection of the mobile phone to different terminals through a universal serial bus (universal serial bus, USB) cable or wireless fidelity (WiFi). For example, the mobile phone projects the audio/video content displayed on the mobile phone to other terminal devices through the USB DisplayPort (DP) interface, or realizes the mobile phone-to-tablet computer, personal computer (personal computer, personal computer) through Miracast or the multi-screen collaboration function of the manufacturer. PC), large-screen monitors and other devices to cast the screen.

Usually, the screen projection business of Android terminals mostly adopts the Android Virtual Display technology, and the mobile terminal has the ability to build multiple virtual displays. With the development of subsequent screen projection technologies, mobile terminals can be used as multi-device screen projection servers to implement one-to-many device projection services, and each device in different devices corresponds to a virtual display. For example, the mobile phone projects the screen to a display terminal through the USB DP interface, and the mobile phone also projects the screen to another terminal device, such as a smart screen, through the multi-screen collaboration function, so that user A can use the display terminal to work through the mobile phone, and at the same time The mobile phone can also be projected to the smart screen to meet the entertainment needs of home user B.

As shown in Figure 1, the user holds a mobile phone, and the mobile phone is connected to a display screen with a speaker through a USB DP cable. For example, a laptop computer enters the office mode. At the same time, the mobile phone can also cast the screen to the smart screen to play video or voice calls. In the screen scenario, since Android is a multi-tasking system, the audio system is a competitive resource. The application follows the Android audio focus calling mechanism. Before playing audio, you need to apply for the focus on the system side. Only the application that applies for the focus can play audio. , pauses playback once it loses focus. If the calling mechanism of the audio focus is followed, in the scenario shown in Figure 1, the sound of the mobile phone can only be sounded on one screen-casting device. For example, if the sound is selected on the smart screen by default, the sound will be paused on the laptop, and then As a result, the system cannot provide simultaneous sound on audio playback APPs running on different displays, that is, the current Android audio system only supports sound playback on one screen-casting device at the same time, and only corresponds to one playback application. In many screencasting scenarios, users have different audio and video demands.

SUMMARY OF THE INVENTION

The present application provides an audio playback method, device, and device, which are used to solve the problem that audio multi-channel concurrent control input and output cannot be supported in the scenario of virtual display one-to-multi-channel screen projection. Specifically, the application discloses the following technical solutions:

In a first aspect, the present application provides an audio playback method, which can be applied to a first management module. The method includes: when a terminal device starts a screencasting service, the first management module receives the first information sent by the screencasting service module. a request, the first request is used to request the creation of a virtual audio device, and the first request includes information about the first device that the user wishes to screen; the first management module sends the first device to the hardware abstraction layer HAL a request, receiving the device identifier of the first virtual audio device sent by the HAL, the first virtual audio device is created according to the relevant information of the first device; the first management module sends the screen casting service module The device identifier of the first virtual audio device, so that the screen projection service module establishes a binding relationship between the virtual display screen display and the first virtual audio device.

Optionally, the first management module is a module in the application framework layer, such as a multi-screen audio management module, or a vendor audio expansion module.

The method provided in this aspect enhances the audio input and output capability of the existing screen projection scene. The first management module establishes a virtual display for each device requesting screen projection, configures an independent virtual audio input and output device based on the virtual display, and establishes a virtual display and a virtual display. The binding relationship between audio devices enables the audio stream to follow the input and output of the audio device on the screen-casting end when the virtual display runs the application. In the one-to-many screen-casting scenario, the display audio of each screen-casting device is independent of each other and does not interfere with each other.

In addition, each screencasting device runs the application to play the sound, and makes a sound on the current screencasting device. The recording of each screencasting device running the application can be recorded by using the recording function of the current screencasting device, so as to realize the simultaneous sound and sound of the multitasking audio device. pickup.

With reference to the first aspect, in a possible implementation manner of the first aspect, the method further includes: the first management module receives a first binding relationship sent by the screen projection service module, where the first binding relationship includes The correspondence between the first virtual display screen display and the device identifier of the first virtual audio device; when the first application requests to play an audio stream, the first management module determines the first virtual display; according to the first The virtual display searches for the device identification of the first virtual audio device in the first binding relationship; sends the device identification of the first virtual audio device to the native audio framework, so that the native audio framework passes the corresponding audio stream through The first virtual audio device is output to the first device.

In this implementation manner, the first binding relationship between the first virtual display screen display and the device identifier of the first virtual audio device is obtained in advance, so that when the first virtual display is obtained, the first binding relationship can be found according to the first binding relationship. The device identifier of the corresponding first virtual audio device, so as to transmit the audio stream to the first device through the first virtual audio device, so as to realize the playback of the audio stream on the first device.

With reference to the first aspect, in another possible implementation manner of the first aspect, determining the first virtual display by the first management module includes: the first management module acquires the identifier of the first application, and sends the identifier to the second management module The identifier of the first application; the first management module receives the first virtual display determined by the second management module according to the identifier of the first application.

With reference to the first aspect, in another possible implementation manner of the first aspect, after the first management module determines the first virtual display, the method further includes: the first management module determines the first virtual display according to the first virtual display. an audio focus, the first audio focus corresponds to the first virtual display; the first management module queries whether the state of the first audio focus is available; if so, sends the information to the native audio framework the first audio focus. This mode implements the distribution of audio focus, and is used to provide audio focus for the device requesting to play the audio stream.

With reference to the first aspect, in yet another possible implementation manner of the first aspect, the method further includes: if the state of the first audio focus is unavailable for querying, determining that the application occupying the first audio focus is the second application ; the first management module sends first indication information to the native audio framework, where the first indication information is used to notify the native audio framework that the first audio focus needs to be occupied by the first application; when receiving When the second application stops using the first audio focus, sending the first audio focus to the native audio framework, so that the first application acquires the first audio focus.

In this implementation, when two applications apply for the same audio focus, a corresponding first audio focus is allocated to the second application according to the virtual display audio focus mechanism, so as to optimize the audio focus preemption mechanism of applications running on different virtual displays, and support based on the virtual display audio focus mechanism. Multitasking audio concurrency control for virtual displays.

With reference to the first aspect, in yet another possible implementation manner of the first aspect, before the first management module inquires whether the state of the first audio focus is available, the method further includes: acquiring, by the first management module, the first The first focus management record associated with the virtual display, the first focus management record includes the state of the first audio focus, wherein the state of the first audio focus includes idle and in use, and the first virtual display uniquely corresponds to the the first audio focus.

The first management module inquiring whether the state of the first audio focus is available includes: the first management module inquiring, in the first focus management record, whether the state of the first audio focus is idle.

With reference to the first aspect, in yet another possible implementation manner of the first aspect, the method further includes: the first management module maintains the first focus management record according to a virtual display audio focus mechanism; the virtual display audio focus mechanism Including: when there are two or more applications in the same virtual display, and when the two or more applications apply for audio focus at the same time, the application of the high-priority service preempts the application of the low-priority service, and, equal Priority business applications follow the rules of applying for applications later and preempting applications first.

In this implementation manner, the state of each audio focus is managed and recorded by the first management module, so that when a terminal device applies for a focus, the focus can be allocated to different terminal devices, thereby realizing the rational allocation and use of the audio focus and improving the audio frequency. Focus on resource distribution efficiency.

In a second aspect, the present application also provides an audio playback method, the method is applied to the second management module, and the method includes:

The second management module receives a second request sent by the screen projection service module, where the second request is used to create a virtual display for screen projection, wherein the second request includes: specifying the name, size and resolution information of the virtual display; The second management module creates a first virtual display according to the specified virtual display name, size and resolution information; the second management module sends the first virtual display to the screen projection service module, so that the virtual display The screen projection service module establishes a first binding relationship between the first virtual display and the first virtual audio device.

Optionally, the second management module is a module in the application framework layer, such as a multi-screen management module.

With reference to the second aspect, in a possible implementation manner of the second aspect, the method further includes: the second management module receives the first binding relationship sent by the screen projection service module; the second management module Send the first binding relationship to the first management module.

With reference to the second aspect, in another possible implementation manner of the second aspect, the method further includes: when the first application requests to play an audio stream, the second management module acquires the identifier of the first application; the second management module The module searches for the first virtual display to which the first application belongs according to the identifier of the first application; the second management module sends the first virtual display to the first management module.

With reference to the second aspect, in yet another possible implementation manner of the second aspect, the second management module acquiring the identifier of the first application includes: the second management module requests the first application to apply for audio focus When the audio focus request is sent by the first management module, the audio focus request includes the identifier of the first application.

With reference to the second aspect, in yet another possible implementation manner of the second aspect, the second management module acquiring the identifier of the first application includes: the second management module acquiring the first application in the native audio framework When playing the audio stream, the identifier of the first application sent by the first management module is received.

In a third aspect, the present application further provides an audio playback method, which can be applied to a screencasting service module, the method comprising: when the screencasting service module acquires an operation of a user initiating a screencasting service, acquiring the first file created by the second management module. A virtual display screen display; the screen projection service module sends a first request to the first management module after establishing a virtual audio link with the first device that expects screen projection, and the first request is used to request the creation of a virtual display Audio equipment, the first request includes the relevant information of the first equipment that the user wishes to screen; the screen casting service module receives the device identification of the first virtual audio equipment sent by the first management module; establishes a first binding The first binding relationship includes the corresponding relationship between the device identifiers of the first virtual display and the first virtual audio device; the screen projection service module sends the first management module the The first binding relationship.

In the method provided by this aspect, the screen projection service module establishes a binding relationship between the first virtual display screen display and the first virtual audio device according to the received device identifiers, so as to provide subsequent selection of virtual audio devices and distribution of audio streams. Provides convenience so that when the virtual display runs the application, the audio stream follows the input and output of the audio device on the screen-casting end, so that in the one-to-many screen-casting scenario, the audio displayed by each screen-casting device is independent of each other and does not interfere with each other.

Optionally, the screen projection service module may be a functional module located at the application layer. For example, the screencasting service APP, or the screencasting service APK package.

With reference to the third aspect, in a possible implementation manner of the third aspect, the acquiring the first virtual display screen display created by the second management module includes: the screen projection service module sends a message to the second management module A second request, where the second request is used to request the second management module to create a virtual display; the screen projection service module receives the first virtual display created by the second management module according to the second request.

With reference to the third aspect, in another possible implementation manner of the third aspect, the method further includes: when the user's operation of disconnecting the screen-casting service is obtained, releasing the first virtual display and the first virtual audio device. The first binding relationship between device identifiers; the screen projection service module sends a first message to the first management module, where the first message is used to notify the first management module to delete the first binding relation.

In a fourth aspect, the present application further provides a management device, which can be applied to a first management module, where the first management module includes: a first receiving unit, configured to receive a screen projection when a terminal device starts a screen projection service The first request sent by the service module, the first request is used to request the creation of a virtual audio device, and the first request includes the relevant information of the first device that the user expects to screen; the first sending unit is used to abstract from the hardware The layer HAL sends the first request; the first receiving unit is further configured to receive the device identifier of the first virtual audio device sent by the HAL, and the first virtual audio device is based on the relevant information of the first device. Create; the first sending unit is further configured to send the device identification of the first virtual audio device to the screen projection service module, so that the virtual screen projection service module establishes a virtual display screen display and the first Binding relationship between virtual audio devices.

With reference to the fourth aspect, in a possible implementation manner of the fourth aspect, the first receiving unit is further configured to receive a first binding relationship sent by the screen projection service module, the first binding relationship including the correspondence between the first virtual display screen display and the device identifier of the first virtual audio device; the first processing unit is used to determine the first virtual display when the first application requests to play the audio stream, according to the first processing unit A virtual display searches for the device identifier of the first virtual audio device in the first binding relationship; the first sending unit is further configured to send the device identifier of the first virtual audio device to the native audio framework, so that the The native audio framework outputs the corresponding audio stream to the first device through the first virtual audio device.

With reference to the fourth aspect, in another possible implementation manner of the fourth aspect, the first receiving unit is further configured to acquire the identifier of the first application; the first sending unit is further configured to send the second management The module sends the identifier of the first application; the first receiving unit is further configured to receive the first virtual display determined by the second management module according to the identifier of the first application.

With reference to the fourth aspect, in yet another possible implementation manner of the fourth aspect, the first processing unit is further configured to determine a first audio focus according to the first virtual display, where the first audio focus is related to the Corresponding to the first virtual display, query whether the state of the first audio focus is available; the first sending unit is further configured to, when the first processing unit queries the availability of the first audio focus, to the The native audio framework sends the first audio focus.

With reference to the fourth aspect, in yet another possible implementation manner of the fourth aspect, the first processing unit is further configured to determine that the first audio focus is occupied when the state of the first audio focus is inquired to be unavailable The application in question is the second application; the first sending unit is further configured to send first indication information to the native audio framework, where the first indication information is used to notify the native audio framework that the first audio focus needs to be being occupied by the first application; and, when the first receiving unit receives an indication that the second application stops using the first audio focus, sending the first audio focus to the native audio framework, so that the first application acquires the first audio focus.

With reference to the fourth aspect, in yet another possible implementation manner of the fourth aspect, before the first management module queries whether the state of the first audio focus is available, the method further includes: the first processing unit, further using obtaining the first focus management record associated with the first virtual Display, and querying the first focus management record whether the state of the first audio focus is idle.

Wherein, the first focus management record includes the state of the first audio focus, the state of the first audio focus includes idle and in use, and the first virtual display uniquely corresponds to the first audio focus.

With reference to the fourth aspect, in yet another possible implementation manner of the fourth aspect, the first processing unit is further configured to maintain the first focus management record according to a virtual display audio focus mechanism; the virtual display audio focus The mechanism includes: when there are two or more applications in the same virtual display, and when the two or more applications apply for audio focus at the same time, the application of the high-priority service preempts the application of the low-priority service, and, Business applications with the same priority follow the rules of applying for applications later and preempting applications first.

In a fifth aspect, the present application further provides another management device, which can be applied to a second management module, where the second management module includes: a second receiving unit configured to receive a second request sent by the screen projection service module, the The second request is used to create a screen-casting virtual display, wherein the second request includes: specifying the name, size, and resolution information of the virtual display; and a second processing unit, configured to specify the name, size, and resolution of the virtual display according to the specified rate information to create a first virtual display; a second sending unit, configured to send the first virtual display to the screen-casting service module, so that the virtual screen-casting service module establishes the first virtual display and the first virtual display The first binding relationship between audio devices.

With reference to the fifth aspect, in a possible implementation manner of the fifth aspect, the second receiving unit is further configured to receive the first binding relationship sent by the screen projection service module; the second sending The unit is further configured to send the first binding relationship to the first management module.

With reference to the fifth aspect, in another possible implementation manner of the fifth aspect, the second receiving unit is further configured to, when the first application requests to play an audio stream, the second management module to obtain the first application's identifier; the second processing unit is further configured to search for the first virtual display to which the first application belongs according to the identifier of the first application; the second sending unit is further configured to send the information to the first management module Describe the first virtual display.

With reference to the fifth aspect, in yet another possible implementation manner of the fifth aspect, the second receiving unit is further configured to receive the data sent by the first management module when the first application requests to apply for audio focus An audio focus request, where the audio focus request includes the identifier of the first application.

With reference to the fifth aspect, in yet another possible implementation manner of the fifth aspect, the second receiving unit is further configured to receive the first audio stream when the native audio framework acquires the audio stream played by the first application The identifier of the first application sent by the management module.

In a sixth aspect, the present application further provides an application device, the application device can be a screen projection service module, and the screen projection service module includes: a third receiving unit for acquiring the operation of the user to start the screen projection service , obtains the first virtual display screen display created by the second management module; the third sending unit is configured to send a first request to the first management module after establishing a virtual audio link with the first device that expects screen projection, The first request is used to request the creation of a virtual audio device, and the first request includes information about the first device that the user expects to screen; the third receiving unit is further configured to receive a message sent by the first management module The device identifier of the first virtual audio device; the third processing unit is used for the screen projection service module to establish a first binding relationship, and the first binding relationship includes the first virtual display and the first virtual Correspondence between device identifiers of audio devices; the third sending unit is further configured to send the first binding relationship to the first management module.

With reference to the sixth aspect, in a possible implementation manner of the sixth aspect, the third sending unit is further configured to send a second request to the second management module, where the second request is used to request the The second management module creates a virtual display; the third receiving unit is further configured to receive the first virtual display created by the second management module according to the second request.

With reference to the sixth aspect, in another possible implementation manner of the sixth aspect, the third processing unit is further configured to release the first virtual display and all the the first binding relationship between the device identifiers of the first virtual audio device; the third receiving unit is further configured to send a first message to the first management module, where the first message is used to notify the The first management module deletes the first binding relationship.

In a seventh aspect, the present application further provides a terminal device, comprising: at least one processor and a memory, wherein the memory is used to store computer program instructions and/or data; the at least one processor is used to execute the The above-mentioned instructions are used to implement the methods in the various implementation manners of the first aspect to the third aspect.

Optionally, the terminal device further includes an interface circuit, where the interface circuit is configured to acquire computer program instructions and/or data in the memory.

Optionally, the at least one processor, the memory and the interface circuit may be integrated into one processing chip or chip circuit.

Optionally, the apparatus is a terminal device, and the terminal device includes but is not limited to a mobile phone, a PC, and a tablet computer.

In an eighth aspect, the present application further provides an audio playback system, the system includes a source device and at least one screen projection device, wherein the source device is the terminal device of the aforementioned seventh aspect, which is used to implement the aforementioned first aspect Methods in various implementations to the third aspect. In addition, the at least one screen projection device may also be the terminal device in the foregoing seventh aspect.

Optionally, the system may further include other devices, such as cloud servers, network devices, and the like.

In a ninth aspect, the present application also provides a computer-readable storage medium, where instructions are stored in the storage medium, so that when the instructions are executed on a computer or a processor, they can be used to execute the foregoing first aspect and each of the first aspects. The method in one implementation manner, and the aforementioned second aspect and the method in various implementation manners of the second aspect, and the foregoing third aspect and the method in various implementation manners of the third aspect.

In addition, the present application also provides a computer program product, the computer program product includes computer instructions, when the instructions are executed by a computer or a processor, the aforementioned first aspect and the method in various implementation manners of the first aspect can be implemented, And, the aforementioned second aspect and methods in various implementations of the second aspect, and the aforementioned third aspect and methods in various implementations of the third aspect.

It should be noted that the beneficial effects corresponding to the technical solutions of the various implementation manners of the second aspect to the ninth aspect are the same as the beneficial effects of the foregoing first aspect and various implementation manners of the first aspect. For details, please refer to the foregoing first aspect. Aspects and descriptions of beneficial effects in various implementation manners of the first aspect will not be repeated.

Description of drawings

Fig. 1 is a scene schematic diagram of a mobile phone terminal screen projection provided by the present application;

FIG. 2 is a schematic diagram of a scenario of a mobile phone terminal screen projection service provided by an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a terminal device according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a software structure of a terminal device according to an embodiment of the present application;

5 is a flowchart of an audio playback method provided by an embodiment of the present application;

6 is a flowchart of another audio playback method provided by an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a source device projecting a screen to two screen projection devices according to an embodiment of the present application;

FIG. 8 is a signaling flowchart of an audio playback method provided by an embodiment of the present application;

FIG. 9 is a signaling diagram for starting a screen projection process according to an embodiment of the present application;

10 is a signaling diagram of a process of disconnecting screen projection provided by an embodiment of the present application;

11 is a schematic structural diagram of requesting audio focus according to an embodiment of the present application;

FIG. 12 is a signaling flow chart of an application acquiring audio focus according to an embodiment of the present application;

13 is a signaling flowchart of an audio focus preemption mechanism provided by an embodiment of the present application;

14a is a schematic diagram of an audio stream distribution provided by an embodiment of the present application;

FIG. 14b is a signaling flow chart of playing an audio stream provided by an embodiment of the application;

FIG. 15 is a schematic structural diagram of a management apparatus provided by an embodiment of the present application;

FIG. 16 is a schematic structural diagram of another management apparatus provided by an embodiment of the present application;

FIG. 17 is a schematic structural diagram of an application device provided by an embodiment of the present application;

FIG. 18 is a schematic structural diagram of a chip system provided by an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more clearly understood, the following describes the technical solutions in the embodiments of the present application with reference to the accompanying drawings. The program is described in further detail.

Before describing the technical solutions of the embodiments of the present application, the application scenarios of the embodiments of the present application are first described with reference to the accompanying drawings.

The technical solution of the present application can be applied to the one-to-many screen projection business scenario of mobile phone devices. Based on the Android virtual display screen (display), the virtualized technology is used to construct distributed audio, so as to realize multi-terminal screen projection display and independent audio input and output, and support audio Simultaneous vocalization and pickup.

In a possible scenario, such as shown in Figure 2, the mobile phone projects the screen to the smart screen, tablet and display screen through the Android Virtual Display (Android Virtual Display) technology. Each device sets up virtual displays, and assigns independent virtual audio devices to each virtual display. The virtual audio devices include speakers, microphones (MIC), and other input devices, etc., as well as multi-screen audio concurrent Management control, so that users can get an independent audio and video experience when using the screen projection display, so that the audio and video of each display are independent of each other and do not interfere with each other.

The mobile phone may also be other terminal equipment, and the terminal equipment may be a portable device, such as a smart phone, a tablet computer, a notebook computer, a personal computer (PC), a foldable terminal, a wireless communication function Wearable devices (such as smart watches or bracelets), user devices (user devices) or user equipment (UE), and augmented reality (AR) or virtual reality (VR) devices, headsets, etc. . In addition, the device to which the screen is projected may also be any one or more of the above-mentioned terminal devices, and the specific device form of the terminal device is not limited in this embodiment.

In addition, the above-mentioned various terminal devices are equipped with an Android (Android) operating system and/or an Android-compatible system.

In this embodiment, the device that projects the screen is called the source device, and the device that the user desires to project the screen is called the device that projects the screen. For example, in FIG. 2, the source device is a mobile phone terminal, and the screen projection device includes any terminal device such as a smart screen, a tablet, and a display screen.

Optionally, the above application scenarios may also include cloud side or cloud network devices, such as cloud servers, data centers, etc., and the source device and the screen projection device are connected through a wireless network, such as a wireless local area network (Wireless Local Area Networks, WLAN). ) or WiFi, etc.

In an implementation manner, the above-mentioned terminal device may include the following structure, as shown in FIG. 3 , which is a schematic structural diagram of a terminal device provided in this embodiment. The terminal device may include a processor 110 and a memory 120, and further include: a USB interface 130, a power management module 140, a battery 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, and a speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, buttons 191, indicator 192, camera 193, display screen 194, and subscriber identification module (Subscriber Identification Module, SIM) card interface 195 and the like.

The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an acceleration sensor 180C, a gravity sensor 180D, and a touch sensor 180E. In addition, the sensor module 180 may also include a fingerprint sensor, a temperature sensor, an ambient light sensor, and the like.

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the communication device. In other embodiments of the present application, the communication device may include more or less components than shown, or some components are combined, or some components are separated, or different components are arranged. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor ( image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 110 . If the processor 110 needs to use the instruction or data again, it can be directly called from the memory, which avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / or USB interface, etc.

Among them, the USB interface 130 is an interface that conforms to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the communication device, and can also be used to transmit data between the communication device and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices.

The power management module 140 is used for connecting the battery 141 and the processor 110 . The power management module 140 supplies power to the processor 110 , the memory 120 , the display screen 194 , the camera 193 , the wireless communication module 160 and the like. In some embodiments, the power management module 140 may be provided in the processor 110 .

The wireless communication function of the terminal device may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, the baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the communication device may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.

The mobile communication module 150 may provide a wireless communication solution including 2G/3G/4G/5G etc. applied on the communication device. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA) and the like. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110 .

The wireless communication module 160 can provide applications on communication devices including WLAN (such as WiFi network), Bluetooth (bluetooth), global navigation satellite system (GNSS), frequency modulation (frequency modulation, FM), short-range wireless communication Technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 can also receive the signal to be sent from the processor 110 , perform frequency modulation on it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2 .

In some embodiments, the antenna 1 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the terminal device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (GLONASS), and a Beidou navigation satellite system (BDS).

The display screen 194 is used to display conversation windows, display interfaces, images, and the like. In some embodiments, the terminal device may include 1 or N display screens, where N is a positive integer greater than 1.

Memory 120 may be used to store computer-executable program code, which includes instructions. In addition, data/information generated by the terminal device or interacted with other devices can also be stored.

Further, the memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like. The processor 110 executes various functional applications and data processing of the terminal device by executing the instructions stored in the internal memory 120 and/or the instructions stored in the memory provided in the processor.

The pressure sensor 180A is used to sense pressure signals, and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be provided on the display screen 194 . The gyroscope sensor 180B can be used to determine the motion attitude of the terminal device, such as measuring the rotation angle of the terminal device. The acceleration sensor 180C can detect the magnitude of the acceleration of the terminal device in various directions (generally three axes). The acceleration sensor 180C further includes a linear acceleration sensor for obtaining the linear acceleration of the terminal device. The gravity sensor 180D is used to measure the acceleration components of the terminal device in the x-axis, y-axis and z-axis directions. The touch sensor 180E is also called "touch device". The touch sensor 180E may be disposed on the display screen 194 , and the touch sensor 180E and the display screen 194 form a touch screen, also referred to as a "touch screen". The touch sensor 180E is used to detect a touch operation on or near it, such as a double-tap operation.

The keys 191 include a power-on key, a volume key, and the like. Keys 190 may be mechanical keys. It can also be a touch key. The indicator 192 can be an indicator light, which can be used to indicate the charging state, the change of the power, and can also be used to indicate a message, a missed call, a notification, and the like.

The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to achieve contact and separation with the communication device.

It should be noted that, the structure of the above-mentioned cloud side or cloud network device may be the same as or different from that of the terminal device, which is not limited in this embodiment.

In addition, this embodiment also provides a software structure of a terminal device. Taking the terminal device of the Android system as an example, the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. Referring to FIG. 4, in some embodiments, the Android system in the software structure of the terminal device is an application layer (APP), an application framework layer and a hardware abstraction layer (HAL) from top to bottom, and in addition , Android runtime (Android Runtime) and system library (not shown in FIG. 4 ) are also included between the program framework layer and the HAL layer.

Further, the application layer may include a series of application packages, such as APP1, APP2, . . . This embodiment includes a screen projection service APP, and the screen projection service APP is used to implement the screen projection function of different terminal devices.

Optionally, the application framework layer is referred to as a "framework layer (Framework)" for short.

The framework layer provides an application programming interface (Application Programming Interface, API) and a programming framework for applications in the application layer. The framework layer includes some predefined functions.

As shown in Figure 4, the framework layer includes: Android open source platform (Android open source platform, AOSP) native audio framework, multi-screen management (Multi-screen management) module and vendor (Vendor) audio extension function module. Wherein, the Vendor audio extension function module is a newly added function module.

Specifically, the functions/roles included in each module are shown in Table 1.

Table 1

The multi-screen management module, for the main screen and the virtual screen management module, maintains and manages the display information used by the screen projection business. In addition, the multi-screen management module is also associated with the audio framework to realize functions such as creation and destruction of virtual display screens (Display) and configuration management of virtual audio devices.

Audio framework, including AOSP native audio framework and Vendor audio extension function, wherein the Vendor audio extension function is a new function module, used to extend audio policy (AudoPolicy), media focus control (MediaFocusControl) and audio scheduling service (AudioFlinger) modules function to establish audio focus management and audio streaming channel policy control centered on virtual Display.

Further, the functions of the Vendor audio extension include at least the following new functions:

(1) AudioPolicyImp, which is used to implement virtual audio device policy control based on virtual Display;

(2) MediaFocusControlImpl, used to realize audio focus control based on virtual Display;

(3) AudioFlingerImpl, used to realize the audio stream path control based on virtual Display;

(4) Multi-screen audio management module, responsible for virtual Display docking virtual audio equipment management, the virtual audio equipment includes virtual Speaker equipment and virtual Mic equipment, is also used to establish the mapping relationship between the virtual Display and the audio equipment HAL, provides based on Virtual Display focus management.

Optionally, other managers may also be included in the framework layer, such as a window manager, a content provider, a view system, a telephony manager, a notification manager, and the like.

In addition, the Android Runtime includes a core library and a virtual machine. Android Runtime is responsible for scheduling and management of the Android system. Specifically, the core library consists of two parts: one part is the function functions that the java language needs to call, and the other part is the core library of Android.

The system library may include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files and the like. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing. 2D graphics engine is a drawing engine for 2D drawing.

The hardware abstraction layer HAL is an interface layer between the operating system kernel and the hardware circuit, and its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable on a variety of platforms. From the perspective of software and hardware testing, the testing of software and hardware can be completed based on the hardware abstraction layer, which makes it possible to perform parallel testing of software and hardware.

In this embodiment, the HAL includes a USB/DP Audio HAL, a primary audio (Primary Audio) HAL, and a virtual audio (Virtual Audio) HAL. Among them, the Virtual Audio HAL is a newly added functional module, and the Virtual Audio HAL can be used for docking with distributed virtual audio devices, and for docking with remote screen projection audio input and output devices. Specifically, the functions of each HAL module are shown in Table 2.

Table 2

The flow of the method provided in this embodiment will be described in detail below.

The embodiment of the present application provides an audio stream playback method, which is used to realize the screen projection display of multi-terminal devices, and the simultaneous sounding and pickup of audio in the scenario of a one-to-multiple screen projection service of a mobile phone device.

As shown in FIG. 5, an audio playback method provided by this embodiment includes:

101: When the terminal device starts the screencasting service, the first management module receives a first request sent by the screencasting service module, where the first request is used to request the creation of a virtual audio device, and the first request includes the user's desired screencasting related information of the first device, such as the identifier of the first device, the device name of the first device, and the like.

Wherein, the first management module is a multi-screen audio management module, located in the framework layer, or a new function module in the Vendor audio extension function.

102: The first management module sends the first request to the hardware abstraction layer HAL.

103: The first management module receives the device identifier of the first virtual audio device sent by the HAL, where the first virtual audio device is created according to the related information of the first device.

104: The first management module sends the device identification of the first virtual audio device to the screen projection service module, so that the screen projection service module establishes a connection between the virtual display screen display and the first virtual audio device. binding relationship.

Correspondingly, the screen projection service module receives the device identifier of the first virtual audio device, and then establishes a first binding relationship according to the acquired first virtual display, where the first binding relationship is the first virtual audio device The corresponding relationship between the device identifier and the first virtual display.

For example, the first virtual display is Virtual display0, the device identifier of the first virtual audio device is Virtual Audio 0, and the first binding relationship is established as <Virtual display0, Virtual Audio 0>.

It should be understood that when multiple applications request screen projection, the first management module receives the device identifiers requesting multiple virtual audio devices from the HAL, and establishes multiple binding relationships, each of which includes a virtual audio device. The correspondence between display and a virtual audio device.

In addition, as shown in Figure 6, the above also includes:

105: The first management module receives the first binding relationship sent by the screen projection service module.

106: When the first application requests to play the audio stream, the first management module determines the first virtual display.

Wherein, the first application has a music playback function, such as

APP.

In an implementation manner, the first management module acquires the identifier of the first application, such as the package name, pid, etc. of the first application, and then sends the identifier of the first application to the second management module. The second management module searches for the first virtual display to which the first application belongs according to the identifier of the first application, and sends the first virtual display to the first management module. The first management module receives the first virtual display determined by the second management module according to the identifier of the first application.

In addition, after step 106, the method further includes:

The first management module determines a first audio focus according to the first virtual display, and the first audio focus corresponds to the first virtual display; inquires whether the state of the first audio focus is available; if so, The first audio focus is then sent to the native audio framework.

If the state of the first audio focus is not available, determine that the application occupying the first audio focus is the second application; the first management module sends first indication information to the native audio framework, and the first An indication message is used to notify the native audio framework that the first audio focus needs to be occupied by the first application; when receiving an indication that the second application stops using the first audio focus, send a notification to the native audio framework. The audio framework sends the first audio focus so that the first application acquires the first audio focus.

Further, before the first management module inquires whether the state of the first audio focus is available, the method further includes: acquiring, by the first management module, a first focus management record associated with the first virtual display, the first focus The management record includes the state of the first audio focus, wherein the state of the first audio focus includes idle and in use, and the first virtual display uniquely corresponds to the first audio focus.

107: The first management module searches for the device identifier of the first virtual audio device in the first binding relationship according to the first virtual display.

108: The first management module sends the device identifier of the first virtual audio device to the native audio framework, so that the native audio framework outputs the corresponding audio stream to the first device through the first virtual audio device.

In addition, the above method further includes: the first management module maintains the first focus management record according to the virtual display audio focus mechanism;

The virtual display audio focus mechanism includes: there are two or more applications in the same virtual display, and when the two or more applications apply for audio focus at the same time, preempt the low priority according to the high priority service application Business applications, and business applications with the same priority follow the rule of applying for applications first and preempting applications later.

In addition, it also includes: each virtual display has an independent audio focus, and the audio focuses corresponding to different virtual displays are independent of each other and interfere with each other.

The method provided by this embodiment enhances the audio input and output capability of the existing screen projection scene. The first management module establishes a virtual display for each device requesting screen projection, configures an independent virtual audio input and output device based on the virtual display, and establishes the virtual display and the virtual display. The binding relationship between the virtual audio devices enables the audio stream to follow the input and output of the audio device on the screen-casting end when the virtual display runs the application. In the one-to-many screen-casting scenario, the display audio of each screen-casting device is independent of each other and does not interfere with each other. .

In addition, this embodiment also provides another audio playback method, which can be applied to a second management module, where the second management module is a multi-screen management module. Specifically, the method includes:

The second management module receives a second request sent by the screen projection service module, where the second request is used to create a virtual display for screen projection, wherein the second request includes: specifying the name, size and resolution information of the virtual display; The second management module creates a first virtual display according to the specified virtual display name, size and resolution information; and sends the first virtual display to the screen projection service module, so that the virtual screen projection service module A first binding relationship between the first virtual display and the first virtual audio device is established.

In addition, the above method further includes: the second management module receives the first binding relationship sent by the screen projection service module, and sends the first binding relationship to the first management module.

Wherein, in a possible implementation manner, acquiring the identifier of the first application by the second management module includes: the second management module receives the first management module when the first application requests to apply for audio focus The audio focus request sent by the module includes the identifier of the first application.

This embodiment builds an audio focus control mechanism centered on a virtual display, assigns a focus to each virtual display, performs audio focus preemption management based on the virtual display where the application is located, and supports coexistence of audio focus and concurrent management of audio playback for different virtual display applications.

Optionally, in another possible implementation manner, the second management module acquiring the identifier of the first application includes: when the second management module acquires the audio stream played by the first application when the native audio framework acquires , receiving the identifier of the first application sent by the first management module.

In addition, for the screen projection service module, the audio playback method provided by this embodiment includes:

The screen projection service module acquires the first virtual display screen display created by the second management module when acquiring the user's operation of starting the screen projection service; the screen projection service module establishes a virtual audio link with the first device that expects screen projection After the route is completed, a first request is sent to the first management module, where the first request is used to request the creation of a virtual audio device, and the first request includes relevant information of the first device on which the user wishes to project the screen.

The screen projection service module receives the device identifier of the first virtual audio device sent by the first management module; the screen projection service module establishes a first binding relationship, and the first binding relationship includes the first virtual audio device. display the corresponding relationship between the device identifier of the first virtual audio device; and send the first binding relationship to the first management module.

The acquiring the first virtual display screen display created by the second management module includes: the screen projection service module sends a second request to the second management module, where the second request is used to request the second The management module creates a virtual display; the screen projection service module receives the first virtual display created by the second management module according to the second request.

When the screen-casting service is ended, the method further includes: when the user's operation of disconnecting the screen-casting service is obtained, releasing the first binding relationship between the device identifiers of the first virtual display and the first virtual audio device; The screen projection service module sends a first message to the first management module, where the first message is used to notify the first management module to delete the first binding relationship. Specifically, the first virtual audio device is deleted through the first management module, and the first virtual audio device is deleted through the HAL.

The methods provided by the above embodiments will be described in detail below with reference to specific examples.

Referring to FIG. 7 , the audio playback system provided in this embodiment includes one source device and two screen projection devices, wherein the source device is a mobile phone 100 , and the screen projection device includes a first device and a second device. Optionally, the first device is the smart screen 200 and the second device is the tablet 300. The following takes the mobile phone 100 and the smart screen 200 as examples to introduce the screen casting process from the source device to the screen casting device, and the establishment of the screen casting connection. The method flow of the post audio stream is transmitted from the mobile phone 100 to the smart screen 200 .

Specifically, referring to FIG. 8 , the method includes the following three parts of the process. The first part of the process is summarized as: start the screen projection process, which is a preparation stage before playing audio, including steps S1 to S4, the main purpose is to establish virtual display and virtual display. A one-to-one binding relationship between audio devices.

The second part of the process is: application and preemption of audio focus. Including steps S5 to S6, because before the source device plays the audio stream, it needs to obtain an audio focus, and each virtual display can provide an audio focus. When two or more applications apply for the audio focus of the same virtual display When the audio focus is preempted, the audio focus can be provided for the requested application by following the virtual display audio focus mechanism.

The third part of the process is: according to the streaming strategy, the audio stream is transmitted to the virtual audio device corresponding to the application, so as to realize the audio transmission and playback from the source device to the screen projection device, which mainly includes steps S7 and S8.

The three parts of the process may be executed in sequence, or the first part of the process and the second part of the process may be executed simultaneously, which is not limited in this embodiment.

Among them, as shown in FIG. 5 , the involved functional modules include: the screen projection service module 10 of the mobile phone 100 , the first management module 21 , the second management module 22 and the virtual audio HAL 30 . The screen projection service module 10 is located in the application layer, the first management module 21 is located in the native audio framework 20, the second management module is located in the multi-screen management module, and the native audio framework 20 and the multi-screen management module are located in the framework layer. The virtual audio HAL 30 is located at the hardware layer.

Optionally, in a possible implementation manner, the screen projection service module 10 is an Android application package (Android application package, APK). In addition, other applications such as the first application 11 and the second application 12 may also be included, and the first application 11 and the second application 12 have an audio playback function.

Optionally, in another possible implementation manner, the screen projection service module 10 may also be located in the framework layer, is associated with the screen projection service application of the application layer, and is used for executing the relevant process of the screen projection service application.

Optionally, the first management module 21 is a multi-screen audio management module, and the second management module 22 is a multi-screen management module.

The method flow of each part is described in detail below.

First, in the preparation stage, the first part of the process: start the screen projection process, as shown in Figure 8, which includes:

S1: The user starts the screen projection service on the terminal device.

Specifically, the user starts the screencasting through the corresponding portal of the screencasting. For example, the user clicks on the screen casting service application on the mobile phone 100 to start the screen casting operation on the first device, where the first device is the smart screen 200 .

S2: Establish a first communication link between the terminal device and the first device.

Wherein, the first communication link may be established between the terminal device and the first device by means of USB or WiFi.

Specifically, after receiving the user's operation of starting the screen projection service, the mobile phone 100 establishes a first communication link with the smart screen 200 . One implementation is,

S2-1: The screen projection service module 10 sends a first message to the smart screen 200, where the first message is used to establish a first communication link with the mobile phone 100. Specifically, the screen projection service module 10 sends the first message to the smart screen 200 through radio resource control (Radio Resource Control, RRC) signaling. The first message includes information such as the identification of the mobile phone 100, the name of the mobile phone device, and the connection request.

S2-2: After receiving the first message, the smart screen 200 sends a first response message to the mobile phone 100.

Wherein, the first response message includes: mobile phone 100 identification, mobile phone device name, related capabilities of the smart screen 200 (size, resolution, support for video coding), audio related capabilities, such as whether to support Speaker or Mic capabilities, support sampling rate, bit width, etc., and related connection authentication information.

Optionally, the first response message is an RRC message. Specifically, the smart screen 200 sends the RRC message to the application layer of the mobile phone 100 .

S2-3: The application layer of the mobile phone 100 (for example, the screen projection service module 10 ) receives the first response message, determines that the terminal device sending the first response message is the smart screen 200 , and establishes the first response message with the smart screen 200 . communication link.

The screen projection service module 10 selects to connect to the smart screen 200 by means of USB or WiFi according to the scenario, and establishes the first communication link.

S3: A virtual audio device link is established between the mobile phone 100 and the first device.

S4: The mobile phone 100 establishes a first binding relationship between the first virtual display and the device identifier of the first virtual audio device.

Wherein, as shown in Figure 9, after step S2, the method includes:

S2': The screen projection service module 10 requests the second management module 22 to create a screen projection virtual display.

Specifically, after the first communication link is established between the mobile phone 100 and the smart screen 200, the screen projection service module 10 of the mobile phone 100 sends a second request to the second management module 22, and the second request is used for creating and casting The virtual display associated with the screen. The second request includes: virtual display name, size, and resolution information.

One embodiment includes: the screen-casting service module 10 calls the interface of the second management module 22, and sends the virtual display name, size and resolution information, creates a virtual display associated with screencasting, and uses the virtual display as the screencasting display source. .

The second management module 22 receives the second request, and creates a first virtual display, such as virtual display 0, according to the information carried in the second request. And, the first virtual display is sent to the screen projection service module 10 .

The screen projection service module 10 receives the first virtual display sent by the second management module 22, and uses the first virtual display to create a second communication link with the smart screen 200. The second communication link is a virtual audio device link.

Specifically, an implementation manner includes: the screen projection service module 10 sends a request message to the smart screen 200 for requesting to establish a second communication link with the smart screen 200 . After receiving the request message sent by the screen projection service module 10, the smart screen 200 sends a feedback result to the screen projection service module 10, where the feedback result indicates whether the second communication link is successfully created. Wherein, the feedback result includes: specified audio device type, device ID and audio channel access information.

Correspondingly, the screen projection service module 10 receives the feedback result, and when the feedback result indicates that the creation is successful, the mobile phone 100 and the smart screen 200 establish the second communication link.

Referring to Figure 9, the above step S4 specifically includes:

S4-1: After successfully establishing the second communication link with the smart screen 200, the screen projection service module 10 sends a first request to the first management module 21, where the first request is used to request the creation of a virtual audio device. Correspondingly, the first management module 21 receives the first request sent by the screen projection service module 10 .

The first request includes the specified audio device type, the device ID of the mobile phone 100, and audio channel access information, where the access information includes USB or WiFi.

S4-2: After receiving the first request, the first management module 21 sends the first request to the virtual audio HAL 30. Correspondingly, the virtual audio HAL 30 receives the first request forwarded by the first management module 21.

S4-3: The virtual audio HAL 30 creates an associated virtual audio device (audio device) through the virtual audio device link between the mobile phone 100 and the smart screen 200, and determines the device identifier of the virtual audio device, such as determining the virtual audio device If the device is the first virtual audio device, the device identifier of the device identifier of the virtual audio device is the device identifier of the first virtual audio device, such as audio device 0.

Specifically, the virtual audio HAL 30 creates an associated virtual audio device for the session link between the mobile phone 100 and the smart screen 200, and generates a virtual audio device identifier. The virtual audio device object includes the virtual audio type, device ID and audio path access information Wait.

S4-4: The virtual audio HAL 30 sends to the first management module 21 a device identifier for creating a virtual audio device (VirtualAudio), that is, the device identifier of the first virtual audio device.

S4-5: After receiving the device identifier of the virtual audio device, the first management module 21 sends the device identifier of the virtual audio device to the screen projection service module 10.

Optionally, the first management module 21 sends the device identifier of the virtual audio device through an instruction or a message.

S4-6: The screen projection service module 10 receives the device identifier of the virtual audio device, and establishes a binding relationship between the device identifier of the virtual audio device (Virtual Audio) and the first virtual display. Wherein, the first virtual display is obtained from the aforementioned S2'.

Specifically, the screen projection service module 10 binds the first virtual display with the device identifier of the newly created virtual audio device, that is, configures a virtual audio device for the virtual display, and establishes a one-to-one binding between the virtual display and the virtual audio device established relationship. Referring to Table 3, it is the first binding relationship between the established first virtual display screen (Virtual display 0) and the device identifier of the first virtual audio device (Virtual Audio 0).

table 3

S4-7: The screen projection service module 10 sends the binding relationship to the second management module 22, and the second management module 22 sends the binding relationship to the first management module 21 after receiving the binding relationship.

S4-8: The first management module 21 receives the binding relationship, and maintains and stores it.

In this embodiment, the mobile phone 100 mirrors the smart screen 200, the mirroring service module 10 creates a virtual display for the smart screen 200, and the mirroring service module 10 notifies the first management module of the binding relationship between the virtual display and the virtual audio device 21 , enabling the first management module 21 to maintain the binding relationship between the virtual display and the virtual audio device, so as to provide a basis for subsequent multi-screen audio focus management and selection of audio devices corresponding to audio streams.

In addition, this embodiment also provides a method for disconnecting the screencasting connection. When the mobile phone 100 receives an operation of disconnecting the screencasting request, the screencasting service module needs to release the binding between the virtual display and the created virtual audio device. relationship, as shown in Figure 10, the specific process is as follows:

201: The user initiates and disconnects the screencasting service through the screencasting service application.

Specifically, the user clicks the screen projection service interface on the mobile phone 100 to disconnect the screen projection connection with the smart screen 200 .

202: The screen projection service module 10 sends a first instruction to the multi-screen management module after receiving the user's disconnection of the screen projection operation, where the first instruction is used to request to disconnect the virtual display associated with the smart screen 200, and the first instruction includes: Including smart screen 200 identification information.

Wherein, the multi-screen management module is the aforementioned multi-screen management module shown in FIG. 4 , or the second management module 22 shown in FIG. 7 , and the second management module 22 is used as an example in this embodiment.

202: The second management module 22 receives the first instruction, and searches for the corresponding virtual display according to the identification information of the smart screen 200 carried in the first instruction.

In this embodiment, it is assumed that the virtual display found by the second management module 22 is the first virtual display.

203 : The second management module 22 releases the first binding relationship after finding the first virtual display, where the first binding relationship is a corresponding relationship between the first virtual display and the device identifiers of the first virtual audio device.

204: The second management module 22 sends a second instruction to the first management module 21, where the second instruction includes: the display screen 200 corresponds to the virtual display identifier (ie the first virtual display), the smart screen 200 identifier and other information. Correspondingly, the first management module 21 receives the second instruction sent by the second management module 22 .

205: The first management module 21 searches the binding relationship record between the virtual display and the virtual audio device according to the virtual display identifier carried in the second instruction, and after finding the binding relationship record, the first management module 21 Clear the binding relationship in the binding relationship record.

In this example, the first management module 21 finds a first binding relationship according to the first virtual display, and then releases the device identification between the first virtual display and the first virtual audio device in the first binding relationship. Correspondence.

206: The screen projection service module 10 sends a third instruction to the first management module 21, where the third instruction is used to notify the first management module 21 to delete the virtual audio device corresponding to the smart screen 200. The third instruction includes information such as the identification information of the smart screen 200, the device identification of the first virtual audio device, and the like.

207: After receiving the third instruction, the first management module 21 sends the third instruction to the virtual audio HAL 30 to notify the virtual audio HAL 30 to delete the first virtual audio device corresponding to the smart screen 200.

208: The virtual audio HAL 30 receives the third instruction, disconnects the relationship between the smart screen 200 and the first virtual audio equipment according to the information such as the device identification of the first virtual audio equipment carried in the third instruction, and Delete the information of the first virtual audio device.

209: The screen projection service module 10 disconnects the communication link with the smart screen 200.

Specifically, the screen projection service module 10 sends a disconnection request to the smart screen 200 , where the disconnection request is used to disconnect the first communication link with the mobile phone 100 and the smart screen 200 . After receiving the disconnection request, the smart screen 200 disconnects the first communication link.

In addition, the above method further includes that the first management module 21 and the second management module 22 update the binding relationship shown in Table 3 above, and release the first virtual display.

This embodiment provides a method for disconnecting the screen projection connection. When the screen projection connection does not need to be maintained between the source device and the screen projection device, the first management module and the second management module contact the virtual display and the virtual audio device. The binding relationship between them, thereby releasing the virtual display and providing convenience for the subsequent establishment of connections with other screen projection devices,

In addition, after the screen projection service module 10 establishes the first binding relationship, according to the above steps S1 to S4, the multi-screen audio management module will internally maintain multiple virtual displays and virtual audios for multiple device scenarios of mobile phone screen projection. The binding relationship between devices, and the audio playback function of applications of different virtual displays will be designed and developed based on the Android audio focus mechanism. This embodiment provides an audio focus mechanism based on virtual display, which is unaware of the application, and the application APP does not need code modification.

In addition, based on the audio focus solution of the virtual display, the multi-screen audio management module also provides a focus management record for each virtual display on the basis of Table 3, which is used to record the current virtual display audio focus state (Audio Focus State), occupation Audio focus application (Owner) and apply for audio focus application stack (APP Stack). Wherein, the virtual display audio focus state includes two states: used (Used) and idle (Idle). Specifically, as shown in Table 4,

Table 4

The status and occupants of the virtual audio focus are recorded in Table 4, which provides a basis for subsequent audio focus preemption and audio stream allocation.

After the preparation stage of the first part of the process is completed, the second part of the process is executed, and the second part of the process is: a virtual display-based audio focus preemption process. As shown in Figure 8, the second part of the process is summarized as:

S5: The user starts the first application to play an audio operation, and the first application has an audio playing function, such as QQ Music.

S6: The mobile terminal applies for the audio focus according to the user's operation, and acquires the first audio focus. The first audio focus uniquely corresponds to the first virtual display.

This process provides a preemption mechanism for the application to play audio focus in the scenario where the mobile phone is connected to multiple devices. This mechanism maintains a focus management record for each virtual display to record whether the current virtual display has audio focus that can be used by the application. , and the audio focus information currently applied by each application.

As shown in Figure 11 above, an application in the application layer applies to the AudioManager of the framework layer for audio focus, and MediaFocusControl searches the multi-screen audio management module according to the virtual display that the application application belongs to, and determines whether the focus applied by the application running on the corresponding virtual display is available. , it returns to apply for the audio focus of the application; if it is unavailable, that is, the focus is used by other applications, it will preempt the audio focus of the application currently in use, return the audio focus available to the application application, and notify the preempted application of losing the audio focus.

The audio focus mechanism based on virtual display is as follows:

First, each virtual display has an independent audio focus, and the audio focuses owned by different virtual displays are independent of each other and do not interfere with each other.

Second, if there are two or more applications in the same virtual display, and two or more applications apply for audio focus at the same time, follow the existing Android audio focus preemption mechanism, that is, high-priority business applications preempt low For the applications of priority services, the rules of application preemption after application are followed for service applications of the same priority.

As shown in Figure 12, the application applies for audio focus, and in the case of audio focus preemption, the process for the application to acquire audio focus is as follows:

Steps S5 to S6 are the process for the first application to apply for audio focus, which specifically includes:

S5: Receive an operation that the user starts the first application to play music.

Specifically, the user opens a first application on the mobile phone 100 to play audio, and the first application is a music playing APP, such as

And the first application corresponds to a virtual display identifier established in the foregoing first part of the process, such as virtual display 0.

S6-1: Before the first application plays audio, it applies to the native audio framework 20 for audio focus through the Android audio focus application interface.

An implementation manner is that the first application sends a request to the native audio framework 20 for requesting to apply for audio focus. The request includes: Android audio focus request message, audio type, and the like. The requested audio focus is used to play the audio stream for the first application.

Optionally, the native audio framework 20 is an AOSP native audio framework.

S6-2: After receiving the request for applying for audio focus, the native audio framework 20 sends the request for applying for audio focus to the first management module 21, where the request includes: Android audio focus request message, audio type, and the first application The relevant information of the first application, etc., the relevant information of the first application includes but is not limited to the process id (process id, pid), package name, etc. that the first application currently starts and runs.

S6-3: After receiving the request, the first management module 21 sends the request for applying for audio focus to the second management module 22, where the request is used to query the virtual display information where the first application is located, and the request includes the Relevant information of the first application, such as pid, package name of the first application, and the like.

S6-4: After receiving the request, the second management module 22 queries the virtual display identifier corresponding to the first application according to the relevant information of the first application in the request.

For example, if the first application is application A or APP1, according to the corresponding relationship shown in Table 5, it is determined that the virtual display identifier corresponding to application A is Virtual display 0.

table 5

Smart screen 200

Virtual display0

Virtual Audio 0

Idle idle

S6-5: The second management module 22 sends the searched virtual display identifier to the first management module 21.

Specifically, the second management module 22 carries the searched virtual display identifier through the response message, that is, the response message includes: the virtual display identifier (Virtual display 0) corresponding to the first application.

S6-6: The first management module 21 receives the virtual display identifier, searches for the virtual display record in the relationship table between the virtual display and the virtual audio device maintained in Table 4 or Table 5, and determines that the virtual display record is in the virtual display record. Whether the state of the audio focus is available, or in other words whether other applications use the audio focus.

In this example, if the audio focus state corresponding to Virtual display0 is Idle (Idle), it is determined that the first audio focus currently applied for by the first application is available, indicating that the first audio focus requested by the first application is available.

S6-7: The first management module 21 returns a query result to the native audio framework 20, where the query result includes indication information requesting that the first audio focus is available.

For example, the first management module 21 sends a response (response message) to the native audio framework 20, where the response includes indication information that the first audio focus is available, allowing application A to authorize access to the first audio focus information.

S6-8: The native audio framework 20 determines that the first audio focus applied by the first application is available according to the query result, that is, the application for the first audio focus is successful, and sends the indication information that the first audio focus is available to application A.

Correspondingly, after receiving the indication information from the native audio framework 20, the first application obtains the first audio focus.

In addition, the above step S6-6 also includes: the first management module 21 updates the virtual display0 record information. Specifically, the first management module 21 sets the state of the first audio focus in the virtual display 0 to "in use", sets the application A to occupy the audio focus of the virtual display 0, and adds the application A to the application stack table, and the updated virtual The record information of display0 is shown in Table 6.

Table 6. The updated virtual display0 record information

In addition, in the above step 207, if the first management module 21 determines that the first audio focus is in the "use" state, it indicates that the first audio focus has been occupied by other applications, and the first application to apply for the audio focus needs to comply with the audio focus preemption mechanism.

The following describes the method flow of application B and application C to apply for audio focus. Specifically, as shown in Figure 13, the method includes:

S5: The mobile phone 100 receives the operation that the user starts the application B to play audio, the application B has an audio playback function, and the virtual display corresponding to the application B is identified as virtual display 1.

After the application B receives the user operation, it calls the interface of the first management module 21 to apply for the audio focus and obtain the second audio focus. The specific process is the same as the foregoing steps S6-1 to S6-8, and details are not repeated here in this embodiment. At this time, application A uses the first audio focus to play the audio stream, and application B uses the second audio focus to play the audio stream.

In this case, the updated virtual display record information is shown in Table 7.

Table 7

When application C applies for audio focus, the above method further includes:

S5': The mobile phone 100 receives the operation that the user starts the application C to play the audio.

Application C applies to the first management module 21 and the second management module 22 for audio focus, and the second management module queries the virtual display corresponding to application C as virtual display 1. For the specific process, reference may be made to the foregoing steps S6-1 to S6-4, which will not be repeated here.

After the second management module 22 queries the virtual display where the application C is located, as shown in FIG. 13 , the method includes:

301: The first management module 21 receives the virtual display identifier where the application C is located and returned by the second management module 22. In this example, the display identifier where the application C is located is the virtual display 1.

302 : The first management module 21 determines, according to the virtual display1 where the application C is located, that the audio focus requested by the application C is the second audio focus, and determines whether the second audio focus is available.

In this example, the virtual display identifiers corresponding to application C and application B are both virtual display 1, and the virtual display 1 corresponds to the second audio focus, wherein the second audio focus is acquired and used by application B first, so as to determine The second audio focus for this application C application is not available.

303: The first management module 21 preempts the application of the low-priority service according to the virtual display audio focus mechanism, that is, the high-priority service application preempts the application of the low-priority service, and applies for the application preemption priority rule after following the service application of the same priority. In this example, because the service priority corresponding to application C is higher than that of application C, and application C is an application applied later, it is determined that application C preempts the second audio focus of application B.

304: The first management module 21 sends indication information to the native audio framework 20, where the indication information is used to instruct the application C to preempt the second audio focus.

305: After receiving the indication information, the native audio framework 20 sends a notification to application B to notify application B to release the currently used second audio focus.

306: After receiving the notification, application B stops using and releases the second audio focus.

In addition, after releasing the second audio focus in step 306, the method further includes:

307: The first management module 21 sends a notification message to the native audio framework 20 to notify the application C to use the second audio focus.

308: The native audio framework 20 sends the communication message to application C to notify application C to obtain the second audio focus.

309: After receiving the notification message, application C obtains and uses the second audio focus.

In this embodiment, when multiple applications apply for the same audio focus, the application is allocated a corresponding focus according to the virtual display audio focus mechanism, thereby optimizing the audio focus preemption mechanism of applications running on different virtual displays, and supporting multitasking based on virtual display. Audio concurrency control.

In addition, the first management module 21 also refreshes the virtual display record information, as shown in Table 8.

Table 8

It should be noted that the foregoing application A, application B, and application C may be the first application or the second application in the foregoing embodiment, which is not limited in this embodiment.

After the application applies for the audio focus, the audio stream is played through the Android SDK interface. The multi-screen audio management module splits the audio stream played by the application to the corresponding virtual audio device for output and playback according to the mapping relationship between the display where the application is located and the virtual audio device.

The third part of the process: define the audio distribution strategy based on the virtual Display, and transmit the audio stream to the virtual audio device corresponding to the application according to the distribution strategy.

Specifically, as shown in Figure 14a, the Android audio stream is distributed through AudioFlinger, and the multi-screen audio management module builds the binding relationship between the virtual display and the device ID of the virtual audio device in the "first part of the process". The current application After obtaining the audio focus, create an AudioTrack to play the audio (or create an AudioRecod recording), AudioFlinger first finds the corresponding virtual audio device according to the virtual display belonging to the current application, and then distributes the application AudioTrack (or AudioRecord) to the corresponding virtual audio device. The virtual audio HAL After receiving the audio stream, distribute it to the corresponding screen projection device.

As shown in Figure 14b, the flow of the method for audio distribution based on virtual display includes:

401: When the startup application A is used to play music, the application A plays the music, calls the Android SDK interface to play the first audio stream, and the application A writes the first audio stream to the native audio framework 20.

402 : After receiving the first audio stream of the application A, the native audio framework 20 sends a query request to the first management module 21 to query the audio playback policy of the application A.

Wherein, the query request includes application A related information, such as application A's package name, pid, identification and other information.

403 : After receiving the query message, the first management module 21 sends the query request to the second management module 22 .

404: After receiving the request message sent from the first management module, the second management module 22 queries the virtual display identifier to which the application A belongs according to the virtual display record information table.

In this example, according to the corresponding relationship in Table 6 to Table 8, it is determined that the virtual display identifier to which application A belongs is virtual display 0.

405 : The second management module 22 sends the query result, that is, the virtual display identifier, to the first management module 21 . For example, the second management module 22 returns the first query result to the first management module 21, and the first query result includes: the virtual display 0 where the application A is located.

406: After receiving the first query result, the first management module 21 searches for the virtual audio device corresponding to the virtual display 0 from the maintained binding relationship according to the virtual display 0 where the application A is located.

The binding relationship is the corresponding relationship between the virtual display and the device identifier of the virtual audio device established in the foregoing "Part 1 of the process". As shown in Table 4, Table 5 and Table 6, the device identification of the virtual audio device bound to application A is determined, and the virtual audio device bound to the virtual display 0 is a smart screen, and the virtual audio device is bound to the virtual display 0. The device identification of the audio device is the smart screen 200 .

407: The first management module 21 sends the device identification of the searched virtual audio device to the native audio architecture 20.

In this example, the first management module 21 sends the second query result, and the second query result includes the device identifier of the virtual audio device corresponding to display0, the smart screen 200 .

408: After receiving the second query result, the native audio framework 20 acquires a first audio stream according to the audio policy, where the first audio stream is an audio stream that needs to be played on the smart screen 200.

409: The native audio framework 20 distributes the first audio stream played by application A and the device identifier of the virtual audio device to the virtual audio HAL 30, that is, sends the information of the first audio stream and the smart screen 200 to the virtual audio HAL.

410: After the virtual audio HAL 30 receives the first audio stream sent by the native audio framework 20 and the device identifier of the virtual audio device, it finds the corresponding audio path according to the device identifier of the virtual audio device, and sends the first audio stream through the audio path. to the audio equipment smart screen 200.

After receiving the first audio stream, the smart screen 200 plays the first audio stream, so as to realize the function of playing the first audio stream of the application A in the mobile phone 100 on the smart screen 200 .

Wherein, the above-mentioned steps 401 to 407 correspond to step S7 in the aforementioned "third part of the flow", and steps 408 to 410 correspond to the aforementioned step S8 in the "third part of the flow".

Similarly, when application B requests to play the second audio stream, the second audio stream is transmitted to the second device, such as the tablet 300, through the device identifier of the virtual audio device found by the first management module 21, thereby realizing the mobile phone 100. The function of the application B to transmit the second audio stream to another screen projection device, the tablet 300.

Specifically, in the process of applying B to transmit and distribute the second media stream, the methods of steps 411 to 420 are the same as the foregoing steps 401 to 410, refer to the description of the foregoing embodiment, and are not described in detail here.

In this embodiment, since the virtual display allocated by application A (that is, virtual display 0) and the virtual display allocated by application B (that is, virtual display 1) are different, and the virtual audio devices corresponding to each display are independent of each other and interfere with each other, so Through the binding relationship established by the "first part of the process", the one-to-many screen projection function can be realized, and the multi-terminal screen projection display and audio input and output are independent of each other. Achieve true multitasking audio concurrency and pickup.

It should be understood that, after obtaining the second audio focus, the above-mentioned application C requests to play the third audio stream and plays the third audio stream on the tablet 300 associated with the virtual display 1. The specific process is the same as the aforementioned steps 401 to 410. See The description of the foregoing embodiment is not repeated in this embodiment.

In addition, the above scheme of transmitting audio streams to the virtual audio device corresponding to the application according to the streaming strategy is also applicable to other scenarios, such as recording scenarios, so that each screen-casting device can use the current screen-casting device for recording when running the application, which satisfies one requirement. Recording requirements for multi-screen projection. Alternatively, it is also applicable to a one-to-many projection screen distributed camera (camera) virtualization environment, which can realize a distributed camera usage scenario in a multi-screen scenario.

In the embodiment of the present application, the mobile terminal can transmit the displayed picture content to each display screen of the screen projection device based on the form of the application window, so as to realize audio concurrency for each application window.

Apparatus embodiments corresponding to the foregoing method embodiments are introduced below.

FIG. 15 is a schematic structural diagram of a management apparatus provided by an embodiment of the present application. The apparatus may be applied to the aforementioned first management module, or may also be a component located in a terminal device. Furthermore, the apparatus can implement all the functions of the first management module 21 in the foregoing embodiments, and execute an audio playing method in the foregoing embodiments.

Wherein, the apparatus includes: a first receiving unit 1501, a first processing unit 1502, and a first sending unit 1503, and may further include other units or modules, such as a storage unit and the like.

Specifically, the first receiving unit 1501 is configured to receive a first request sent by the screen projection service module when the terminal device starts the screen projection service, where the first request is used to request the creation of a virtual audio device, and the first request includes Relevant information of the first device that the user wishes to screen; the first sending unit 1503 is configured to send the first request to the hardware abstraction layer HAL; the first receiving unit 1501 is further configured to receive the first virtual audio device sent by the HAL The device identifier of the first virtual audio device is created according to the relevant information of the first device. The first sending unit 1503 is further configured to send the device identifier of the first virtual audio device to the screen projection service module, so that the virtual screen projection service module establishes a relationship between the virtual display screen display and the first virtual audio device. binding relationship between.

Optionally, in a specific implementation of this embodiment, the first receiving unit 1501 is further configured to receive the first binding relationship sent by the screen projection service module, where the first binding relationship includes the first binding relationship. The corresponding relationship between the virtual display screen display and the device identifier of the first virtual audio device; the first processing unit 1502 is configured to determine the first virtual display when the first application requests to play the audio stream, according to the first virtual display Find the device identifier of the first virtual audio device in the first binding relationship; the first sending unit 1503 is further configured to send the device identifier of the first virtual audio device to the native audio framework, so that the native audio framework The corresponding audio stream is output to the first device through the first virtual audio device.

Optionally, in another specific implementation of this embodiment, the first receiving unit 1501 is further configured to acquire the identifier of the first application; the first sending unit 1503 is further configured to send the first application to the second management module An identifier of an application; the first receiving unit 1501 is further configured to receive the first virtual display determined by the second management module according to the identifier of the first application.

Optionally, in yet another specific implementation of this embodiment, the first processing unit 1502 is further configured to determine a first audio focus according to the first virtual display, where the first audio focus is the same as the first audio focus. Corresponding to the virtual display, query whether the state of the first audio focus is available; the first sending unit 1503 is further configured to send to the native audio framework when the first processing unit 1502 queries that the first audio focus is available the first audio focus.

Optionally, in yet another specific implementation of this embodiment, the first processing unit 1502 is further configured to, when querying the status of the first audio focus is unavailable, determine that the application occupying the first audio focus is: The second application; the first sending unit 1503 is further configured to send first indication information to the native audio framework, where the first indication information is used to notify the native audio framework that the first audio focus needs to be The first application is occupied; and, when the first receiving unit receives an instruction that the second application stops using the first audio focus, sending the first audio focus to the native audio framework, so that all The first application acquires the first audio focus.

Optionally, in another specific implementation of this embodiment, before the first management module queries whether the state of the first audio focus is available, the method further includes: the first processing unit 1502 is further configured to obtain the first audio focus. A first focus management record associated with a virtual display, and querying the first focus management record whether the state of the first audio focus is idle. Wherein, the first focus management record includes the state of the first audio focus, the state of the first audio focus includes idle and in use, and the first virtual display uniquely corresponds to the first audio focus.

Optionally, in another specific implementation of this embodiment, the first processing unit 1502 is further configured to maintain the first focus management record according to the virtual display audio focus mechanism. Wherein, the virtual display audio focus mechanism includes: there are two or more applications in the same virtual display, and when the two or more applications apply for audio focus at the same time, preempt the low priority service application according to the high priority The applications of the priority services, and, the service applications of the same priority follow the rule of applying for the application first and preempting the application first.

In addition, this embodiment further provides another management device. As shown in FIG. 16 , the device can be applied to a second management module to implement all or part of the functions of the second management module 22, wherein the second management module The management module includes: a second receiving unit 1601, a second processing unit 1602, and a second sending unit 1603, and may also include other units or modules, such as a storage unit and the like.

The second receiving unit 1601 is configured to receive a second request sent by the screen-casting service module, the second request is used to create a virtual display for screen-casting, and the second request includes: specifying the name, size and resolution of the virtual display rate information. The second processing unit 1602 is configured to create a first virtual display according to the specified virtual display name, size and resolution information; the second sending unit 1603 is configured to send the first virtual display to the screen projection service module, so that The virtual screen projection service module establishes a first binding relationship between the first virtual display and the first virtual audio device.

Optionally, in a specific implementation of this embodiment, the second receiving unit 1601 is further configured to receive the first binding relationship sent by the screen projection service module; the second sending unit 1603 is further configured to Send the first binding relationship to the first management module.

Optionally, in another specific implementation of this embodiment, the second receiving unit 1601 is further configured to, when the first application requests to play the audio stream, the second management module to obtain the identifier of the first application; The second processing unit 1602 is further configured to search for the first virtual display to which the first application belongs according to the identifier of the first application; the second sending unit 1603 is further configured to send the first virtual display to the first management module.

Optionally, in another specific implementation of this embodiment, the second receiving unit 1601 is further configured to receive an audio focus request sent by the first management module when the first application requests to apply for audio focus , the audio focus request includes the identifier of the first application.

Optionally, in yet another specific implementation of this embodiment, the second receiving unit 1601 is further configured to receive, when the native audio framework acquires the audio stream played by the first application, a message sent by the first management module. the identifier of the first application.

In addition, this embodiment also provides an application device. As shown in FIG. 17 , the device can be applied to a screen projection service module to realize part or all of the functions of the aforementioned screen projection service module 10. The application device It includes: a third receiving unit 1701, a third processing unit 1702, and a third sending unit 1703, and may also include other units or modules, such as a storage unit and the like.

Specifically, the third receiving unit 1701 is configured to obtain the first virtual display screen display created by the second management module when obtaining the user's operation of starting the screen projection service; the third sending unit 1703 is configured to obtain the first virtual display screen display created by the second management module; After the device establishes a virtual audio link, it sends a first request to the first management module, where the first request is used to request the creation of a virtual audio device, and the first request includes information about the first device that the user wishes to screen. information.

The third receiving unit 1701 is further configured to receive the device identifier of the first virtual audio device sent by the first management module; the third processing unit 1702 is configured to establish a first binding relationship for the screen projection service module, the first The binding relationship includes the corresponding relationship between the first virtual display and the device identifier of the first virtual audio device; the third sending unit 1703 is further configured to send the first binding to the first management module relation.

Optionally, in a specific implementation of this embodiment, the third sending unit 1703 is further configured to send a second request to the second management module, where the second request is used to request the second management module The module creates a virtual display; the third receiving unit 1701 is further configured to receive the first virtual display created by the second management module according to the second request.

Optionally, in another specific implementation of this embodiment, the third processing unit 1702 is further configured to release the first virtual display and the first virtual display when acquiring the operation of the user disconnecting the screencasting service. The first binding relationship between the device identifiers of the virtual audio device; the third receiving unit 1703 is further configured to send a first message to the first management module, where the first message is used to notify the first management module to delete the first binding relationship.

It should be noted that, each of the above-mentioned management apparatuses and application apparatuses may be implemented by software or a combination of software and hardware. Wherein, the functions of the first receiving unit to the third receiving unit and the first sending unit to the third sending unit can be realized through the internal port of the terminal device, and the functions of the first processing unit to the third processing unit can be realized through The processor 110 shown in FIG. 3 is implemented, or jointly implemented by the processor 110 and the memory 120 . Data transmission between the terminal device and each screen projection device, such as audio streaming transmission, may be implemented through the mobile communication module 150 or the wireless communication module 160 .

When using the terminal device shown in Figure 3, such as a mobile phone, a one-to-many projection scenario is realized through the functions of the projection screen service, multi-frequency audio management and multi-screen management module. The multi-terminal projection device display and audio input and output are independent of each other. A screen-casting device displays the running application playback sound on the current screen-casting device, and each screen-casting device displays a running application recording that can be recorded using the current screen-casting device to achieve multi-task audio concurrency and pickup.

In addition, an embodiment of the present application also provides an audio playback system, the system includes at least one source device and at least two screen projection devices, and both the source device and the screen projection device may be any of the foregoing terminal devices, and the The structure of the terminal device may be the same as or different from the structure of the apparatus shown in FIG. 3 . In addition, a network device, such as a server, a cloud platform, etc., may also be included, and the network device is used to assist the terminal device to realize the one-to-many screen projection service function.

In addition, an embodiment of the present application also provides a chip system. As shown in FIG. 18 , the chip system includes a processor 1801 to support the above-mentioned device to implement the functions involved in the above-mentioned methods. For example, the foregoing first part of the flow to the third part Method steps in part of the process. In a possible design, the chip system further includes a memory 1802 for storing the first management device, the second management device, and necessary computer instructions and data for each module. The chip system may be composed of chips, or may include chips and other discrete devices.

The memory 1802 includes a computer storage medium, wherein the computer storage medium may store program instructions, and when the program instructions are executed, may include part or all of the steps of the audio playback method provided by the present application. The storage medium can be a magnetic disk, an optical disk, a read-only storage memory ROM or a random access storage memory RAM, and the like.

In the above-mentioned embodiments, all or part of the implementation may be implemented by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product.

The above-mentioned computer storage medium may be a computer program product, and the computer program product includes one or more computer instructions, such as an instruction requesting to apply for audio focus, an instruction requesting to create a virtual display, a query instruction, etc., which are loaded and executed on a computer. When computer program instructions are used, all or part of the method processes or functions described in the above-mentioned embodiments of the present application are generated.

Also, in the description of this application, unless stated otherwise, "plurality" means two or more than two. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first", "second", "third" are used to describe the same items or items with basically the same function and effect. Similar items are distinguished. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second", "third" and the like are not necessarily different.

The above-described embodiments of the present application do not limit the protection scope of the present application.

Claims

An audio playback method, characterized in that the method comprises:

When the terminal device starts the screencasting service, the first management module receives the first request sent by the screencasting service module, the first request is used to request the creation of a virtual audio device, and the first request includes the first request that the user wishes to screencast. information about the device;

The first management module sends the first request to the hardware abstraction layer HAL;

The first management module receives the device identifier of the first virtual audio device sent by the HAL, and the first virtual audio device is created according to the relevant information of the first device;

The first management module sends the device identification of the first virtual audio device to the screen projection service module, so that the screen projection service module establishes a binding between the virtual display screen display and the first virtual audio device. established relationship.
The method of claim 1, further comprising:

The first management module receives the first binding relationship sent by the screen projection service module, where the first binding relationship includes the correspondence between the first virtual display screen display and the device identifier of the first virtual audio device relation;

When the first application requests to play the audio stream, the first management module determines the first virtual display;

The first management module searches for the device identifier of the first virtual audio device in the first binding relationship according to the first virtual display;

The first management module sends the device identification of the first virtual audio device to the native audio framework, so that the native audio framework outputs the corresponding audio stream to the first device through the first virtual audio device.
The method according to claim 2, wherein the first management module determines the first virtual display, comprising:

The first management module obtains the identifier of the first application;

The first management module sends the identifier of the first application to the second management module;

The first management module receives the first virtual display determined by the second management module according to the identifier of the first application.
The method according to claim 2 or 3, wherein after the first management module determines the first virtual display, the method further comprises:

The first management module determines a first audio focus according to the first virtual display, and the first audio focus corresponds to the first virtual display;

The first management module inquires whether the state of the first audio focus is available;

If so, send the first audio focus to the native audio framework.
The method of claim 4, further comprising:

If the state of the querying the first audio focus is unavailable, determining that the application occupying the first audio focus is the second application;

The first management module sends first indication information to the native audio framework, and the first indication information is used to notify the native audio framework that the first audio focus needs to be occupied by the first application;

When receiving the instruction that the second application stops using the first audio focus, the first audio focus is sent to the native audio framework, so that the first application acquires the first audio focus.
The method according to claim 4, wherein before the first management module inquires whether the state of the first audio focus is available, the method further comprises:

The first management module acquires a first focus management record associated with the first virtual display, the first focus management record includes the state of the first audio focus, wherein the state of the first audio focus includes idle and in use, The first virtual display uniquely corresponds to the first audio focus;

The first management module inquires whether the status of the first audio focus is available, including:

The first management module inquires in the first focus management record whether the state of the first audio focus is idle.
The method of claim 6, further comprising:

The first management module maintains the first focus management record according to the virtual display audio focus mechanism;

The virtual display audio focus mechanism includes: there are two or more applications in the same virtual display, and when the two or more applications apply for audio focus at the same time, preempt the low priority according to the high priority service application Business applications, and business applications with the same priority follow the rule of applying for applications first and preempting applications later.
An audio playback method, characterized in that the method comprises:

The second management module receives a second request sent by the screen-casting service module, where the second request is used to create a virtual display for screen-casting, wherein the second request includes: specifying the name, size, and resolution information of the virtual display;

The second management module creates a first virtual display according to the specified virtual display name, size and resolution information;

The second management module sends the first virtual display to the screen-casting service module, so that the virtual screen-casting service module establishes a first binding between the first virtual display and the first virtual audio device relation.
The method of claim 8, further comprising:

receiving, by the second management module, the first binding relationship sent by the screen projection service module;

The second management module sends the first binding relationship to the first management module.
The method of claim 8 or 9, further comprising:

When the first application requests to play the audio stream, the second management module obtains the identifier of the first application;

The second management module searches for the first virtual display to which the first application belongs according to the identifier of the first application;

The second management module sends the first virtual display to the first management module.
The method according to claim 10, wherein the second management module obtains the identification of the first application, comprising:

When the first application requests to apply for audio focus, the second management module receives an audio focus request sent by the first management module, where the audio focus request includes the identifier of the first application.
The method according to claim 10, wherein the second management module obtains the identifier of the first application, comprising:

The second management module receives the identifier of the first application sent by the first management module when the native audio framework acquires the audio stream played by the first application.
An audio playback method, characterized in that the method comprises:

The screen projection service module acquires the first virtual display screen display created by the second management module when acquiring the operation of the user initiating the screen projection service;

The screen projection service module sends a first request to the first management module after establishing a virtual audio link with the first device that expects screen projection, where the first request is used to request the creation of a virtual audio device, and the first request is used to request the creation of a virtual audio device. A request includes information about the first device on which the user wishes to project the screen;

The screen projection service module receives the device identifier of the first virtual audio device sent by the first management module;

The screen projection service module establishes a first binding relationship, and the first binding relationship includes a corresponding relationship between the device identifiers of the first virtual display and the first virtual audio device;

The screen projection service module sends the first binding relationship to the first management module.
The method according to claim 13, wherein the acquiring the first virtual display screen display created by the second management module comprises:

The screen projection service module sends a second request to the second management module, where the second request is used to request the second management module to create a virtual display;

The screen projection service module receives the first virtual display created by the second management module according to the second request.
The method of claim 13 or 14, further comprising:

When acquiring the user's operation of disconnecting the screencasting service, releasing the first binding relationship between the device identifiers of the first virtual display and the first virtual audio device;

The screen projection service module sends a first message to the first management module, where the first message is used to notify the first management module to delete the first binding relationship.
A management device, characterized in that it is applied to a first management module, and the first management module includes:

The first receiving unit is configured to receive a first request sent by the screencasting service module when the terminal device starts the screencasting service, the first request is used to request the creation of a virtual audio device, and the first request includes the user's desire to cast a screen. Information about the first device on the screen;

a first sending unit, configured to send the first request to the hardware abstraction layer HAL;

The first receiving unit is further configured to receive the device identifier of the first virtual audio device sent by the HAL, and the first virtual audio device is created according to the relevant information of the first device;

The first sending unit is further configured to send the device identification of the first virtual audio device to the screen-casting service module, so that the virtual screen-casting service module establishes a virtual display screen display and the first virtual audio Binding relationship between devices.
The apparatus of claim 16, wherein:

The first receiving unit is further configured to receive a first binding relationship sent by the screen projection service module, where the first binding relationship includes a first virtual display screen display and a device identifier of the first virtual audio device Correspondence between;

a first processing unit, configured to determine a first virtual display when the first application requests to play an audio stream, and search for the device identifier of the first virtual audio device in the first binding relationship according to the first virtual display;

The first sending unit is further configured to send the device identification of the first virtual audio device to the native audio framework, so that the native audio framework outputs the corresponding audio stream to the first virtual audio device through the first virtual audio device. equipment.
The apparatus of claim 17, wherein:

The first receiving unit is further configured to obtain the identifier of the first application;

The first sending unit is further configured to send the identifier of the first application to the second management module;

The first receiving unit is further configured to receive the first virtual display determined by the second management module according to the identifier of the first application.
The device according to claim 17 or 18, characterized in that,

The first processing unit is further configured to determine a first audio focus according to the first virtual display, the first audio focus corresponds to the first virtual display, and query whether the state of the first audio focus is available ;

The first sending unit is further configured to send the first audio focus to the native audio framework when the first processing unit queries that the first audio focus is available.
The apparatus of claim 19, wherein:

The first processing unit is further configured to determine that the application occupying the first audio focus is the second application when the state of the first audio focus is inquired to be unavailable;

The first sending unit is further configured to send first indication information to the native audio framework, where the first indication information is used to notify the native audio framework that the first audio focus needs to be occupied by the first application ; and, when the first receiving unit receives an instruction that the second application stops using the first audio focus, sending the first audio focus to the native audio framework, so that the first application Acquire the first audio focus.
The apparatus according to claim 19, wherein before the first management module inquires whether the state of the first audio focus is available, the method further comprises:

The first processing unit is further configured to acquire a first focus management record associated with the first virtual display, and query whether the state of the first audio focus is idle in the first focus management record;

Wherein, the first focus management record includes the state of the first audio focus, the state of the first audio focus includes idle and in use, and the first virtual display uniquely corresponds to the first audio focus.
The device of claim 21, wherein:

The first processing unit is further configured to maintain the first focus management record according to the virtual display audio focus mechanism;

The virtual display audio focus mechanism includes: there are two or more applications in the same virtual display, and when the two or more applications apply for audio focus at the same time, preempt the low priority according to the high priority service application Business applications, and business applications with the same priority follow the rule of applying for applications first and preempting applications later.
A management device, characterized in that it is applied to a second management module, the second management module comprising:

The second receiving unit is configured to receive a second request sent by the screen projection service module, where the second request is used to create a virtual display for screen projection, wherein the second request includes: specifying the name, size and resolution of the virtual display information;

a second processing unit, configured to create a first virtual display according to the specified virtual display name, size and resolution information;

a second sending unit, configured to send the first virtual display to the screen-casting service module, so that the virtual screen-casting service module establishes a first binding between the first virtual display and the first virtual audio device established relationship.
The apparatus of claim 23, wherein:

The second receiving unit is further configured to receive the first binding relationship sent by the screen projection service module;

The second sending unit is further configured to send the first binding relationship to the first management module.
The device according to claim 23 or 24, characterized in that,

The second receiving unit is further configured to, when the first application requests to play the audio stream, the second management module to obtain the identifier of the first application;

The second processing unit is further configured to search for the first virtual display to which the first application belongs according to the identifier of the first application;

The second sending unit is further configured to send the first virtual display to the first management module.
The apparatus of claim 25, wherein:

The second receiving unit is further configured to receive an audio focus request sent by the first management module when the first application requests to apply for audio focus, where the audio focus request includes the identifier of the first application.
The apparatus of claim 25, wherein:

The second receiving unit is further configured to receive the identifier of the first application sent by the first management module when the native audio framework acquires the audio stream played by the first application.
An application device, characterized in that it is applied to a screen projection service module, and the screen projection service module includes:

a third receiving unit, configured to acquire the first virtual display screen display created by the second management module when acquiring the operation of the user initiating the screen projection service;

A third sending unit, configured to send a first request to the first management module after establishing a virtual audio link with the first device expected to screen the screen, where the first request is used to request the creation of a virtual audio device, the The first request includes relevant information of the first device on which the user wishes to project the screen;

The third receiving unit is further configured to receive the device identifier of the first virtual audio device sent by the first management module;

a third processing unit, used for the screen projection service module to establish a first binding relationship, where the first binding relationship includes a corresponding relationship between the device identifiers of the first virtual display and the first virtual audio device ;

The third sending unit is further configured to send the first binding relationship to the first management module.
The apparatus of claim 28, wherein:

The third sending unit is further configured to send a second request to the second management module, where the second request is used to request the second management module to create a virtual display;

The third receiving unit is further configured to receive the first virtual display created by the second management module according to the second request.
The device according to claim 28 or 29, characterized in that,

The third processing unit is further configured to release the first binding relationship between the device identifiers of the first virtual display and the first virtual audio device when the user's operation of disconnecting the screen projection service is obtained;

The third receiving unit is further configured to send a first message to the first management module, where the first message is used to notify the first management module to delete the first binding relationship.
A terminal device, characterized by comprising: at least one processor and a memory,

the memory for storing computer program instructions;

The at least one processor for executing the instructions to implement the method of any one of claims 1 to 15.
A computer-readable storage medium, wherein computer program instructions are stored in the computer-readable storage medium;

The computer program instructions, when executed, implement a method as claimed in any one of claims 1 to 15.