CN113923305B

CN113923305B - Multi-screen cooperative communication method, system, terminal and storage medium

Info

Publication number: CN113923305B
Application number: CN202111519288.XA
Authority: CN
Inventors: 朱吉男
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2021-12-14
Filing date: 2021-12-14
Publication date: 2022-06-21
Anticipated expiration: 2041-12-14
Also published as: CN113923305A; CN116264598A

Abstract

The embodiment of the application relates to the technical field of terminals, and provides a multi-screen collaborative conversation method, a multi-screen collaborative conversation system, a multi-screen collaborative conversation terminal and a storage medium. At this time, if the user starts the recording function, the recording file includes the sound of the other party of the call sent by the device at the opposite end of the call and the sound of the user collected by the second terminal.

Description

Multi-screen cooperative communication method, system, terminal and storage medium

Technical Field

The embodiment of the application relates to the field of terminals, in particular to a multi-screen collaborative call method, a multi-screen collaborative call system, a multi-screen collaborative call terminal and a storage medium.

Background

With the development of electronic technology and mobile internet, a user or a home may have multiple terminals, such as a mobile phone, a notebook computer, a tablet computer, and a smart home device, at the same time. In the prior art, multiple terminals can cooperate with each other to realize cross-device multi-screen cooperation, so that continuous service experience is provided for users.

In the prior art, multiple terminals are in multi-screen coordination, and only the terminal with the SIM card can be used for making a call, for example, a mobile phone and a tablet computer are in multi-screen coordination, and if the SIM card is arranged in the mobile phone, a user can only make a call with the mobile phone, and the tablet computer cannot hear the sound of the other party.

Disclosure of Invention

An object of the embodiments of the present application is to provide a multi-screen collaborative call method, a multi-screen collaborative call system, a multi-screen collaborative terminal, and a storage medium, so as to solve a problem how to switch a call sound from a mobile phone to a tablet/computer when an operator calls in a multi-screen collaborative state.

In a first aspect, an embodiment of the present application provides a multi-screen collaborative call method, where the method includes: under the condition that a first terminal is connected with a second terminal and the second terminal displays a first terminal interface, the first terminal receives an incoming call of a device at the opposite end of a call; responding to an incoming call answering operation of a user on the first terminal interface in the second terminal, wherein the first terminal receives call downlink data sent by a call opposite-end device and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data, wherein the first terminal comprises an audio processor, a first PCM device and a virtual call hardware management module, the audio processor is used for receiving the call downlink data, the first PCM device is used for acquiring the call downlink data from the audio processor, and the virtual call hardware management module is used for reading the call downlink data from the first PCM device and sending the call downlink data to the second terminal; the second terminal collects first call uplink data and sends the first call uplink data to the call opposite terminal equipment through the first terminal; responding to a recording starting operation of a user, and obtaining recording data by the first terminal, wherein the recording data comprises the call downlink data and the first call uplink data.

According to the multi-screen cooperative conversation method provided by the embodiment of the application, when the first terminal and the second terminal are in multi-screen cooperation, if a user operates the first terminal interface displayed by the second terminal to answer an incoming call, conversation sound can be switched from the first terminal to the second terminal, namely, the sound of the other party in conversation is played through the second terminal, and meanwhile, the sound of the user is collected by the second terminal and is sent to the opposite-end conversation equipment through the first terminal, so that the user can be in conversation with the other party in conversation through the second terminal. At this time, if the user starts the recording function, the recording file includes the sound of the other party of the call sent by the device at the opposite end of the call and the sound of the user collected by the second terminal.

Optionally, the method further comprises: under the condition that a first terminal is connected with a second terminal and the second terminal displays a first terminal interface, responding to the dialing operation of a user on the first terminal interface in the second terminal, and calling the opposite-end conversation equipment by the first terminal; after the opposite-end call device is connected, the first terminal receives call downlink data sent by the opposite-end call device and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data; the second terminal collects first call uplink data and sends the first call uplink data to the call opposite terminal equipment through the first terminal.

In the embodiment of the application, when the first terminal and the second terminal are in multi-screen cooperation, if a user operates the first terminal interface displayed by the second terminal to make a call, the call sound can be switched from the first terminal to the second terminal, so that the user can communicate with the other party through the second terminal.

Optionally, the method further comprises: under the condition that the first terminal is in communication with the opposite-end communication device, the first terminal displays a popup frame, wherein the popup frame is used for prompting a user to cooperate the first terminal and the second terminal in a multi-screen mode; responding to the operation of a user on a connection icon in the bullet frame, connecting the first terminal with the second terminal, displaying a first terminal interface by the second terminal, receiving call downlink data sent by a call opposite terminal device by the first terminal and sending the call downlink data to the second terminal, and playing the call downlink data by the second terminal; the second terminal collects first call uplink data and sends the first call uplink data to the call opposite terminal equipment through the first terminal.

In the embodiment of the application, when a user communicates with the other party of the communication through the first terminal, if the user connects the first terminal and the second terminal and starts multi-screen cooperation, the communication sound can be switched from the first terminal to the second terminal, so that the user communicates with the other party of the communication through the second terminal.

Optionally, the method further comprises: under the conditions that the first terminal is in communication with the opposite-end communication device, the first terminal is connected with the second terminal, and the second terminal displays a first terminal interface, the first terminal displays a first notification panel; responding to the operation of a user on an audio and video switching icon in the first notification panel in the first terminal, receiving call downlink data sent by a call opposite-end device by the first terminal and sending the call downlink data to the second terminal, and playing the call downlink data by the second terminal; the second terminal collects first call uplink data and sends the first call uplink data to the call opposite terminal equipment through the first terminal.

In the embodiment of the application, when a user communicates with the other party of the communication through the first terminal and the first terminal is in multi-screen cooperation with the second terminal, if the user operates the first terminal to perform audio and video switching, the communication sound can be switched from the first terminal to the second terminal, so that the user communicates with the other party of the communication through the second terminal.

Optionally, the method further comprises: under the conditions that the first terminal is in communication with the opposite-end communication device, the first terminal is connected with the second terminal, and the second terminal displays a first terminal interface, the second terminal displays a first notification panel; responding to the operation of a user on an audio and video switching icon in the first notification panel in the second terminal, the first terminal receives call downlink data sent by a call opposite terminal device and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data; the second terminal collects first call uplink data and sends the first call uplink data to the call opposite terminal equipment through the first terminal.

In the embodiment of the application, when a user communicates with the other party of the communication through the first terminal and the first terminal is in multi-screen cooperation with the second terminal, if the user operates the second terminal to perform audio and video switching, the communication sound can be switched from the first terminal to the second terminal, so that the user communicates with the other party of the communication through the second terminal.

Optionally, the first terminal further includes: the system comprises a call application program, a coordination module, a transmission module, a second PCM device and an audio device; the second terminal includes: a transmission module and an audio device;

the first terminal receives call downlink data sent by a call opposite terminal device and sends the call downlink data to the second terminal, and the step of playing the call downlink data by the second terminal comprises the following steps: the conversation application program responds to the operation of a user and sends a virtual conversation starting instruction to the cooperation module; the cooperation module sends the virtual call starting instruction to the virtual call hardware management module; in response to receiving the virtual call starting instruction, the virtual call hardware management module disables audio equipment of the first terminal, opens the first PCM equipment, and disables the first PCM equipment from acquiring uplink data; the audio processor receives the call downlink data sent by the call opposite-end equipment; the first PCM equipment acquires the call downlink data from the audio processor; the virtual call hardware management module reads the call downlink data from the first PCM device and sends the call downlink data to a transmission module of the first terminal; the transmission module of the first terminal sends the call downlink data to the transmission module of the second terminal, and the audio equipment of the second terminal plays the call downlink data;

the second terminal collects first call uplink data and sends the first call uplink data to the call opposite terminal equipment through the first terminal, and the steps comprise: the audio equipment of the second terminal collects the first call uplink data, and the transmission module of the second terminal sends the first call uplink data to the transmission module of the first terminal; the transmission module of the first terminal sends the first call uplink data to the virtual call hardware management module; the virtual call hardware management module writes the first call uplink data into the second PCM equipment; and the audio processor reads the first call uplink data from the second PCM equipment and sends the first call uplink data to the call opposite-end equipment.

In the embodiment of the application, the cooperation module, the virtual call hardware management module and the transmission module are added to the first terminal, and the transmission module is added to the second terminal, so that the call sound is switched from the first terminal to the second terminal, and a user can communicate with the other party through the second terminal.

Optionally, the first terminal further includes: the system comprises an audio service class, an audio hardware management module and a virtual call cache region; the step of the first terminal obtaining the recording data in response to the recording starting operation of the user includes: the call application program responds to the recording starting operation and sends a recording starting instruction to the audio service class; the audio service class sends the recording starting instruction to the audio hardware management module; and responding to the received recording starting instruction, the audio hardware management module acquires the recording data from the virtual call cache region, wherein the recording data is stored after the virtual call hardware management module performs sound mixing on the call downlink data and the first call uplink data.

In the embodiment of the application, the virtual call hardware management module is used for mixing the sound of the other party of the call and the sound of the user collected by the second terminal and storing the mixed sound into the virtual call cache region, and the audio hardware management module is used for reading the sound of the two parties of the call from the virtual call cache region, so that the first terminal can record the sound of the two parties of the call when the user calls the other party of the call through the second terminal.

Optionally, the method further comprises: under the conditions that a first terminal is connected with a second terminal, the second terminal displays a first terminal interface, and a user communicates with a communication opposite-end device through the second terminal, the first terminal plays communication downlink data and collects second communication uplink data to send the second communication uplink data to the communication opposite-end device in response to the closing operation of the user on the first terminal interface in the second terminal; the first terminal obtains recording data, wherein the recording data comprises the call downlink data and the second call uplink data.

In the embodiment of the application, when the user communicates with the other party of the call through the second terminal and the first terminal and the second terminal are in multi-screen cooperation, if the user operates the second terminal to disconnect the first terminal, the call sound can be switched back to the first terminal from the second terminal, so that the user communicates with the other party of the call through the first terminal.

Optionally, the method further comprises: the method comprises the steps that under the conditions that a first terminal is connected with a second terminal, the second terminal displays a first terminal interface, and a user communicates with a communication opposite-end device through the second terminal, the first terminal displays a second notification panel; and responding to the operation that the user breaks off the icon in the first terminal on the second notification panel, and the first terminal plays the call downlink data, collects the second call uplink data and sends the second call uplink data to the call opposite-end equipment.

In the embodiment of the application, when the user communicates with the other party of the call through the second terminal and the first terminal and the second terminal are in multi-screen cooperation, if the user operates the first terminal to disconnect the second terminal, the call sound can be switched back to the first terminal from the second terminal, so that the user communicates with the other party of the call through the first terminal.

Optionally, the method further comprises: the method comprises the steps that under the conditions that a first terminal is connected with a second terminal, the second terminal displays a first terminal interface, and a user communicates with a communication opposite-end device through the second terminal, the first terminal displays a second notification panel; and responding to the operation of the user on the audio and video switching icon in the second notification panel in the first terminal, and the first terminal plays the call downlink data, collects the second call uplink data and sends the second call uplink data to the call opposite terminal equipment.

In the embodiment of the application, when a user communicates with the other party of the communication through the second terminal and the first terminal is in multi-screen cooperation with the second terminal, if the user operates the first terminal to perform audio and video switching, the communication sound can be switched back to the first terminal from the second terminal, so that the user communicates with the other party of the communication through the first terminal.

Optionally, the method further comprises: under the conditions that a first terminal is connected with a second terminal, the second terminal displays a first terminal interface, and a user communicates with the opposite-end communication device through the second terminal, the second terminal displays a second notification panel; and responding to the operation of the user on the audio and video switching icon in the second notification panel in the second terminal, the first terminal plays the call downlink data, collects the second call uplink data and sends the second call uplink data to the call opposite terminal equipment.

In the embodiment of the application, when a user communicates with the other party of the communication through the second terminal and the first terminal is in multi-screen cooperation with the second terminal, if the user operates the second terminal to perform audio and video switching, the communication sound can be switched back to the first terminal from the second terminal, so that the user communicates with the other party of the communication through the first terminal.

Optionally, the first terminal further includes: the system comprises a call application program, a cooperation module, a transmission module and audio equipment;

the first terminal plays the call downlink data and collects second call uplink data to be sent to the call opposite terminal device, and the steps comprise: the communication application program responds to the operation of a user and sends a virtual communication closing instruction to the cooperation module; the cooperation module sends the virtual call closing instruction to the virtual call hardware management module; in response to receiving the virtual call closing instruction, the virtual call hardware management module cancels the forbidding of the audio equipment of the first terminal and cancels the forbidding of the first PCM equipment to acquire uplink data; the audio processor receives the call downlink data and plays the call downlink data through the audio equipment of the first terminal, acquires the second call uplink data collected by the audio equipment of the first terminal and sends the second call uplink data to the call opposite terminal equipment.

In the embodiment of the application, the communication sound is switched back to the first terminal from the second terminal through the cooperation module and the virtual communication hardware management module which are additionally arranged in the first terminal, so that a user can communicate with the other party through the first terminal. Optionally, the first terminal further includes an audio hardware management module; the step of the first terminal obtaining the recording data includes: and the audio hardware management module reads the recording data from the first PCM device, wherein the recording data is obtained by the first PCM device acquiring the call downlink data and the second call uplink data from the audio processor and mixing the audio data.

In the embodiment of the application, after the call sound is switched back to the first terminal from the second terminal, the sound of the other party of the call and the sound of the user collected by the first terminal are obtained from the audio processor through the first PCM device, and the sound of the two parties of the call is read from the first PCM device by the audio hardware management module, so that call recording is realized.

Optionally, the first terminal further includes: the system comprises a call application program, audio service classes, an audio hardware management module and a virtual call buffer;

the step of the first terminal obtaining the recording data in response to the recording starting operation of the user includes: the call application program responds to the recording starting operation and sends a recording starting instruction to the audio service class; the audio service class sends the recording starting instruction to the audio hardware management module; in response to receiving the recording starting instruction, the audio hardware management module judges whether the first terminal carries out virtual communication currently; if the first terminal currently performs a virtual call, the audio hardware management module acquires the recording data from the virtual call cache region, wherein the recording data is stored after the virtual call hardware management module performs audio mixing on the call downlink data and the first call uplink data; if the second terminal does not currently perform virtual call, the audio hardware management module opens the first PCM device and reads the recording data from the first PCM device, wherein the recording data is obtained by the first PCM device acquiring the call downlink data and the second call uplink data from the audio processor and performing sound mixing.

In the embodiment of the application, when the first terminal starts recording, if a user is in a call with the other party through the second terminal, the recording is the sound of the other party and the sound of the user collected by the second terminal; recording the sound of the other party and the user sound collected by the first terminal if the user is in a call with the other party through the first terminal; thereby ensuring normal recording of the sound of both parties of the call.

Optionally, the method further comprises: the call application program responds to the recording closing operation of the user and sends a recording closing instruction to the audio service class; the audio service class sends the recording closing instruction to the audio hardware management module; in response to receiving the recording closing instruction, the audio hardware management module judges whether the first terminal carries out virtual communication currently; if the first terminal is currently in a virtual call, the audio hardware management module does not close the first PCM device and stops acquiring the recording data from the virtual call cache region; if the first terminal does not perform virtual call currently, the audio hardware management module closes the first PCM device to read the recording data, and stops acquiring the recording data from the first PCM device.

In the embodiment of the application, when the first terminal closes the recording, if the user is in a conversation with the other party through the second terminal, the first PCM equipment is not closed; if the user is talking with the other party through the first terminal, closing the first PCM equipment; thereby ensuring the normal conversation between the user and the other party of the conversation.

Optionally, the virtual call hardware management module maintains a virtual call flag bit, where the virtual call flag bit has a first state and a second state, the first state indicates that the first terminal currently performs a virtual call, and the second state indicates that the first terminal does not currently perform a virtual call; the step of judging whether the first terminal carries out virtual communication currently by the audio hardware management module comprises the following steps: the audio hardware management module detects the virtual call flag bit; if the virtual call marking bit is in the first state, the first terminal carries out virtual call currently; and if the virtual call marking bit is in the second state, the first terminal does not perform virtual call currently.

In the embodiment of the application, when the first terminal starts or closes the recording, the audio hardware management module determines whether the user is in a conversation with the other party of the conversation through the first terminal or the second terminal by detecting the virtual conversation mark bit, so that the normal recording of the sound of the two parties of the conversation is ensured.

Optionally, the audio hardware management module maintains a recording flag bit; the method further comprises the following steps: in response to receiving the recording starting instruction, the audio hardware management module sets the recording mark position as a first identifier, and the first identifier represents that the first terminal currently carries out call recording; and responding to the received recording closing instruction, and enabling the audio hardware management module to set the recording mark position as a second identifier, wherein the second identifier represents that the first terminal does not record the call currently.

In the embodiment of the application, when the first terminal starts or closes the recording, the audio hardware management module resets the recording mark bit, so that whether a user records the sound or not can be accurately judged by the virtual communication hardware management module when the communication sound is switched from the first terminal to the second terminal or the communication sound is switched from the second terminal to the first terminal, and the sound of both parties of the communication is ensured to be recorded normally.

Optionally, the first terminal further includes: the system comprises a call application program, a coordination module, a transmission module, second PCM equipment, a virtual call cache region and audio equipment;

the method comprises the steps that the first terminal receives call downlink data sent by call opposite-end equipment and sends the call downlink data to the second terminal, the second terminal plays the call downlink data, the second terminal collects first call uplink data, and the first call uplink data are sent to the call opposite-end equipment through the first terminal, and the method comprises the following steps:

the communication application program responds to the operation of a user and sends a virtual communication starting instruction to the cooperation module;

the cooperation module sends the virtual call starting instruction to the virtual call hardware management module;

in response to receiving the virtual call starting instruction, the virtual call hardware management module judges whether the first terminal carries out call recording currently;

if the first terminal currently carries out call recording, the virtual call hardware management module disables audio equipment of the first terminal, opens the first PCM equipment and prohibits the first PCM equipment from acquiring uplink data, reads the call downlink data from the first PCM equipment and sends the call downlink data to a transmission module of the first terminal, receives the first call uplink data sent by the transmission module of the first terminal and writes the first call uplink data into the second PCM equipment, and carries out sound mixing on the call downlink data and the first call uplink data to obtain the recording data and stores the recording data into the virtual call cache region;

if the first terminal does not record the call currently, the virtual call hardware management module disables the audio device of the first terminal, opens the first PCM device and prohibits the first PCM device from acquiring uplink data, reads the call downlink data from the first PCM device, sends the call downlink data to the transmission module of the first terminal, receives the first call uplink data sent by the transmission module of the first terminal, and writes the first call uplink data into the second PCM device.

In the embodiment of the application, when the call sound is switched from the first terminal to the second terminal, if the user is calling and recording with the other party, the sound of the other party and the sound of the user collected by the second terminal are mixed by the virtual call hardware management module and then stored in the virtual call buffer, so that the sound of the two parties of the call is read from the virtual call buffer by the audio hardware management module, and the first terminal can continue recording while the user calls with the other party through the second terminal.

Optionally, the step of playing, by the first terminal, the call downlink data, collecting second call uplink data, and sending the second call uplink data to the call peer device includes: the communication application program responds to the operation of a user and sends a virtual communication closing instruction to the cooperation module; the cooperation module sends the virtual call closing instruction to the virtual call hardware management module; in response to receiving the virtual call closing instruction, the virtual call hardware management module judges whether the first terminal carries out call recording currently; if the first terminal currently carries out call recording, the virtual call hardware management module removes the forbidding of the audio equipment of the first terminal and cancels the forbidding of the first PCM equipment to obtain uplink data; and if the first terminal does not record the call currently, the virtual call hardware management module removes the forbidding of the audio equipment of the first terminal and closes the first PCM equipment.

In the embodiment of the application, when the call sound is switched back to the first terminal from the second terminal, if the user is calling and recording with the other party, the virtual call hardware management module cancels the prohibition of the first PCM device for acquiring the uplink data, so that the first PCM device can acquire the sound of the other party and the sound of the user acquired by the first terminal from the audio processor, and the audio hardware management module can read the sounds of the two parties of the call from the first PCM device, thereby realizing that the first terminal can continue recording while the user calls with the other party of the call through the first terminal.

Optionally, the first terminal further includes an audio hardware management module, where the audio hardware management module maintains a recording flag bit, the recording flag bit has a first identifier and a second identifier, the first identifier indicates that the first terminal currently performs call recording, and the second identifier indicates that the first terminal does not currently perform call recording; the virtual call hardware management module judges whether the first terminal carries out call recording currently or not, and the method comprises the following steps: the virtual call hardware management module detects the recording mark bit; if the recording mark bit is in the first identifier, the first terminal currently carries out call recording; if the recording mark bit is in the second identifier, the first terminal does not record the call currently.

In the embodiment of the application, when the call sound is switched from the first terminal to the second terminal or the call sound is switched back from the second terminal to the first terminal, the virtual call hardware management module determines whether the user is recording or not by detecting the recording mark bit, so that the sound of both parties of the call can be continuously recorded after the terminal is switched.

Optionally, the virtual call hardware management module maintains a virtual call flag bit, and the method further includes: in response to receiving the virtual call starting instruction, the virtual call hardware management module sets the virtual call mark position to be a first state, and the first state represents that the first terminal carries out virtual call currently; and in response to receiving the virtual call closing instruction, the virtual call hardware management module sets the virtual call mark position to be in a second state, wherein the second state represents that the first terminal does not perform virtual call currently.

In the embodiment of the application, when the call sound is switched from the first terminal to the second terminal or the call sound is switched back from the second terminal to the first terminal, the virtual call hardware management module resets the virtual call flag bit, so that when the first terminal starts or closes the recording, the audio hardware management module can accurately judge whether the user is calling with the other party through the first terminal or the second terminal, and normal recording of the sound of the two parties of the call is ensured.

In a second aspect, an embodiment of the present application further provides a multi-screen collaborative call method, which is applied to a first terminal, and the method includes: the method comprises the steps that a first terminal receives an incoming call of a device at the opposite end of a call, wherein the first terminal is connected with a second terminal, and the second terminal displays a first terminal interface; responding to an incoming call answering operation of a user on the first terminal interface in the second terminal, receiving call downlink data sent by a call opposite-end device and sending the call downlink data to the second terminal so that the second terminal plays the call downlink data; receiving first call uplink data collected by the second terminal, and sending the first call uplink data to the call opposite-end equipment; and responding to the recording starting operation of a user to obtain recording data, wherein the recording data comprises the call downlink data and the first call uplink data.

In a third aspect, an embodiment of the present application further provides a multi-screen collaborative call method, which is applied to a second terminal, and the method includes: receiving and playing call downlink data sent by a first terminal, wherein the call downlink data is sent to the first terminal by the first terminal after the first terminal is connected with a second terminal and the second terminal displays a first terminal interface and receives an incoming call of a call opposite terminal device and responds to an incoming call answering operation of a user on the first terminal interface in the second terminal; the second terminal collects first call uplink data and sends the first call uplink data to the call opposite-end equipment through the first terminal, so that the first terminal responds to the recording starting operation of a user and obtains recording data, wherein the recording data comprise the call downlink data and the first call uplink data.

In a fourth aspect, an embodiment of the present application further provides a terminal, where the terminal includes: one or more processors; a memory; an audio device; a display screen; wherein the memory is to store computer program code comprising computer instructions; when the processor executes the computer instruction, the terminal executes the multi-screen collaborative call method executed by the first terminal in the first aspect, or the terminal executes the multi-screen collaborative call method executed by the second terminal in the first aspect.

In a fifth aspect, an embodiment of the present application further provides a multi-screen collaborative call system, where the system includes a first terminal and a second terminal, the first terminal performs the multi-screen collaborative call method performed by the first terminal in the first aspect, and the second terminal performs the multi-screen collaborative call method performed by the second terminal in the first aspect.

In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes a computer program, and when the computer program runs on a terminal, the terminal is enabled to execute the multi-screen cooperative call method executed by the first terminal in the first aspect, or the terminal is enabled to execute the multi-screen cooperative call method executed by the second terminal in the first aspect.

Advantageous effects of the second to sixth aspects and implementations thereof described above reference may be made to the description of the advantageous effects of the method of the first aspect and implementations thereof.

Drawings

Fig. 1 is a schematic architecture diagram of a multi-screen collaborative call system according to an embodiment of the present disclosure;

fig. 2 is a first schematic structural diagram of a terminal according to an embodiment of the present disclosure;

fig. 3 is a block diagram of a software structure of a terminal according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram of a telephone function of a terminal provided in the prior art;

fig. 5 is a first flowchart illustrating a multi-screen collaborative call method according to an embodiment of the present application;

fig. 6 is a first application scenario diagram of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 7 is a schematic view of an application scenario of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 8 is a schematic application scenario diagram three of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 9 is a fourth application scenario diagram of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 10 is a schematic view of an application scenario of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 11 is a first data flow diagram of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 12 is a first interaction diagram of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 13 is a second flowchart illustrating a multi-screen collaborative call method according to an embodiment of the present application;

fig. 14 is a schematic application scenario six of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 15 is a seventh application scenario diagram of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 16 is an application scenario diagram eight of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 17 is a schematic view nine of an application scenario of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 18 is a second interaction diagram of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 19 is a data flow diagram of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 20 is a data flow chart of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 21 is a data flow chart of a multi-screen cooperative call method according to an embodiment of the present application;

fig. 22 is a third interaction schematic diagram of a multi-screen collaborative call method according to an embodiment of the present application;

fig. 23 is a fourth interaction diagram of a multi-screen collaborative call method according to the embodiment of the present application;

fig. 24 is a schematic structural diagram of a terminal according to an embodiment of the present application.

Detailed Description

The multi-screen cooperation is a distributed technology, and multiple terminals can be mutually matched to realize cross-equipment and cross-system multi-screen cooperation. The mobile phone is in multi-screen cooperation with the tablet/computer, and a user can control the mobile phone by operating the tablet/computer, such as returning mails, answering/making calls, and the like.

At present, when a mobile phone and a tablet/computer are in multi-screen coordination, if the mobile phone and a call opposite-end device carry out operator call, because the operator call needs to depend on an SIM card on the mobile phone, in the call process, the sound of the call other party can only be sent to the mobile phone of a user through a mobile network, and the sound of the user can also only be sent to the call other party through the mobile phone. That is, although the user can complete the operation of receiving/making a call by operating the tablet/computer, the user cannot hear the voice of the other party of the call through the tablet/computer.

Therefore, when an operator calls in a multi-screen coordination state, how to switch the call sound from the mobile phone to the tablet/computer is a technical problem to be solved by the embodiment of the present application.

In order to solve the foregoing technical problem, an embodiment of the present application provides a multi-screen collaborative call method, where a mobile phone can send a sound of another party of a call to a tablet/computer for playing, and meanwhile, the tablet/computer can collect a sound of a user and send the sound to a device at a call opposite end through the mobile phone, so as to switch the call sound from the mobile phone to the tablet/computer.

Embodiments of the present application will be described in detail below with reference to the accompanying drawings.

The multi-screen collaborative call method provided by the embodiment of the present application can be applied to the multi-screen collaborative call system 100 shown in fig. 1. As shown in fig. 1, the multi-screen collaborative call system 100 may include a first terminal 101 and a second terminal 102.

The first terminal 101 may be a device having a display function and a phone function, for example, a mobile phone, a tablet computer, a wearable electronic device, a notebook computer, a smart television, and the like, which is not limited in this embodiment of the present application.

The second terminal 102 may be a device having a display function and a radio/audio playing function, such as a mobile phone, a tablet Computer, a smart television, a notebook Computer, an Ultra-mobile Personal Computer (UMPC), a handheld Computer, a netbook, a Personal Digital Assistant (PDA), a wearable electronic device, an in-vehicle device, a virtual reality device, and the like, which is not limited in this embodiment.

The first terminal 101 may perform multi-screen coordination with the second terminal 102, and the first terminal 101 may perform a call with a call peer device. While the first terminal 101 and the second terminal 102 are in multi-screen coordination, if the first terminal 101 is in a call with a call peer device, the first terminal 101 may switch the call sound from the first terminal 101 to the second terminal 102, so that the call sound is emitted from the tablet/computer.

For example, the first terminal 101 may be a mobile phone, and the second terminal 102 may be a tablet computer or a notebook computer, or the first terminal 101 may be a tablet computer, and the second terminal 102 may be a notebook computer or a smart television, or both the first terminal 101 and the second terminal 102 may be mobile phones, or both the first terminal 101 and the second terminal 102 may be tablet computers, and the like.

Taking the first terminal 101 as a mobile phone and the second terminal 102 as a tablet computer for example, in some scenarios, the call sound may be switched from the mobile phone to the tablet computer, so that the sound of the other party of the call is emitted from the tablet computer.

The opposite-end call device is a device of the other party to the call, and the opposite-end call device can be a device with a telephone function, such as a mobile phone, a tablet computer, a wearable electronic device, a notebook computer, a smart television and the like.

Referring to fig. 2, which is a schematic structural diagram of a terminal 200 according to an embodiment of the present disclosure, the terminal 200 may be a first terminal 101, such as a mobile phone, in the multi-screen collaborative call system 100; the second terminal 102 may also be the second terminal in the multi-screen collaborative phone system 100, such as a tablet computer. The hardware structure of the terminal 200 will be described below by taking a mobile phone as an example.

As shown in fig. 2, the terminal 200 may include a processor 210, an external memory interface 220, an internal memory 221, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, keys 290, a motor 291, an indicator 292, a camera 293, a display 294, a SIM card interface 295, and the like. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity optical sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, a rotation axis sensor 280N, and the like.

It is to be understood that the illustrated structure of the present embodiment does not constitute a specific limitation of the terminal 200. In other embodiments, terminal 200 may include more or fewer components than shown, or combine certain components, or split certain components, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Processor 210 may include one or more processing units, such as: the processor 210 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors.

A memory may also be provided in processor 210 for storing instructions and data. In some embodiments, the memory in processor 210 is a cache memory. The memory may hold instructions or data that have just been used or recycled by processor 210. If the processor 210 needs to use the instruction or data again, it can be called directly from the memory. Avoiding repeated accesses reduces the latency of the processor 210, thereby increasing the efficiency of the system.

The charge management module 240 is configured to receive a charging input from a charger. The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and/or the charging management module 240, and provides power to the processor 210, the internal memory 221, the external memory, the display 294, the camera 293, and the wireless communication module 260.

The wireless communication function of the terminal 200 may be implemented by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, a modem processor, a baseband processor, and the like.

The

antennas

1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in terminal 200 may be used to cover a single or multiple communication bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 250 may provide a solution including 2G/3G/4G/5G wireless communication and the like applied on the terminal 200. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a Low Noise Amplifier (LNA), and the like. The mobile communication module 250 can receive the electromagnetic wave from the antenna 1, filter, amplify, etc. the received electromagnetic wave, and transmit the electromagnetic wave to the modem processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modem processor, and convert the signal into electromagnetic wave through the antenna 1 to radiate the electromagnetic wave. In some embodiments, at least some of the functional modules of the mobile communication module 250 may be disposed in the processor 210. In some embodiments, at least some of the functional modules of the mobile communication module 250 may be disposed in the same device as at least some of the modules of the processor 210.

The wireless communication module 260 may provide solutions for wireless communication applied on the terminal 200, including wireless-local area networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), bluetooth (bluetooth, BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), and the like. The wireless communication module 260 may be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on electromagnetic wave signals, and transmits the processed signals to the processor 210. The wireless communication module 260 may also receive a signal to be transmitted from the processor 210, frequency-modulate and amplify the signal, and convert the signal into electromagnetic waves via the antenna 2 to radiate the electromagnetic waves.

In some embodiments, antenna 1 of terminal 200 is coupled to mobile communication module 250 and antenna 2 is coupled to wireless communication module 260, such that terminal 200 may communicate with networks and other devices via wireless communication techniques. The wireless communication technology may include global system for mobile communications (GSM), General Packet Radio Service (GPRS), code division multiple access (code division multiple access, CDMA), Wideband Code Division Multiple Access (WCDMA), time-division code division multiple access (time-division code division multiple access, TD-SCDMA), Long Term Evolution (LTE), LTE, BT, GNSS, WLAN, NFC, FM, and/or IR technologies, etc. The GNSS may include a Global Positioning System (GPS), a global navigation satellite system (GLONASS), a beidou satellite navigation system (BDS), a quasi-zenith satellite system (QZSS), and/or a Satellite Based Augmentation System (SBAS).

The terminal 200 implements a display function through the GPU, the display screen 294, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 294 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 294 is used to display images, video, and the like. For example, in the embodiment of the present application, the display screen 294 is used to display content such as an application icon, a shortcut menu option, and a delay setting window, so as to guide a user to set a delay time duration. The display screen 294 includes a display panel. The display panel may be a Liquid Crystal Display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode, AMOLED), a flexible light-emitting diode (FLED), a miniature, a Micro-o led, a quantum dot light-emitting diode (QLED), or the like. In some embodiments, terminal 200 may include 1 or N display screens 294, N being a positive integer greater than 1.

The terminal 200 may implement a photographing function through the ISP, the camera 293, the video codec, the GPU, the display screen 294, and the application processor.

The ISP is used to process the data fed back by the camera 293. For example, when a photo is taken, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converting into an image visible to naked eyes. The ISP can also carry out algorithm optimization on the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in camera 293.

The camera 293 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signal in standard RGB, YUV and other formats. In some embodiments, terminal 200 may include 1 or N cameras 293, N being a positive integer greater than 1.

The external memory interface 220 may be used to connect an external memory card, such as a Micro SD card, to extend the storage capability of the terminal 200. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, files such as music, video, etc. are saved in an external memory card.

Internal memory 221 may be used to store computer-executable program code, including instructions. The processor 210 executes various functional applications of the terminal 200 and data processing by executing instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The storage data area may store data (e.g., audio data, a phonebook, etc.) created during use of the terminal 200, and the like. In addition, the internal memory 221 may include a high-speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), and the like.

The terminal 200 may implement an audio function through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the earphone interface 270D, and the application processor. Such as music playing, recording, etc.

Audio module 270 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. Audio module 270 may also be used to encode and decode audio signals. In some embodiments, the audio module 270 may be disposed in the processor 210, or some functional modules of the audio module 270 may be disposed in the processor 210.

The speaker 270A, also called a "horn", is used to convert an audio electrical signal into an acoustic signal. The terminal 200 can listen to music through the speaker 270A or listen to a handsfree call.

The receiver 270B, also called "earpiece", is used to convert the electrical audio signal into an acoustic signal. When the terminal 200 receives a call or voice information, it is possible to receive voice by bringing the receiver 270B close to the human ear.

The microphone 270C, also referred to as a "microphone," is used to convert acoustic signals into electrical signals. When making a call or sending voice information, the user can input a voice signal into the microphone 170C by speaking the user's mouth near the microphone 270C. The terminal 200 may be provided with at least one microphone 270C. In other embodiments, the terminal 200 may be provided with two microphones 270C to achieve a noise reduction function in addition to collecting sound signals. In other embodiments, the terminal 200 may further include three, four or more microphones 270C to collect sound signals, reduce noise, identify sound sources, implement directional recording functions, and the like.

The headphone interface 270D is used to connect wired headphones. The headset interface 270D may be a USB interface 230, or may be a 3.5mm open mobile electronic device platform (OMTP) standard interface, a cellular telecommunications industry association (cellular telecommunications industry association) standard interface of the USA.

The SIM card interface 295 is used to connect a SIM card. The SIM card can be attached to and detached from the terminal 200 by being inserted into the SIM card interface 295 or being pulled out from the SIM card interface 295. The terminal 200 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 may support a Nano SIM card, a Micro SIM card, a SIM card, etc. Multiple cards can be inserted into the same SIM card interface 295 at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 295 may also be compatible with different types of SIM cards. The SIM card interface 295 may also be compatible with external memory cards. The terminal 200 interacts with the network through the SIM card to implement functions such as communication and data communication. In some embodiments, the terminal 200 employs esims, namely: an embedded SIM card. The eSIM card may be embedded in the terminal 200 and cannot be separated from the terminal 200.

It should be noted that the hardware structure of the terminal 200 is described by taking a mobile phone as an example, and those skilled in the art should understand that in practical applications, the hardware structures of the first terminal 101 and the second terminal 102 may be increased or decreased appropriately based on the above hardware structure, and the embodiment of the present application does not limit this.

The software system of the terminal 200 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application exemplifies a software structure of the terminal 200 by taking an Android system with a layered architecture as an example. Fig. 3 is a block diagram of a software configuration of the terminal 200 according to the embodiment of the present invention.

The layered architecture divides the software into several layers, each layer having a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, an application layer, an application framework layer, an Android runtime (Android runtime) and system library, and a kernel layer from top to bottom.

The application layer may include a series of application packages.

As shown in fig. 3, the application package may include Applications (APPs) such as camera, gallery, calendar, phone call, map, navigation, WLAN, bluetooth, music, video, short message, etc. The application framework layer provides an Application Programming Interface (API) and a programming framework for the application program of the application layer. The application framework layer includes a number of predefined functions.

Wherein, the conversation application program comprises: the function of making a call in response to a call-making operation or a call-receiving operation of the user (for example, clicking an answer key of a caller identification interface), the function of making a call recording in response to an open recording operation of the user (for example, clicking a record key of a call interface), and the function of making an emergency call in response to a call-making emergency operation of the user (for example, selecting an emergency contact to dial in the emergency call interface).

When the user operates the terminal 200 to answer a call, or make a call, or record a call, or make an emergency call, the call application generates a corresponding command to send to the application framework layer in response to the user's operation.

As shown in FIG. 3, the application framework layers may include a window manager, content provider, view system, phone manager, resource manager, notification manager, and the like.

The window manager is used for managing window programs. The window manager can obtain the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like.

The content provider is used to store and retrieve data and make it accessible to applications. The data may include video, images, audio, calls made and answered, browsing history and bookmarks, phone books, etc.

The view system includes visual controls such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide a communication function of the terminal 200. Such as management of call status (including on, off, etc.).

Wherein the telephony manager provides classes related to the call, for example, the Audio Service class (Audio Service). When the user operates the terminal 200 to answer a call, or make a call, or record a call, or make an emergency call, the call application generates a corresponding instruction to send to the audio service class of the application framework layer in response to the user's operation. The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and the like.

The notification manager enables the application to display notification information in the status bar, can be used to convey notification-type messages, can disappear automatically after a short dwell, and does not require user interaction. Such as a notification manager used to inform download completion, message alerts, etc. The notification manager may also be a notification that appears in the form of a chart or scroll bar text at the top status bar of the system, such as a notification of a background running application, or a notification that appears on the screen in the form of a dialog window. Such as prompting for text messages in the status bar, sounding a prompt tone, vibrating the device, flashing an indicator light, etc.

The Android Runtime comprises a core library and a virtual machine. The Android runtime is responsible for scheduling and managing an Android system.

The core library comprises two parts: one part is a function which needs to be called by java language, and the other part is a core library of android.

The application layer and the application framework layer run in a virtual machine. And executing java files of the application program layer and the application program framework layer into a binary file by the virtual machine. The virtual machine is used for performing the functions of object life cycle management, stack management, thread management, safety and exception management, garbage collection and the like.

The system library may include a plurality of functional modules. For example: surface managers (surface managers), Media Libraries (Media Libraries), three-dimensional graphics processing Libraries (e.g., OpenGL ES), Audio Libraries (Audio Libraries), and the like.

The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

The media library supports a variety of commonly used audio, video format playback and recording, and still image files, among others. The media library may support a variety of audio-video encoding formats, such as MPEG4, h.264, MP3, AAC, AMR, JPG, PNG, and the like.

The three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like.

The telephone manager only provides a bridge for accessing the core library for the call application program, and the specific function implementation is completed in the audio library. For example, the Audio library has an Audio Service class (Audio Service) therein corresponding to the Audio Service class of the application framework layer.

The HAL layer encapsulates a Linux kernel driver, provides an interface upwards and shields implementation details of bottom hardware.

The HAL layer may include Wi-Fi HAL, audio HAL (audio HAL), camera HAL (camera HAL), etc.

The Audio HAL defines a standard interface that the handset must implement to ensure the Audio hardware functions operate properly when called by the Audio Service, and is responsible for truly associating the Audio Flinger/Audio Policy Service with the hardware device.

The kernel layer is a layer between hardware and software. The inner core layer at least comprises a display driver, a camera driver, an audio driver and a sensor driver.

The audio driver is mainly responsible for interaction with hardware, including audio devices such as speakers, microphones, and the like.

To facilitate understanding of the embodiment of the present application more clearly, before a multi-screen collaborative call method in the embodiment of the present application is introduced, a telephone function of the first terminal 101 in the prior art is introduced based on the Android system introduced in fig. 3, taking the first terminal 101 as a mobile phone as an example.

The telephone function of the first terminal 101 includes a call, call recording, and an emergency call, and a flow of the first terminal 101 implementing the telephone function is exemplarily described below with reference to fig. 4.

In one implementation manner, as shown in fig. 4, a flow of a call between a first terminal 101 and a device at a peer-to-peer end of the call is as follows:

when the user answers the incoming call, the call application responds to the incoming call answering operation of the first terminal 101 by the user (for example, clicking an answer key of an incoming call display interface, etc.), generates a call opening instruction, calls the audio service class of the call manager and the audio service class of the audio library, sends the call opening instruction to the audio hardware management module of the audio HAL, and then sends the call opening instruction to the audio processor by the audio hardware management module. After receiving the call starting instruction, the audio processor receives call downlink data sent by the call opposite terminal device and sends the call downlink data to the loudspeaker for playing; and acquiring the call uplink data collected by the microphone and sending the call uplink data to the call opposite terminal equipment.

When the user makes a call, the call application responds to the call-making operation of the user to the first terminal 101 (for example, a call entering the main interface, selecting a relevant contact, clicking to make a call, etc.), and generates a call-opening instruction after the call-opposite terminal device is connected. Then, the call application program calls the audio service class of the telephone manager and the audio service class of the audio library, and sends the call starting instruction to the audio hardware management module of the audio HAL, and then the audio hardware management module sends the call starting instruction to the audio processor. After receiving the call starting instruction, the audio processor receives call downlink data sent by the call opposite-end equipment and sends the call downlink data to the loudspeaker for playing; and acquiring the call uplink data collected by the microphone and sending the call uplink data to the call opposite terminal equipment.

It should be noted that, in some scenarios, the audio processor may also send the call downlink data to the receiver for playing, or send the call downlink data to the wired earphone for playing through the earphone interface, or send the call downlink data to the wireless earphone for playing through bluetooth. Meanwhile, the call uplink data acquired by the audio processor may be acquired by a wireless earphone or a wired earphone connected to the terminal 200. The embodiment of the present application does not set any limit to this.

An Audio processor, i.e., an Audio DSP, is a Digital Signal Processor (DSP) that processes Audio. The Audio DSP is responsible for processing Audio signals and interacting with the Audio DSP of the opposite-end device for communication.

In another implementation manner, as shown in fig. 4, the flow of the first terminal 101 performing call recording is as follows:

in the process of a call between the first terminal 101 and the opposite-end device, when a user records a call, the call application responds to a recording start operation of the user (for example, clicking a recording key of a call interface), generates a recording start instruction, calls an audio service class in a telephone manager and an audio service class in an audio library, and sends the recording start instruction to an audio hardware management module of an audio HAL.

And after receiving the recording starting instruction, the audio hardware management module opens the first PCM equipment and designates an audio scene as an uplink channel and a downlink channel. And then, the first PCM equipment acquires the call uplink data and the call downlink data from the audio processor and performs sound mixing on the call uplink data and the call downlink data to obtain the recording data. And the audio hardware management module reads the recording data from the first PCM equipment, and sends the recording data to the call application program through the audio hardware management module and the audio service class, and the call application program stores the recording data.

The PCM device is a PCM device file, in a Linux system, all devices are finally abstracted into one or more device files accessible to a user space, and processes of the user space achieve the purpose of controlling hardware by reading and writing the device files. For playing or recording sound, the audio HAL is implemented by reading or writing to the PCM device file.

Recording data are required to be acquired from a designated PCM device in an Android system, the recording data comprise sounds of both parties of a call, the PCM device can be designated by selecting an audio scene (namely userase), and one PCM device can correspond to a plurality of usescases.

The first PCM device is an incall record PCM, that is, a PCM device for realizing call recording. The incall record PCM corresponds to three usectaes which are respectively as follows: an uplink path (usecast _ incall _ rec _ downlink), a downlink path (usecast _ incall _ rec _ uplink), an uplink path and a downlink path (usecast _ incall _ rec _ uplink _ and _ downlink). By selecting the three usechases, the first PCM device can respectively acquire call downlink data, call uplink data and call downlink data.

Because call uplink data and call downlink data are needed for realizing call recording, a call recording scene needs to select usecast _ initial _ rec _ uplink _ and _ downlink to acquire the call uplink data and the call downlink data. That is, the first PCM device is designated by selecting usecast _ include _ rec _ uplink _ and _ downlink, thereby realizing call recording.

In yet another implementation manner, as shown in fig. 4, the procedure of the first terminal 101 making the emergency call is as follows:

when a user dials an emergency call, the call application program responds to the operation of dialing the emergency call (for example, clicking the emergency call on a screen locking interface, clicking personal emergency information, selecting an emergency contact person to dial and the like), writes emergency call data into an audio service class of a telephone manager, writes the emergency call data into the audio service class of an audio library by the audio service class of the telephone manager, and writes the emergency call data into an audio hardware management module of an audio HAL by the audio service class of the audio library. And then, the audio hardware management module writes the emergency call data into the second PCM equipment, and the audio processor reads the emergency call data from the second PCM equipment and sends the emergency call data to the equipment at the opposite end of the call.

If the user makes an emergency call operation: and clicking an emergency call on a screen locking interface, clicking personal emergency information, selecting an emergency contact person for dialing, and then calling the equipment at the opposite end as the equipment of the emergency contact person. And after receiving the emergency call data, the equipment at the opposite end of the call plays the emergency call data, so that the emergency contact can help the user to make a call for help through the equipment.

The second PCM device is an incall music PCM, i.e., a PCM device for implementing an emergency call function. The incall music PCM also has a corresponding userase, and is different from the incall record PCM. For example, the userase corresponding to INCALL MUSIC PCM may be USECASE _ INCALL _ MUSIC _ UPLINK.

As can be seen from the above, in the existing call flow, the audio processor receives the call downlink data and sends the call downlink data to the speaker for playing, the Android system does not provide an interface for directly obtaining the call downlink data, and the audio system cannot directly obtain the call downlink data from the audio processor. However, in the existing call recording procedure, the first PCM device may obtain the call uplink data and the call downlink data from the audio processor.

Therefore, the multi-screen cooperative call method provided in the embodiment of the present application improves the phone function of the first terminal 101 in the prior art, so that the first terminal 101 receives the call downlink data of the call peer device and sends the call downlink data to the second terminal 102 for playing, and receives the first call uplink data collected by the second terminal 102 and sends the first call uplink data to the call peer device. As described in detail below.

The multi-screen cooperative communication method according to the embodiment of the present application will be described in detail below by taking the first terminal 101 as a mobile phone and the second terminal 102 as a tablet computer as an example.

Fig. 5 is a schematic flow chart of a multi-screen cooperative communication method according to an embodiment of the present disclosure. The multi-screen collaborative call method is applied to the multi-screen collaborative call system 100 shown in fig. 1, and may include S301 to S304.

S301, under the condition that the first terminal is connected with the second terminal and the second terminal displays the first terminal interface, the first terminal receives an incoming call of the opposite-end call device.

S302, responding to the operation of answering the incoming call to the interface of the first terminal in the second terminal by the user, the first terminal receives the call downlink data sent by the equipment of the opposite call terminal and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data.

And S303, the second terminal collects the first call uplink data and sends the first call uplink data to the call opposite terminal equipment through the first terminal.

The first terminal 101 is connected with the second terminal 102, and the second terminal 102 displays a first terminal interface, that is, the first terminal 101 and the second terminal 102 are performing multi-screen coordination. The first terminal interface refers to a window of the first terminal 101 which is displayed on the same screen as the second terminal 102 in a mirror image manner.

When the first terminal 101 and the second terminal 102 are in multi-screen coordination, if an incoming call comes from a device at the opposite end of the call, the first terminal 101 displays an incoming call display interface, and simultaneously, the first terminal interface in the second terminal 102 synchronously displays the incoming call display interface.

At this time, if the user operates the incoming call display interface displayed by the second terminal 102 to answer an incoming call, it can be considered that the user tends to make a call with the other party of the call through the second terminal 102, so that the call sound is switched from the first terminal 101 to the second terminal 102.

That is, the first terminal 101 receives downlink communication data sent by the call peer device and sends the downlink communication data to the second terminal 102, and meanwhile, the first terminal 101 receives first uplink call data collected by the second terminal 102 and sends the first uplink call data to the call peer device. The call downstream data refers to the voice of the other party of the call. The first call uplink data refers to a user voice collected by a sound receiving device (e.g., a microphone) of the second terminal 102.

That is, the first terminal 101 receives the sound of the other party of the call sent by the opposite-party device of the call and sends it to the second terminal 102, and the sound reproduction device (e.g., speaker) of the second terminal 102 reproduces the sound of the other party of the call. Meanwhile, the sound receiving device (e.g., a microphone) of the second terminal 102 collects the sound of the user and sends the sound to the opposite-end device through the first terminal 101.

For example, referring to fig. 6, the mobile phone and the tablet computer are in multi-screen cooperation, that is, the mobile phone is connected to the tablet computer, and the tablet computer displays the mobile phone interface 401. The method comprises the steps that a call is made on a mobile phone, a call display interface is displayed on a mobile phone interface 1021 in the tablet personal computer, if a user clicks an answering key in the call display interface of the tablet personal computer to answer, the sound of the other party to be communicated is sent out from the tablet personal computer, and meanwhile the sound of the user is collected by the tablet personal computer and sent to the other party to be communicated through the mobile phone.

S304, responding to the recording starting operation of the user, and the first terminal obtains recording data, wherein the recording data comprises call downlink data and first call uplink data.

The recording operation is started, the user may click the recording icon in the call interface of the first terminal 101, or the user may click the recording icon in the call interface synchronously displayed on the first terminal interface of the second terminal 102.

That is, under the multi-screen coordination of the first terminal 101 and the second terminal 102, if the first terminal 101 performs recording after opening the virtual call, the recording data includes call downlink data and first call uplink data. The virtual call means that the first terminal 101 and the opposite-end device are in a call state, but a call sound is sent from the second terminal 102, that is, the user makes a conversation with the other party of the call through the second terminal 102. The virtual call is opened, that is, the call sound is switched from the first terminal 101 to the second terminal 102.

In addition to the above application scenarios, in other scenarios, the first terminal 101 may also open the virtual call, and the following description is given by way of example.

In a possible implementation manner, on the basis of fig. 3, the multi-screen collaborative call method provided in the embodiment of the present application may further include S301-1, S302-1, S303, and S304.

S301-1, under the condition that the first terminal is connected with the second terminal and the second terminal displays the first terminal interface, responding to the dialing operation of the user on the first terminal interface in the second terminal, and calling the opposite-end equipment by the first terminal.

S302-1, after the call opposite-end equipment is connected, the first terminal receives call downlink data sent by the call opposite-end equipment and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data.

When the first terminal 101 and the second terminal 102 are in multi-screen coordination, if a user operates the first terminal interface displayed by the second terminal 102 to make a call, it may be considered that the user tends to make a call with the other party of the call through the second terminal 102, so the first terminal 101 opens a virtual call, and the call sound is switched from the first terminal 101 to the second terminal 102.

For example, referring to fig. 7, the mobile phone and the tablet computer are in multi-screen cooperation, that is, the mobile phone is connected to the tablet computer, and the tablet computer displays the mobile phone interface 401. If the user operates the mobile phone interface 401 displayed by the tablet computer to make a call, for example, a phone icon in the mobile phone interface 401 is clicked, after the call is entered, the relevant contact person is selected, and a dial key is clicked to dial, so that the other party of the call is called. After the other party of the call is connected, the sound of the other party of the call is sent out from the tablet personal computer, and meanwhile, the tablet personal computer collects the sound of the user and sends the sound to the other party of the call through the mobile phone.

In another possible implementation manner, on the basis of fig. 3, the multi-screen cooperative call method provided in the embodiment of the present application may further include S301-2, S302-2, S303, and S304.

S301-2, under the condition that the first terminal is in communication with the opposite-end communication device, the first terminal displays a popup frame, wherein the popup frame is used for prompting a user to cooperate the first terminal and the second terminal in a multi-screen mode.

S302-2, responding to the operation of the user on the connection icon in the popup frame, connecting the first terminal with the second terminal, displaying a first terminal interface by the second terminal, receiving call downlink data sent by the call opposite-end equipment by the first terminal and sending the call downlink data to the second terminal, and playing the call downlink data by the second terminal.

When a user communicates with a call peer device through a first terminal 101, if the user operates the first terminal 101 and the second terminal 102 to enable the first terminal 101 and the second terminal 102 to perform multi-screen coordination, it may be considered that the user tends to communicate with the other party of the call through the second terminal 102, so that the first terminal 101 opens a virtual call and switches call sound from the first terminal 101 to the second terminal 102.

For example, referring to fig. 8, when a user is talking with another party through a mobile phone, a pop-up frame 402 prompting the user to connect the mobile phone with a tablet computer is displayed in the mobile phone, and if the user clicks a connection icon in the pop-up frame 402, the first terminal is connected with the second terminal and the second terminal displays a first terminal interface, that is, the mobile phone is connected with the tablet computer, and multi-screen collaboration is started. At the moment, the voice of the other party to be communicated is sent out from the tablet personal computer, meanwhile, the tablet personal computer collects the voice of the user and sends the voice to the other party to be communicated through the mobile phone, and the user can communicate with the other party to be communicated through the tablet personal computer. In yet another possible implementation manner, on the basis of fig. 3, the multi-screen collaborative call method provided in the embodiment of the present application may further include S301-3, S302-3, S303, and S304.

S301-3, under the conditions that the first terminal is in communication with the opposite-end communication device, the first terminal is connected with the second terminal, and the second terminal displays a first terminal interface, the first terminal displays a first notification panel.

S302-3, responding to the operation of the user on the audio and video switching icon in the first notification panel in the first terminal, connecting the first terminal with the second terminal, displaying a first terminal interface by the second terminal, receiving call downlink data sent by the call opposite-end equipment by the first terminal and sending the call downlink data to the second terminal, and playing the call downlink data by the second terminal.

When a user communicates with a communication peer device through the first terminal 101 and the second terminal 102 are in multi-screen cooperation, if the user operates the first terminal 101 to switch audio and video to the second terminal, the first terminal 101 starts a virtual communication and switches communication sound from the first terminal 101 to the second terminal 102.

For example, referring to fig. 9, when a user is talking with another party through a mobile phone, and the mobile phone and a tablet pc are in multi-screen coordination, the user slides the status bar of the mobile phone downward to open the first notification panel 403, and if the user clicks the "audio/video switch to tablet" icon in the first notification panel 403, the talking sound is switched from the mobile phone to the tablet pc, that is, the sound of the other party is sent from the tablet pc, and at the same time, the tablet pc collects the sound of the user and sends the sound to the other party, and the user talks with the other party through the tablet pc.

In yet another possible implementation manner, on the basis of fig. 3, the multi-screen collaborative call method provided in the embodiment of the present application may further include S301-4, S302-4, S303, and S304.

S301-4, when the first terminal is in communication with the opposite-end communication device, the first terminal is connected with the second terminal, and the second terminal displays the first terminal interface, the second terminal displays the first notification panel.

S302-4, responding to the operation of the user on the audio and video switching icon in the first notification panel in the second terminal, connecting the first terminal with the second terminal, displaying a first terminal interface by the second terminal, receiving call downlink data sent by the call opposite-end equipment by the first terminal and sending the call downlink data to the second terminal, and playing the call downlink data by the second terminal.

When a user communicates with a communication peer device through the first terminal 101 and the second terminal 102 are in multi-screen cooperation, if the user operates the second terminal 102 to switch audio and video to the second terminal 102, the first terminal 101 starts a virtual communication and switches communication sound from the first terminal 101 to the second terminal 102.

For example, referring to fig. 10, when a user is talking with another party through a mobile phone and the mobile phone and a tablet pc are in multi-screen coordination, the user slides the status bar of the tablet pc downward to open the first notification panel 403, if the user clicks the "audio/video switch to tablet" icon in the first notification panel 403, the voice of the other party is switched from the mobile phone to the tablet pc, that is, the voice of the other party is emitted from the tablet pc, and at the same time, the tablet pc collects the voice of the user and sends the voice to the other party, and the user talks with the other party through the tablet pc.

Of course, besides the above-described possible implementation manners, a person skilled in the art may also design other implementation manners to open the virtual call, and the embodiment of the present application does not limit this.

The above introduces a complete call process in a virtual call scenario. However, as can be seen from the implementation of the existing telephone function described in fig. 4, the Android system does not provide an interface for directly acquiring the call downlink data, and meanwhile, the audio processor directly acquires the call uplink data acquired by the microphone of the first terminal 101 and sends the call uplink data to the device at the opposite end of the call. However, the first PCM device may obtain the call upstream data and the call downstream data from the audio processor, and the audio processor may read the emergency call data from the second PCM device.

Therefore, to implement the virtual call function in the embodiment of the present application, that is, the first terminal 101 receives the call downlink data of the call peer device and sends the call downlink data to the second terminal 102 for playing, and receives the first call uplink data collected by the second terminal 102 and sends the first call uplink data to the call peer device, the functions of the first PCM device and the second PCM device may be considered.

As can be seen from the description in fig. 4, in the call recording scenario, the first PCM device may obtain the call uplink data and the call downlink data from the audio processor, and therefore, in the virtual call scenario, the call downlink data may be obtained from the audio processor through the first PCM device.

Meanwhile, because call recording may be required in a virtual call scene, the first PCM device is set to include an uplink path and a downlink path, that is, the user _ include _ rec _ uplink _ and _ downlink is selected to acquire call uplink data and call downlink data.

In addition, as can be seen from the description in fig. 4, in an emergency call scenario, the audio hardware management module may write emergency call data into the second PCM device, and the audio processor reads the emergency call data from the second PCM device and sends the emergency call data to the opposite-end-to-call device. Therefore, the first call uplink data sent by the second terminal 102 may be written into the second PCM device, and then, the audio processor reads the first call uplink data from the second PCM device and sends the first call uplink data to the call peer device.

Therefore, the audio processor has both the call downlink data sent by the call peer device and the first call uplink data read from the second PCM device.

However, in a virtual call scenario, the first PCM device is configured to include an uplink and a downlink, so that the first PCM device may obtain call downlink data and first call uplink data from the audio processor, which inevitably causes echo to occur when the second terminal 102 plays. Therefore, it is desirable to close the upstream path of the first PCM device to prohibit the first PCM device from acquiring the first call upstream data from the audio processor.

Meanwhile, in a virtual call scenario, a user communicates with another party of the call through the second terminal 102, so that in order to avoid an echo, the first terminal 101 needs to disable its own audio device when starting the virtual call. The audio equipment can comprise playback equipment and radio equipment, the playback equipment can be a loudspeaker, a receiver, a wired earphone, a wireless earphone and the like, and the radio equipment can be a microphone, a wired earphone, a wireless earphone and the like. This is not limited in the embodiments of the present application, and the following embodiments are described by taking a speaker and a microphone as examples.

On the basis of the above analysis, compared to fig. 4, as shown in fig. 11, in order to implement a virtual call, a coordination module, a virtual call hardware management module, a transmission module, and a virtual call buffer may be added to the first terminal 101, and accordingly, a transmission module may be added to the second terminal 102, where the virtual call hardware management module is in the audio HAL shown in fig. 3.

The coordination module is started when the first terminal 101 and the second terminal 102 perform multi-screen coordination connection, and is mainly responsible for monitoring a call state and controlling call logic, for example, when a virtual call is monitored to be opened, a virtual call opening instruction is sent to the virtual call hardware management module; and when the virtual call closing is monitored, sending a virtual call closing instruction to the virtual call hardware management module.

It should be noted that, the second terminal 102 also has a corresponding coordination module (not shown in fig. 9), and the coordination module in the second terminal 102 may monitor the operation of the tablet computer by the user and generate a corresponding instruction. For example, when the user operation shown in fig. 6 or fig. 7 is monitored, a call request is generated and sent to the coordination module of the first terminal 101. In another example, the user operation shown in fig. 10 is monitored, and a call handover request is generated and sent to the coordination module of the first terminal 101.

The transmission module, which may be denoted as virModem transmisson, is mainly responsible for audio data transmission between the first terminal 101 and the second terminal 102.

The virtual call hardware management module can be represented as virModem hal and is mainly used for: and when receiving a virtual call opening instruction or a virtual call closing instruction sent by the cooperation module, controlling the opening and closing of the PCM equipment, setting an audio scene, reading and writing audio data, and mixing audio of call downlink data and first call uplink data.

The virtual talk cache is used for: and the storage virtual call hardware management module mixes the call downlink data and the first call uplink data to obtain mixed data.

The following describes a detailed implementation of call and call recording in a virtual call scenario with reference to fig. 11 and 12.

Referring to fig. 12, the above-described processes of the first terminal receiving the call downlink data sent by the call peer device and sending the call downlink data to the second terminal in S302, S302-1, S302-2, S302-3, and the second terminal playing the call downlink data, and the process of the second terminal acquiring the first call uplink data and sending the first call uplink data to the call peer device through the first terminal in S303 may include:

and S11, the call application program responds to the virtual call opening operation of the user and sends a virtual call opening instruction to the cooperation module. The virtual call operation may be any one of fig. 6 to 10, and is used to switch the call sound from the mobile phone to the tablet computer.

And S12, the cooperation module sends the virtual call starting instruction to the virtual call hardware management module.

S13, in response to receiving the virtual call starting instruction, the virtual call hardware management module disables the audio device of the first terminal, opens the first PCM device, and disables the first PCM device from acquiring uplink data.

When the call sound is switched from the mobile phone to the tablet personal computer, in order to ensure that the call sound is not played and received from the loudspeaker and the microphone of the mobile phone, the virtual call hardware management module needs to disable the loudspeaker and the microphone of the mobile phone, so that the loudspeaker of the mobile phone does not play the sound of the other party of the call, and the microphone of the mobile phone does not collect the sound of the user.

S15-1, the audio processor receives call downlink data sent by the call opposite-end device;

s15-2, the first PCM equipment acquires call downlink data from the audio processor;

s15-3, the virtual call hardware management module reads call downlink data from the first PCM device and sends the call downlink data to a transmission module of the first terminal;

s15-4, the transmission module of the first terminal sends the call downlink data to the transmission module of the second terminal;

s15-5, the transmission module of the second terminal sends the call downlink data to the audio equipment of the second terminal;

and S15-6, the audio equipment of the second terminal plays the call downlink data.

S15-1 to S15-6 are the implementation processes of playing the sound of the other party of the call by the tablet computer.

S16-1, the audio equipment of the second terminal collects the first call uplink data and transmits the first call uplink data to the transmission module of the second terminal;

s16-2, the transmission module of the second terminal sends the first call uplink data to the transmission module of the first terminal;

s16-3, the transmission module of the first terminal sends the first call uplink data to the virtual call hardware management module;

s16-4, the virtual call hardware management module writes the first call uplink data into the second PCM equipment;

and S16-5, the audio processor reads the first call uplink data from the second PCM equipment and sends the first call uplink data to the call opposite-end equipment.

S16-1-S16-5 is an implementation process that the tablet computer collects the voice of a user and sends the voice to the opposite-end communication device through the mobile phone.

The interaction process among the modules is the call flow under the virtual call scene. The call flow in the virtual call scene needs to use the first PCM device, and the virtual call hardware management module prohibits the first PCM device from acquiring uplink data from the audio processor, so that the audio hardware management module cannot read recording data from the first PCM device. Therefore, in order to implement call recording in the virtual call scenario, the call flow further includes:

s17-1, the virtual call hardware management module performs sound mixing on the call downlink data and the first call uplink data;

and S17-2, the virtual call hardware management module stores the mixed data in a virtual call buffer area.

As shown in fig. 11, the virtual call hardware management module mixes the call downlink data and the first call uplink data, and stores the mixed data, i.e., the recording data, in the virtual call cache region, and the audio hardware management module can read the recording data from the virtual call cache region, thereby realizing call recording in the virtual call scene.

Therefore, referring to fig. 12, the process of the first terminal obtaining the recording data in response to the recording start operation of the user in S304 may include:

s21, the call application program responds to the recording starting operation and sends a recording starting instruction to the audio service class;

s22, the audio service class sends the recording starting instruction to the audio hardware management module;

s17-3, in response to receiving the recording starting instruction, the audio hardware management module acquires recording data from the virtual call cache region;

s17-4, the audio hardware management module sends the recording data to an audio service class;

s17-5, the audio service class sends the recorded sound data to the communication application.

And S17-6, the communication application program stores the recorded data.

In some scenarios, the first terminal 101 may also close the virtual call to switch the call sound from the second terminal 102 back to the first terminal 101.

Therefore, referring to fig. 13 based on fig. 5, after S304, the multi-screen cooperative communication method according to the embodiment of the present application may further include S305 to S306.

S305, under the conditions that the first terminal is connected with the second terminal, the second terminal displays the first terminal interface, and the user communicates with the opposite-end communication device through the second terminal, the first terminal plays the downlink communication data and collects the uplink second communication data to send to the opposite-end communication device in response to the closing operation of the user on the first terminal interface in the second terminal.

When the first terminal 101 and the second terminal 102 cooperate in a multi-screen manner and a user makes a call with a call peer device through the second terminal 102, if the user operates the second terminal 102 to disconnect the first terminal 101, the first terminal 101 closes a virtual call, and the call sound is switched back to the first terminal 101 from the second terminal 102.

That is, the first terminal 101 receives and plays the downlink communication data sent by the peer-to-peer communication device, and collects the uplink second communication data and sends the second communication data to the peer-to-peer communication device. The second communication upstream data refers to user voice collected by the sound receiving device (e.g., microphone) of the first terminal 101.

That is, the first terminal 101 receives the sound of the other party of the call sent by the opposite-end device of the call, and plays the sound through a sound playing device (e.g., a speaker), and at the same time, collects the sound of the user through a sound receiving device (e.g., a microphone) and sends the sound to the opposite-end device of the call.

For example, referring to fig. 14, the user is talking to the other party via the tablet computer, and the mobile phone and the tablet computer are cooperating with each other in a multi-screen manner, that is, the mobile phone is connected to the tablet computer and the tablet computer displays the mobile phone interface 401. The user firstly clicks the multi-screen cooperation disconnection icon in the mobile phone interface 401 of the tablet computer, the tablet computer displays the popup frame 404 to prompt the user to confirm whether to disconnect the mobile phone from the tablet computer, and then the user clicks the disconnection icon in the popup frame 404, so that the first terminal is disconnected from the second terminal, and the multi-screen cooperation is stopped. At the moment, the voice of the other party is sent out from the mobile phone, meanwhile, the mobile phone collects the voice of the user and sends the voice to the other party, and the user communicates with the other party through the mobile phone.

S306, the first terminal obtains recording data, wherein the recording data comprise call downlink data and second call uplink data.

After the first terminal 101 closes the virtual call, the call sound is switched back to the first terminal 101 from the second terminal 102, and the user makes a conversation with the other party through the second terminal 102, which is the call flow of the first terminal 101 in the prior art. In this case, if the first terminal 101 is still recording the call, the recording data includes call downlink data and second call uplink data.

In addition to the above application scenarios, in some other scenarios, the first terminal 101 may also close the virtual call, and the following description is given by way of example.

In a possible implementation manner, on the basis of fig. 13, after S304, the multi-screen cooperative conversation method provided in the embodiment of the present application may further include S305-1 to S306.

S305-1, under the conditions that the first terminal is connected with the second terminal, the second terminal displays a first terminal interface, and a user communicates with a communication opposite-end device through the second terminal, the first terminal displays a second notification panel;

and responding to the operation that the user breaks the icon in the first terminal on the second notification panel, playing the call downlink data by the first terminal, collecting the second call uplink data and sending the second call uplink data to the call opposite-end equipment.

S306, the first terminal obtains the recording data, wherein the recording data comprises call downlink data and second call uplink data.

When the first terminal 101 and the second terminal 102 are in multi-screen cooperation and a user communicates with a call peer device through the second terminal 102, if the user operates the first terminal 101 to disconnect from the second terminal 102, the first terminal 101 closes the virtual call and switches call sound from the second terminal 102 back to the first terminal 101.

For example, referring to fig. 15, when a user is talking with another party through a tablet pc, and a mobile phone and the tablet pc are multi-screen cooperating, the user slides the status bar of the mobile phone downward to open the second notification panel 405, and if the user clicks the "off" icon in the second notification panel 405, the voice of the talking party is switched back from the tablet to the mobile phone, that is, the voice of the talking party is sent from the mobile phone, and at the same time, the mobile phone collects the voice of the user and sends the voice to the talking party, and the user talks with the talking party through the mobile phone.

In another possible implementation manner, on the basis of fig. 13, after S304, the multi-screen cooperative conversation method provided in the embodiment of the present application may further include S305-2 to S306.

S305-2, under the conditions that the first terminal is connected with the second terminal, the second terminal displays a first terminal interface, and a user communicates with a communication opposite-end device through the second terminal, the first terminal displays a second notification panel;

and responding to the operation of the user on the audio and video switching icon in the second notification panel in the first terminal, playing the call downlink data by the first terminal, collecting the second call uplink data and sending the second call uplink data to the call opposite-end equipment.

When the first terminal 101 and the second terminal 102 cooperate in a multi-screen mode and a user communicates with a device at a call end through the second terminal 102, if the user operates the first terminal 101 to switch audio and video back to the first terminal 101, the first terminal 101 closes a virtual call and switches call sound from the second terminal 102 back to the first terminal 101.

For example, referring to fig. 16, a user is talking with another party through a tablet computer, and a mobile phone and the tablet computer are in multi-screen cooperation, the user slides the status bar of the mobile phone downward to open the second notification panel 405, if the user clicks the "audio/video switch to mobile phone" icon in the second notification panel 405, the voice of the talking party is switched back to the mobile phone from the tablet, that is, the voice of the talking party is sent out from the mobile phone, and at the same time, the mobile phone collects the voice of the user and sends the voice to the talking party, and the user talks with the talking party through the mobile phone.

In yet another possible implementation manner, on the basis of fig. 13, after S304, the multi-screen cooperative call method provided in the embodiment of the present application may further include S305-3 to S306.

S305-3, when the first terminal is connected with the second terminal, the second terminal displays the first terminal interface, and a user communicates with a communication opposite terminal device through the second terminal, the second terminal displays a second notification panel;

and responding to the operation of the user on the audio and video switching icon in the second notification panel in the second terminal, the first terminal plays the call downlink data, collects the second call uplink data and sends the second call uplink data to the call opposite-end equipment.

When the first terminal 101 and the second terminal 102 cooperate in a multi-screen mode and a user communicates with a device at a call end through the second terminal 102, if the user operates the second terminal 102 to switch audio and video back to the first terminal 101, the first terminal 101 closes a virtual call and switches call sound from the second terminal 102 back to the first terminal 101.

For example, referring to fig. 17, when a user is talking with another party through a tablet computer and a mobile phone and the tablet computer are in multi-screen coordination, the user slides the status bar of the tablet computer downward to open the second notification panel 405, if the user clicks the "audio/video switch to mobile phone" icon in the second notification panel 405, the voice of the talking party is switched back to the mobile phone from the tablet, that is, the voice of the talking party is sent from the mobile phone, and at the same time, the mobile phone collects the voice of the user and sends the voice to the talking party, and the user talks with the talking party through the mobile phone.

Of course, besides the above-described possible implementation manners, a person skilled in the art may also design other implementation manners to close the virtual call, for example, if the first terminal 101 detects that the first terminal 101 is close to the ear of the person through a built-in proximity sensor, the virtual call is closed, and this embodiment of the present application does not limit this. It should be noted that, on the basis of fig. 5, an application scenario in which the first terminal 101 closes the virtual call is described. It should be understood by those skilled in the art that the above-mentioned S305-S306, or S305-1-S306, or S305-2-S306, or S305-3-S306 can also be applied after the above-mentioned S304 in S301-1-S304, or S301-2-S304, or S301-3-S304, or S304-4-S304, and the embodiments of the present application do not limit this.

After the first terminal 101 closes the virtual call, call sound needs to be played and received from the speaker and the microphone of the first terminal 101, and therefore the virtual call hardware management module releases the disabling of the speaker and the microphone of the first terminal 101. Meanwhile, in the virtual call scenario, the virtual call hardware management module prohibits the first PCM device from acquiring the uplink data from the audio processor, so after the first terminal 101 closes the virtual call, the virtual call hardware management module also cancels the prohibition of the first PCM device from acquiring the uplink data from the audio processor.

The detailed implementation of closing the virtual call by the first terminal 101, and the call and call recording after closing the virtual call will be described below with reference to fig. 18 and 19.

Referring to fig. 18, the above-described process of playing the call downlink data by the first terminal in S305, S305-1, S302-2, S302-3, and collecting the second call uplink data to send to the call peer device may include:

and S31, the call application program responds to the operation of the user and sends a virtual call closing instruction to the cooperation module.

The operation of closing the virtual call may be any one of fig. 14 to 17, and is used to switch the call sound from the tablet computer back to the mobile phone.

And S32, the cooperation module sends the virtual call closing instruction to the virtual call hardware management module.

And S33, in response to receiving the instruction of closing the virtual call, the virtual call hardware management module releases the forbidding of the audio equipment of the first terminal and cancels the forbidding of the first PCM equipment to acquire the uplink data.

S35-1, the audio equipment of the first terminal collects second communication uplink data and transmits the second communication uplink data to the audio processor;

and S35-2, the audio processor sends the second communication uplink data to the communication opposite-end equipment.

S36-1, the audio processor receives call downlink data sent by the call opposite-end device and transmits the call downlink data to the audio device of the first terminal;

and S36-2, the audio equipment of the first terminal plays the call downlink data.

Referring to fig. 19, the speaker of the first terminal 101 plays the voice of the other party of the call, i.e., the call downlink data sent by the opposite-end device of the call, and the microphone of the first terminal 101 collects the voice of the user, i.e., the second call uplink data.

The interaction process among the modules is a call flow after the first terminal 101 closes the virtual call. The first terminal 101 closes the call recording process after the virtual call, and on the basis of the call process, the method further includes:

and S37-1, the first PCM equipment acquires the call downlink data and the second call uplink data from the audio processor and performs sound mixing.

As shown in fig. 19, the audio hardware management module reads the data obtained by mixing the call downlink data and the second call uplink data from the first PCM device, that is, the recording data, and then the call recording after closing the virtual call can be realized.

Therefore, referring to fig. 18, the process of the first terminal obtaining the recorded sound data in S306 may include:

s37-2, the audio hardware management module reads the recording data from the first PCM equipment;

s37-3, the audio hardware management module sends the recording data to an audio service class;

s37-4, the audio service class sends the recorded sound data to the communication application.

And S37-5, the communication application program stores the recorded data.

As can be seen from the implementation of the conventional telephone function described in fig. 4, when the first terminal 101 records a call, the audio hardware management module needs to open the first PCM device, and read call uplink data and call downlink data from the first PCM device. As can be seen from fig. 11, in the virtual call scenario, the virtual call hardware management module prohibits the first PCM device from acquiring the uplink data from the audio processor, and the virtual call hardware management module reads the call downlink data from the first PCM device. Obviously, both call recording and virtual call require the use of the first PCM device and the data to be acquired is different, so there may be a conflict. Therefore, based on the above, the following describes a procedure of the first terminal 101 turning on or off the recording and a procedure of the first terminal 101 turning on or off the virtual call.

First, a process of starting recording by the first terminal 101 is described, where in the step S304, in response to the recording starting operation of the user, the process of obtaining the recording data by the first terminal may include steps S3041 to S3045.

S3041, the call application sends a recording start instruction to the audio service class in response to the recording start operation.

S3042, the audio service class sends the recording start instruction to the audio hardware management module.

In this embodiment, the audio hardware management module may maintain a recording flag bit, where the recording flag bit has a first identifier (e.g., true) and a second identifier (e.g., false), where the first identifier indicates that the first terminal 101 is currently performing call recording, and the second identifier indicates that the first terminal 101 is not currently performing call recording.

For example, when the first terminal 101 is turned on, the audio hardware management module may initialize the recording flag position, and set the recording flag position to the second state. Meanwhile, when the first terminal 101 starts call recording, the audio hardware management module may set the recording mark position to a first state; and when the first terminal 101 closes the call recording, the audio hardware management module may reset the recording flag to the second state.

S3043, in response to receiving the recording start instruction, the audio hardware management module determines whether the first terminal currently performs a virtual call.

The audio hardware management module responds to the recording starting instruction, and first determines whether the first terminal 101 is currently performing a virtual call. For example, the audio hardware management module determines whether the audio device of the first terminal 101 is disabled, and if the audio device is disabled, it indicates that the first terminal 101 is currently performing a virtual call; if not, it indicates that the first terminal 101 is not currently engaged in the virtual call. If the second PCM device is turned on, the audio hardware management module indicates that the first terminal 101 currently performs a virtual call; if not, it indicates that the first terminal 101 is not currently performing the virtual call. The embodiment of the present application does not set any limit to this.

The audio hardware management module may further determine whether the first terminal 101 performs a virtual call currently by detecting the virtual call flag bit. In this embodiment, the virtual call hardware management module maintains a virtual call flag bit, where the virtual call flag bit has a first state (e.g., true) and a second state (e.g., false), where the first state indicates that the first terminal 101 is currently performing a virtual call, and the second state indicates that the first terminal 101 is not currently performing a virtual call.

For example, when the first terminal 101 starts the multi-screen protocol, the virtual call hardware management module may initialize the virtual call flag bit, and set the virtual call flag bit to the second state. Meanwhile, when the first terminal 101 starts a virtual call, the virtual call hardware management module may set the virtual call flag position to a first state; and when the first terminal 101 closes the virtual call, the virtual call hardware management module may reset the virtual call flag to the second state.

Therefore, the process of the audio hardware management module determining whether the first terminal currently performs the virtual call in S3043 may include:

detecting the virtual call marking bit by an audio hardware management module; if the virtual call marking bit is in the first state, the first terminal carries out virtual call currently; and if the virtual call marking bit is in the second state, the first terminal does not carry out the virtual call currently.

S3044, if the first terminal currently performs a virtual call, the audio hardware management module obtains recording data from the virtual call buffer, where the recording data is stored after the virtual call hardware management module mixes the call downlink data and the first call uplink data.

If the user starts call recording, the first terminal 101 is performing a virtual call, that is, as shown in fig. 20, the first PCM device obtains call downlink data from the audio processor, and the audio processor reads the first call uplink data from the second PCM device. Since the first PCM device is in use and the virtual talk hardware management module prohibits the first PCM device from obtaining the upstream data from the audio processor, the audio hardware management module will not be able to read the recorded data from the first PCM device. Therefore, as shown in fig. 11, the virtual call hardware management module needs to mix the call downlink data and the first call uplink data, and store the mixed data, i.e., the recording data, in the virtual call buffer, and the audio hardware management module can read the recording data from the virtual call buffer.

S3045, if the second terminal does not currently perform the virtual call, the audio hardware management module opens the first PCM device, and reads the recording data from the first PCM device, where the recording data is obtained by the first PCM device obtaining the call downlink data and the second call uplink data from the audio processor and performing audio mixing.

If the user starts call recording, the first terminal 101 is performing a call, that is, as shown in fig. 21, the audio processor receives call downlink data sent by the device at the opposite end of the call and sends the call downlink data to the speaker for playing, and the audio processor acquires second call uplink data collected by the microphone and sends the second call uplink data to the device at the opposite end of the call. As the first PCM device is not used, as shown in fig. 15, the audio hardware management module needs to open the first PCM device, set the first PCM device to obtain the call downlink data and the second call uplink data from the audio processor and mix the call downlink data and the second call uplink data, and then the audio hardware management module reads the recorded sound data from the first PCM device.

To better explain the processes of S3041 to S3045, the following description is provided by using the interaction diagram shown in fig. 22, and as shown in fig. 22, the process of starting recording by the first terminal 101 may include:

s21, when the user executes the record starting operation, for example, the user clicks the record icon in the call interface, the call application responds to the record starting operation to generate a record starting instruction, and the record starting instruction is issued to the audio service class;

s22, the audio service class issues the record starting instruction to the audio hardware management module;

s23, the audio hardware management module opens the first PCM device, wherein the audio hardware management module can select usecast _ incall _ rec _ uplink _ and _ downlink to designate the first PCM device;

s24, the audio hardware management module marks the recording to true;

s25, the audio hardware management module detects the virtual call flag bit;

s26, if the audio hardware management module detects that the virtual call flag bit is true, the audio hardware management module acquires the recording data from the virtual call buffer area, sends the recording data to an audio service class, sends the recording data to a call application program by the audio service class, and stores the recording data by the call application program;

the audio data in the virtual call buffer area is stored after the virtual call hardware management module mixes the call downlink data and the first call uplink data, and the detailed implementation can be seen in fig. 12, which is not described herein again.

And S27, if the audio hardware management module detects that the virtual call flag bit is false, the audio hardware management module reads the recording data from the first PCM device, sends the recording data to an audio service class, sends the recording data to a call application program by the audio service class, and stores the recording data by the call application program.

The audio data in the first PCM device is obtained by mixing the call downlink data and the second call uplink data by the first PCM device, and the detailed implementation may be referred to fig. 18, which is not described herein again.

Further, a process of closing the recording by the first terminal 101 is introduced, and after the above S3041 to S3045, the multi-screen cooperative communication method provided in the embodiment of the present application may further include S304a to S304 e.

S304a, the call application program sends a record closing instruction to the audio service class in response to the record closing operation of the user.

In the process of recording the call by the first terminal 101, if the user clicks the "record" icon in the call interface again, the process of closing the call record is triggered.

S304b, the audio service class sends the instruction of closing the recording to the audio hardware management module.

S304c, in response to receiving the recording closing instruction, the audio hardware management module judges whether the first terminal is currently in virtual communication.

In this embodiment, the process of the audio hardware management module in S304c determining whether the first terminal is currently performing the virtual call is similar to that in S3043, and is not described herein again.

S304d, if the first terminal is currently performing the virtual call, the audio hardware management module does not close the first PCM device, and stops obtaining the recording data from the virtual call buffer.

If the user closes the call recording, the first terminal 101 is performing a virtual call, and the virtual call needs to use the first PCM device, so the audio hardware management module does not close the first PCM device, and stops reading the recorded data from the virtual call buffer, that is, switches from the data flow diagram shown in fig. 11 to the data flow diagram shown in fig. 20.

S304e, if the first terminal does not currently perform the virtual call, the audio hardware management module closes the first PCM device to read the recording data, and stops obtaining the recording data from the first PCM device.

If the user closes the call recording and the first terminal 101 is in a call, the audio hardware management module closes the first PCM device because the call does not need to use the first PCM device, and stops reading the recording data from the first PCM device, that is, switches from the data flow diagram shown in fig. 19 to the data flow diagram shown in fig. 21.

To better describe the processes of S304a to S304e, please refer to fig. 22, the process of closing the recording by the first terminal 101 may include:

s41, when the user executes the recording closing operation, for example, the user clicks the recording icon again in the call interface, the call application responds to the operation to generate a recording closing instruction, and the recording closing instruction is sent to the audio service class;

s42, the audio service issues the recording closing instruction to the audio hardware management module;

s43, the audio hardware management module detects the virtual call flag bit;

s44, if the audio hardware management module detects that the virtual call marking position is true, the audio hardware management module marks the recording position as false;

s45, if the audio hardware management module detects that the virtual talk flag bit is false, the audio hardware management module closes the first PCM device, and then marks the recording flag bit as false.

The content described in fig. 22 relates to a scenario that the first terminal 101 may already be in a virtual call when starting call recording; and when the first terminal 101 closes the call recording, the scene may still be in the virtual call.

Those skilled in the art should understand that, in some scenarios, the first terminal 101 may already be in the call recording when the virtual call is opened; alternatively, the first terminal 101 may still be in the call record when the virtual call is closed. For example, in conjunction with fig. 8, a user is talking to the other party of the call through a mobile phone, and the mobile phone is recording, the user clicks a "connect" icon displayed on the mobile phone to perform a multi-screen cooperative connection, and the mobile phone opens a virtual call. As another example, in conjunction with fig. 15, a mobile phone and a tablet computer are in multi-screen coordination, and a user is talking to another party through the mobile phone while the mobile phone is recording, the user clicks a "disconnect" icon displayed on the mobile phone, and the mobile phone closes the virtual conversation.

Therefore, the flow of the first terminal 101 to open or close the virtual talk needs to be considered.

A process of starting a virtual call by a first terminal 101 is introduced, a process of receiving call downlink data sent by a call peer device and sending the call downlink data to a second terminal in S302, S302-1, S302-2, S302-3, and S302-3 by the first terminal, playing the call downlink data by the second terminal, and a process of acquiring first call uplink data by the second terminal in S303 and sending the first call uplink data to the call peer device by the first terminal may include S3021 to S3025.

And S3021, the call application program responds to the operation of the user and sends a virtual call starting instruction to the cooperation module.

And S3022, the cooperation module sends the virtual call starting instruction to a virtual call hardware management module. And S3023, in response to receiving the virtual call starting instruction, the virtual call hardware management module judges whether the first terminal currently performs call recording.

The virtual call hardware management module may determine whether the first terminal 101 currently performs call recording through the recording flag bit introduced in S3042, that is, the process of determining whether the first terminal currently performs call recording by the virtual call hardware management module in S3023 may include:

the virtual call hardware management module detects a recording mark bit; if the recording mark position is in the first mark, the first terminal carries out call recording currently; and if the recording mark bit is in the second identifier, the first terminal does not record the call currently. S3024, if the first terminal currently performs call recording, disabling the audio device of the first terminal by the virtual call hardware management module, opening the first PCM device and disabling the first PCM device from acquiring uplink data, reading call downlink data from the first PCM device and sending the call downlink data to the transmission module of the first terminal, receiving first call uplink data sent by the transmission module of the first terminal and writing the first call uplink data into the second PCM device, and performing audio mixing on the call downlink data and the first call uplink data to obtain recorded data and store the recorded data in the virtual call cache region.

If the user starts a virtual call, the first terminal 101 is recording the call, that is, as shown in fig. 19, the audio processor receives call downlink data sent by the device at the opposite end of the call and sends the call downlink data to the speaker for playing, the audio processor acquires second call uplink data collected by the microphone and sends the second call uplink data to the device at the opposite end of the call, and the first PCM device acquires the call downlink data and the second call uplink data from the audio processor.

Since the virtual call needs to switch the call sound from the first terminal 101 to the second terminal 102, the call recording needs the audio hardware management module to obtain the recording data, that is, as shown in fig. 11, the virtual call hardware management module needs to disable the local audio device and prohibit the first PCM device from obtaining the uplink data from the audio processor, and then the virtual call hardware management module reads the call downlink data from the first PCM device, writes the first call uplink data into the second PCM device, and mixes the call downlink data and the first call uplink data to obtain the recording data and stores the recording data into the virtual call buffer.

S3025, if the first terminal does not record a call currently, the virtual call hardware management module disables the audio device of the first terminal, opens the first PCM device and disables the first PCM device from acquiring uplink data, reads call downlink data from the first PCM device, sends the call downlink data to the transmission module of the first terminal, receives the first call uplink data sent by the transmission module of the first terminal, and writes the first call uplink data into the second PCM device.

If the user opens the virtual call, the first terminal 101 is performing a call, that is, as shown in fig. 21, the audio processor receives call downlink data sent by the device at the opposite end of the call and sends the call downlink data to the speaker for playing, and the audio processor acquires second call uplink data collected by the microphone and sends the second call uplink data to the device at the opposite end of the call.

Since the virtual call needs to switch the call sound from the first terminal 101 to the second terminal 102, that is, as shown in fig. 14, the virtual call hardware management module needs to disable the local audio device and disable the first PCM device from acquiring the upstream data from the audio processor, and then, the virtual call hardware management module reads the call downstream data from the first PCM device and writes the first call upstream data into the second PCM device.

To better explain the processes of S3021 to S3025, the following description is given by using an interaction diagram shown in fig. 23, and as shown in fig. 23, the process of starting a virtual call by the first terminal 101 may include:

s11, when the user performs the operation of opening the virtual phone call, for example, the user performs any one of the operations shown in fig. 6 to 10, the phone call application generates a virtual phone call opening instruction in response to the operation of opening the virtual phone call, and sends the virtual phone call opening instruction to the coordination module;

s12, the cooperation module sends the virtual call starting instruction to the virtual call hardware management module;

s1a, the virtual call hardware management module marks the virtual call to true;

s13, the virtual call hardware management module forbids the audio device of the first terminal, opens the first PCM device and forbids the first PCM device to obtain the uplink data; the virtual call hardware management module can select usecast _ include _ rec _ uplink _ and _ downlink to designate the first PCM equipment, and forbid the first PCM equipment from acquiring uplink data;

s15-1 to S15-3, the audio processor receives call downlink data sent by the call opposite-end device, the first PCM device obtains the call downlink data from the audio processor, and the virtual call hardware management module reads the call downlink data from the first PCM device;

S16-1-S16-3, the audio equipment of the second terminal collects first call uplink data and transmits the first call uplink data to a transmission module of the second terminal, the transmission module of the second terminal transmits the first call uplink data to the transmission module of the first terminal, and the transmission module of the first terminal transmits the first call uplink data to a virtual call hardware management module;

s1b, detecting a recording mark position by a virtual communication hardware management module;

s1c, if the virtual call hardware management module detects that the recording mark bit is true, executing:

s17-1 to S17-2, mixing the first call uplink data and the call downlink data by the virtual call hardware management module, and storing the mixed data in a virtual call buffer area; S15-4-S15-6, the transmission module of the first terminal sends the call downlink data to the transmission module of the second terminal, the transmission module of the second terminal sends the call downlink data to the audio equipment of the second terminal, and the audio equipment of the second terminal plays the call downlink data; and S16-4~ S16-5, the virtual call hardware management module writes the first call uplink data into the second PCM device, and the audio processor reads the first call uplink data from the second PCM device and sends the first call uplink data to the call opposite-end device.

S1d, if the virtual call hardware management module detects that the recording mark bit is false, executing:

S15-4-S15-6, the transmission module of the first terminal sends the call downlink data to the transmission module of the second terminal, the transmission module of the second terminal sends the call downlink data to the audio equipment of the second terminal, and the audio equipment of the second terminal plays the call downlink data; and S16-4~ S16-5, the virtual call hardware management module writes the first call uplink data into the second PCM device, and the audio processor reads the first call uplink data from the second PCM device and sends the first call uplink data to the call opposite-end device.

Optionally, the virtual call hardware management module may write the call downlink data and the first call uplink data into the two buffer areas, and then sequentially read the ith call downlink data and the ith first call uplink data from the two buffer areas in sequence to add, so as to complete sound mixing.

After the foregoing S3021 to S3025, the multi-screen cooperative communication method provided in the embodiment of the application may further include S302a to S302 e.

S302a, the call application sends a command to close the virtual call to the collaboration module in response to the user' S operation.

S302b, the coordination module sends the instruction to close the virtual call to the virtual call hardware management module.

S302c, in response to receiving the instruction of closing the virtual call, the virtual call hardware management module determines whether the first terminal currently performs call recording.

In this embodiment, the process that the virtual session hardware management module in S302c determines whether the first terminal currently performs the session recording is similar to that in S3023, and is not described herein again.

S302d, if the first terminal currently performs the call recording, the virtual call hardware management module releases the disabling of the audio device of the first terminal, and cancels the prohibition of the first PCM device for obtaining the uplink data.

If the user closes the virtual call, the first terminal 101 is performing call recording, and since the call recording needs to use the first PCM device and the first PCM device needs to acquire the upstream data and the downstream data, the virtual call hardware management module releases the disabling of the local audio device and cancels the prohibition of the first PCM device for acquiring the upstream data from the audio processor, that is, switches from the data flow diagram shown in fig. 11 to the data flow diagram shown in fig. 19.

S302e, if the first terminal does not record the call currently, the virtual call hardware management module disables the audio device of the first terminal, and closes the first PCM device. If the user closes the virtual call and the first terminal 101 is in a call, the virtual call hardware management module deactivates the local audio device and closes the first PCM device, that is, switches from the data flow diagram shown in fig. 20 to the data flow diagram shown in fig. 21, since the call does not need to use the first PCM device.

To better explain the processes of S302 a-S302 e, please refer to fig. 23, the process of closing the virtual talk by the first terminal 101 may include:

s31, when the user performs the operation of closing the virtual phone call, for example, the user performs any one of the operations shown in fig. 14 to 17, the phone call application program responds to the operation to generate a virtual phone call closing instruction, and sends the virtual phone call closing instruction to the coordination module;

s32, the cooperation module sends the instruction of closing the virtual call to the virtual call hardware management module;

s3a, detecting the recording mark position by the virtual communication hardware management module;

s3b, if the virtual call hardware management module detects that the recording mark bit is true, executing:

s33, the virtual call hardware management module cancels the forbidding of the first PCM equipment to obtain the uplink data and cancels the forbidding of the audio equipment of the first terminal; s35-1 to S35-2, the audio equipment of the first terminal collects second communication uplink data and transmits the second communication uplink data to the audio processor, and the audio processor sends the second communication uplink data to the communication opposite terminal equipment; s36-1 to S36-2, the audio processor receives call downlink data sent by the call opposite-end device and transmits the call downlink data to the audio device of the first terminal for playing; s37-1 to S37-2, the first PCM equipment acquires the call downlink data and the second call uplink data from the audio processor and performs sound mixing; and S3c, the virtual call hardware management module marks the virtual call flag position as false.

S3d, if the virtual call hardware management module detects that the recording mark bit is false, executing:

s3e, the virtual call hardware management module closes the first PCM device; s33, the virtual call hardware management module releases the forbidding of the microphone and the loudspeaker, S35-1-S35-2, the audio equipment of the first terminal collects second call uplink data and transmits the second call uplink data to the audio processor, and the audio processor sends the second call uplink data to the call opposite-end equipment; s36-1 to S36-2, the audio processor receives call downlink data sent by the call opposite-end device and transmits the call downlink data to the audio device of the first terminal for playing; and S3f, the virtual call hardware management module marks the virtual call as false.

The multi-screen collaborative call method applied to the multi-screen collaborative call system 100 in the embodiment of the present application is described above, and then the multi-screen collaborative call methods applied to the first terminal 101 and the second terminal 102 are respectively described below.

The embodiment of the present application further provides a multi-screen collaborative call method, where the multi-screen collaborative call method is applied to the first terminal 101, for example, a mobile phone, and the multi-screen collaborative call method may include:

s401, a first terminal receives an incoming call of a device at a call opposite end, wherein the first terminal is connected with a second terminal, and the second terminal displays a first terminal interface;

s402, responding to the incoming call answering operation of a user to the interface of the first terminal in the second terminal, receiving call downlink data sent by the call opposite-end equipment and sending the call downlink data to the second terminal so that the second terminal plays the call downlink data;

s403, receiving first call uplink data collected by a second terminal, and sending the first call uplink data to a call opposite terminal device;

s404, responding to the recording starting operation of the user, and obtaining recording data, wherein the recording data comprises call downlink data and the first call uplink data.

For the detailed implementation of the above-mentioned S401 to S404, reference may be made to the description of the foregoing embodiments, which are not described herein again.

The embodiment of the present application further provides a multi-screen collaborative call method, where the multi-screen collaborative call method is applied to the second terminal 102, such as a tablet computer, and the multi-screen collaborative call method may include:

s501, receiving and playing call downlink data sent by a first terminal, wherein the call downlink data is sent to the first terminal by the first terminal after the first terminal is connected with a second terminal and the second terminal displays a first terminal interface and receives an incoming call of a call opposite terminal device and responds to an incoming call answering operation of a user on the first terminal interface in the second terminal;

s502, the second terminal collects first call uplink data and sends the first call uplink data to the call opposite-end equipment through the first terminal, so that the first terminal responds to the recording starting operation of a user and obtains recording data, wherein the recording data comprises call downlink data and the first call uplink data.

For the detailed implementation of the above-mentioned S501-S502, reference may be made to the description of the foregoing embodiments, which are not described herein again.

An embodiment of the present application further provides a terminal, as shown in fig. 24, where the terminal 600 may include: one or more processors 601, memory 602, audio devices 603, and display screens 604, the processors 601, memory 602, audio devices 603, and display screens 604 being coupled. Wherein the terminal 600 comprises the first terminal 101 or the second terminal 102, the memory 602 is adapted to store computer program code comprising computer instructions. When the processor executes the computer instructions, the terminal 600 may perform various functions or steps performed by the first terminal 101 in the above-described method embodiment, or performed by the second terminal 102 in the above-described method embodiment.

The Memory 602 may include a Random Access Memory (RAM) and may also include a non-volatile Memory (NVM).

The processor 601 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method embodiments may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 601. The processor 601 may be a general-purpose processor, and includes a Central Processing Unit (CPU), a Micro Control Unit (MCU), a Complex Programmable Logic Device (CPLD), a Field Programmable Gate Array (FPGA), and an embedded ARM.

Embodiments of the present application further provide a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit may be interconnected by wires. For example, the interface circuit may be used to receive signals from other devices (e.g., the memory of the terminal). As another example, the interface circuit may be used to send signals to other devices (e.g., a processor). Illustratively, the interface circuit may read instructions stored in the memory and send the instructions to the processor. The instructions, when executed by the processor, may cause the terminal to perform the various steps in the embodiments described above. Of course, the chip system may further include other discrete devices, which is not specifically limited in this embodiment of the present application.

An embodiment of the present application further provides a computer storage medium, where the computer storage medium includes computer instructions, and when the computer instructions are run on the terminal, the terminal is caused to perform each function or step performed by the first terminal 101 in the above method embodiment, or each function or step performed by the second terminal 102 in the above method embodiment.

The present embodiment also provides a computer program product, which when running on a computer, causes the computer to execute each function or step executed by the first terminal 101 in the above method embodiment, or each function or step executed by the second terminal 102 in the above method embodiment.

Each functional unit in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or partially contributed to by the prior art, or all or part of the technical solutions may be embodied in the form of a software product, where the software product is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a variety of media that can store program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only a specific implementation of the embodiments of the present application, but the scope of the embodiments of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the embodiments of the present application should be covered by the scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

1. A multi-screen cooperative conversation method is characterized by comprising the following steps:

under the condition that a first terminal is connected with a second terminal and the second terminal displays a first terminal interface, the first terminal receives an incoming call of a device at the opposite end of a call;

responding to an incoming call answering operation of a user on the first terminal interface in the second terminal, wherein the first terminal receives call downlink data sent by a call opposite-end device and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data, wherein the first terminal comprises an audio processor, a first PCM device and a virtual call hardware management module, the audio processor is used for receiving the call downlink data, the first PCM device is used for acquiring the call downlink data from the audio processor, the virtual call hardware management module is used for reading the call downlink data from the first PCM device and sending the call downlink data to the second terminal, and the virtual call hardware management module prohibits the first PCM device from acquiring uplink data;

the second terminal collects first call uplink data and sends the first call uplink data to the call opposite terminal equipment through the first terminal;

responding to the recording starting operation of a user, the first terminal obtains recording data, wherein the recording data comprise call downlink data and first call uplink data, and the first terminal further comprises a virtual call buffer area which is used for storing the call downlink data and the first call uplink data.

2. The method of claim 1, wherein the method further comprises:

under the condition that a first terminal is connected with a second terminal and the second terminal displays a first terminal interface, responding to the call making operation of a user on the first terminal interface in the second terminal, and calling the opposite-end call device by the first terminal;

after the opposite-end call device is connected, the first terminal receives call downlink data sent by the opposite-end call device and sends the call downlink data to the second terminal, and the second terminal plays the call downlink data;

the second terminal collects first call uplink data and sends the first call uplink data to the call opposite terminal device through the first terminal.

3. The method of claim 1, wherein the method further comprises:

under the condition that the first terminal is in communication with the opposite-end communication device, the first terminal displays a popup frame, wherein the popup frame is used for prompting a user to cooperate the first terminal and the second terminal in a multi-screen mode;

responding to the operation of a user on a connection icon in the bullet frame, connecting the first terminal with the second terminal, displaying a first terminal interface by the second terminal, receiving call downlink data sent by a call opposite terminal device by the first terminal and sending the call downlink data to the second terminal, and playing the call downlink data by the second terminal;

the second terminal collects first call uplink data and sends the first call uplink data to the call opposite terminal equipment through the first terminal.

4. The method of claim 1, wherein the method further comprises:

under the conditions that the first terminal is in communication with the opposite-end communication device, the first terminal is connected with the second terminal, and the second terminal displays a first terminal interface, the first terminal displays a first notification panel;

responding to the operation of a user on an audio and video switching icon in the first notification panel in the first terminal, receiving call downlink data sent by a call opposite-end device by the first terminal and sending the call downlink data to the second terminal, and playing the call downlink data by the second terminal;

5. The method of claim 1, wherein the method further comprises:

under the conditions that the first terminal is in communication with the opposite-end communication device, the first terminal is connected with the second terminal, and the second terminal displays a first terminal interface, the second terminal displays a first notification panel;

responding to the operation of a user on an audio and video switching icon in the first notification panel in the second terminal, receiving call downlink data sent by a call opposite-end device by the first terminal and sending the call downlink data to the second terminal, and playing the call downlink data by the second terminal;

6. The method of any of claims 1-5, wherein the first terminal further comprises: the system comprises a call application program, a coordination module, a transmission module, a second PCM device and an audio device; the second terminal includes: a transmission module and an audio device;

the first terminal receives call downlink data sent by a call opposite terminal device and sends the call downlink data to the second terminal, and the step of playing the call downlink data by the second terminal comprises the following steps:

in response to receiving the virtual call starting instruction, the virtual call hardware management module disables audio equipment of the first terminal, opens the first PCM equipment, and disables the first PCM equipment from acquiring uplink data;

the audio processor receives the call downlink data sent by the call opposite-end equipment;

the first PCM equipment acquires the call downlink data from the audio processor;

the virtual call hardware management module reads the call downlink data from the first PCM device and sends the call downlink data to a transmission module of the first terminal;

the transmission module of the first terminal sends the call downlink data to the transmission module of the second terminal, and the audio equipment of the second terminal plays the call downlink data;

the second terminal collects first call uplink data and sends the first call uplink data to the call opposite terminal equipment through the first terminal, and the steps comprise:

the audio equipment of the second terminal collects the first call uplink data, and the transmission module of the second terminal sends the first call uplink data to the transmission module of the first terminal;

the transmission module of the first terminal sends the first call uplink data to the virtual call hardware management module;

the virtual call hardware management module writes the first call uplink data into the second PCM equipment;

and the audio processor reads the first call uplink data from the second PCM equipment and sends the first call uplink data to the call opposite-end equipment.

7. The method of claim 6, wherein the first terminal further comprises: an audio service class and an audio hardware management module;

the step of the first terminal obtaining the recording data in response to the recording start operation of the user includes:

the call application program responds to the recording starting operation and sends a recording starting instruction to the audio service class;

the audio service class sends the recording starting instruction to the audio hardware management module;

and responding to the received recording starting instruction, the audio hardware management module acquires the recording data from the virtual call cache region, wherein the recording data is stored after the virtual call hardware management module performs sound mixing on the call downlink data and the first call uplink data.

8. The method of claim 1, wherein the method further comprises:

under the conditions that a first terminal is connected with a second terminal, the second terminal displays a first terminal interface, and a user communicates with a communication opposite-end device through the second terminal, the first terminal plays communication downlink data and collects second communication uplink data to send the second communication uplink data to the communication opposite-end device in response to the closing operation of the user on the first terminal interface in the second terminal;

the first terminal obtains recording data, wherein the recording data comprises the call downlink data and the second call uplink data.

9. The method of claim 8, wherein the method further comprises:

the method comprises the steps that under the conditions that a first terminal is connected with a second terminal, the second terminal displays a first terminal interface, and a user communicates with a communication opposite-end device through the second terminal, the first terminal displays a second notification panel;

and responding to the operation that the user breaks off the icon in the first terminal on the second notification panel, and the first terminal plays the call downlink data, collects the second call uplink data and sends the second call uplink data to the call opposite-end equipment.

10. The method of claim 8, wherein the method further comprises:

and responding to the operation of the user on the audio and video switching icon in the second notification panel in the first terminal, and the first terminal plays the call downlink data, collects the second call uplink data and sends the second call uplink data to the call opposite terminal equipment.

11. The method of claim 8, wherein the method further comprises:

the method comprises the steps that under the conditions that a first terminal is connected with a second terminal, the second terminal displays a first terminal interface, and a user communicates with a communication opposite-end device through the second terminal, the second terminal displays a second notification panel;

and responding to the operation of the user on the audio and video switching icon in the second notification panel in the second terminal, the first terminal plays the call downlink data, collects the second call uplink data and sends the second call uplink data to the call opposite terminal equipment.

12. The method of any of claims 8-11, wherein the first terminal further comprises: the system comprises a call application program, a cooperation module, a transmission module and audio equipment;

the first terminal plays the call downlink data and collects second call uplink data to be sent to the call opposite terminal device, and the steps comprise:

the communication application program responds to the operation of a user and sends a virtual communication closing instruction to the cooperation module;

the cooperation module sends the virtual call closing instruction to the virtual call hardware management module;

in response to receiving the virtual call closing instruction, the virtual call hardware management module cancels the forbidding of the audio equipment of the first terminal and cancels the forbidding of the first PCM equipment to acquire uplink data;

the audio processor receives the call downlink data and plays the call downlink data through the audio equipment of the first terminal, acquires the second call uplink data collected by the audio equipment of the first terminal and sends the second call uplink data to the call opposite terminal equipment.

13. The method of claim 12, wherein the first terminal further comprises an audio hardware management module;

the step of the first terminal obtaining the recording data includes:

and the audio hardware management module reads the recording data from the first PCM device, wherein the recording data is obtained by the first PCM device acquiring the call downlink data and the second call uplink data from the audio processor and mixing the audio data.

14. The method of any of claims 8-11, wherein the first terminal further comprises: a call application program, an audio service class and an audio hardware management module;

in response to receiving the recording starting instruction, the audio hardware management module judges whether the first terminal carries out virtual communication currently;

if the first terminal currently performs a virtual call, the audio hardware management module acquires the recording data from the virtual call cache region, wherein the recording data is stored after the virtual call hardware management module performs audio mixing on the call downlink data and the first call uplink data;

if the second terminal does not currently perform virtual call, the audio hardware management module opens the first PCM device and reads the recording data from the first PCM device, wherein the recording data is obtained by the first PCM device acquiring the call downlink data and the second call uplink data from the audio processor and performing sound mixing.

15. The method of claim 14, wherein the method further comprises:

the call application program responds to the recording closing operation of the user and sends a recording closing instruction to the audio service class;

the audio service class sends the recording closing instruction to the audio hardware management module;

in response to receiving the recording closing instruction, the audio hardware management module judges whether the first terminal carries out virtual communication currently;

if the first terminal is currently in a virtual call, the audio hardware management module does not close the first PCM device and stops acquiring the recording data from the virtual call cache region;

if the first terminal does not currently perform virtual call, the audio hardware management module closes the first PCM device to read the recording data, and stops acquiring the recording data from the first PCM device.

16. The method of claim 15, wherein the virtual talk hardware management module maintains a virtual talk flag bit having a first state and a second state, the first state indicating that the first terminal is currently engaged in a virtual talk, the second state indicating that the first terminal is not currently engaged in a virtual talk;

the step of judging whether the first terminal carries out virtual communication currently by the audio hardware management module comprises the following steps:

the audio hardware management module detects the virtual call flag bit;

if the virtual call marking bit is in the first state, the first terminal carries out virtual call currently;

and if the virtual call marking bit is in the second state, the first terminal does not perform virtual call currently.

17. The method of claim 16, wherein the audio hardware management module maintains a recording flag bit;

the method further comprises the following steps:

in response to receiving the recording starting instruction, the audio hardware management module sets the recording mark position as a first identifier, and the first identifier represents that the first terminal currently carries out call recording;

and responding to the received recording closing instruction, and enabling the audio hardware management module to set the recording mark position as a second identifier, wherein the second identifier represents that the first terminal does not record the call currently.

18. The method of any of claims 8-11, wherein the first terminal further comprises: the system comprises a call application program, a coordination module, a transmission module, a second PCM device and an audio device;

the conversation application program responds to the operation of a user and sends a virtual conversation starting instruction to the cooperation module;

if the first terminal currently carries out call recording, the virtual call hardware management module disables the audio equipment of the first terminal, opens the first PCM equipment and prohibits the first PCM equipment from acquiring uplink data, reads the call downlink data from the first PCM equipment and sends the call downlink data to the transmission module of the first terminal, receives the first call uplink data sent by the transmission module of the first terminal and writes the first call uplink data into the second PCM equipment, and carries out sound mixing on the call downlink data and the first call uplink data to obtain the recording data and stores the recording data into the virtual call cache region;

if the first terminal does not record the call currently, the virtual call hardware management module disables the audio equipment of the first terminal, opens the first PCM equipment, prohibits the first PCM equipment from acquiring uplink data, reads the call downlink data from the first PCM equipment, sends the call downlink data to the transmission module of the first terminal, receives the first call uplink data sent by the transmission module of the first terminal, and writes the first call uplink data into the second PCM equipment.

19. The method of claim 18, wherein the step of playing the call downlink data by the first terminal and collecting the second call uplink data to send to the call peer device comprises:

in response to receiving the virtual call closing instruction, the virtual call hardware management module judges whether the first terminal carries out call recording currently;

if the first terminal currently carries out call recording, the virtual call hardware management module removes the forbidding of the audio equipment of the first terminal and cancels the forbidding of the first PCM equipment to obtain uplink data;

and if the first terminal does not record the call currently, the virtual call hardware management module removes the forbidding of the audio equipment of the first terminal and closes the first PCM equipment.

20. The method of claim 19, wherein the first terminal further comprises an audio hardware management module that maintains a recording flag bit having a first flag that indicates that the first terminal is currently recording a call and a second flag that indicates that the first terminal is not currently recording a call;

the virtual call hardware management module judges whether the first terminal carries out call recording currently or not, and the method comprises the following steps:

the virtual call hardware management module detects the recording mark bit;

if the recording mark bit is in the first identifier, the first terminal currently carries out call recording;

and if the recording mark bit is in the second identifier, the first terminal does not record the call currently.

21. The method of claim 20, wherein the virtual telephony hardware management module maintains a virtual telephony flag bit, the method further comprising:

in response to receiving the virtual call starting instruction, the virtual call hardware management module sets the virtual call mark position to a first state, wherein the first state represents that the first terminal currently carries out virtual call;

and in response to receiving the virtual call closing instruction, the virtual call hardware management module sets the virtual call mark position to be in a second state, wherein the second state represents that the first terminal does not perform virtual call currently.

22. A multi-screen cooperative conversation method is applied to a first terminal, and comprises the following steps:

the method comprises the steps that a first terminal receives an incoming call of a call opposite-end device, wherein the first terminal is connected with a second terminal, and the second terminal displays a first terminal interface;

responding to an incoming call answering operation of a user on the first terminal interface in the second terminal, receiving call downlink data sent by a call opposite-end device and sending the call downlink data to the second terminal so as to enable the second terminal to play the call downlink data, wherein the first terminal comprises an audio processor, a first PCM device and a virtual call hardware management module, the audio processor is used for receiving the call downlink data, the first PCM device is used for acquiring the call downlink data from the audio processor, the virtual call hardware management module is used for reading the call downlink data from the first PCM device and sending the call downlink data to the second terminal, and the virtual call hardware management module prohibits the first PCM device from acquiring uplink data;

receiving first call uplink data collected by the second terminal, and sending the first call uplink data to the call opposite-end equipment;

responding to a recording starting operation of a user, and obtaining recording data, wherein the recording data comprises the call downlink data and the first call uplink data, and the first terminal further comprises a virtual call cache region, and the virtual call cache region is used for storing the call downlink data and the first call uplink data.

23. A multi-screen cooperative conversation method is applied to a second terminal, and comprises the following steps:

receiving and playing call downlink data sent by a first terminal, wherein the call downlink data is sent to the first terminal by the first terminal after the first terminal is connected with a second terminal and the second terminal displays a first terminal interface and receives an incoming call of a call opposite terminal device and responds to an incoming call answering operation of a user on the first terminal interface in the second terminal; the first terminal comprises an audio processor, first PCM equipment and a virtual call hardware management module, wherein the audio processor is used for receiving call downlink data, the first PCM equipment is used for acquiring the call downlink data from the audio processor, the virtual call hardware management module is used for reading the call downlink data from the first PCM equipment and sending the call downlink data to the second terminal, and the virtual call hardware management module prohibits the first PCM equipment from acquiring uplink data;

the second terminal collects first call uplink data and sends the first call uplink data to the call opposite-end equipment through the first terminal, so that the first terminal responds to the recording starting operation of a user and obtains recording data, wherein the recording data comprise the call downlink data and the first call uplink data, the first terminal further comprises a virtual call buffer area, and the virtual call buffer area is used for storing the call downlink data and the first call uplink data.

24. A terminal, characterized in that the terminal comprises:

one or more processors;

a memory;

an audio device;

a display screen;

wherein the memory is to store computer program code comprising computer instructions; when the processor executes the computer instructions, the terminal performs the multi-screen collaborative call method performed by the first terminal in any one of claims 1 to 21, or the terminal performs the multi-screen collaborative call method performed by the second terminal in any one of claims 1 to 21.

25. A multi-screen collaborative calling system, the system comprising a first terminal and a second terminal, the first terminal performing the multi-screen collaborative calling method executed by the first terminal in any one of claims 1 to 21, the second terminal performing the multi-screen collaborative calling method executed by the second terminal in any one of claims 1 to 21.

26. A computer-readable storage medium, comprising a computer program, which when executed on a terminal, causes the terminal to execute the multi-screen collaborative session method executed by a first terminal in any one of claims 1-21, or cause the terminal to execute the multi-screen collaborative session method executed by a second terminal in any one of claims 1-21.