CN111447398A - Multi-screen interaction method for audio and video real-time talkback - Google Patents
Multi-screen interaction method for audio and video real-time talkback Download PDFInfo
- Publication number
- CN111447398A CN111447398A CN202010329780.XA CN202010329780A CN111447398A CN 111447398 A CN111447398 A CN 111447398A CN 202010329780 A CN202010329780 A CN 202010329780A CN 111447398 A CN111447398 A CN 111447398A
- Authority
- CN
- China
- Prior art keywords
- audio
- video
- mobile phone
- client
- equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
- H04N21/4363—Adapting the video or multiplex stream to a specific local network, e.g. a IEEE 1394 or Bluetooth® network
- H04N21/43637—Adapting the video or multiplex stream to a specific local network, e.g. a IEEE 1394 or Bluetooth® network involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11]
Abstract
The invention relates to a multi-screen interaction method for audio and video real-time talkback, which comprises the following steps: (1) the video talkback or video conference server supports the login of a master client and a slave client of the same user ID; the intelligent equipment end is started at a background, logs in a server and starts an equipment searching module; (2) when the mobile phone client performs video talkback or video conference and finds that the network state is a WIFI environment, the device searching module is called to perform device searching, and when related intelligent devices or slave clients are found, a user is reminded of whether to switch the response function to the slave client; (3) when a user performs function switching operation, relevant functions are switched to corresponding intelligent equipment from a mobile phone end; (4) when the user manually switches back, the related functions are switched back to the mobile phone end from the corresponding intelligent equipment; (5) and when the mobile phone leaves the WIFI environment, the function switching is automatically carried out and the function is returned. The invention can ensure the convenience and the mobility of the mobile phone for carrying out the video conference.
Description
Technical Field
The invention relates to multi-screen interaction of a mobile phone, a PC, an intelligent sound box, an intelligent television and a set top box, in particular to a multi-screen interaction processing method for audio and video real-time talkback and audio and video conferences.
Background
In the existing audio and video real-time intercom system, a certain user client can only use one device at the same time, for example, when a mobile phone is used for video conference or video intercom, if a television or a set top box is in the vicinity, the user needs to quit the mobile phone client, and then the user uses the television or the set top box to enter the system again, so that the effect of multi-screen interaction is achieved.
Therefore, only one device can be independently used, and when multiple devices are switched, a login and switching process is inevitable, and the advantages of multiple devices, such as a large screen effect of a television and a playing effect of an intelligent sound box, cannot be fully utilized.
Disclosure of Invention
Object of the Invention
The invention aims to solve the following technical problems:
1. the real-time performance of multi-screen interaction switching in audio and video real-time talkback (including audio and video conferences) is improved, and the user experience is improved.
2. The large-screen playing characteristic of a common intelligent television (set top box) in audio and video real-time talkback (including audio and video conferences) is solved, but the defect of no video acquisition module is overcome.
3. Various audio and video acquisition and playing devices are fully utilized and are concurrent, and better audio and video talkback experience is achieved.
4. And performing background login on a plurality of devices by using a mode of simultaneously logging in a master client and a slave client.
The specific technical scheme of the invention is as follows:
a multi-screen interaction method for audio and video real-time talkback,
a video talkback (video conference) server supports the login of a master client and a slave client with the same user ID;
when the mobile phone client performs video talkback (during video conference), when the network state is found to be a WIFI environment, the equipment searching module is called to perform equipment searching, and when relevant intelligent equipment (slave client) is searched, a user is reminded of whether to switch the response function to the slave client;
when a user performs function switching operation, relevant functions are switched to corresponding intelligent equipment from a mobile phone end;
when the user manually switches back, the related functions are switched back to the mobile phone end from the corresponding intelligent equipment;
and when the mobile phone leaves the WIFI environment, the function switching is automatically carried out and the function is returned.
When the server receives the data stream, only the user ID verification of the packet header is carried out, and the source address verification is not carried out, so that the transmission continuity of the audio and video data is ensured when the mobile phone end is switched to the WIFI environment from the 3G \4G \5G network environment.
The invention can complete all functions of audio and video intercommunication (audio and video conference) by using a plurality of devices as one client simultaneously. During switching, when the server receives the first data sent by the switched equipment, the data of the previous equipment is closed, so that seamless switching is ensured.
When the mobile phone end is used for video intercommunication, when the mobile phone end enters the WIFI environment from 4G \5G, the connection-free advantage of UDP is utilized, and the fact that real-time audio and video transmission experience is not influenced when the mobile phone end is switched between the 4G \5G environment and the WIFI environment can be guaranteed.
When the mobile phone terminal enters a WIFI environment, the background automatically searches for the equipment, finds the audio acquisition and playing equipment (intelligent sound box), the large-screen playing equipment (set top box and intelligent television) and the video acquisition equipment (IPCamera), and reminds a user whether to change corresponding video acquisition, video playing, audio acquisition and audio playing.
When a user changes, the WIFI network is used for communication (faster) of the intranet to inform related equipment to start a corresponding function, for example, the smart television plays video, receives the notification, joins video and audio talkback (or video conference) from the client, and requests a video and audio stream to play.
When the smart television receives the video stream and starts playing, the mobile phone end is informed to stop playing the video and the audio, but the audio and video acquisition part of the mobile phone end still continues to operate.
Similarly, when the smart speaker device is notified, the smart speaker device joins a video-audio intercom (or video conference) from the client to perform audio acquisition, transmission, and audio request and play.
In the switching process, in order to ensure seamless switching, for a client, such as an intelligent sound box device, when starting audio acquisition and sending a first frame of audio data, a mobile phone end is informed to stop an audio acquisition (encoding and sending) module, when the first frame of audio is received and the audio is started to be played, the mobile phone end is informed to stop the audio playing, and for a server, when the first frame of audio data from the intelligent sound box is received, the audio data of a subsequent mobile phone end is not received any more. Similarly, for the smart television (set top box), when the first frame of video is received, the mobile phone end is informed to stop video playing, so that the loss of playing content caused in the switching process is reduced.
When a main client (mobile phone) leaves a WIFI network, the mobile phone detects that the network environment changes, automatically sends a switching request to a server, and automatically starts audio and video acquisition and audio and video playing. And when the server receives the switching request, stopping receiving the audio and video data from the client and stopping sending the audio and video data to the slave client. Or the switching process can be manually performed by the user through a host client (mobile phone) interface.
The mobile phone client comprises a video acquisition module, an audio acquisition module, a video playing module, an audio playing module, an equipment searching module and a network communication module.
And each intelligent equipment terminal (slave client) is provided with a corresponding audio-visual acquisition module or playing module, an equipment searching module and a network communication module according to different equipment types.
And the intelligent equipment end is started at the background, logs in a server and starts an equipment searching module.
Technical effects
1. The playing advantages or the collecting advantages of different devices can be fully utilized, such as the video playing advantage of a television, the audio playing advantage and the audio collecting advantage of an intelligent sound box.
2. The functional defects of different devices, such as a smart television (or a set-top box), can be avoided, and some devices are provided with a camera and a microphone, but the mainstream devices do not have the acquisition devices.
3. Mobile phone end acquisition and play and audio and video processing equipment such as a smart television (or a set top box), a smart sound box, an IPCamera and the like are switched rapidly and seamlessly, so that video intercommunication (video conference experience) is improved
4. The advantages are fully utilized, and convenience and mobility of the mobile phone for video conference are guaranteed.
Drawings
Fig. 1 is a flow chart of switching functions of a mobile phone terminal to an equipment terminal:
fig. 2 is a flow chart of the device side functionality switching back to the handset side.
Detailed Description
The following further describes the embodiments of the present invention with reference to the drawings:
as shown in fig. 1 and 2, the video talk-back (video conference) server supports master-slave client login for the same user ID.
The mobile phone client comprises a video acquisition module, an audio acquisition module, a video playing module, an audio playing module, an equipment searching module and a network communication module.
Each intelligent device end (slave client) is provided with a corresponding audio-visual acquisition module or playing module, a device searching module and a network communication module according to different device types.
And starting the background of the intelligent equipment terminal, logging in the server and starting the equipment searching module.
When the mobile phone client performs video talkback (video conference), and the network state is found to be a WIFI environment, the equipment searching module is called to perform equipment searching, and when relevant intelligent equipment (slave client) is searched, a user is reminded whether to switch the response function to the slave client.
And when the user performs function switching operation, the related functions are switched to the corresponding intelligent equipment from the mobile phone end.
And when the user manually switches back, the related functions are switched back to the mobile phone end from the corresponding intelligent equipment.
And when the mobile phone leaves the WIFI environment, the function switching is automatically carried out and the function is returned.
When the server receives the data stream, only the user ID verification of the packet header is carried out, and the source address verification is not carried out, so that the transmission continuity of the audio and video data is ensured when the mobile phone end is switched to the WIFI environment from the 3G \4G \5G network environment.
The invention can complete all functions of audio and video intercommunication (audio and video conference) by using a plurality of devices as one client simultaneously. During switching, when the server receives the first data sent by the switched equipment, the data of the previous equipment is closed, so that seamless switching is ensured.
Claims (9)
1. A multi-screen interaction method for audio and video real-time talkback is characterized by comprising the following steps:
(1) the video talkback or video conference server supports the login of a master client and a slave client of the same user ID; the intelligent equipment end is started at a background, logs in a server and starts an equipment searching module;
(2) when the mobile phone client performs video talkback or video conference and finds that the network state is a WIFI environment, the device searching module is called to perform device searching, and when related intelligent devices or slave clients are found, a user is reminded of whether to switch the response function to the slave client;
(3) when a user performs function switching operation, relevant functions are switched to corresponding intelligent equipment from a mobile phone end;
(4) when the user manually switches back, the related functions are switched back to the mobile phone end from the corresponding intelligent equipment;
(5) and when the mobile phone leaves the WIFI environment, the function switching is automatically carried out and the function is returned.
2. The multi-screen interaction method for audio and video real-time intercom according to claim 1, wherein when the server receives the data stream, only user ID verification of a packet header is performed, and source address verification is not performed, so that consistency of transmission of audio and video data is guaranteed when a mobile phone end is switched from a 3G \4G \5G network environment to a WIFI environment.
3. The multi-screen interaction method for audio-video real-time intercom according to claim 1, wherein all functions of audio-video intercom or audio-video conference are completed by using multiple devices as one client at the same time; during switching, when the server receives the first data sent by the switched equipment, the data of the previous equipment is closed, so that seamless switching is ensured.
4. The multi-screen interaction method for audio-video real-time intercom according to claim 1, wherein when the mobile phone end is used for video intercom and enters a WIFI environment from 4G \5G, a connection-free method of UDP is used to ensure that real-time audio-video transmission experience is not affected when the mobile phone end is switched between the 4G \5G environment and the WIFI environment.
5. The multi-screen interaction method for audio and video real-time intercom according to claim 1, wherein when a mobile phone enters a WIFI environment, a background automatically searches for equipment, finds audio acquisition and play equipment or an intelligent sound box, large-screen play equipment and video acquisition equipment, and reminds a user whether to change corresponding video acquisition, video play, audio acquisition and audio play.
6. The multi-screen interaction method for audio-video real-time intercom according to claim 1, wherein when a user changes, the WIFI network is used for communicating with an intranet to inform related equipment to start a corresponding function, the function is used as a client to join in a video-audio intercom or video conference, and a video-audio stream is requested to be played;
when the smart television receives the video stream and starts playing, the smart television informs the mobile phone end to stop playing the video and audio, but the audio and video acquisition part of the mobile phone end still continues to operate;
similarly, when the intelligent sound box device is notified, the intelligent sound box device is used as a client to join in video-audio talkback or video conference, and audio acquisition, transmission, audio request and playing are carried out.
7. A multi-screen interaction method for real-time audio and video intercom according to claim 1, wherein in the process of switching, in order to ensure seamless switching, for the client, when starting audio acquisition and sending a first frame of audio data, the mobile phone is notified to stop the audio acquisition module, when the first frame of audio is received and the playing is started, the mobile phone is notified to stop the audio playing, for the server, when the first frame of audio data from the smart speaker is received, the audio data of the mobile phone end coming subsequently is not received any more; similarly, for the smart television, when the first frame of video is received, the mobile phone end is notified to stop video playing, so that the loss of playing content caused in the switching process is reduced.
8. The multi-screen interaction method for audio and video real-time talkback according to claim 1, wherein when a main client leaves a WIFI network, a mobile phone terminal detects that a network environment changes, automatically sends a switching request to a server, and automatically starts audio and video acquisition and audio and video playing; when the server receives the switching request, stopping receiving the audio and video data from the client and sending the audio and video data to the slave client; or the switching process is performed manually by the user through the host client interface.
9. The multi-screen interaction method for audio and video real-time intercom according to claim 1, wherein the mobile phone client comprises a video acquisition module, an audio acquisition module, a video playing module, an audio playing module, an equipment searching module or a network communication module;
and each intelligent equipment end is provided with a corresponding audio-visual acquisition module or playing module, an equipment searching module and a network communication module according to different equipment types.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010329780.XA CN111447398A (en) | 2020-04-23 | 2020-04-23 | Multi-screen interaction method for audio and video real-time talkback |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010329780.XA CN111447398A (en) | 2020-04-23 | 2020-04-23 | Multi-screen interaction method for audio and video real-time talkback |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111447398A true CN111447398A (en) | 2020-07-24 |
Family
ID=71653429
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010329780.XA Pending CN111447398A (en) | 2020-04-23 | 2020-04-23 | Multi-screen interaction method for audio and video real-time talkback |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111447398A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112261339A (en) * | 2020-10-15 | 2021-01-22 | Oppo广东移动通信有限公司 | Video communication method, video communication device, electronic equipment and computer-readable storage medium |
-
2020
- 2020-04-23 CN CN202010329780.XA patent/CN111447398A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112261339A (en) * | 2020-10-15 | 2021-01-22 | Oppo广东移动通信有限公司 | Video communication method, video communication device, electronic equipment and computer-readable storage medium |
| CN112261339B (en) * | 2020-10-15 | 2023-04-11 | Oppo广东移动通信有限公司 | Video communication method, video communication device, electronic equipment and computer-readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110535871B (en) | WebRTC-based classroom real-time video projection method and system | |
| CN102209230B (en) | Video conference system with main venue control module | |
| WO2013113190A1 (en) | Method and system for implementing multi-screen synchronous display | |
| US20150365453A1 (en) | Media projection method and device, control terminal and cloud server | |
| US20140118468A1 (en) | Method and system for adapting a television for multimedia conferencing | |
| KR20140066641A (en) | Server, multimedia apparatus, and control method of thereof | |
| KR20160019479A (en) | Methods, systems, and media for managing output of an hdmi source | |
| EP2924998A1 (en) | Method, apparatus and system for acquiring playback data stream of real-time video communication | |
| US11323661B2 (en) | Multi-device recording synchronization method and system, and conference system | |
| CN105681845A (en) | Method and system for synchronously watching television program through mobile terminal | |
| CN105763905A (en) | Method and system for sharing camera between mobile phone and television set | |
| CN108924603B (en) | Multi-terminal streaming media playing system and method based on WiFi (Wireless Fidelity) defined by software | |
| CN103338346A (en) | Method and system for realizing multimedia digital conference | |
| CN106488255B (en) | A kind of method and device of switching media stream | |
| JP5428734B2 (en) | Network device, information processing apparatus, stream switching method, information processing method, program, and content distribution system | |
| CN103561311A (en) | Screen transmission method and television | |
| CN111447398A (en) | Multi-screen interaction method for audio and video real-time talkback | |
| JP6536716B2 (en) | Transmission system, first transmission terminal, transmission method, display method and program | |
| JP2007166572A (en) | Method, computing system, and computer readable medium for group reproduction applied on network | |
| WO2016169228A1 (en) | Fusion equipment supporting multimedia telephony, communication method and user equipment | |
| WO2008141542A1 (en) | A method, video device and system for displaying information at the time of switching channels | |
| JP2008271197A (en) | Television system | |
| CN104917996A (en) | Method and system for switching video call | |
| US11818185B2 (en) | Audio streaming from host Bluetooth device to multiple receiving Bluetooth devices | |
| CN106375933A (en) | Projection control method of multiple intelligent hardware devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |