WO2020253754A1

WO2020253754A1 - Multi-terminal multimedia data communication method and system

Info

Publication number: WO2020253754A1
Application number: PCT/CN2020/096679
Authority: WO
Inventors: 杨枭; 黎椿键
Original assignee: 华为技术有限公司
Priority date: 2019-06-19
Filing date: 2020-06-18
Publication date: 2020-12-24
Also published as: JP7416519B2; CN112118062B; CN112118062A; JP2022537012A

Abstract

Embodiments of the present application relate to the technical field of multimedia, and provided thereby is a multi-terminal multimedia data communication method, which may use a wireless connection to achieve the transmission of multimedia data between a plurality of terminals. The specific solution is as follows: a master electronic device establishes a connection with a first slave electronic device and a second slave electronic device, respectively. The first slave electronic device receives a playback instruction and sends the playback instruction to the master electronic device. The master electronic device responds to the playback instruction, plays back first multimedia data, and simultaneously sends at least part of the first multimedia data to the first slave electronic device and the second slave electronic device, so that the master electronic device, the first slave electronic device and the second slave electronic device simultaneously play back the first multimedia data. The first slave electronic device receives a first human voice and sends same to the master device, while the second slave electronic device receives a second human voice and sends same to the master device. The master electronic device mixes the first human voice, the second human voice and the first multimedia data so as to generate and play back second multimedia data.

Description

Multi-terminal multimedia data communication method and system

This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office of China with the application number 201910533498.0, and the Chinese patent application with the title of “a multi-terminal multimedia data communication method and system” on June 19, 2019 Priority, the entire content of which is incorporated in this application by reference.

Technical field

The embodiments of the present application relate to the field of multimedia technology, and in particular, to a method and system for multi-terminal multimedia data communication.

Background technique

With the popularization and intelligent development of mobile terminal audio equipment such as mobile phones, tablets, personal computers (PC) and wireless speakers, there are usually multiple smart devices in the home at the same time, and they are connected to the same In a local area network, cross-device interaction and collaborative entertainment have become an important development direction for enhancing user entertainment experience.

In the current multi-terminal collaborative uplink and downlink multimedia applications, taking multi-machine K song as an example, it generally includes multiple electronic devices (such as mobile phones), one of which is the host and the other electronic devices are the slaves. After the host establishes the karaoke room through the host, turn on the KTV function, and other slave computers can join the karaoke room. Homeowners can order songs online. The songs are the same as common MVs in KTV karaoke rooms, including pictures, subtitles and accompaniment. The audience can apply for the microphone through the slave machine and be approved by the host, then order songs online and start the microphone arrangement. When it is the turn to play the song ordered by the audience, the audience becomes the singer. The audience can adjust the accompaniment and human voice volume through the slave machine, but at this time the song control authority (song pause, song cut) still belongs to the host. During this process, the "microphone" will be delivered to different mic audiences in sequence, and the singer has always been one person, which cannot meet the real-life use scenario of multi-person KTV with multiple people karaoke at the same time.

Summary of the invention

The embodiments of the present application provide a multi-terminal multimedia data communication method and system, which can enable multiple terminals to collaborate to establish new applications to provide new experiences, and also enable convenient interconnection and sharing of multimedia content between devices, thereby Make full use of the advantages of each device. In addition, the present invention is also beneficial to construct an ecological and application system based on multi-machine coordination.

In order to achieve the foregoing objectives, the following technical solutions are adopted in the embodiments of this application:

In the first aspect, an embodiment of the present application provides a multi-terminal multimedia data communication method, which is applied to multiple electronic devices, and the multiple electronic devices include a master electronic device, a first slave electronic device, and a second slave electronic device. The method includes: the master electronic device establishes connections with the first slave electronic device and the second slave electronic device respectively. The first slave electronic device receives the first play instruction, and the first slave electronic device sends the first play instruction to the master electronic device. In response to the first play instruction, the master electronic device plays the first multimedia data, and at the same time sends at least a part of the first multimedia data to the first slave electronic device and the second slave electronic device, so that the master electronic device and the first The slave electronic device and the second slave electronic device synchronously play the first multimedia data. The first slave electronic device receives the first human voice, while the second slave electronic device receives the second human voice. The master electronic device receives the first human voice sent by the first slave electronic device and receives the second human voice sent by the second slave electronic device. The master electronic device combines the first human voice, the second human voice, and the first multimedia data. Mixing is performed to generate second multimedia data and play.

In this solution, the master electronic device can establish connections with multiple slave electronic devices to realize multimedia data interaction between the master electronic device and the multiple slave electronic devices. The user can send the first play instruction to the master electronic device through the first slave electronic device. The main electronic device responds to the first play instruction to play the first multimedia data. At the same time, the master electronic device can send at least a part of the first multimedia data to multiple slave electronic devices, and use a synchronization algorithm to monitor the data synchronization between the master electronic device and the multiple slave electronic devices in real time, so that the master electronic device The device and multiple slave electronic devices synchronously play the first multimedia data. Then multiple slave electronic devices can simultaneously receive the human voice of their respective users, for example, the first slave electronic device receives the first human voice, while the second slave electronic device receives the second human voice. Multiple slave electronic devices send the received human voices of their respective users to the master electronic device, and the master electronic device mixes the received human voices with the first multimedia data, performs anti-howling and sound mixing processing, and generates the first 2. Multimedia data and play.

With reference to the first aspect, in the first embodiment of the first aspect, the master electronic device plays the first multimedia data synchronously with the first slave electronic device and the second slave electronic device, specifically: the master electronic device determines that the master electronic device The first clock deviation with the first slave electronic device; the master electronic device determines the second clock deviation between the master electronic device and the second slave electronic device; the master electronic device determines the first slave electronic device according to the first clock deviation The first starting time of playing the first multimedia data; the first starting time of playing is used to indicate the starting time of the first slave electronic device to play the first multimedia data; the master electronic device determines the first time according to the second clock deviation 2. The second initial playing moment of the slave electronic device playing the first multimedia data; the second initial playing moment is used to indicate the starting moment of the second slave electronic device playing the first multimedia data.

In this solution, in order to enable multiple terminal devices to play the first multimedia data synchronously, the master electronic device determines the first clock deviation between the master electronic device and the first slave electronic device to determine that the first slave electronic device plays the first multimedia data. A first initial playing moment of multimedia data, where the first initial playing moment is used to indicate a starting moment when the first slave electronic device plays the first multimedia data. The master electronic device uses the second clock deviation between the master electronic device and the second slave electronic device to determine the second initial playback time at which the second slave electronic device plays the first multimedia data. The second initial playback time is used Yu represents the start time when the second slave electronic device plays the first multimedia data. Therefore, the master electronic device can adjust the first multimedia data to be played by the first slave electronic device and the second slave electronic device through the start time of the master electronic device playing the first multimedia data, the first clock deviation, and the second clock deviation. At the initial playing time of, the master electronic device, the first slave electronic device, and the second slave electronic device synchronously play the first multimedia data.

It should be noted that the clock deviation can also be determined by the slave electronic device. At this time, the master electronic device sends the starting time of playing the first multimedia data to the first slave electronic device and the second slave electronic device. The first slave electronic device determines the first clock deviation between the master electronic device and the first slave electronic device, and determines the first slave electronic device according to the start time when the master electronic device plays the first multimedia data and the first clock deviation The start time of playing the first multimedia data. Similarly, the second slave electronic device determines the second clock deviation between the master electronic device and the second slave electronic device, and determines the second clock deviation according to the start time when the master electronic device plays the first multimedia data and the second clock deviation. The starting time of playing the first multimedia data from the electronic device.

In another possible situation, the master electronic device determines the clock deviation, and sends its own start playback time and clock deviation to the slave electronic device, and the slave electronic device determines the start playback time. The master electronic device determines the first clock deviation between the master electronic device and the first slave electronic device; the master electronic device determines the second clock deviation between the master electronic device and the second slave electronic device; the master electronic device plays the master electronic device The start time of the first multimedia data and the first clock deviation are sent to the first slave electronic device, and the first slave electronic device determines the first time according to the start time of the master electronic device playing the first multimedia data and the first clock deviation. A slave electronic device plays the start time of the first multimedia data; the master electronic device sends the start time of the master electronic device to play the first multimedia data and the second clock deviation to the second slave electronic device, the second slave The electronic device determines the start time of the second slave electronic device to play the first multimedia data according to the start time of the master electronic device playing the first multimedia data and the second clock deviation.

With reference to the first aspect or the first embodiment of the first aspect, in the second embodiment of the first aspect, the master electronic device establishes connections with the first slave electronic device and the second slave electronic device respectively, including: the master electronic device and The first slave electronic device is in the same wireless local area network, the master electronic device displays a WiFi connection identification, and the first slave electronic device establishes a connection with the master electronic device by identifying the WiFi connection identification.

In this solution, the master electronic device and the first slave electronic device are in the same wireless local area network, that is, the master electronic device and the first slave electronic device are in the same location, the master electronic device and the first slave electronic device establish a connection through WiFi. At this time, the main electronic device displays a WiFi connection identification, and the WiFi connection identification carries WiFi networking information, including: WiFi name, WiFi password, address and port number of the main electronic device. The first slave electronic device can establish a connection with the master electronic device by identifying the WiFi identifier.

With reference to the first aspect or the first embodiment of the first aspect, in the third embodiment of the first aspect, the master electronic device establishes connections with the first slave electronic device and the second slave electronic device respectively, including: the master electronic device and The second slave electronic device is not in the same wireless local area network, the master electronic device sends networking information to the second slave electronic device, and the second slave electronic device establishes a connection with the master electronic device by analyzing the networking information.

In this solution, the master electronic device and the second slave electronic device can be in different geographic locations. At this time, the master electronic device and the second slave electronic device are not in the same local area network. At this time, the master electronic device can send networking information to the second The slave electronic device, the second slave electronic device establishes a connection with the master electronic device by analyzing networking information. The networking information may include: the IP address and port number of the main electronic device. The master electronic device and the second slave electronic device can establish a connection with the data communication network through WiFi.

In combination with the third embodiment of the first aspect, in the fourth embodiment of the first aspect, the main electronic device mixes the first human voice, the second human voice, and multimedia data to generate and play the second multimedia data. , The method further includes: the master electronic device sends the second multimedia data to the second slave electronic device; the master electronic device and the second slave electronic device play the second multimedia data synchronously.

In this solution, since the second slave electronic device and the master electronic device are not in the same location, the master electronic device receives the first voice sent by the first slave electronic device and receives the second person sent by the second slave electronic device After the voice, the first voice, the second voice, and the first multimedia data are mixed. After generating the second multimedia data, in order to share the second multimedia data with the second slave electronic device that is not in the same location, the master electronic device sends the second multimedia data to the second slave electronic device, so that the second The second multimedia data can be played from the electronic device. In this process, the master electronic device needs to monitor the synchronization problem between the master electronic device and the second slave electronic device in real time, so that the master electronic device and the second slave electronic device synchronously play the second multimedia data.

With reference to the first aspect, in a fifth embodiment of the first aspect, at least a part of the first multimedia data includes audio, video, or lyrics of the first multimedia data.

In this solution, at least a part of the first multimedia data may be one of audio, video, lyrics, audio and video, audio and lyrics, video and lyrics, audio and video, and lyrics.

With reference to the first aspect, in a sixth embodiment of the first aspect, the method further includes: the main electronic device receives a second play instruction input by the user. In response to the second play instruction, the master electronic device plays the third multimedia data, and at the same time sends at least a part of the third multimedia data to the first slave electronic device and the second slave electronic device, so that the master electronic device and the first The slave electronic device and the second slave electronic device synchronously play the third multimedia data. The first slave electronic device receives the first human voice, while the second slave electronic device receives the second human voice. The master electronic device receives the first human voice sent by the first slave electronic device and receives the second human voice sent by the second slave electronic device. The master electronic device combines the first human voice, the second human voice, and the third multimedia data Mixing is performed to generate fourth multimedia data and play.

In this solution, the user can directly issue a play instruction to the main electronic device, and the main electronic device responds to the play instruction to play the third multimedia data. It should be noted that the third multimedia data may be the same as the first multimedia data or different from the first multimedia data.

With reference to the first aspect or the sixth embodiment of the first aspect, in a seventh embodiment of the first aspect, the method further includes: first receiving the first human voice from the electronic device, and simultaneously receiving the second voice from the electronic device. Two human voices, while the main electronic device receives a third human voice. The master electronic device receives the first human voice sent by the first slave electronic device and the second human voice sent by the second slave electronic device. The master electronic device converts the first human voice, the second human voice, the third human voice, and the fourth human voice. The multimedia data is mixed, and a fifth multimedia file is generated and played.

In the solution of the present application, the master electronic device can also receive the user's human voice. At this time, the first electronic device, the second electronic device and the master electronic device can simultaneously receive the human voice of their respective users. For example, the first slave electronic device receives the first human voice. At the same time, the second slave electronic device receives the second human voice, while the master electronic device receives the third human voice. The first slave electronic device sends the received first human voice to the master electronic device, and the second slave electronic device sends the received second human voice to the master electronic device. The main electronic device mixes the first voice, the second voice, the third voice, and the fourth multimedia data to generate a fifth multimedia file and play it.

In the second aspect, embodiments of the present application provide a multi-terminal multimedia data communication system, including a master electronic device, a first slave electronic device, and a second slave electronic device: the master electronic device is used to communicate with the first slave electronic device, The second slave electronic devices establish connections respectively. The first slave electronic device is used for receiving the first play instruction, and the first electronic device is also used for sending the first play instruction to the master electronic device. The master electronic device is also used to respond to the first play instruction to play the first multimedia data, and at the same time send at least a part of the first multimedia data to the first slave electronic device and the second slave electronic device, so that the master electronic device The first multimedia data is played synchronously with the first slave electronic device and the second slave electronic device. The first slave electronic device is also used to receive the first human voice, while the second slave electronic device is used to receive the second human voice. The master electronic device is used to receive the first human voice sent by the first slave electronic device and the second human voice sent by the second slave electronic device. The master electronic device is also used to transmit the first human voice, the second human voice, and the second human voice. One multimedia data is mixed, and second multimedia data is generated and played.

In this solution, the master electronic device can establish connections with multiple slave electronic devices to realize multimedia data interaction between the master electronic device and the multiple slave electronic devices. The user can send the first play instruction to the master electronic device through the first slave electronic device. The main electronic device responds to the first play instruction to play the first multimedia data. At the same time, the master electronic device can send at least a part of the first multimedia data to multiple slave electronic devices, and use a synchronization algorithm to monitor the data synchronization between the master electronic device and the multiple slave electronic devices in real time, so that the master electronic device The device and multiple slave electronic devices synchronously play the first multimedia data. Then multiple slave electronic devices can simultaneously receive the human voice of their respective users, for example, the first slave electronic device receives the first human voice, while the second slave electronic device receives the second human voice. Multiple slave electronic devices send the received human voices of their respective users to the master electronic device, and the master electronic device mixes the received human voices with the first multimedia data, performs anti-howling and mixing processing, and generates the first 2. Multimedia data and play.

With reference to the second aspect, in the first embodiment of the second aspect, the master electronic device plays the first multimedia data synchronously with the first slave electronic device and the second slave electronic device, specifically: the master electronic device is used to determine the master The first clock deviation between the electronic device and the first slave electronic device; the master electronic device is also used to determine the second clock deviation between the master electronic device and the second slave electronic device; the master electronic device is also used to determine the second clock deviation between the master electronic device and the second slave electronic device; Deviation, which determines the first initial playback moment when the first slave electronic device plays the first multimedia data; the first initial playback moment is used to indicate the starting moment when the first slave electronic device plays the first multimedia data; the master electronic device It is also used to determine the second initial playback time of the second slave electronic device to play the first multimedia data according to the second clock deviation; the second initial playback time is used to indicate that the second slave electronic device plays the first multimedia The start time of the data.

With reference to the second aspect or the first embodiment of the second aspect, in the second embodiment of the second aspect, the master electronic device establishes connections with the first slave electronic device and the second slave electronic device respectively, including: the master electronic device and The first slave electronic device is in the same wireless local area network, the master electronic device is used to display a WiFi connection identification, and the first slave electronic device is used to establish a connection with the master electronic device by identifying the WiFi connection identification.

With reference to the second aspect or the first embodiment of the second aspect, in the third embodiment of the second aspect, the master electronic device establishes connections with the first slave electronic device and the second slave electronic device respectively, including: the master electronic device and The second slave electronic device is not in the same wireless local area network, the master electronic device is used to send networking information to the second slave electronic device, and the second slave electronic device is used to establish a connection with the master electronic device by analyzing the networking information.

With reference to the third embodiment of the second aspect, in the fourth embodiment of the second aspect, the main electronic device mixes the first human voice, the second human voice, and multimedia data to generate and play the second multimedia data. The system further includes: the master electronic device is also used to send second multimedia data to the second slave electronic device; the master electronic device and the second slave electronic device play the second multimedia data synchronously.

With reference to the second aspect, in a fifth embodiment of the second aspect, at least a part of the first multimedia data includes audio, video, or lyrics of the first multimedia data.

With reference to the second aspect, in a sixth embodiment of the second aspect, the system further includes: the main electronic device is configured to receive a second play instruction input by the user. The master electronic device is also used to respond to the second play instruction to play the third multimedia data, and at the same time send at least a part of the third multimedia data to the first slave electronic device and the second slave electronic device, so that the master electronic device The third multimedia data is played synchronously with the first slave electronic device and the second slave electronic device. The first slave electronic device is used for receiving the first human voice, and the second slave electronic device is used for receiving the second human voice. The master electronic device is also used to receive the first human voice sent by the first slave electronic device and the second human voice sent by the second slave electronic device. The master electronic device is also used to transmit the first human voice, the second human voice, and The third multimedia data is mixed, and the fourth multimedia data is generated and played.

With reference to the second aspect or the sixth embodiment of the second aspect, in a seventh embodiment of the second aspect, the system further includes: the first slave electronic device is further configured to receive the first human voice, and the second slave electronic device The device is also used to receive the second human voice, and the main electronic device is also used to receive the third human voice. The master electronic device is also used to receive the first human voice sent by the first slave electronic device and the second human voice sent by the second slave electronic device. The master electronic device is also used to transmit the first human voice, the second human voice, and the second human voice. The three voices and the fourth multimedia data are mixed to generate a fifth multimedia file and play it.

On the other hand, an embodiment of the present application provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes computer instructions, and when the computer instructions are executed on a computer, the computer executes any of the above aspects. A possible implementation of a multi-terminal multimedia data communication method.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following will briefly introduce the drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative work, other drawings can be obtained based on these drawings.

FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the application;

2 is a schematic diagram of the software structure of an electronic device provided by an embodiment of the application;

3 is a schematic diagram of a local application of multi-terminal multimedia data communication provided by an embodiment of the application;

4 is a schematic diagram of a remote application of multi-terminal multimedia data communication provided by an embodiment of this application;

5 is a hardware and software system architecture of a multi-terminal multimedia data communication system host provided by an embodiment of the application;

FIG. 6 is a hardware and software system architecture of a slave machine of a multi-terminal multimedia data communication system provided by an embodiment of the application;

FIG. 7 is a flowchart of data interaction between a host and a slave according to an embodiment of the application;

FIG. 8 is a flowchart of another host-slave data interaction provided by an embodiment of this application;

FIG. 9 is a flowchart of another host-slave data interaction provided by an embodiment of this application;

FIG. 10 is a flowchart of another group of master-slave data interaction provided by an embodiment of this application;

FIG. 11 is a schematic diagram of a local application of multi-machine K song provided by an embodiment of the application;

FIG. 12A is a schematic diagram of a display interface provided by an embodiment of this application;

FIG. 12B is a schematic diagram of another display interface provided by an embodiment of the application;

FIG. 13 is a schematic diagram of another display interface provided by an embodiment of the application;

FIG. 14 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

15 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

FIG. 16 is a schematic diagram of another display interface provided by an embodiment of the application;

FIG. 17 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

18 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

19 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

20 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

21 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

FIG. 22 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

FIG. 23 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

24 is a schematic diagram of a remote application of multi-machine K song provided by an embodiment of the application;

FIG. 25 is a schematic diagram of a display interface provided by an embodiment of the application;

FIG. 26 is a schematic diagram of another display interface provided by an embodiment of the application;

FIG. 27 is a schematic diagram of another display interface provided by an embodiment of the application;

FIG. 28 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

FIG. 29 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

FIG. 30 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

FIG. 31 is a schematic diagram of another set of display interfaces provided by an embodiment of the application;

Figure 32 is a schematic diagram of a multi-terminal multimedia data communication system provided by an embodiment of the application

Detailed ways

In order to make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

FIG. 1 shows a schematic structural diagram of an electronic device 100.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2. , Mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber identification module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include pressure sensor 180A, gyroscope sensor 180B, air pressure sensor 180C, magnetic sensor 180D, acceleration sensor 180E, distance sensor 180F, proximity light sensor 180G, fingerprint sensor 180H, temperature sensor 180J, touch sensor 180K, ambient light Sensor 180L, bone conduction sensor 180M, etc.

It can be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU), etc. Among them, the different processing units may be independent devices or integrated in one or more processors.

The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.

In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include integrated circuit (I2C) interfaces, integrated circuit built-in audio (inter-integrated circuit sound, I2S) interfaces, pulse code modulation (PCM) interfaces, universal asynchronous transmitters and receivers (universal asynchronous transmitters). receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / Or Universal Serial Bus (USB) interface, etc.

The I2C interface is a two-way synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple sets of I2C buses. The processor 110 may be coupled to the touch sensor 180K, charger, flash, camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may couple the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through an I2C bus interface to implement the touch function of the electronic device 100.

The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple sets of I2S buses. The processor 110 may be coupled with the audio module 170 through an I2S bus to realize communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may transmit audio signals to the wireless communication module 160 through an I2S interface, so as to realize the function of answering calls through a Bluetooth headset.

The PCM interface can also be used for audio communication to sample, quantize and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 may also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a two-way communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 may transmit audio signals to the wireless communication module 160 through a UART interface, so as to realize the function of playing music through a Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with the display screen 194, the camera 193 and other peripheral devices. The MIPI interface includes camera serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through a CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate through a DSI interface to realize the display function of the electronic device 100.

The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and so on. GPIO interface can also be configured as I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 130 is an interface that complies with the USB standard specification, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and so on. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through the headphones. This interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely a schematic description, and does not constitute a structural limitation of the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection modes in the foregoing embodiments, or a combination of multiple interface connection modes.

The charging management module 140 is used to receive charging input from the charger. Among them, the charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive the charging input of the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 may receive the wireless charging input through the wireless charging coil of the electronic device 100. While the charging management module 140 charges the battery 142, it can also supply power to the electronic device through the power management module 141.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be provided in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may also be provided in the same device.

The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna can be used in combination with a tuning switch.

The mobile communication module 150 can provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform processing such as filtering, amplifying and transmitting the received electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.

The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is processed by the baseband processor and then passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), and global navigation satellites. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, perform frequency modulation, amplify it, and convert it into electromagnetic wave radiation via the antenna 2.

In some embodiments, the antenna 1 of the electronic device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite-based augmentation systems (SBAS).

The electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode). AMOLED, flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, and N is a positive integer greater than one.

The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

The ISP is used to process the data fed back from the camera 193. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transfers the electrical signal to the ISP for processing and is converted into an image visible to the naked eye. ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.

The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it to the photosensitive element. The light-sensing element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats. In some embodiments, the electronic device 100 may include 1 or N cameras 193, and N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in a variety of encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, and so on.

NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, for example, the transfer mode between human brain neurons, it can quickly process input information and can continuously learn by itself. The NPU can realize applications such as intelligent cognition of the electronic device 100, such as image recognition, face recognition, voice recognition, text understanding, and so on.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, save music, video and other files in an external memory card.

The internal memory 121 may be used to store computer executable program code, where the executable program code includes instructions. The internal memory 121 may include a storage program area and a storage data area. Among them, the storage program area can store an operating system, at least one application program (such as a sound playback function, an image playback function, etc.) required by at least one function. The data storage area can store data (such as audio data, phone book, etc.) created during the use of the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. For example, music playback, recording, etc.

The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110, or part of the functional modules of the audio module 170 may be provided in the processor 110.

The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music through the speaker 170A, or listen to a hands-free call.

The receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a call or voice message, it can receive the voice by bringing the receiver 170B close to the human ear.

The microphone 170C, also called "microphone", "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can approach the microphone 170C through the mouth to make a sound, and input the sound signal to the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which can implement noise reduction functions in addition to collecting sound signals. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions.

The earphone interface 170D is used to connect wired earphones. The earphone interface 170D may be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be provided on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors and so on. The capacitive pressure sensor may include at least two parallel plates with conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations that act on the same touch location but have different touch operation strengths may correspond to different operation instructions. For example: when a touch operation whose intensity of the touch operation is less than the first pressure threshold is applied to the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

The gyro sensor 180B may be used to determine the movement posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the shake angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shake of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude based on the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip holster. In some embodiments, when the electronic device 100 is a flip machine, the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, features such as automatic unlocking of the flip cover are set.

The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of electronic devices, and used in applications such as horizontal and vertical screen switching, pedometers, etc.

Distance sensor 180F, used to measure distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light to the outside through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and the pocket mode will automatically unlock and lock the screen.

The ambient light sensor 180L is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived brightness of the ambient light. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket to prevent accidental touch.

The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application locks, fingerprint photographs, fingerprint answering calls, etc.

The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold value, the electronic device 100 executes to reduce the performance of the processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

Touch sensor 180K, also called "touch device". The touch sensor 180K may be disposed on the display screen 194, and the touch screen is composed of the touch sensor 180K and the display screen 194, which is also called a “touch screen”. The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. The visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100, which is different from the position of the display screen 194.

The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can obtain the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 180M can also contact the human pulse and receive the blood pressure pulse signal. In some embodiments, the bone conduction sensor 180M may also be provided in the earphone, combined with the bone conduction earphone. The audio module 170 can parse the voice signal based on the vibration signal of the vibrating bone block of the voice obtained by the bone conduction sensor 180M, and realize the voice function. The application processor may analyze the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M, and realize the heart rate detection function.

The button 190 includes a power button, a volume button, and so on. The button 190 may be a mechanical button. It can also be a touch button. The electronic device 100 may receive key input, and generate key signal input related to user settings and function control of the electronic device 100.

The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration notification, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as photographing, audio playback, etc.) can correspond to different vibration feedback effects. Acting on touch operations in different areas of the display screen 194, the motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminding, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

The indicator 192 may be an indicator light, which may be used to indicate the charging status, power change, or to indicate messages, missed calls, notifications, and so on.

The SIM card interface 195 is used to connect to the SIM card. The SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to achieve contact and separation with the electronic device 100. The electronic device 100 may support 1 or N SIM card interfaces, and N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. The same SIM card interface 195 can insert multiple cards at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 may also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as call and data communication. In some embodiments, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present invention takes an Android system with a layered architecture as an example to exemplify the software structure of the electronic device 100.

FIG. 2 is a software structure block diagram of an electronic device 100 according to an embodiment of the present invention.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Communication between layers through software interface. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

The application layer can include a series of application packages.

As shown in Figure 2, the application package can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

The application framework layer provides application programming interfaces (application programming interface, API) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 2, the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, and a notification manager.

The window manager is used to manage window programs. The window manager can obtain the size of the display, determine whether there is a status bar, lock the screen, take a screenshot, etc.

The content provider is used to store and retrieve data and make these data accessible to applications. The data may include video, image, audio, phone calls made and received, browsing history and bookmarks, phone book, etc.

The view system includes visual controls, such as controls that display text and controls that display pictures. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface that includes a short message notification icon may include a view that displays text and a view that displays pictures.

The phone manager is used to provide the communication function of the electronic device 100. For example, the management of the call status (including connecting, hanging up, etc.).

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and it can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify the download completion, message reminder, etc. The notification manager can also be a notification that appears in the status bar at the top of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, text messages are prompted in the status bar, prompt sounds, electronic devices vibrate, and indicator lights flash.

Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.

The core library consists of two parts: one part is the function functions that the java language needs to call, and the other part is the core library of Android.

The application layer and the application framework layer run in a virtual machine. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

The system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), three-dimensional graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc.

The surface manager is used to manage the display subsystem and provides a combination of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to realize 3D graphics drawing, image rendering, synthesis, and layer processing.

The 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

In the following, the workflow of the software and hardware of the electronic device 100 will be exemplified in conjunction with capturing a photo scene.

When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into original input events (including touch coordinates, time stamps of touch operations, etc.). The original input events are stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer. The camera 193 captures still images or videos.

The embodiments of this application provide a multi-terminal multimedia data communication method and system, which can use wireless connections (such as Wi-Fi, Bluetooth, mobile communication systems, etc.) to achieve low-latency synchronization of multimedia data streams between multiple terminal devices. Point transmission. The types of mobile terminal devices in the embodiments of this application are not limited, and can be mobile phones, portable computers, PDAs (personal digital assistants), tablet computers, smart TVs, smart speakers, PCs, smart homes, wireless terminal devices, communications Equipment and other electronic equipment. The multi-terminal multimedia data communication method in the embodiment of the present application can be used in application scenarios such as multi-machine playback, multi-machine K song, multi-machine conversation, and multi-machine recording. The above-mentioned multimedia applications based on the multi-terminal multimedia data communication method, such as multi-machine playback, multi-machine karaoke, multi-machine conversation, multi-machine recording, etc., include local applications and remote applications. The embodiments of this application have no special requirements on the device type and the number of networking devices, as long as it is a smart device with WiFi networking capabilities, it can be supported.

Exemplarily, FIG. 3 shows a schematic diagram of a local application of multi-terminal multimedia data communication. As shown in Figure 3, three slaves are connected to a master to realize multimedia sharing. In this embodiment, the three slaves are only for illustrative purposes. In actual use, 4 or more slaves can be connected to the master. . Among them, multiple slaves can collect the voice information of their respective users. For example, the microphone slave 1 can collect the voice information of user 1, the microphone slave 2 can collect the voice information of user 2, and the microphone slave 3 can collect the voice of user 3. information. In order to realize multimedia data sharing, the host and the slave need to be networked first. When the local application is networked, the host interface can display a QR code to provide connection parameters, or activate other short-distance communication functions such as NFC (such as WiFi, Bluetooth, etc.) Provide connection parameters to initiate networking. The QR code may carry the WiFi name, WiFi password, address and port number of the main electronic device. The slave machine obtains the connection parameters by scanning the QR code or NFC touch or other short-distance communication methods and joins the network to establish a connection.

Smart phones, smart TVs and smart speakers can all be used as hosts. When cooperating with karaoke applications, the host downloads karaoke accompaniment and video images from the cloud, and mixes the vocal and accompaniment sent from each slave to eliminate howling and play. During multi-person calls or multi-person conference speech applications, the host can download background music from the cloud, mix the human voice sent from each slave with the background music, eliminate howling and play it.

Mobile terminals such as smart phones, tablets, smart watches, wearable devices, etc. can be used as slaves. In the collaborative K song application, the slave can be used as a microphone to pick up the user's singing or speech and send it to the host via short-distance communication such as WiFi. On the other hand, the slave also receives the downstream data stream from the master to display the K song's picture and lyrics. In a possible implementation, in addition to displaying the K song and lyrics, the slave can also play the mixed audio of the vocal and accompaniment sent by the master. This function is turned on by default when the slave is connected to the headset .

Among them, the host and slave can be connected to various peripherals. For example, speakers (wired or Bluetooth) or amplifying devices are used to amplify the sound, and headphones can also be connected.

Exemplarily, FIG. 4 shows a schematic diagram of a remote application of multi-terminal multimedia data communication. As shown in Figure 4, four slaves are connected to a master to realize multimedia sharing. The master and user 1 and user 2 are at location A, user 3 is at location B, and user 4 is at location C. At this time, the master and slaves Located in the same or different regions. In order to realize multimedia file sharing, the host and slave need to be networked first. For local slaves, you can connect to the master via WiFi, Bluetooth or NFC. The host can display the QR code to provide connection parameters or provide connection parameters by activating short-range communication functions such as NFC to initiate networking, and the slave can scan the QR code or other short-range communication methods such as NFC to obtain networking information to join the network. For remote slaves, you can connect to the host via WiFi or a data communication network (2G/3G/4G/5G). The host can send the QR code to the remote slave, or send networking information to the remote slave, and the QR code or networking information carries the host's IP address and port number. The remote slave machine participates in the networking by identifying the QR code or analyzing the networking information, thereby connecting to the host.

Smart phones, smart TVs and smart speakers can all be used as hosts. The host downloads K song accompaniment and video screen or background sound from the cloud, and mixes the vocals sent from each slave with the accompaniment or background sound to eliminate howling and play it locally. At the same time, the master sends the mixed audio stream and video stream to the local slave through short-distance communication such as WiFi. The host sends the mixed audio and video streams to remote slaves via WiFi or data communication network.

As a slave, the mobile device can be used as a microphone to pick up the user's voice information and send it to the host. On the other hand, the slave also receives the downstream data stream from the master, such as K song pictures, lyrics or accompaniment during conference speech.

The local microphone slave can be used as a microphone to pick up the user’s singing or speech and send it to the host via short-distance communication such as WiFi. On the other hand, it receives the downstream data stream from the host to display the K song's picture and lyrics.

The remote microphone slave can be used as a microphone to pick up the user's voice information and send it to the host through the data communication network. On the other hand, it receives the downstream data stream from the host, so as to play the audio containing the accompaniment and the sound information of itself and other users, display the picture and lyrics of K song, or make the user hear other users during calls and meetings Voice.

In order to ensure a good user experience, the host needs to control the synchronization of the audio and video playback between the host and each slave. The host needs to determine the starting time of each slave to play the audio and video by determining the clock deviation between the host and each slave. For example, the master determines the first initial playback time of the multimedia data played by the slave 1 by determining the first clock deviation between the master and the slave 1, and the first initial playback time is used to indicate the start time of the multimedia data played by the slave . The master determines the second initial playback time of the multimedia data played by the slave 2 through the second clock deviation between the master and the slave 2, and the second initial playback time is used to indicate the start time of the slave 2 to play the multimedia data. Therefore, the master can adjust the initial playback time of the multimedia data played by the slave 1 and the second slave 2 through the start time of the master playing multimedia data and the first clock deviation and the second clock deviation, so that the master and the slave 1, the slave 2Play the multimedia data synchronously.

It should be noted that the clock deviation can also be determined by the slave. At this time, the host sends the starting time of playing multimedia data to the slave 1 and the slave 2. The slave 1 determines the first clock deviation between the master and the slave 1, and determines the start time of the multimedia data played by the slave 1 according to the start time of the master playing multimedia data and the first clock deviation. Similarly, the slave 2 determines the second clock deviation between the master and the slave 2, and determines the start time of the multimedia data played by the slave 2 according to the start time of the master playing multimedia data and the second clock deviation.

In another possible situation, the host determines the clock deviation, and sends its own start playing time and clock deviation to the slave, and the slave determines the start playing time. The host determines the first clock deviation between the host and the slave 1; the host determines the second clock deviation between the host and the slave 2; the host sends the start time and the first clock deviation of the multimedia data to the slave 1 , The slave 1 determines the start time of the multimedia data played by the slave 1 according to the start time of the master playing the multimedia data and the first clock deviation; the master sends the start time of the multimedia data played by the host and the second clock deviation to the slave 2 The slave 2 determines the start time of the multimedia data played by the slave 2 according to the start time of the host playing the multimedia data and the second clock deviation.

Next, the software and hardware system structure of a multi-terminal multimedia data communication system provided by this application is introduced. Applications such as multi-machine playback, multi-machine K song, multi-machine conversation, and multi-machine recording can all be implemented through the software and hardware system structure of the multi-terminal multimedia data communication system provided in this application. The embodiment of the application realizes that the software and hardware system architecture of the multi-terminal cooperative playing application system is unified, so a specific device can be used as a master to initiate networking, or as a slave to join the networking. When acting as a master or as a slave, the only difference is whether a specific function module is turned on.

Exemplarily, FIG. 5 shows a schematic diagram of a hardware and software system architecture 500 of a multi-terminal multimedia data communication system host. The host system architecture 500 may include a multi-machine cooperative slave data receiving module 510, a multi-machine cooperative downlink data receiving module 520, an audio decoding module 530, an anti-howling module 540, a sound mixing module 550, a sound effect processing module 551, and a cooperative audio output control Module 560, multi-machine cooperative data sending module 570, Bluetooth protocol stack 571, USB interface 572, recording algorithm module 573, Codec574, display interface 575, display 580, cooperative APP581, Modem interface 590, WiFi protocol stack/chip 591, Bluetooth chip /Antenna 592, TypeC digital interface 593, 3.5mm headphone jack/TypeC analog interface 594, earpiece or speaker 595, MIC array 596. The cooperative audio output control module 560 may include sampling rate and bit width conversion 560A, volume control 560B, channel selection 560C, channel combination 560D, and so on. The collaborative APP can include multi-machine play 581A, multi-machine K song 581B, multi-machine call 581C, multi-machine recording 581D, etc.

It can be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the hardware and software system architecture 500 of the multi-terminal multimedia data communication system host. In other embodiments of the present application, the hardware and software system architecture 500 of the multi-terminal multimedia data communication system host may include more or fewer modules or components than shown. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

Specifically, the functions between the modules are as follows:

The multi-machine coordinated slave data receiving module 510 is used to receive the uplink data sent from each slave with a very low time delay.

The downlink data receiving module 520 is configured to receive K song accompaniment and screen video from the cloud with extremely low latency, and read K song accompaniment and screen video in local storage.

The audio decoding module 530 is used to decode the audio data stream. If it is a channel encoding format, output the PCM code stream of each channel. If it is an audio object encoding format, output audio object information and channel PCM code stream.

The anti-howling module 540 is used to eliminate the echo mixed into the host MIC and the slave MIC and avoid howling. The anti-howling reference signal is provided by the audio decoding module.

The sound mixing module 550 is used for mixing the upward human voice with the accompaniment sound.

The sound effect processing module 551 is used for rendering effects, such as reverberation and 3D surround sound.

It should be noted that the channel PCM code stream processed by the mixing module 550 can be directly transmitted to the collaborative audio output control module 560. In addition, the user can choose to perform further sound effects on the channel PCM code stream after the mixing process. In processing 551, the rendered channel PCM code stream is obtained and then transmitted to the cooperative audio output control module.

The cooperative audio output control module 560 has multiple input terminals, and can receive inputs from multiple modules, such as a multi-machine cooperative downlink data receiving module 520, a sound mixing module 550, and a sound effect processing module 551. At the same time, it has the following low-latency sub-function modules inside:

Sampling rate and bit width conversion 560A: Convert the sampling rate so that the sampling rate and bit width meet the output requirements;

Volume control 560B: used to control the output volume;

Channel selection 560C: used to select which channel to output;

Channel Merging 560D: The user merges each channel into one channel for playback to increase the volume.

The cooperative audio output control module 560 can also input audio streams to the multi-machine cooperative data sending module 570, the Bluetooth protocol stack 571, the USB interface 572, the recording algorithm 573 and the Codec574. In addition, in order to maintain the synchronization of the media image and sound, the cooperative audio output control module 570 sends the delay value to the display interface 575 so as to be able to align the delay and achieve synchronization of accompaniment and lyrics.

The collaboration APP 581 is used to display collaboration functions on the screen and receive user input. The user controls the multi-machine coordinated data transmission 570 and coordinated audio output control 560 modules through the operation interface of the cooperative APP.

The multi-machine cooperative data sending module 570 is docked with the Modem interface 590 and the WiFi protocol stack/chip 591, and can send audio streams and video streams to each slave with extremely low delay.

In other embodiments, the Bluetooth protocol stack 571 can also be docked with the Bluetooth chip/antenna 593, and the USB interface 572 can be docked with the TypeC (digital) 593 to send audio streams and video streams to each slave.

Codec574 is connected with earpiece or speaker 595 and 3.5mm earphone holder/TypeC (analog) 595, and is used to play the PCM code stream that has been mixed 550 or processed by sound effect 551. In some embodiments, the host can also be used as a microphone. For example, when a smart phone is used as a host, it can also be used as a microphone to receive user singing. At this time, the Codec574 is docked with the MIC array to receive user input received by the host microphone.

Exemplarily, FIG. 6 shows a schematic diagram of a software and hardware system architecture 600 of a slave of a multi-terminal multimedia data communication system.

Among them, the multi-machine cooperative host data receiving module is used to receive the media data stream sent by the host with a very low delay

The slave MIC610 is connected to the Codec620 (or other interface chip), and enters the multi-machine cooperative data sending module through cooperative audio output control to send to the host.

For ease of understanding, the collaborative K song is taken as an example to introduce the data interaction process between the host and the slave of the multi-terminal multimedia data communication system proposed in this embodiment. The host in the collaborative K song application can be a smart TV or a smart phone. For smart phones, both the master and slave can be connected to external speakers for amplifying speakers, and the slave can also be connected to headphones. Therefore, cooperative karaoke can be divided into four modes: TV (master) + multiple local slaves, TV (master) + multiple local and remote slaves, smart phone (as master and microphone) + multiple local slaves, Smart phone (as master and microphone) + multiple local and remote slaves.

Exemplarily, FIG. 7 shows a data interaction process between the host and the slave when a smart TV (TV) is used as a master and multiple smart phones are used as slaves.

Step 701: The slave selects a song, and the song information is transmitted to the host via a short-distance communication method such as WiFi or Bluetooth.

Step 702: Multi-machine coordinated downlink data receiving module to receive K song accompaniment and screen video from the cloud with extremely low latency, and read K song accompaniment and screen video in local storage, and send the audio data stream to the audio decoding module , Send the audio data stream that can be played directly to the cooperative audio output control module, and send the video data stream to the display interface.

Step 703: The host sends the K song picture and lyrics to the slave through the multi-machine collaborative data sending module, and the lyrics and video information of the song are obtained from the machine.

Step 704: The audio decoding module decodes the audio data stream, and the display interface sends the video data stream to the display for display. The accompaniment audio processed by the audio decoding module is sent to the anti-howling module as an anti-howling reference signal, and at the same time to the mixing module and the sound effect processing module.

Step 705: The user sings, and the slave MIC obtains the user's voice information, and transmits the singing voice to the multi-machine coordinated slave data receiving module.

Step 706: The multi-machine coordinated slave data receiving module sends the MIC data stream of the slave human voice to the host through a short-distance communication method such as WiFi or Bluetooth. Since the human voice received by the slave MIC is mixed with the TV accompaniment sound, the anti-howling module is needed for anti-howling processing.

Step 707: Perform anti-howling processing to eliminate the echo mixed into the MIC of the slave and avoid howling.

Step 708: sound mixing processing, a clean human voice is obtained after anti-howling processing, and then the clean human voice and the accompaniment sound are mixed through the mixing module. The channel PCM code stream after mixing processing is sent to the cooperative audio output control module.

Optionally, step 709: the user can render the mixed sound channel PCM code stream to generate reverberation and 3D surround sound.

Step 710: The cooperative audio output control module can perform sampling rate and bit width conversion, volume control, channel selection, or channel combination on the channel PCM code stream after mixing processing or the channel PCM code after sound effect processing, etc. Processing, the processed channel PCM code stream is sent to Codec. In addition, in order to keep the lyrics and accompaniment music synchronized, the "cooperative audio output control" sends the delay value to the "display interface" to determine and adjust the relative delay between the lyrics and the music.

Step 711: Play the singing voice sung by the user through the host receiver or speaker.

As another example, FIG. 8 shows the interaction process between the host and the slave when the smart TV is used as the host and the mobile device is used as the local slave and the remote slave.

The difference from Figure 7 is that when a remote slave machine joins K song,

Step 801: Both the master and the slave can select songs. Take the local slave selection of songs as an example.

Step 802: The host WiFi or data communication network (2G/3G/4G/5G) sends the K song screen and lyrics to the remote slave machine, and the remote slave obtains the song lyrics and video information.

Step 803: The multi-machine coordinated slave data receiving module sends the MIC data stream of the remote slave human voice to the master via WiFi or a data communication network (4G/5G) for anti-howling processing.

Step 804: The multi-machine coordinated data sending module of the host sends the processed PCM code stream to the remote slave.

Step 805: The remote slave handset or speaker plays the accompaniment sound transmitted from the host, and simultaneously plays the singing voices sung by the local user and the remote user.

Exemplarily, as shown in FIG. 9, when a mobile device serves as both a master and a microphone, and multiple mobile devices serve as local slaves, the interaction process between the master and the slaves is shown.

The difference from Figure 7 is that at this time, a mobile phone acts as a host and a microphone.

Step 901: The host MIC receives the user's voice information.

Step 902: Send the host MIC data stream to the anti-howling processing module through the host Codec.

Step 903: Since the speaker of the mobile phone is relatively small, when the mobile phone is used as the host, the mobile phone can be used to connect external speakers such as external speakers to amplify the host sound and create a KTV effect.

Exemplarily, as shown in Figure 10, when a mobile device acts as both a host and a microphone, and multiple mobile devices are used as local slaves and remote slaves, the interaction process between the host and the slaves can be seen in Figures 8, 9 and 9 Figure 10 is combined.

Next, the UI operation interface and operation flow of the local application of multi-machine K song are explained in detail. Exemplarily, a smart TV is used as the master and multiple smart phones are used as slaves as an example for introduction. It should be noted that the number of slaves can be multiple, and this embodiment takes two slaves as an example for description.

Figure 11 shows a local application system for multi-machine K song, including a master smart TV, microphone slave 1 and microphone slave 2. Both slaves are smart phones, and the master and two slaves are in the same Within the local area network.

FIG. 12A shows a graphical user interface (GUI) on the TV side, and the GUI is the network configuration interface 1201 of the TV. FIG. 12B shows a graphical user interface of a mobile phone, and the GUI is a wireless local area network configuration interface 1202 of the mobile phone. As shown in Figure 12A and Figure 12B, at this time, the host TV and the two local mobile phone slaves are both connected to the same local area network.

Exemplarily, FIG. 13 shows a GUI of the TV. The GUI can become the TV desktop 1301. When the TV detects that the user controls the remote control to select the icon 1302 of the K song application (application, APP) on the TV desktop 1301, , You can start the TV version of the K song application, and display another GUI as shown in FIG. The recommendation interface 1401 may include a song-song console control 1402 for entering the song-song interface. When the TV detects that the user controls the remote control to click the karaoke station control 1402 on the recommendation interface 1401, a GUI as shown in FIG. 15 is displayed, and the GUI may be called the karaoke station interface 1501. The karaoke platform interface 1501 may include a control 1502 for entering the mobile phone scan code karaoke interface. When the TV detects that the user’s remote control clicks the control 1502 on the song order interface 1501, another GUI as shown in Figure 16 is displayed. This GUI can be called the mobile phone code scanning interface 1601, and the mobile phone code scanning interface 1601 1601 can include a song two-dimensional code 1602. By scanning and identifying the QR code 1602 of the song on the mobile phone's karaoke interface 1601 through the K song software on the mobile phone, the song can be ordered through the K song software on the mobile phone, and the mobile phone can be used as a microphone to receive the user's voice information. For multiple local mobile phones, K song can be added by scanning the song QR code 1602, so the KTV effect can be constructed through TV and mobile phones.

Next, the operation process of adding K song from the mobile phone is briefly explained.

Exemplarily, (a) in FIG. 17 shows a GUI of a mobile phone, and the GUI is a desktop 1701 of the mobile phone. At this time, the same K song software is installed on the mobile phone and the TV. When the mobile phone detects that the user clicks the icon 1702 of the K song application on the desktop 1701, the K song application can be started. (b) in Figure 20 shows the K song application startup interface 1703. After the K song application is successfully opened, Another GUI as shown in (c) in FIG. 17 is displayed. The GUI may include a home page interface 1704, and the home page interface 1704 displays recommendation information. The GUI may also include a menu bar 1705, and the menu bar 1705 has controls 1706 for entering my interface. When the mobile phone detects that the user has clicked the operation of the control 1706 for entering my interface, another GUI as shown in (a) in FIG. 18 is displayed. This GUI may be called my interface 1801. The GUI may include controls 1802. When the mobile phone detects that the user clicks on the control 1802, another GUI as shown in (b) of FIG. 18 is displayed. The GUI may include a control 1803 for scanning a two-dimensional code. Thereafter, if the mobile phone detects that the user has clicked on the control 1803, it can display an interface 1804 for scanning a two-dimensional code as shown in (c) in FIG. 18. As shown in (c) of FIG. 18, the interface 1804 for scanning the two-dimensional code is now aligned with the two-dimensional code for song on the TV. When the QR code is successfully recognized, the GUI 1901 shown in (a) in Figure 19 is displayed. The GUI includes a Koke console interface 1902. The GUI also includes a menu bar 1903. The menu bar 1903 includes The song’s karaoke console control 1904, the controller control 1905 for the user to sing, and the control 1906 for displaying the song that has been ordered, the song karaoke console control 1904 for the song is now lit. The karaoke station interface 1902 includes a disconnect control 1908, and the user can click the disconnect control 1908 to disconnect the mobile phone from the TV. The karaoke station interface 1902 may also include a control 1909 for singing a song, and a control 1910 for adding a song to the song list. The karaoke station interface 1902 also includes a search bar 1907. The search bar 1907 can search for songs by the user’s input of the name of the song and the name of the author. For example, as shown in (a) in Figure 19, the user selects the name of the song For the song of "Song 123", various versions of the song will appear under the search bar 1907 at this time. Illustratively, the user clicks the control 1909 to sing the selected song. At this time, the host TV can cache the songs in the cloud or locally. After the cache is successful, the host TV displays as shown in (b) in Figure 19, and the host TV plays the song "Song 123" selected by the user.

At this time, the user can click on the controller control on the mobile phone for the user to sing to enter the TV controller interface and perform the singing. Exemplarily shown in Figure 20 (a), when the mobile phone detects that the user clicks on the controller controls for the user to sing, it displays the GUI shown in Figure 20 (b), which contains the GUI that can be used for singing The microphone control 2001, the control 2002 used to start the original singing, the control 2003 used to pause, the control 2004 used to cut the song, the control 2005 used to sing again, the control used to select more 2006, and also include Control 2007 for sending emoji barrage. The user can open the microphone control 2001 by clicking or sliding. As shown in (c) in Figure 20, this is the GUI when the user is singing at this time. The microphone of the mobile phone can receive the user's singing voice and play the singing voice of the user through the TV speaker.

In another embodiment, when the mobile phone detects that the user clicks on the controller control for the user to sing, it enters the TV controller interface and displays the GUI as shown in Figure 21. At this time, the TV console interface displays both the song video screen and the Lyrics are convenient for users to use. The GUI contains a control 2101 for opening and closing the barrage. When the control 2101 is turned on, if the user clicks on the emoticon barrage, the emoticon barrage will appear on both the TV and mobile video screens. When the control 2102 is closed, If the user clicks on the emoji barrage, it will only appear on the TV screen and not on the mobile phone. Similarly, the GUI can also include controls 2102 for starting the original singing, controls 2103 for pausing, microphone controls 2104 for singing, controls 2105 for cutting songs, controls 2106 for re-singing, and selection There are more controls 2107, and it also includes controls 2108 for sending emoji barrage.

Exemplarily, as shown in Figure 22 (a), when the mobile phone detects that the user clicks on the control used to send emoticon barrage, the TV host displays the emoticon barrage, as shown in Figure 22 (b) Show. Exemplarily, the appearance mode of the emoji barrage may be sliding out from the top, bottom, left and right of the TV screen, or may be faded in and out, which is not limited here.

Exemplarily, as shown in Figure 23 (a), when the mobile phone detects that the user clicks to select more controls, it displays the GUI shown in Figure 23 (b), which may include a scan Code control 2301, control 2302 for sharing, control 2303 for editing barrage. Exemplarily, when the mobile phone detects that the user clicks on the operator of the control 2302 for sharing, the control shown in (c) in FIG. 23 is displayed, and the K song audio can be shared to other third-party applications.

The control 2303 for editing barrage can realize the opening and closing of emoticon barrage, and supports users to edit text and send text barrage.

Next, the UI operation interface and operation flow of the remote application of multi-machine K song are briefly explained. Exemplarily, a smart TV is used as the master and multiple smart phones are used as slaves as an example for introduction. It should be noted that the number of slaves can be multiple, and this embodiment takes two slaves as an example for description.

Figure 24 shows a remote application system for multi-machine K song, including a master smart TV, local microphone slave 1, remote microphone slave 2, and remote microphone slave 3. The three slaves are all smart phones. At this time, the host and the local slave 1 are in the same wireless LAN, and the remote slave 2 and the remote slave 3 are not in the same place with the host.

For the local slave device, the above-mentioned local application system can scan the QR code or other short-distance communication methods to join the K song. This embodiment focuses on the operation flow of adding the K song from the remote slave device. Exemplarily, as shown in Figure 25, at this time, the QR code scanning and song ordering interface contains a sharing control 2501 in addition to the QR code of the song, through which the K song networking information can be sent to the remote slave Machine, realize remote K song. Exemplarily, when the TV detects that the user controls the remote control to select the sharing control on the TV desktop, it displays the GUI as shown in FIG. 26, and at this time, the friend list 2601 of user 1 is displayed on the GUI. In order to explain the details of the buddy list 2601, the buddy list part is enlarged, and the display is shown in Fig. 27. As can be seen from (a) in Fig. 27, the buddy list contains a control 2701 for closing sharing. A control 2702 for searching friends, a multi-select control 2703 for selecting multiple friends, a K song friend list of user 1, and a control 2704 for viewing up and down. Exemplarily, when the TV detects that the user controls the remote control to select the multi-select control 2703, it displays the GUI as shown in (b) in Figure 27. At this time, the control 2705 for selection appears in front of the friend's profile picture, and the GUI also displays Contains a control 2706 for completing selection and a control 2707 for canceling multiple selections. Exemplarily, the user sends the K song invitation to user 2 and user 3 in different places.

Exemplarily, after user 1 sends the karaoke invitation, users 2 and 3 will receive the karaoke invitation message and display (a) as shown in Figure 28. When the mobile phone detects that the user clicks on the karaoke invitation message Then, as shown in (b) in Figure 28, the QR code used to scan the code to order songs on the host is sent to the remote slave. Exemplarily, as shown in FIG. 29, user 2 and user 3 can recognize the two-dimensional code to join K song by long pressing or other means. It should be noted that the K song invitation sent by the host to the slave by sharing can be a QR code or a link, which can be sent to remote users in the form of a message in the K song application, or it can be in the form of SMS.

For another example, the mobile phone can also be used as the host to initiate collaborative K song. Exemplarily, as shown in Figure 30 (a), when the mobile phone detects that the user clicks on the icon of the K song application on the desktop, the K song application can be started, and Figure 30 (b) shows The start interface of the K song application. After the K song application is successfully opened, the mobile phone detects that the user clicks on the song control, and displays the GUI shown in Figure 30 (c), which contains the controls for multi-person collaborative K song 3001. When the mobile phone detects that the user clicks on the multi-person K song control 3001, it displays the GUI as shown in Figure 31, and the mobile phone enters the karaoke table interface. The bottom of the GUI karaoke table interface contains the menu bar that contains the code scanning points for song ordering. Song control 3101. When the mobile phone detects that the user clicks on the control 3101, it displays a GUI as shown in (b) in Figure 31. The GUI contains a two-dimensional code for a song and a sharing control. For local users, you can add K songs by scanning the QR code. For remote users, the host may have used the sharing control to share the QR code with remote users.

The embodiment of the application provides a multi-terminal multimedia data communication method, which can use wireless connections (such as Wi-Fi, Bluetooth, mobile communication systems, etc.) to realize low-latency simultaneous multipoint transmission of multimedia data streams between multiple terminal devices , Can make multiple terminals work together to establish new applications to provide new experiences, and can also facilitate the interconnection and sharing of multimedia content between devices, so as to make full use of the advantages of each device. The multi-terminal multimedia data communication method provided by the embodiments of the present application can be used in application scenarios such as multi-machine playback, multi-machine K song, multi-machine conversation, and multi-machine recording. The above-mentioned multimedia applications based on multi-terminal cooperative play, such as multi-machine playback, multi-machine K song, multi-machine conversation, multi-machine recording, etc., include two ways of local application and remote application.

FIG. 32 is a schematic diagram of a multimedia data communication system of a terminal provided by an embodiment of the application. Referring to FIG. 32, the system includes: a master electronic device 11, a first slave electronic device 12, and a second slave electronic device 13, wherein:

The master electronic device is used to establish connections with the first slave electronic device and the second slave electronic device respectively;

The first slave electronic device is configured to receive a first play instruction, and the first electronic device is also configured to send the first play instruction to the master electronic device;

The master electronic device is also used to respond to the first play instruction to play the first multimedia data, and at the same time send at least a part of the first multimedia data to the first slave electronic device and the second slave electronic device, so that the master electronic device Playing the first multimedia data synchronously with the first slave electronic device and the second slave electronic device;

The first slave electronic device is also used to receive the first human voice, while the second slave electronic device is used to receive the second human voice;

The master electronic device is also used to receive the first human voice sent by the first slave electronic device and the second human voice sent by the second slave electronic device. The master electronic device is also used to transmit the first human voice, the second human voice, and The first multimedia data is mixed, and the second multimedia data is generated and played.

The present application also provides a storage medium, including: a readable storage medium and a computer program, the computer program is used to implement the multi-terminal multimedia data communication method provided by any of the foregoing embodiments.

All or part of the steps in the foregoing method embodiments can be implemented by a program instructing relevant hardware. The aforementioned program can be stored in a readable memory. When the program is executed, it executes the steps of the foregoing method embodiments; and the foregoing memory (storage medium) includes: read-only memory (English: read-only memory, ROM), RAM, flash memory, hard disk, and solid state hard disk , Magnetic tape, floppy disk, optical disc, and any combination thereof

The above content is only the specific implementation of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Covered in the scope of protection of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

A multi-terminal multimedia data communication method, characterized in that it is applied to multiple electronic devices, the multiple electronic devices including a master electronic device, a first slave electronic device, and a second slave electronic device, and the method includes:

Establishing connections between the master electronic device and the first slave electronic device and the second slave electronic device respectively;

The first slave electronic device receives a first play instruction, and the first slave electronic device sends the first play instruction to the master electronic device;

In response to the first play instruction, the master electronic device plays the first multimedia data, and at the same time sends at least a part of the first multimedia data to the first slave electronic device and the second slave electronic device Device, so that the master electronic device, the first slave electronic device, and the second slave electronic device synchronously play the first multimedia data;

The first slave electronic device receives a first human voice, while the second slave electronic device receives a second human voice;

The master electronic device receives the first human voice sent by the first slave electronic device and receives the second human voice sent by the second slave electronic device, and the master electronic device transmits the first human voice The human voice, the second human voice, and the first multimedia data are mixed to generate and play the second multimedia data.
The method according to claim 1, wherein the master electronic device, the first slave electronic device, and the second slave electronic device synchronously play the first multimedia data, specifically:

Determining, by the master electronic device, a first clock deviation between the master electronic device and the first slave electronic device;

Determining, by the master electronic device, a second clock deviation between the master electronic device and the second slave electronic device;

The master electronic device determines, according to the first clock deviation, a first initial playback time at which the first slave electronic device plays the first multimedia data; the first initial playback time is used to indicate the first 1. The starting time of playing the first multimedia data from the electronic device;

The master electronic device determines, according to the second clock deviation, a second initial playback time at which the second slave electronic device plays the first multimedia data; the second initial playback time is used to indicate all The start time of the second slave electronic device playing the first multimedia data.
The method according to claim 1 or 2, wherein the establishment of the connection between the master electronic device and the first slave electronic device and the second slave electronic device respectively comprises:

The master electronic device and the first slave electronic device are in the same wireless local area network, the master electronic device displays a WiFi connection identification, and the first slave electronic device recognizes the WiFi connection identification with the master electronic device establish connection.
The method according to claim 1 or 2, wherein the establishment of the connection between the master electronic device and the first slave electronic device and the second slave electronic device respectively comprises:

The master electronic device and the second slave electronic device are not in the same wireless local area network, the master electronic device sends networking information to the second slave electronic device, and the second slave electronic device analyzes the group The network information establishes a connection with the main electronic device.
The method according to claim 4, wherein the main electronic device mixes the first human voice, the second human voice, and the multimedia data to generate and play the second multimedia data. The method also includes:

Sending, by the master electronic device, the second multimedia data to the second slave electronic device;

The master electronic device and the second slave electronic device synchronously play the second multimedia data.
The method according to claim 1, wherein at least a part of the first multimedia data includes audio, video, or lyrics of the first multimedia data.
The method of claim 1, wherein the method further comprises:

The main electronic device receives the second play instruction input by the user;

Responding to the second play instruction, the master electronic device plays third multimedia data, and at the same time sends at least a part of the third multimedia data to the first slave electronic device and the second slave electronic device, So that the master electronic device, the first slave electronic device, and the second slave electronic device synchronously play the third multimedia data;

The first slave electronic device receives a first human voice, while the second slave electronic device receives a second human voice;

The master electronic device receives the first human voice sent by the first slave electronic device and receives the second human voice sent by the second slave electronic device, and the master electronic device transmits the first human voice The human voice, the second human voice, and the third multimedia data are mixed to generate and play fourth multimedia data.
The method according to claim 1 or 7, wherein the method further comprises:

The first slave electronic device receives a first human voice, while the second slave electronic device receives a second human voice, while the master electronic device receives a third human voice;

The master electronic device receives the first human voice sent by the first slave electronic device and receives the second human voice sent by the second slave electronic device, and the master electronic device sends the first human voice The second voice, the second voice, the third voice, and the fourth multimedia data are mixed to generate a fifth multimedia file and play it.
A multi-terminal multimedia data communication system, characterized in that the system includes a master electronic device, a first slave electronic device and a second slave electronic device, and the system includes:

The master electronic device is used to establish connections with the first slave electronic device and the second slave electronic device respectively;

The first slave electronic device is used to receive a first play instruction, and the first electronic device is also used to send the first play instruction to the master electronic device;

The master electronic device is further configured to play the first multimedia data in response to the first play instruction, and at the same time send at least a part of the first multimedia data to the first slave electronic device, the second Two slave electronic devices, so that the master electronic device, the first slave electronic device, and the second slave electronic device synchronously play the first multimedia data;

The first slave electronic device is also used to receive a first human voice, while the second slave electronic device is used to receive a second human voice;

The master electronic device is further configured to receive the first human voice sent by the first slave electronic device and receive the second human voice sent by the second slave electronic device, and the master electronic device also uses The first human voice, the second human voice, and the first multimedia data are mixed to generate second multimedia data and play.
The system according to claim 9, wherein the master electronic device, the first slave electronic device, and the second slave electronic device synchronously play the first multimedia data, specifically:

The master electronic device is used to determine a first clock deviation between the master electronic device and the first slave electronic device;

The master electronic device is also used to determine a second clock deviation between the master electronic device and the second slave electronic device;

The master electronic device is further configured to determine, according to the first clock deviation, a first initial playback moment at which the first slave electronic device plays the first multimedia data; the first initial playback moment is used to indicate The start time of the first slave electronic device playing the first multimedia data;

The master electronic device is further configured to determine, according to the second clock deviation, a second initial playback time at which the second slave electronic device plays the first multimedia data; the second initial playback time is used At represents the start time of the second slave electronic device playing the first multimedia data.
The system according to claim 9 or 10, wherein the master electronic device is configured to establish connections with the first slave electronic device and the second electronic device, respectively, comprising:

The master electronic device and the first slave electronic device are in the same wireless local area network, the master electronic device is used to display a WiFi connection identifier, and the first slave electronic device is used to identify the WiFi connection identifier and the The main electronic device establishes a connection.
The system according to claim 9 or 10, wherein the master electronic device is configured to establish connections with the first slave electronic device and the second electronic device, respectively, comprising:

The master electronic device and the second slave electronic device are not in the same wireless local area network, the master electronic device is used to send networking information to the second slave electronic device, and the second slave electronic device is used to pass Analyze the networking information to establish a connection with the main electronic device.
The system according to claim 12, wherein the main electronic device mixes the first human voice, the second human voice, and the multimedia data to generate and play the second multimedia data. The system also includes:

The master electronic device is also used to send the second multimedia data to the second slave electronic device;

The master electronic device and the second slave electronic device synchronously play the second multimedia data.
The system according to claim 9, wherein at least a part of the first multimedia data includes audio, video, or lyrics of the first multimedia data.
The system of claim 9, wherein the system further comprises:

The main electronic device is used to receive a second play instruction input by a user;

The master electronic device is also used to respond to the second play instruction, play third multimedia data, and at the same time send at least a part of the third multimedia data to the first slave electronic device and the second slave An electronic device, so that the master electronic device, the first slave electronic device, and the second slave electronic device synchronously play the third multimedia data;

The first slave electronic device is used to receive a first human voice, and the second slave electronic device is used to receive a second human voice;

The master electronic device is further configured to receive the first human voice sent by the first slave electronic device and receive the second human voice sent by the second slave electronic device, and the master electronic device also uses The first human voice, the second human voice and the third multimedia data are mixed to generate and play fourth multimedia data.
The system according to claim 9 or 15, wherein the system further comprises:

The first slave electronic device is also used to receive the first human voice, while the second slave electronic device is also used to receive the second human voice, and the master electronic device is also used to receive a third human voice;

The master electronic device is further configured to receive the first human voice sent by the first slave electronic device and receive the second human voice sent by the second slave electronic device, and the master electronic device is also configured to The first voice, the second voice, the third voice, and the fourth multimedia data are mixed to generate a fifth multimedia file and play it.
A computer-readable storage medium, wherein the computer-readable storage medium includes computer instructions, which when the computer instructions run on a computer, cause the computer to execute the multiplex according to any one of claims 1-8. The multimedia data communication method of the terminal.