WO2022089088A1 - Display device, mobile terminal, screen-casting data transmission method, and transmission system - Google Patents

Abstract

Disclosed in some embodiments of the present application are a display device, a mobile terminal, a screen-casting data transmission method, and a transmission system. In practical applications, a display device can receive a screen-casting connection request sent by a mobile terminal, and after the screen-casting connection request is obtained, the type of screen-casting applications is detected, so that when the type of screen-casting applications is a third-party application, the mobile terminal is connected by means of a Bluetooth module to receive an audio data stream in the mobile terminal. The display device can obtain an audio signal of the mobile terminal by means of Bluetooth connection when the third-party screen-casting application cannot acquire the audio data stream from the mobile terminal, and synchronously play the audio signal.

Description

Display device, mobile terminal, screen projection data transmission method and transmission system

This application requires a Chinese patent filed on October 27, 2020 with application number 202011163037.8; filed on December 21, 2020 with application number 202011517483.4; and filed on February 9, 2021 with application number 202110175799.8 priority to the application, the entire contents of which are incorporated herein by reference.

technical field

The present application relates to display device technology, and in particular, to a display device, a mobile terminal, a screen projection data transmission method, and a transmission system.

Background technique

Display devices such as smart TVs can be connected to mobile terminals such as mobile phones to display the content in the mobile terminal by using a screen projection method. For example, DLNA (full name in English) pushes the multimedia on the mobile phone to the smart TV in the form of URL for playback, and does not support mirror projection; AirPlay screen mirroring is suitable for wireless projection of iOS and Mac systems; Miracast multi-screen interaction It is suitable for wireless screen projection of Android system; WiDi screen is suitable for wireless screen projection of Windows system. All three screencasting methods need to ensure that the screencasting device is of the same manufacturer or has the same specific version of the operating system to complete the screencasting. In order to realize screen projection between mobile phones and smart TVs of different manufacturers or different operating systems, APP screen projection software is usually used. Among them, the APP screen mirroring software is to install the screen mirroring software client on the mobile phone, and push the screen image to the smart TV for playback. However, the screencasting software client usually does not have access to audio data, so the screencasting process completed by the APP screencasting software can only transmit video data, but not audio data, which reduces the user experience.

SUMMARY OF THE INVENTION

Some embodiments of the present application provide a display device including a display, a communicator, a Bluetooth module, and a controller. Wherein, the display is used to display a user interface and a screen-cast video image; the communicator is configured to establish a screen-cast connection with the mobile terminal; and the Bluetooth module is configured to transmit an audio data stream. The controller is configured to perform the following program steps: receiving a screen-casting connection request sent by the mobile terminal; in response to the screen-casting connection request, detecting a screen-casting application type; if the screen-casting application type is a third-party application, through the screen-casting application type The bluetooth module is connected to the mobile terminal to obtain the audio data stream.

Some embodiments of the present application further provide a mobile terminal, including: a display unit, a communication unit, a Bluetooth circuit, and a processor. The display unit is configured to display a user interface to perform user interaction; the communication unit is configured to establish a screen projection connection with the display device; and the Bluetooth circuit is configured to transmit audio data streams. The processor is configured to execute the following program steps: obtaining a control instruction input by a user for establishing a screen-casting connection; sending a screen-casting connection request to the display device through the communication unit in response to the control instruction; detecting screen-casting Application type; if the screen projection application type is a third-party application, connect the display device through a Bluetooth circuit to send an audio data stream.

Some embodiments of the present application further provide a screen projection data transmission method, which is applied to a screen projection data transmission system, where the screen projection data transmission system includes a mobile terminal and a display device, and the method includes: the mobile terminal obtains a user input for A control instruction for establishing a screencasting connection, and sending a screencasting connection request to the display device according to the control instruction; the display device receives the screencasting connection request; and establishing a screencasting connection with the mobile terminal according to the screencasting connection request ; The mobile terminal and/or the display device detect the screencasting application type; if the screencasting application type is a third-party application, establish a Bluetooth connection between the mobile terminal and the display device to transmit audio data streams.

Some embodiments of the present application provide a screen projection data transmission system, including a mobile terminal and a display device; wherein the mobile terminal has a built-in communication unit and a Bluetooth circuit; the display device has a built-in communicator and a Bluetooth module; the mobile terminal is is configured to: obtain a control instruction input by the user for establishing a screen-casting connection, and send a screen-casting connection request to a display device according to the control instruction; the display device is configured to: receive the screen-casting connection request; and according to the control instruction The screen-casting connection request establishes a screen-casting connection with the mobile terminal; the mobile terminal and/or the display device are further configured to: detect a screen-casting application type; if the screen-casting application type is a third-party application, A Bluetooth connection is established between the mobile terminal and the display device to transfer audio data streams.

Some embodiments of the present application provide a display device, including: a receiving module for receiving a transmission instruction; a driving module for determining a transmission mode in which the display device is located according to the transmission instruction; wherein the transmission mode is network transmission The main mode or the screen-casting transmission-based mode; the network transmission and the screen-casting transmission both perform data transmission through the same wireless communication module in the display device; The transmission mode determines the transmission duration adjustment instruction and sends the transmission duration adjustment instruction to the wireless communication firmware; the display screen is used to display the content of the network transmission and/or the screen projection transmission. In some embodiments, the embodiments of the present application provide a display device, comprising:

Some embodiments of the present application provide a display device, including a display screen for displaying images; a controller connected to the display screen, the controller being configured to: receive a device search message broadcast by a mobile terminal; The mobile terminal sends a feedback message generated based on the device search message, where the feedback message includes the screencasting matching code and device address of the display device; receives the screencasting connection request sent by the mobile terminal, and sends the screencasting connection request according to the A screen connection request establishes a screen-casting connection with the mobile terminal. When the screen-casting connection request is that the screen-casting matching code of the mobile terminal is the same as the screen-casting matching code of the display device, the mobile terminal will perform screencasting according to the display The device address of the device is generated.

Some embodiments of the present application provide a mobile terminal, including: a near field communication (Near Field Communication, NFC for short) response component for responding to an NFC card that meets a preset distance; a controller connected to the NFC response component , the controller is configured to: when the NFC response component responds to the NFC card, broadcast a device search message in the local area network currently accessed by the mobile terminal; receive a feedback message sent by the display device based on the device search message , the feedback message includes the screencasting matching code and device address of the display device; obtain the screencasting matching code of the mobile terminal, when the screencasting matching code of the mobile terminal matches the screencasting of the display device When the codes are the same, a screen-casting connection is established with the display device according to the device address.

Description of drawings

1 is a schematic diagram of a usage scenario of a display device according to one or more embodiments of the present application;

FIG. 2 is a block diagram of a hardware configuration of a display device 200 according to one or more embodiments of the present application;

FIG. 3 is a schematic diagram of software configuration in a display device 200 according to one or more embodiments of the present application;

FIG. 4 is a schematic diagram of displaying an icon control interface of an application in a display device 200 according to one or more embodiments of the present application;

FIG. 5 is a schematic structural diagram of a mobile terminal 300 according to one or more embodiments of the present application;

6 is a block diagram of a software structure of a mobile terminal 300 according to one or more embodiments of the present application;

7 is a schematic diagram of a user interface of a mobile terminal according to one or more embodiments of the present application;

8a-8c are schematic diagrams of screen projection interaction according to one or more embodiments of the present application;

9a-9b are schematic diagrams of audio transmission according to one or more embodiments of the present application;

10 is a schematic flowchart of judging an application type according to one or more embodiments of the present application;

11a-11b are schematic diagrams of Bluetooth connections according to one or more embodiments of the present application;

12a-12b are schematic diagrams of screen projection connections according to one or more embodiments of the present application;

FIG. 13 is a schematic diagram of a synchronization flowchart of a display device according to one or more embodiments of the present application;

14 is a schematic diagram of a screen projection connection structure of a display device according to one or more embodiments of the present application;

FIG. 15 is a schematic diagram of a screen projection connection structure of a mobile terminal according to one or more embodiments of the present application;

16a-16b are schematic diagrams of a first scenario of establishing a screen projection connection according to one or more embodiments of the present application;

17 is a schematic diagram of a display interface of a mobile terminal when playing a video according to one or more embodiments of the present application;

FIG. 18 is a second scenario diagram of establishing a screen projection connection according to one or more embodiments of the present application;

FIG. 19 is a third scenario diagram of establishing a screen projection connection according to one or more embodiments of the present application;

20 is a schematic diagram of a scenario of setting a screen projection matching code of a display device according to one or more embodiments of the present application;

21 is a schematic diagram of a scenario of setting a screencasting matching code of a mobile terminal according to one or more embodiments of the present application;

22 is a schematic flowchart of a method for establishing a screen projection connection according to one or more embodiments of the present application;

23 is a schematic diagram of a frame sequence of a video stream according to one or more embodiments of the present application;

FIG. 24 is a schematic flowchart of a method for preventing a blurry screen from appearing in a screen projection process provided according to one or more embodiments of the present application;

25 is a possible P2P device discovery process according to one or more embodiments of the present application;

26 is a process for adjusting a duration in a network transmission-based mode according to one or more embodiments of the present application;

FIG. 27 is a process for adjusting a duration in a mode based on screen projection transmission according to one or more embodiments of the present application;

FIG. 28 is a possible P2P device discovery process according to one or more embodiments of the present application.

Detailed ways

In order to make the objectives, implementations and advantages of the present application clearer, the exemplary embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the exemplary embodiments of the present application. Obviously, the exemplary embodiments described It is only a part of the embodiments of the present application, but not all of the embodiments.

Based on the exemplary embodiments described in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the appended claims of this application. Furthermore, although the disclosures in this application have been presented in terms of illustrative example or instances, it should be understood that various aspects of this disclosure may also constitute a complete embodiment in isolation. It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the embodiments described below, rather than intended to limit the embodiments of the present application. Unless otherwise specified, these terms are to be understood according to their ordinary and ordinary meanings.

FIG. 1 is a schematic diagram of a usage scenario of a display device according to one or more embodiments of the present application. As shown in FIG. 1 , the display device 200 also performs data communication with the server 400, and the user can operate the display device 200 through the mobile terminal 300 or the control device 100. In some embodiments, the control device 100 may be a remote control, and the communication between the remote control and the display device includes at least one of infrared protocol communication or Bluetooth protocol communication, and other short-range communication methods, and the display is controlled wirelessly or wiredly. device 200. In some embodiments, the mobile terminal 300 may include any one of a mobile phone, a tablet computer, a computer, a notebook computer, an AR/VR device, and the like. Communication of data may also be performed using the mobile terminal 300 and the display device. In some embodiments, the display device 200 may also be controlled in a manner other than the control apparatus 100 and the mobile terminal 300 ; the display device 200 also performs data communication with the server 400 . The display device 200 may be allowed to communicate via local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200 . The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

FIG. 2 is a block diagram of a hardware configuration of a display device 200 according to one or more embodiments of the present application. As shown in FIG. 2, the display device 200 includes a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. at least one. The controller includes a central processing unit, a video processing unit, an audio processing unit, a graphics processing unit, a RAM, a ROM, and a first interface to an nth interface for input/output. The display 260 may be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and may also be a projection device and a projection screen. The tuner-demodulator 210 receives broadcast television signals through wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals. The detector 230 is used to collect external environment or external interaction signals. The controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

In some embodiments, the controller 250, through various software control programs stored on the memory, controls the operation of the display device and responds to user operations. The controller 250 controls the overall operation of the display apparatus 200 . A user may input a user command on a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

In some embodiments, a "user interface" is a medium interface for interaction and information exchange between an application program or an operating system and a user, which enables conversion between an internal form of information and a form acceptable to the user. The commonly used form of user interface is Graphical User Interface (GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It can be an icon, window, control and other interface elements displayed on the display screen of the electronic device, wherein the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc. at least one of the visual interface elements.

FIG. 3 is a schematic diagram of software configuration in the display device 200 according to one or more embodiments of the present application. As shown in FIG. 3 , the system is divided into four layers. Layer"), the Application Framework layer (referred to as the "framework layer"), the Android runtime (Android runtime) and the system library layer (referred to as the "system runtime layer"), and the kernel layer. The kernel layer contains at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power supply drive etc.

FIG. 4 is a schematic diagram showing an icon control interface of an application program in a display device 200 according to one or more embodiments of the present application. As shown in FIG. 4 , the application program layer includes at least one application program that can display a corresponding icon control on the display. , such as: live TV application icon control, video on demand application icon control, media center application icon control, application center icon control, game application icon control, etc. Live TV app that can provide live TV from different sources. Video-on-demand application that can provide video from different storage sources. Unlike live TV applications, video-on-demand provides a display of video from certain storage sources. A media center application that can provide a variety of multimedia content playback applications. The application center can provide storage of various applications. .

FIG. 5 is a schematic structural diagram of a mobile terminal 300 according to one or more embodiments of the present application. The embodiment will be described in detail below by taking the mobile terminal 300 as an example. It should be understood that the mobile terminal 300 shown in FIG. 5 is only an example, and the mobile terminal 300 may have more or less components than those shown in FIG. 5, two or more components may be combined, or Different component configurations are possible. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits. As shown in FIG. 5 , the mobile terminal 300 includes: a radio frequency (RF) circuit 310, a memory 320, a display unit 330, a camera 340, a sensor 350, an audio circuit 360, and a Wireless Fidelity (Wi-Fi) circuit 370, the processor 380, the Bluetooth circuit 381, and the power supply 390 and other components.

The RF circuit 310 can be used for receiving and sending signals during sending and receiving information or during a call, can receive downlink data from the base station and then hand it over to the processor 380 for processing; and can send uplink data to the base station. Memory 320 may be used to store software programs and data. The processor 380 executes various functions and data processing of the mobile terminal 300 by running software programs or data stored in the memory 320 . The display unit 330 may be configured to receive input numerical or character information, and generate signal input related to user settings and function control of the mobile terminal 300 . The display unit 330 may also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the mobile terminal 300. Camera 340 may be used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 380 for conversion into a digital image signal. The mobile terminal 300 may also include at least one sensor 350 . The audio circuit 360 , the speaker 361 , and the microphone 362 may provide an audio interface between the user and the mobile terminal 300 . The processor 380 is the control center of the mobile terminal 300, uses various interfaces and lines to connect various parts of the entire terminal, and executes the mobile terminal by running or executing the software programs stored in the memory 320 and calling the data stored in the memory 320. 300 various functions and processing data. In some embodiments, the processor 380 may include one or more processing units; the Bluetooth circuit 381 is used to exchange information with other Bluetooth devices having Bluetooth circuits through the Bluetooth protocol. The mobile terminal 300 also includes a power source 390 (eg, a battery) for powering the various components. The power supply can be logically connected to the processor 380 through a power management system, so that functions such as managing charging, discharging, and power consumption are implemented through the power management system. The mobile terminal 300 may also be configured with a power button, which is used to power on and off the terminal, and lock the screen and other functions.

FIG. 6 is a block diagram of a software structure of a mobile terminal 300 according to one or more embodiments of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, which are, from top to bottom, an application layer, an application framework layer, an Android runtime (Android runtime) and a system library, and a kernel layer. The application layer can include a series of application packages.

The workflow of the software and hardware of the mobile terminal 300 is exemplarily described below in conjunction with the capturing and photographing scene. When the touch screen 331 receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into raw input events (including touch coordinates, timestamps of touch operations, etc.). Raw input events are stored at the kernel layer. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon, for example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer. Camera 340 captures still images or video. In some specific implementations, FIG. 7 is a schematic diagram of a user interface of a mobile terminal according to one or more embodiments of the present application. As shown in FIG. 7 , a user can open a corresponding application by touching an application icon on the user interface, or by touching an application icon on the user interface. Touch a folder icon on the user interface to open the corresponding folder.

Based on the above-mentioned display device 200 and the mobile terminal 300 , the display device 200 can play and display the content in the mobile terminal 300 by establishing a screen projection connection. 8a-8c are schematic diagrams of screen projection interaction according to one or more embodiments of the present application, as shown in FIG. 8a. In some embodiments, as shown in FIG. 8b , in order to establish a screen-casting connection relationship, the user may send a screen-casting connection request through the mobile terminal 300 , and send the screen-casting connection request to the display device 200 through a WiFi network. The display device 200 then completes the configuration of the transmission protocol according to the screen projection connection request, thereby establishing a transmission channel for the screen projection data with the mobile terminal 300 .

The display device 200 can also realize the screen projection connection with the mobile terminal 300 through different connection manners. For example, when the display device 200 and the mobile terminal 300 access the same wireless local area network, a screen projection connection may be established based on a WiFi network. For another example, when both the display device 200 and the mobile terminal 300 are provided with an NFC (Near Field Communication, near field communication) component, a screen projection connection relationship can be established through the NFC component. Obviously, other wired or wireless connection methods may also be used between the display device 200 and the mobile terminal 300 to establish a screen projection connection relationship, such as RF radio frequency connection, infrared connection, cellular network, and connections provided by those skilled in the art in the above embodiments Other connection methods that can be thought of on the basis of the method.

After the screen projection connection is established, the display device 200 can receive the relevant data of the mobile terminal 300 through the screen projection data channel, so as to play the relevant content in the display device 200 . Under different screen projection data transmission protocols, the screen projection data acquired by the display device 200 is in different forms. For example, under the DLNA (Digital Living Network Alliance) protocol, the display device 200 can obtain the link address of a resource in the mobile terminal 300 through the screen projection data channel, and obtain the media asset content data by accessing the link address. , so as to play the media asset data. At the same time, the display device 200 may also acquire control instructions sent by the mobile terminal 300 through the screen projection data channel, such as fast-forward, pause, and stop projection, when playing the media asset data. For example, after the connection is established through the communication protocol, the mobile terminal 300 such as the mobile phone and the display device 200 will negotiate the device function and network conditions to select a suitable video and audio transmission format. After establishing the connection protocol (Session) of the audio and video stream, a series of real-time streaming protocol (RTSP) control commands are used to control the playback and termination of the audio and video stream.

Under other screencasting connection protocols such as Miracast screencasting, Airplay screencasting, NFC mirror screencasting, and LeBao screencasting, the display device 200 can directly obtain the video data stream from the mobile terminal 300 . The video data stream may be the video data stream corresponding to the final display screen of the display unit 310 of the mobile terminal 300 , that is, the video data stream includes the media content screen, UI interface and other contents of the mobile terminal 300 that can be displayed in the display unit 310 content of the video screen. The video data stream may also be only video data that can be acquired in the mobile terminal 300, such as the video data of the media asset being played. At this time, the mobile terminal 300 can be used as a signal source of the display device 200, and the display device 200 can continuously obtain video data through the screen projection data channel and display it on the display 275 in real time to achieve the effect of synchronous display with the mobile terminal 300.

The mobile terminal 300 can acquire different screen projection data according to different screen projection protocols. For example, for system applications or direct screen projection at the system level, the mobile terminal 300 may use Miracast screen projection, Airplay screen projection or WiDi screen projection protocol to establish a screen projection data channel.

Among them, Miracast screen projection is a wireless display standard based on Wi-Fi direct connection developed by the Wi-Fi Alliance. Miracast screencasting can be applied to Android devices, that is, Android devices can share video images through Miracast screencasting. Generally, smart devices with Android 5.0 and above have built-in Miracast protocol, so you can directly perform wireless screen projection through the WiFiP2P function between the screen-casting device and the WiFi network card of the receiving device without installing any software. In addition, Miracast screencasting also supports simultaneous transmission of audio and video. However, because there are too many Android devices, Miracast cannot be fully compatible with all Android devices, and Miracast-based screencasting does not have a screencasting code function, which cannot prevent malicious or mistaken screencasting.

Airplay screen projection is a wireless playback technology launched by Apple, which is suitable for wireless screen projection of iOS and Mac systems. Apple series devices have built-in Airplay screen projection function. When used in conjunction with screen projection devices that support AirPlay server services, the effect is clear and stable. The protocol has its own screen projection code function. However, it is necessary to ensure that the Apple device and the screen caster are used in the same local area network (same network segment), and cannot be used across network segments or VLANs.

WiDi screen projection is a wireless playback technology dominated by Intel and is also based on WiFi Direct technology. Windows 8/10 laptops natively support WiDi screen projection, which can be directly projected without installing any software, and supports USB reverse control function.

The above three screen projection protocols can not only acquire the video data stream in the mobile terminal 300, but also acquire the audio data stream in the mobile terminal 300, so that the video data and audio data can be synchronously sent to the display device 200, so that the display device 200 can Simultaneous playback of video and audio. Since the above-mentioned Miracast screencasting protocol and Airplay screencasting protocol require system-level application permissions, the Miracast screencasting protocol and Airplay screencasting protocol are usually only applicable to system applications, or the display device 200 and the mobile terminal 300 of the same manufacturer pass the built-in rules of the operating system Directly establish a screencasting connection.

Because the sound of the mobile terminal 300 can only be obtained by system-level applications, or some related system-level modifications are required within the same manufacturer of the sender (Source) and the receiver (Sink) to transmit the sound, such as NFC mirroring At the same time, the mobile terminal 300 can transmit the video and audio streams to the display device 200 at the same time. However, if the mobile terminal 300 uses an application not developed by the current manufacturer, it can only transmit the video stream to the display device 200 for playback, and cannot transmit audio data. That is, some third-party applications do not have system-level permissions and cannot use the Miracast or Airplay projection protocol, and can only use other projection protocols to establish a projection connection. For example, the NFC mirroring application adopts the NFC screen mirroring protocol and so on. Since these screencasting protocols do not have system-level application permissions, they cannot directly obtain the audio data of the mobile terminal 300 after the screencasting connection is established. Therefore, when a third-party application is used for screencasting, only screencasting video will appear. The problem of not being able to pass audio.

In order to transmit audio data, as shown in FIG. 8c , in some embodiments of the present application, a screen projection data transmission system is provided, and the system includes a display device 200 and a mobile terminal 300 . The display device 200 includes a display 275 , a communicator 220 , a Bluetooth module and a controller 250 ; the mobile terminal 300 includes a display unit 330 , a communication unit, a Bluetooth circuit 381 and a processor 380 .

In practical applications, the user may perform interactive operations through the mobile terminal 300 to input control instructions for various purposes. For example, input control commands can be used to establish a screencasting connection. The mobile terminal 300 can input the control instruction for establishing the screen projection connection in various ways according to different interaction principles. For example, the display unit 330 of the mobile terminal 300 includes a touch control component, and the user interacts by inputting touch commands through the touch control component to enter the screen casting application, and after clicking the screen casting button, a screen casting connection is established.

The screen projection application refers to an application program installed in the mobile terminal 300, which may be a system application or a third-party application. Usually, the system application is an application program that comes with the mobile terminal 300 from the factory, and cannot be uninstalled in a normal operation mode. It should be noted that, in this embodiment, the system application has a broad sense, including not only application programs whose developers are system operators, but also other applications having system-level data acquisition rights.

The third-party application refers to an application that can be installed in the mobile terminal 300 and run in the mobile terminal 300 to realize some specific functions. For example, media asset playback applications, game applications, and the like. Similarly, the third-party applications also have a generality, including not only applications developed by non-system operators, but also applications developed by system operators but without system-level data acquisition rights.

In addition, the user can also input control instructions for establishing a screen-casting connection in other ways. For example, for the mobile terminal 300 with a built-in intelligent voice system, the user can input control commands for establishing a screencasting connection by inputting control voices, such as "start screencasting" and "start music broadcast screencasting". After the user inputs the above control voice, the mobile terminal 300 can automatically start the screen projection application and establish a screen projection connection relationship with the display device 200 to display the content in the mobile terminal 300 on the display device 200 .

It should be noted that, in the screen projection data transmission system provided in this embodiment, the screen projection connection relationship can be established in both directions, that is, the user can input control commands through the mobile terminal 300, so that the mobile terminal 300 can actively initiate screen projection to the display device 200. connect. The user may also input a control instruction through the display device 200 , and the specific control instruction input manner is similar to that of the mobile terminal 300 , which will not be repeated here. After the control instruction is input, the display device 200 may also initiate a screen projection connection to the mobile terminal 300 , so that the content in the display device 200 is displayed on the mobile terminal 300 by screen projection.

After acquiring the control instruction for establishing the screen-casting connection, the mobile terminal 300 may, in response to the control instruction, send a screen-casting connection request to the display device through the communication unit to establish the screen-casting connection. The screen projection connection request can be read by the display device 200 , and a response action is made according to the specific content in the screen projection connection request, so as to establish a screen projection data transmission channel between the mobile terminal 300 and the display device 200 .

To this end, the screencasting connection request needs to include information related to establishing a screencasting data transmission channel. For example, the connection form of the transmission channel, the protocol, the address of the mobile terminal 300, and the like. After receiving the screen-casting connection request, the display device 200 parses information such as the connection form, protocol, and address of the mobile terminal 300 from the screen-casting connection request, and negotiates the connection information in a manner corresponding to the transmitted protocol, so as to complete the negotiation after the negotiation is completed. Then establish a data transmission channel.

After the data transmission channel is established, the screen projection application can acquire the video data stream of the mobile terminal 300 in real time, and send the video data stream to the display device 200 through the screen projection data transmission channel. After receiving the video data stream, the display device 200 can Real-time display is performed on its display 275 to realize screen projection display of video content. Obviously, since there is a certain difference between the shape of the display area of the display 275 of the display device 200 and the display area of the display unit of the mobile terminal 300, when the display device 200 displays the projected video image, it is also necessary to check the display state. make adjustments. For example, the scale of the video picture is scaled, so that the projected video picture can be completely displayed on the display 275 . In addition, the display device 200 may also perform further optimization processing on the screen projection video data, for example, perform an interpolation algorithm on the screen projection video data, etc., to obtain a better display effect.

Since the screencasting application does not have access to audio data when transferring screencasting data through some third-party screencasting applications, only video signals can be transmitted through the established screencasting data transmission channel. After the connection, the mobile terminal 300 can also detect the type of the screen projection application, and determine whether the screen projection application currently used for the screen projection operation is a system application or a third-party application.

According to the detection result of the screen-casting application type, different screen-casting data transmission methods can be determined. If the screencasting application type is a system application, that is, the current screencasting application has the right to obtain the audio data stream, it can directly send the audio data stream to the display device 200 through the screencasting data transmission channel, so as to realize the synchronous playback of video images in the display device 200 and audio signal. If the screencasting application type is a third-party application, that is, the current screencasting application does not have the permission to obtain the audio data stream, and only video data, but not audio data, can be transmitted through the screencasting data transmission channel. Therefore, the display device 200 can be connected through the Bluetooth circuit to transmit the audio data stream.

It can be seen that through the Bluetooth connection between the display device 200 and the mobile terminal 300, the display device 200 is used as the receiving end (sink end) of the Bluetooth connection, and the mobile terminal 300 is used as the sending end (source end) of the Bluetooth connection, that is, the display device 200 As the audio playback device of the mobile terminal 300, the sound in the mobile terminal 300 is output in real time, and the synchronous playback of video data and audio data is realized.

In some embodiments, the display device 200 may also detect the screen-casting application type when establishing the screen-casting connection, that is, the display device 200 may respond to the screen-casting connection request after receiving the screen-casting connection request sent by the mobile terminal 300 . to detect the screencasting application type. If the type of the screen projection application is a third-party application, the mobile terminal 300 is connected through a Bluetooth module to obtain an audio data stream. That is, in this embodiment, the screen projection application type can be detected by the display device 200, so that no additional configuration is required for the mobile terminal 300, so that the above-mentioned screen projection data transmission is implemented between the mobile terminals 300 of different manufacturers and the display device 200. Way.

According to the above-mentioned embodiment, when a screen-casting connection is established, the sending method of relevant data can be determined according to the type of screen-casting application, and for third-party applications, a Bluetooth connection can be used to transmit audio data, so as to alleviate the problem that third-party applications can only transmit video data and cannot transmit them. Problems with audio data. For system applications, since they have access rights to audio data streams, they can directly transmit audio data through the established screen-casting connection relationship. Figures 9a-9b are schematic diagrams of audio transmission according to one or more embodiments of the present application. As shown in Figures 9a and 9b, in some embodiments, if the screen projection application type is a system application, it can communicate with The mobile terminal 300 establishes a screen projection connection to transmit the video data stream and the audio data stream.

For example, when an application needs to obtain the audio stream recording permission of the Android system, it can first obtain the sound acquisition permission, and this permission belongs to the system permission and does not allow third-party apps to use it. If it belongs to the system application, it can obtain the audio stream through the interface. The system authority is increased at the level, and the signature verification is performed at the same time, so that the system application can use the interface to obtain the audio stream. For system applications, the original data transmission method can still be used to directly transmit the video data stream and audio data stream. This data transmission method can reduce the amount of data processing, and can alleviate the occasional transmission delay when Bluetooth transmits audio data, which is conducive to maintaining audio Drawing synchronization.

In order to detect the type of screencasting application, in some embodiments of the present application, FIG. 10 is a schematic flowchart of judging the application type according to one or more embodiments of the present application. As shown in FIG. 10 , the display device 200 or the mobile terminal 300 may First obtain the package name information of the screencasting application, and then extract the judgment attribute value from the package name information. Thus, the type of the current screencasting application is determined according to the judgment attribute value. If the judgment attribute value is equal to 0, the screencasting application is marked as a third-party application; if the judgment attribute value is not equal to 0, the screencasting application is marked as a system application.

After the screen-casting connection is established through the screen-casting application, if the screen-casting application is a third-party application, a Bluetooth connection needs to be established between the display device 200 and the mobile terminal 300 . 11a-11b are schematic diagrams of Bluetooth connection according to one or more embodiments of the present application. As shown in FIGS. 11a and 11b, in some embodiments, the step of connecting the display device 200 to the mobile terminal 300 through a Bluetooth module further includes: Sending the address information of the display device 200 to the mobile terminal 300 by applying the screen projection protocol, so that the mobile terminal 300 performs Bluetooth pairing according to the address information; then receiving the pairing request sent by the mobile terminal 300 through the Bluetooth transmission protocol; and in response to the pairing request, The Bluetooth module is set to receive mode, and a Bluetooth connection is established with the mobile terminal 200 .

In order to establish a Bluetooth connection between the display device 200 and the mobile terminal 300, after detecting that the application type of the screencasting application is a third-party application, the display device 200 can send the current device address information to the mobile terminal 300, such as the display device 200 The Bluetooth Mac address is: 94:17:00:xx:64:8c, so that the mobile terminal 300 can perform Bluetooth pairing through the received address information, automatically with the Bluetooth address of 94:17:00:xx:64:8c The device establishes a Bluetooth connection. Since there is no Bluetooth connection relationship established between the display device 200 and the mobile terminal 300 at this time, the address information of the display device 200 may be sent to the mobile terminal 300 through a protocol corresponding to the screen projection application.

In some embodiments, the address information sent by the display device 200 may further include a control instruction, such as a control instruction used to control the mobile terminal to perform trust verification, to control the mobile terminal 300 so that after the mobile terminal 300 receives the address information , a pairing request is sent through the Bluetooth transmission protocol, and the display device 200 can set the Bluetooth module to receive mode in response to the pairing request, and establish a Bluetooth connection with the mobile terminal 300 .

After receiving the address information, the mobile terminal 300 can also perform local management on the address information for direct calling when establishing a Bluetooth connection subsequently. For example, the mobile terminal 300 matches the scan list with the Bluetooth Mac address received from the sink (ie, matches with the display device 200 ), and if it is negotiated, Bluetooth pairing and connection can be performed. When mirroring the screen, the video data can be transmitted to the display device 200 through the screen projection connection, and the audio data can be transmitted to the display device 200 through the Bluetooth connection to realize synchronous transmission of video and audio.

In order to facilitate the establishment of a Bluetooth connection relationship between the mobile terminal 300 and the display device 200, some embodiments of the present application also provide a Bluetooth pairing method, which can automatically establish a Bluetooth connection without the user's perception, which ensures that both Security, and no need for users to manually click to confirm, reducing the number of user interactions.

In the Bluetooth pairing connection method, a server can be set in the display device 200 of the receiver (sink) first, which is responsible for monitoring the Bluetooth information of the receiver and synchronously receiving the Bluetooth request information of the mobile terminal 300 of the transmitter (source). Since network communication (such as a screen-casting connection) has been established between the display device 200 and the mobile terminal 300, the mobile terminal 300 can transmit relevant information to the display device 200 through the protocol layer, and the display device 200 can transmit the device-related information synchronously to the mobile terminal 300 .

In the process of transmitting relevant information through the protocol layer, a device mutual trust flag can be established between the display device 200 and the mobile terminal 300 to indicate that the display device 200 and the mobile terminal 300 are safe, and can be used to implement non-perceptual Bluetooth pairing connect. That is, after the mutual trust mark is recorded respectively, if the mobile terminal 300 and the display device 200 establish a Bluetooth connection, trust verification can be performed by detecting the mutual trust mark, so that after passing the trust verification, Bluetooth pairing and direct connection can be performed directly.

For example, after the screen projection connection is established between the mobile terminal 300 and the display device 200, the relevant information of the trusted device may be recorded in the display device 200 and the mobile terminal 300 respectively, that is, the current display device 200 may be recorded in the mobile terminal 300 as a trusted device. , record in the display device 200 that the current mobile terminal 300 is a trusted device. When the mobile terminal 300 initiates a Bluetooth connection request to the display device 200, the display device 200 may determine the trusted device after receiving the Bluetooth connection request. If the corresponding information of the mobile terminal 300 that initiates the Bluetooth connection request is the same as the recorded trusted device information, the Bluetooth pairing connection can be directly performed.

Similarly, when the display device 200 initiates a Bluetooth connection request to the mobile terminal 300, the mobile terminal 300 may also perform trusted device verification on the display device 200 after receiving the Bluetooth connection request. When the device information corresponding to the display device 200 is the same as one of the recorded trusted device information, it is determined that the current display device 200 has passed the trust verification, so the Bluetooth connection relationship can be established directly without the need for the user to manually confirm the pairing code and other information.

It can be seen that in this embodiment, by recording the trusted device information, when the display device 200 and the mobile terminal 300 establish a Bluetooth connection, the pairing connection can be directly performed at the system level, without the need for manual operation by the user, and on the premise of ensuring security, reducing The number of user interactions enables rapid transfer of video data and audio data between the display device 200 and the mobile terminal 300, which facilitates the synchronous transfer of audio and video.

In some embodiments, in order to be able to record the trusted device information in the display device 200 and the mobile terminal 300 , a list of trusted devices can be created in the display device 200 and the mobile terminal 300 respectively, and the trust device list can be established in the display device 200 and the mobile terminal 300 respectively. After the screen is connected, the device information of the display device 200 and the mobile terminal 300 are both saved in the trusted device list.

For example, for the display device 200, a trusted device list can be established in it first, and the trusted list can include table items such as the device name, the device network address, and the device's Bluetooth mac address, as shown in the following table:

device name	website address	bluetooth mac address
xx mobile phone 1	Fe80::6917:ff:xxxx:8984	94:17:00:xx:64:01
xx mobile phone 2	Fe80::6917:ff:xxxx:8985	94:17:00:xx:64:8c
...	...	...

When the display device 200 establishes a network communication connection such as a screen projection connection with the mobile terminal 300, the display device 200 can extract the device name, the device network address and the Bluetooth mac of the device related to the mobile terminal 300 from the device information when the screen projection connection is established. address and other information, and add it to the trusted list to complete the recording of trusted device information. Since the security of the device has been verified when the network communication connection is established, that is, the display device 200 and the mobile terminal 300 that can establish the network communication connection have already met the security requirements, so other forms of connection may not be used when connecting. Then perform security verification and pair directly.

In some embodiments, in order to realize the trust verification between the display device 200 and the mobile terminal 300, after receiving the Bluetooth connection request, the display device 200 and the mobile terminal 300 may, according to the device information included in the Bluetooth connection request, The trusted device list is matched to determine whether the device that currently initiates the Bluetooth connection request is a trusted device.

For example, when the display device 200 receives the Bluetooth connection request sent by the mobile terminal 300, the display device 200 can extract the device information of the mobile terminal 300 in the Bluetooth connection request, including the device name, the device network address, and the device's Bluetooth mac address. The extracted device information is then matched in the trusted device list, and when all or part of the information such as the device name, the device network address, and the device's Bluetooth mac address are recorded in the trusted device list, it is determined that the current mobile terminal 300 device information passes through. Trust verification, so the Bluetooth connection request can be directly responded to, and the Bluetooth connection pairing can be automatically completed without the need for the user to manually complete the pairing operation.

Similarly, for the mobile terminal 300, the device information of the display device 200 can be recorded in the trusted device list in the same way, and when the display device 200 sends a Bluetooth connection request to the mobile terminal 300, according to the device information of the display device 200 Matching is performed in the trusted device list, so that when the match hits any entry, it is determined that the current display device 200 is a trusted device, so that the Bluetooth connection pairing is automatically completed, and the user does not need to manually complete the pairing operation.

In addition, in order to realize the user's non-sensing Bluetooth pairing operation, after the trust verification of the display device 200 or the mobile terminal 300 is passed, the mobile terminal 300 or the display device 200 can directly complete the pairing connection at the system level. Among them, the pairing connection at the system level means that after the current application has the permission of the Bluetooth connection at the system layer, the Bluetooth pairing is directly completed at the system layer, without calling the display dialog box through the application layer and other processing processes.

For example, a system service can be built into the display device 200. After the display device 200 establishes a screen projection connection with the mobile terminal 300, the display device 200 can monitor the change of the Bluetooth state through the system service, and register the relevant broadcast information of the Bluetooth state. After listening to the "scanning" information, it searches the list of Bluetooth devices and matches with trusted devices. After listening to the pairing request "pairing request" information, the broadcast information is processed, and the trusted device is paired at the system level. The whole process will not call the application layer to display the pairing interface, and the user does not need to actively click to confirm. Pairing connection of bluetooth devices.

In some embodiments, during the process of establishing the screen projection connection between the display device 200 and the mobile terminal 300, the security verification of the other party's device may also be performed. For example, after receiving the screen projection connection request sent by the mobile terminal 300, the display device 200 may perform security verification on the mobile terminal 300 in response to the screen projection connection request. The specific method of security verification can be as follows: through the built-in security verification program, the hardware information, network information and data transmission authority of the mobile terminal 300 are extracted and matched with the data rules stored in the security database. If the information corresponding to the terminal 300 is not in the blacklist, or the information corresponding to the mobile terminal 300 has not been maliciously tampered with, it can be determined that the mobile terminal 300 has passed the security verification.

For the mobile terminal 300 that has passed the security verification, the display device 200 completes operations such as address matching and protocol configuration by performing network communication connection pairing, thereby establishing a network communication connection between the display device 200 and the mobile terminal 300 . For example, the display device 200 may configure a screencasting protocol for the mobile terminal 300 that has passed the security verification, that is, configure the screencasting protocol according to the screencasting application, and form screencasting data transmission through multiple handshakes between the mobile terminal 300 and the display device 200 . aisle.

It should be noted that, in the process of establishing a screen projection connection, FIGS. 12a-12b are schematic diagrams of screen projection connections according to one or more embodiments of the present application. As shown in FIGS. 12a and 12b , the mobile terminal 300 The screen-casting application type is detected, so that when it is detected that the screen-casting application is a third-party application, a Bluetooth device search is actively performed to obtain a list of all connectable Bluetooth devices. Then, according to the Bluetooth Mac and other information fed back by the display device 200, matching is performed in the Bluetooth device list to determine the display device 200 for which the screen projection connection is established, so as to directly initiate a Bluetooth connection pairing to the display device 200.

When performing a Bluetooth pairing connection, in some usage scenarios, the Bluetooth function switch of the display device 200 or the mobile terminal 300 is not turned on. At this time, the display device 200 or the mobile terminal 300 cannot use the Bluetooth function to establish a Bluetooth connection. relation. Therefore, in some embodiments, in the step of connecting the Bluetooth module to the mobile terminal, the controller is further configured to:

Detect the switch status of the Bluetooth module. If the switch state is on, perform the step of acquiring address information; if the switch state is off, send an on command to the Bluetooth module to enable the Bluetooth module. For example, the display device 200 and the mobile terminal 300 can set the Bluetooth state of the current device through the Bluetooth interface, so as to search for the Bluetooth device after turning on the Bluetooth.

After the screen projection connection and the Bluetooth connection are established, the display device 200 can acquire video data and audio data from the mobile terminal 300 in real time, and play the video data and audio data through the display 275 and speakers to realize screen projection. FIG. 13 is a schematic diagram of a synchronization process of a display device according to one or more embodiments of the present application. As shown in FIG. 13 , in some embodiments, the step of connecting the mobile terminal through the Bluetooth module further includes: obtaining through the Bluetooth module The audio data stream in the mobile terminal, and the video data stream in the mobile terminal obtained through the communicator. After acquiring the audio data stream and the video data stream, play the audio data stream and the video data stream synchronously.

In practical applications, the display device 200 may acquire video data through the screen projection data channel after establishing the screen projection connection and the Bluetooth connection, and display the content of the video data on the display 275 . Meanwhile, through the Bluetooth connection, the display device 200 can be used as an audio output device of the mobile terminal 300, thereby outputting the sound in the mobile terminal 300 through the audio output device of the display device 200, such as a speaker. Since the display device 200 can directly play the sound signal in the process of audio output through the Bluetooth connection, it can also play the content played in the mobile terminal 300 in real time even if the screen projection application does not have the right to acquire audio data.

It should be noted that due to the audio output mode of the Bluetooth connection, the audio and video are not synchronized due to the delay of the Bluetooth data transmission. Therefore, in the process of synchronously playing the audio data stream and the video data stream, the mobile terminal can The audio data stream is marked, for example, marking the start playing time and so on. After the display device 200 receives the audio data stream and the video data stream, it can first buffer the audio data stream and the video data stream, identify the mark, and start playing according to the mark time to ensure the synchronization of audio and video.

It can be known from the above technical solutions that the screen projection data transmission system provided in the above embodiments includes a display device 200 and a mobile terminal 300 . Wherein, the mobile terminal 300 has a built-in communication unit and a bluetooth circuit 381; the display device 200 has a built-in communicator 220 and a bluetooth module, and the mobile terminal 300 is configured to: obtain a control command input by the user for establishing a screen-casting connection, and sending a screen projection connection request to the display device 200 according to the control instruction. The display device 200 is configured to: receive a screen-casting connection request; and establish a screen-casting connection with the mobile terminal 300 according to the screen-casting connection request; the mobile terminal and/or the display device are further configured to: detect the screen-casting application type; The type is a third-party application that establishes a Bluetooth connection between the mobile terminal and the display device to transmit audio data streams.

The screen-casting data transmission system may detect the type of screen-casting application when the display device 200 and the mobile terminal 300 establish a screen-casting connection through the screen-casting application. And when the screencasting application does not have the right to acquire audio data of the mobile terminal, the audio data stream is transmitted through the Bluetooth connection, so that the display device 200 can synchronously play the image and audio in the mobile terminal, solving the problem that the screencasting connection cannot transmit audio data.

The display device 200 can receive the screencasting connection request sent by the mobile terminal 300, and after acquiring the screencasting connection request, detect the screencasting application type, so that when the screencasting application type is a third-party application, connect the mobile phone through a Bluetooth module. terminal to receive the audio data stream in the mobile terminal. The display device can acquire the audio signal of the mobile terminal through the Bluetooth connection when the screen-casting application cannot acquire the audio data stream from the mobile terminal, and play it synchronously. The display device can transmit audio data when it is not of the same manufacturer as the mobile terminal to improve user experience.

Based on the above-mentioned screen projection data transmission system, FIG. 14 is a schematic diagram of a screen projection connection structure of a display device according to one or more embodiments of the present application. As shown in FIG. 14 , a display device 200 is also provided in some embodiments of the present application. A display 275, a communicator 220, a Bluetooth module, and a controller 250 are included. Wherein, the display 275 is used for displaying the user interface and displaying the screen-casting video image; the communicator 220 is configured to establish a screen-casting connection with the mobile terminal 300; and the Bluetooth module is configured to transmit the audio data stream. The controller 250 is configured to perform the following program steps: receiving a screen-casting connection request sent by the mobile terminal 300; in response to the screen-casting connection request, detecting a screen-casting application type; if the screen-casting application type is a third-party application, connecting the mobile terminal through a Bluetooth module terminal to obtain the audio data stream.

It can be seen that the display device 200 provided in this embodiment includes: a display 275 , a communicator 220 , a Bluetooth module, and a controller 250 . In practical applications, the display device 200 may receive a screen-casting connection request sent by the mobile terminal 300, and after acquiring the screen-casting connection request, detect the screen-casting application type, so that when the screen-casting application type is a third-party application, the The Bluetooth module is connected to the mobile terminal to receive the audio data stream in the mobile terminal 300 . The display device 200 can acquire the audio signal of the mobile terminal through a Bluetooth connection when the screen-casting application cannot acquire the audio data stream from the mobile terminal, and play it synchronously. The display device can transmit audio data when it is not of the same manufacturer as the mobile terminal to improve user experience.

FIG. 15 is a schematic diagram of a screen projection connection structure of a mobile terminal according to one or more embodiments of the present application. As shown in FIG. 15 , in some embodiments, the type of screen projection application can be detected by the mobile terminal 300 and determined according to the detection result. The screen projection method of the audio data stream. To this end, some embodiments of the present application further provide a mobile terminal 300 , including: a display unit 330 , a communication unit, a Bluetooth circuit 381 and a processor 380 . The display unit 330 is configured to display a user interface to perform user interaction; the communication unit is configured to establish a screen projection connection with the display device 200; the Bluetooth circuit 381 is configured to transmit audio data streams. The processor 380 is configured to perform the following program steps: acquiring a control instruction for establishing a screen-casting connection input by a user, and in response to the control instruction, sending a screen-casting connection request to the display device through the communication unit. By detecting the screen projection application type, if the screen projection application type is a third-party application, the display device 200 is connected through the Bluetooth circuit 381 to send an audio data stream.

It can be seen that the mobile terminal 300 provided in the embodiment of the present application can initiate a screen projection connection to the display device 200 after acquiring the control instruction input by the user, and at the same time detect the current screen projection application type, so that the screen projection application type is a third party During application, a Bluetooth connection relationship is established with the mobile terminal 300 to send audio data streams. The mobile terminal 300 can complete the screen-casting connection with the display device 200 through the screen-casting application, and when the screen-casting application does not have the right to acquire audio data, send the audio data through the Bluetooth connection, so as to solve the problem that the non-system application cannot transmit the audio data.

In some embodiments, in the step of connecting the display device 200 through the Bluetooth circuit 381 when the screen projection application type is detected by the mobile terminal 300, the processor 380 is further configured to: first receive the address information of the display device 200 through the application screen projection protocol , and perform device scanning. For example, the mobile terminal 300 sets the current Bluetooth state through the Bluetooth interface, and after turning on Bluetooth, searches for Bluetooth devices to generate a scan list. Then search the target display device 200 in the scan list according to the address information, set the Bluetooth circuit 381 to the sending mode, and send a pairing request to the target display device 200 through the Bluetooth transmission protocol.

In this embodiment, the display device 200 may feed back address information to the mobile terminal 300 after receiving the screencast connection request sent by the mobile terminal 300, so that the mobile terminal 300 can directly execute the process according to the feedback address information during the device scanning process. The Bluetooth connection matching operation enables the mobile terminal 300 to establish a connection relationship with the display device 200 through a Bluetooth connection, so as to send an audio data stream to the display device 200 .

In some embodiments of the present application, a screen projection data transmission method is also provided, which is applied to a screen projection data transmission system. The screen projection data transmission system includes a mobile terminal 300 and a display device 200, and the method includes the following steps: The mobile terminal 300 acquires a control instruction input by the user for establishing a screen-casting connection, and sends a screen-casting connection request to the display device 200 according to the control instruction. The display device 200 receives the screen-casting connection request; and establishes a screen-casting connection with the mobile terminal 300 according to the screen-casting connection request; the mobile terminal 300 and/or the display device 200 detect the screen-casting application type; if the screen-casting application type is a third-party application, in the A Bluetooth connection is established between the mobile terminal 300 and the display device 200 to transmit audio data streams.

In this embodiment, the method for transmitting the screen projection data may detect the type of the screen projection application through the mobile terminal 300 and/or the display device 200 when the screen projection application is used for the screen projection connection. If the type of the screen projection application is a third-party application, A Bluetooth connection is established between the mobile terminal 300 and the display device 200 to transmit the audio data stream from the mobile terminal 300 to the display device 200, so that the display device 200 can simultaneously display the displayed content of the mobile terminal 300 and simultaneously play the mobile terminal 300 sound, to solve the problem that the screen casting connection cannot transmit audio data.

The following uses NFC screen projection as an example to introduce how to establish a connection between a mobile terminal and a display device. Referring to FIG. 16a and FIG. 16b, FIG. 16a and FIG. 16b are schematic diagrams of scenario 1 of establishing a screen projection connection according to one or more embodiments of the present application. Assuming that there are four display devices connected to the above local area network at the same time, the device names of the four display devices are display device A, display device B, display device C, and display device D, and these four display devices all support screen projection. As shown in FIG. 16a, after the mobile terminal 100 searches, the display interface of the mobile terminal 100 displays the device names of the above four display devices in a list: display device A, display device B, display device C, and display device D. As shown in FIG. 16b , when the user needs to project the video in the mobile terminal 100 to the display device C, the user can select and click “display device C” in the display interface of the mobile terminal 100 to establish a project with the display device C. screen connection.

In some embodiments, referring to FIG. 17 , FIG. 17 is a schematic diagram of a display interface when a mobile terminal plays a video according to one or more embodiments of the present application. As shown in FIG. 17 , when the mobile terminal 100 is watching a video, a control 701 for triggering screen projection may be displayed on the display interface of the mobile terminal 100 . When the mobile terminal 100 detects that the control 701 is triggered, it automatically searches the currently accessed local area network for a display device that can perform screen projection, and then displays each display device that can perform screen projection in a list on the display interface of the mobile terminal 701 device name. After the user selects and clicks the device name of the display device to be projected on the display interface of the mobile terminal 100 according to actual screen projection needs, the mobile terminal 100 can establish a screen projection connection with the display device selected by the user.

In some embodiments of the present application, the mobile terminal can support the NFC function, and the display device supporting screen projection will distribute a specific NFC card, and the NFC card can be placed in the remote control of the display device. The P2P MAC address of the display device is stored in the NFC card. When performing screen projection, the user moves the mobile terminal close to the NFC card of the display device that needs to be mirrored, or moves the NFC card of the display device that needs to be mirrored close to the mobile terminal. The terminal, when the mobile terminal detects the NFC card, reads the stored P2P MAC address from the NFC card, and then establishes a screen projection connection with the above-mentioned display device that needs to be screened according to the P2P MAC address. After the mobile terminal is connected to the display device for screen projection, the mobile terminal can send the video played on the mobile terminal to the display device, so that the video in the mobile terminal can be viewed on the display device.

In some embodiments, referring to FIG. 18 , FIG. 18 is a second scenario diagram of establishing a screencasting connection according to one or more embodiments of the present application. In FIG. 18 , it is assumed that the NFC card corresponding to the display device 200 is set in the control device 300 (such as a remote controller) corresponding to the display device 200 , then when performing screen projection, the user moves the mobile terminal 100 close to the control device 300 to read the control The P2P MAC address stored in the NFC card in the device 300, and then establishes a screen projection connection with the display device 200 according to the P2P MAC address.

In some embodiments of the present application, the mobile terminal supports the NFC function, and a display device that supports screen projection is configured with a specific NFC card. When the display device starts a wireless hotspot, the NFC card stores the wireless hotspot information of the display device, and the wireless hotspot information is written to the NFC card by the operating system of the display device when the display device starts the wireless hotspot. When the user needs to cast the screen through the mobile terminal, the mobile terminal can be close to the NFC card on the display device. When the mobile terminal detects the NFC card on the display device, the wireless hotspot information of the display device is read from the NFC card on the display device. , and connect to the wireless hotspot of the display device based on the wireless hotspot information. After the connection is successful, the mobile terminal can establish a screen projection connection with the display device. After the mobile terminal is connected to the display device for screen projection, the mobile terminal can send the video played on the mobile terminal to the display device, so that the video in the mobile terminal can be viewed on the display device.

In some embodiments, referring to FIG. 19 , FIG. 19 is a third scenario diagram of establishing a screen projection connection according to one or more embodiments of the present application. In FIG. 19 , assuming that the NFC card corresponding to the display device 200 is set in the lower left corner of the display device 200 , when performing screen projection, the user moves the mobile terminal 100 to the lower left corner of the display device 200 to read the NFC corresponding to the display device 200 The wireless hotspot information stored in the card is connected to the wireless hotspot of the display device 200 based on the wireless hotspot information. After the connection is successful, the mobile terminal 100 can establish a screen projection connection with the display device 200 . It should be noted that, the NFC card corresponding to the display device 200 may also be set at any other position of the display device 200, which is not limited in this embodiment of the present application.

In some embodiments, the present application provides a mobile terminal, the mobile terminal includes an NFC response component for responding to an NFC card that meets a preset distance; and a controller connected with the NFC response component, the controller is It is configured to: when the NFC response component responds to the NFC card, broadcast a device search message in the local area network currently accessed by the mobile terminal; receive a feedback message sent by the display device based on the device search message, where the feedback message includes the screen projection matching of the display device code and device address; obtain the screencasting matching code of the mobile terminal, when the screencasting matching code of the mobile terminal is the same as the screencasting matching code of the display device, establish a screencasting connection with the above-mentioned display device according to the device address of the above-mentioned display device.

In some embodiments, the above-mentioned NFC response component can use an NFC read-write component with an NFC card sensing function. When the distance between an NFC card and the NFC response component is less than a preset distance, the NFC response component can sense the NFC card. , and respond. The above-mentioned NFC card may be any card with NFC function, such as a bus card and an access control card.

In some embodiments, in order to ensure that the content displayed in the mobile terminal can be accurately projected to the display device that needs to be projected, when the mobile terminal detects any NFC card, it first broadcasts a device search message in the currently accessed local area network; After receiving the device search message, the display device sends a feedback message to the mobile terminal, and the feedback message includes the screen projection matching code and device address of the display device; after receiving the feedback message sent by the display device, the mobile terminal determines that the mobile terminal Whether the screencasting matching code saved locally and the screencasting matching code fed back by the display device are the same, if they are the same, then the mobile terminal establishes a screencasting connection with the above-mentioned display device according to the device address of the above-mentioned display device, so that the screen displayed in the mobile terminal can be displayed. The content is accurately projected to the above display devices. For example, suppose a user has two TVs installed in his home, one in the bedroom and the other in the living room. When the user needs to project the content displayed in the mobile terminal to the TV in the bedroom, the projections stored in the mobile terminal can be projected in advance. The screen matching code is set to the screen matching code pre-stored by the TV in the bedroom, and then use any NFC card to approach the mobile terminal. At this time, the mobile terminal will automatically send a device search message to the TV in the bedroom and the living room respectively, and then receive the bedroom. The projection matching code and device address respectively fed back by the TV in the living room, since the projection matching code of the TV in the bedroom is the same as the projection matching code currently stored in the mobile terminal, the mobile terminal will automatically , and establish a screen projection connection with the TV in the bedroom.

It is understandable that if the user keeps selecting the TV in the bedroom as the TV that needs to be projected, in the subsequent projection process, the user only needs to use any NFC card to approach the mobile terminal, and the mobile terminal can automatically communicate with the mobile terminal. The TV in the bedroom is connected to the screen projection connection, and the user does not need to perform any other operations.

In some embodiments of the present application, when the display device starts the screen-casting application for the first time, the user can configure and save the screen-casting matching code of the display device by triggering a preset configuration operation. Alternatively, when the display device is started for the first time, the display device randomly generates a screen projection matching code and saves it.

In some embodiments, the user can also modify and save the screen projection matching code of the display device at any time through the setting interface of the display device. Referring to FIG. 20 , FIG. 20 is a schematic diagram of a scenario in which a screen projection matching code of a display device is set according to one or more embodiments of the present application. In FIG. 20 , the user can set the screencasting matching code of the display device 200 as "1234" in the screencasting matching code setting interface of the display device 200, and then save it.

When the mobile terminal needs to establish a screencasting connection with a certain display device, the user can first check the screencasting matching code of the display device to be screencasted, and then set the screencasting matching code of the mobile terminal as the screencasting matching code of the display device . For example, referring to FIG. 21 , FIG. 21 is a schematic diagram of a scenario of setting a screencasting matching code of a mobile terminal according to one or more embodiments of the present application. In FIG. 21 , when the screencasting match code of the display device 200 to which the mobile terminal 100 needs to screencast is “1234”, the user can match the screencasting of the mobile terminal 100 through the screencasting matching code setting interface of the mobile terminal 100 The code is also set to "1234".

After setting the screen projection matching code between the mobile terminal and the display device, if the user needs to project the video in the mobile terminal to the display device, he can use any NFC card to trigger the NFC response component of the mobile terminal, and the mobile terminal can automatically search A display device capable of screencasting, and then according to the screencasting matching code pre-saved locally on the mobile terminal and the screencasting matching code pre-saved by the searched display device to determine the display device that needs to be screened from the searched display devices , and automatically establish a screen-casting connection with the display device that needs to perform screen-casting, without requiring the user to perform other operations, which can effectively simplify the process of establishing a screen-casting connection and improve the efficiency of establishing a screen-casting connection.

In the following embodiments, referring to FIG. 22 , FIG. 22 is a schematic flowchart of a method for establishing a screen projection connection according to one or more embodiments of the present application. The above method for establishing a screen projection connection includes: 1201. Use any NFC card to touch the mobile terminal. Wherein, the above-mentioned NFC card may be any card with NFC function, such as a bus card, an access control card, etc., and it is not necessary to store information about the display device in the card. In this embodiment, the above-mentioned NFC card is only used to start the screen projection application in the mobile terminal.

1202. When the NFC response component of the mobile terminal responds to the NFC card, broadcast a device search message in the local area network currently accessed by the mobile terminal.

1203. After receiving the device search message broadcast by the mobile terminal, the display device obtains its own device information, reads the pre-saved screencasting matching code from the relevant file, and then assembles its own device information and the screencasting matching code as The feedback message is sent to the mobile terminal. In some embodiments, the above-mentioned device address may be the IP address of the display device. Assuming that the IP address of the display device is 192.168.1.100 and the projection matching code is 1234, the above feedback message can be assembled as: ip_192.168.1.100#deviceinfo_1234#.

1204. The mobile terminal receives the feedback message sent by the display device, parses the screen-casting matching code and device information of the display device from the feedback message, and then determines whether the screen-casting matching code of the mobile terminal and the screen-casting matching code of the display device are the same.

1205. When the screencasting matching code of the mobile terminal is the same as the screencasting matching code of the display device, the mobile terminal sends a screencasting connection request to the display device according to the parsed device information of the display device to establish a screencasting connection with the display device .

In some embodiments, when the mobile terminal receives feedback messages sent by two or more display devices, it can respectively determine whether the screencasting matching code in each received feedback message is the same as the screencasting matching code of the mobile terminal ; If the screencasting matching code in the currently received target feedback message is the same as the mobile terminal's screencasting matching code, obtain the target device address in the target feedback message, and establish a screencasting connection with the display device corresponding to the target device address . For example, suppose that there are three display devices connected to the local area network accessed by the mobile terminal, namely: display device A, whose IP address is: 192.168.1.100, and the screen projection code is 1234; display device B, whose IP address is: 192.168.1.101, The screencasting code is 1135; the IP address of display device C is: 192.168.1.102, and the screencasting code is 1136. After the mobile terminal broadcasts the device search information "Search mirror device" to the local area network, it will immediately receive the feedback messages sent by the above three display devices, which are: ip_192.168.1.100#deviceinfo_1234; ip_192.168.1.101#deviceinfo_1135; ip_192.168.1.102#deviceinfo_1136.

Among them, it is assumed that the screencasting match code pre-stored by the mobile terminal is 1234, then when the mobile terminal receives the feedback message "ip_192.168.1.100#deviceinfo_1234", the screencasting match code parsed out is "1234", which is different from that of the mobile terminal. The screencasting binding code saved in advance by the terminal is the same, so the mobile terminal uses the resolved IP address "192.168.1.100" to establish a screencasting connection with display device A. However, when the mobile terminal receives the feedback message "ip_192.168.1.101#deviceinfo_1135" or "ip_192.168.1.102#deviceinfo_1136", the parsed screencasting matching code is different from the screencasting binding code pre-saved by the mobile terminal. , so the mobile terminal will not establish a screen projection connection with display device B or display device C.

In the method for establishing a screen-casting connection provided by the present application, when the mobile terminal searches for multiple display devices capable of screen-casting, the user does not need to select the display device that needs to be screen-casted from the searched display devices, and the mobile terminal can automatically connect with the screen-casting display device. The display device that needs to perform screen projection establishes a screen projection connection, which can effectively simplify the process of establishing a screen projection connection and improve the efficiency of establishing a screen projection connection.

In some embodiments, when the mobile terminal and the display device are in a screen-casting connection state, if the mobile terminal receives a feedback message sent by another display device, it may not perform processing. It can be understood that in the process of screen projection between the mobile terminal and the display device, the data transmission between the mobile terminal and the display device is transmitted through the wireless network, and most of them are transmitted through the User Datagram Protocol (User Datagram Protocol). , UDP), a network protocol that does not guarantee the quality of data transmission, exchanges data, so the problem of network data packet loss will inevitably occur, which will cause the display device to appear blurry.

Among them, the most commonly used encoding and transmission methods in the screen projection process are: inserting P frames (forward predictive coding frames) between key frames (I frames) for transmission. For example, a mobile terminal can send a type of "IPPPPIPPPPIPPPP". Image sequence to display device. Among them, when the display device decodes the I frame, it can decode a whole picture without comparing it with other frames; when decoding the P frame, it needs to compare it with the previous I frame or P frame, Calculate the difference between them to decode a complete picture.

In order to prevent the display device from having a blurry screen during decoding, in some embodiments of the present application, after the mobile terminal establishes a screen projection connection with the display device, the mobile terminal collects image data displayed in the mobile terminal based on a preset collection frequency, and collects the collected image data. The image data is sequentially sent to the display device. Wherein, the image data includes a serial number, and a preset condition is satisfied between the serial number of the image data currently sent by the mobile terminal and the serial number of the image data sent last time.

When the display device receives the above-mentioned image data, if the serial number of the currently received image data and the serial number of the last received image data do not satisfy the preset condition, then determine the Whether the type is P frame; when the type of the currently received image data is P frame, send an I frame request message to the mobile terminal. When receiving the I-frame request message sent by the display device, the mobile terminal collects image data whose type is I-frame and sends it to the display device. After receiving the image data of the type I frame, the display device decodes the image data of the type I frame.

It is understandable that when decoding a P frame, if an I frame or P frame in front of it is lost, then the parameter frame it depends on will become the other I frame or P frame in front of it, and then there will be a blurry screen. Phenomenon.

Referring to FIG. 23, FIG. 23 is a schematic diagram of a frame sequence of a video stream according to one or more embodiments of the present application. In Figure 23, assuming that the 4th frame is not received due to network reasons, when decoding the 5th frame by the display device, it can only refer to the 3rd frame, so that the display device will appear blurry when displaying the 5th frame. At the same time, because the decoded image of the 5th frame is a blurry screen, when the sixth frame and the subsequent P frames are decoded, there will be a blurry screen, until the n+1th frame is decoded, since the n+1 frame is an I frame, so the screen is blurred. phenomenon will disappear. Therefore, the display device can determine whether a preset condition is satisfied between the serial number of the currently received image data and the serial number of the last received image data, and whether the type of the currently received image data is a P frame. Whether there is packet loss on the outgoing data network.

In some embodiments, the above preset condition is that the difference between the serial number of the image data currently received by the display device and the serial number of the image data received last time is 1. Referring to FIG. 24 , FIG. 24 is a schematic flowchart of a method for preventing a blurry screen from appearing during screen projection according to one or more embodiments of the present application.

In some embodiments, the above-mentioned method for preventing smudged screens from appearing during screen projection:

1401. After the mobile terminal establishes a screen projection connection with the display device, collects image data displayed in the mobile terminal based on a preset collection frequency, performs multimedia encoding in real time, encapsulates it into a TS stream and stores it in a data cache buffer, and the buffer Contains the sequence number seqNo and TS stream data of the packet.

1402. The mobile terminal sends the above-mentioned data contained in the buffer to the display device through the established UDP data channel. Among them, each time a buffer is sent, the sequence number seqNo in the buffer is incremented by 1.

1403. The display device receives the image data sent by the mobile terminal, and obtains its serial number from the received image data, and determines whether the difference between the serial number of the currently received image data and the serial number of the last received image data is equal to 1.

1404. If the display device determines that the difference between the serial number of the currently received image data and the serial number of the last received image data is greater than 1, it means that there is a difference between the currently received image data and the last received image data. Part of the image data is lost. At this time, it is determined whether the type of the currently received image data is a P frame.

1405. When the type of the currently received image data is P frame, the display device sends an I frame request message to the mobile terminal.

Wherein, when the type of image data currently received by the display device is I-frame data, the loss of previous data will not affect the decoding of the frame of data, so the frame of image data can be directly decoded.

In some embodiments, the display device discards the received image data until the image data of type I frame sent by the mobile terminal is received.

1406. After receiving the I-frame request message sent by the display device, the mobile terminal immediately collects image data whose type is I-frame.

1407. The mobile terminal sends the collected image data of type I frame to the display device.

1408. The display device decodes based on the received image data of type I frame.

That is, in the embodiment of the present application, the display device can judge whether the screen projection connection has packet loss through the serial number of the image data. A blurry screen appears during projection.

When a smart device A uses multiple network cards, or in other words, when the smart device A can work on multiple networks at the same time, the above interaction method between smart devices can only make the smart device A work in a default network. For example, if smart device A enables a hotspot, the smart device A has both a wired network and a hotspot network. If smart device B and smart device C are connected to the wired network and the hotspot network respectively, the smart device A can only communicate with the smart device. One of device B and smart device C interacts normally, and cannot interact with smart device B and smart device C at the same time. It can be seen that when the smart device is a multi-NIC device, it is difficult to ensure that the smart device can interact normally in all connected networks. Therefore, the current smart device interaction method has certain limitations.

The following takes P2P (Peer To Peer, peer-to-peer) screen projection as an example to introduce wireless network resource allocation. When using WiFi connection, connect the mobile terminal and display device to the same wireless network, use P2P service to realize screen projection, and at the same time ensure the normal WLAN function. The WLAN function is limited, and the packet loss rate increases.

In some embodiments, FIG. 25 is a possible P2P device discovery process according to one or more embodiments of the present application. As shown in FIG. 25 , in the device discovery phase of P2P pairing, the P2P mobile terminal 100A and The display device 200 should constantly switch between scanning and monitoring on channels 1, 6, and 11. When the mobile terminal 100A and the display device 200 happen to be in the same channel, and the mobile terminal 100A scans, that is, sends a detection request, and displays the When the device 200 monitors the detection request sent by the mobile terminal 100A, the display device 200 responds to the detection request and replies to the detection response to complete the device discovery process (this process can also be performed by the display device 200 under the same channel). scanning, the mobile terminal 100A monitors). Based on the above process requirements of the device discovery stage, if the user is to perform screen projection at any time, the background of the display device 200 needs to continuously perform P2P scanning and monitoring, which will cause the channel to keep on 1, 6, 11. Switching greatly affects the normal use of WLAN ports. It not only occupies antenna resources, but also frequently switches channels, which increases network latency and makes the network more stuck. To this end, the following solutions are proposed, multiple working modes are set, and corresponding working modes are selected according to different situations.

In some embodiments, FIG. 26 is a process of adjusting a duration in a network transmission-based mode according to one or more embodiments of the present application. As shown in FIG. 26 , when some network-dependent applications such as download applications , audio applications, video applications, etc. request network data, the network data request is transmitted to the driver module through the framework layer and the core layer, and the driver module determines that the transmission mode of the display device is the network-based mode, and sends The first transmission duration adjustment command is sent to wireless communication firmware such as WiFi firmware.

In some embodiments, the first transmission duration adjustment instruction may be a specific duration allocation instruction, or may be a Δt adjustment instruction as shown in FIG. 8 , where Δt is the difference between the time slice occupied by network transmission and screencast transmission. difference. Taking the transmission time slice provided by the wireless communication module as 1s as an example, when the first transmission duration adjustment command is Δt=800, the WiFi firmware determines the first transmission duration to provide transmission services for network transmission according to the first transmission duration adjustment command: 900ms, and the second transmission duration for providing transmission services for screencast transmission is 100ms. Therefore, network transmission is performed within 900ms of each second, and the WLAN port transmits and receives data packets, and the data packets are transmitted to the network application through the driver module, the core layer and the framework layer.

In some embodiments, the display device further includes a detection module 225 for detecting network transmission quality and triggering the driving module according to the transmission quality. In the network transmission-based mode, the detection module is the drive module. When the network environment deteriorates, the detection module detects that the WiFi packet loss rate becomes higher and the delay is higher than the preset threshold, it sends the packet loss command to the driver module, and the driver module can adjust the command to the size of Δt through the third transmission duration Make mandatory adjustment and gradually adjust Δt, for example, gradually increase the absolute value of Δt in units of 10ms, and continuously compare the delay of WLAN until it is adjusted to the level that satisfies the basic sending and receiving packets of WLAN (for example, the delay is lower than 200ms) , and at the same time ensure that the time that P2P occupies the antenna will not affect the normal monitoring of P2P, for example, not less than 50ms/1s. For example, when the driver module detects that the delay is higher than the preset threshold and adjusts Δt=810, the WiFi firmware determines that the first transmission duration for network transmission is 905ms, and the second transmission duration for screencast transmission is determined to be 905ms. The transmission time is 95ms. If the delay is still higher than the preset threshold, adjust Δt=820, and so on.

In some embodiments, FIG. 27 is a possible P2P device discovery process according to one or more embodiments of the present application. As shown in FIG. 27 , combined with the usage scenario of the display device, the P2P device discovery device of the display device is The scanning and monitoring of the stage is changed to monitoring only on fixed channels (such as 1, 6, and 11), which can reduce the occupation of antenna resources caused by channel switching. At the same time, because the display device is always in the monitoring state, it can also improve the P2P discovery. equipment efficiency.

In some embodiments, FIG. 28 is a process of adjusting a duration in a mode mainly based on screen projection transmission according to one or more embodiments of the present application. As shown in FIG. 28 , when providing transmission services for screen projection transmission Within the second transmission duration of , for example, when the screen-casting application initiates P2P monitoring, the monitoring command is transmitted to the WiFi firmware via the framework layer, the adaptation layer and the driver module, and the WiFi firmware switches to the monitoring channel such as channel 6 for monitoring.

In some embodiments, when the WiFi firmware is monitoring, a P2P detection request is received, and the detection request is sent to the driver module, and the driver module parses the message containing the P2P IE information, thereby determining that the transmission mode of the display device is Screen-based mode.

In some embodiments, the driving module sends a second transmission duration adjustment instruction to the WiFi firmware after determining that the transmission mode in which the display device is located is the screen-casting mode. In some embodiments, the second transmission duration adjustment instruction may be a specific duration allocation instruction, or may be an Δt adjustment instruction, where Δt is the difference between the time slices occupied by network transmission and screen projection transmission. Taking the transmission time slice provided by the wireless communication module as 1s as an example, when the second transmission duration adjustment command is Δt=-600, the WiFi firmware determines the first transmission duration to provide transmission services for network transmission according to the duration adjustment command to be 200ms, which is The second transmission duration for the screencast transmission to provide the transmission service is 800ms.

Therefore, the screen projection transmission is performed within 800ms of each second, and the P2P port sends and receives data packets, and the data packets are transmitted to the screen projection application through the driver module, the adaptation layer and the framework layer. If it is video data, it can be played through the screen casting application.

In some embodiments, the detection module is a screen-casting application when the screen-casting transmission is the main mode. When the screencasting application detects that the P2P packet loss rate has increased and exceeds the preset threshold, for example, the number of lost packets exceeds 100 packets, the screencasting will show a vase freeze problem. The screen-casting application will send the packet loss command to the driver module, and the driver module can forcefully adjust the size of Δt through the third transmission duration adjustment command, and gradually adjust Δt, for example, gradually increase the absolute value of ΔT in units of 10ms, and pass Constantly compare the packet loss rate of the screen projection until it is adjusted to the level that meets the basic P2P sending and receiving packets, and at the same time ensure that the time that the WLAN occupies the antenna will not affect the normal progress of network transmission, for example, not less than 50ms/1s. The above embodiment takes P2P screen projection as an example, and can also be applied to other screen projection technologies, which is not limited thereto.

By dividing the network transmission and screen projection transmission into time slices, the network transmission and the screen projection transmission work in their respective time slices, thus solving the problem of competition between the two for resources and ensuring the orderly and orderly network transmission and screen projection. At the same time, the working efficiency of the display device is improved.

For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above discussion in some embodiments is not intended to be exhaustive or to limit implementations to the specific forms disclosed above. Numerous modifications and variations are possible in light of the above teachings. The above embodiments have been chosen and described to better explain the principles and practical applications, so as to enable those skilled in the art to better utilize the embodiments and various modified embodiments suitable for specific use considerations.

Claims (10)

1. A display device comprising:

   monitor;

   a communicator, configured to establish a screen-casting connection with the mobile terminal;

   a Bluetooth module, configured to pass audio data streams;

   Controller, configured as:

   Receive a screencast connection request sent by the mobile terminal;

   In response to the screen-casting connection request, detecting the screen-casting application type;

   If the type of the screen projection application is a third-party application, connect the mobile terminal through the Bluetooth module to obtain an audio data stream.
2. The display device of claim 1, the controller is further configured to:

   In the step of detecting the screen-casting application type, if the screen-casting application type is a system application, a screen-casting connection is established with the mobile terminal through the system screen-casting protocol to transmit the video data stream and the audio data stream;

   If the type of the screen-casting application is a third-party application, a screen-casting connection is established with the mobile terminal through an application screen-casting protocol to transmit the video data stream.
3. The display device of claim 1, the controller is further configured to:

   In the step of detecting the screen-casting application type, obtain the package name information of the screen-casting application;

   Extract the judgment attribute value from the package name information;

   If the judging attribute value is equal to 0, marking the screen-casting application as a third-party application;

   If the judgment attribute value is not equal to 0, the screen projection application is marked as a system application.
4. The display device of claim 1, the controller is further configured to:

   In the step of connecting the mobile terminal through the bluetooth module, the display device address information is sent to the mobile terminal through the application screen projection protocol, so that the mobile terminal performs bluetooth pairing according to the address information;

   receiving a pairing request sent by the mobile terminal through the Bluetooth transmission protocol;

   In response to the pairing request, the Bluetooth module is set to receive mode, and a Bluetooth connection is established with the mobile terminal.
5. The display device of claim 4, the controller is further configured to:

   In the step of connecting the mobile terminal through the bluetooth module, the switch state of the bluetooth module is detected;

   If the switch state is on, execute the step of acquiring address information;

   If the switch state is off, an on command is sent to the bluetooth module to enable the bluetooth module.
6. The display device of claim 1, the controller is further configured to:

   After the step of connecting the mobile terminal through the bluetooth module, obtain the audio data stream in the mobile terminal through the bluetooth module;

   Obtain the video data stream in the mobile terminal through the communicator;

   The audio data stream and the video data stream are played synchronously.
7. A mobile terminal, comprising:

   Display unit;

   a communication unit, configured to establish a screen projection connection with the display device;

   Bluetooth circuitry configured to transmit audio data streams;

   The processor, configured as:

   Obtain the control command input by the user for establishing a screen-casting connection;

   In response to the control instruction, send a screen projection connection request to the display device through the communication unit;

   Detect the type of screencasting application;

   If the type of the screen projection application is a third-party application, connect the display device through a Bluetooth circuit to send an audio data stream.
8. The mobile terminal of claim 7, the processor is further configured to:

   In the step of connecting the display device through the Bluetooth circuit, receiving the address information of the display device through the application screen projection protocol;

   perform a device scan to generate a scan list;

   Find a target display device in the scan list according to the address information;

   The bluetooth circuit is set to a sending mode, and a pairing request is sent to the target display device through a bluetooth transmission protocol.
9. A method for transmitting screen projection data, comprising:

   The mobile terminal acquires a control instruction input by the user for establishing a screen-casting connection, and sends a screen-casting connection request to the display device according to the control instruction;

   The display device receives the screen projection connection request; and establishes a screen projection connection with the mobile terminal according to the screen projection connection request;

   The mobile terminal and/or display device detects the type of screencasting application;

   If the type of the screen-casting application is a third-party application, a Bluetooth connection is established between the mobile terminal and the display device to transmit an audio data stream.
10. A screen projection data transmission system, comprising a mobile terminal and a display device; wherein, the mobile terminal has a built-in communication unit and a Bluetooth circuit; the display device has a built-in communicator and a Bluetooth module;

    The mobile terminal is configured to: obtain a control instruction input by the user for establishing a screen-casting connection, and send a screen-casting connection request to the display device according to the control instruction;

    The display device is configured to: receive the screen-casting connection request; and establish a screen-casting connection with the mobile terminal according to the screen-casting connection request;

    The mobile terminal and/or the display device are further configured to: detect a screen-casting application type; if the screen-casting application type is a third-party application, establish a Bluetooth connection between the mobile terminal and the display device, to pass the audio data stream.

Images