WO2023011058A1

WO2023011058A1 - Display device, communication terminal, and projected-screen image dynamic display method

Info

Publication number: WO2023011058A1
Application number: PCT/CN2022/102439
Authority: WO
Inventors: 宋子全; 马晓燕; 刘美玉; 庞秀娟; 张娜
Original assignee: 海信视像科技股份有限公司
Priority date: 2021-08-02
Filing date: 2022-06-29
Publication date: 2023-02-09
Also published as: CN117837150A

Abstract

The present application provides a display device (200), a communication terminal (300), and a projected-screen image dynamic display method. The method comprises: when a user triggers a change in the number of communication terminals (300) connected to a display device (200), measuring the number of connections of the communication terminals (300) to plan a layout mode of a projected-screen image; and then, sending to the communication terminals (300) a notification message including a projected-screen image display parameter, such that the communication terminals (300) can generate, according to the display parameter, screen projection data conforming to the layout mode, and display the screen projection data in the display device (200).

Description

Display device, communication terminal, and dynamic display method for projected screen images

Cross References to Related Applications

This application is required to be submitted on August 2, 2021, with the application number 202110881496.8; to be submitted on August 2, 2021, with the application number 202110881335.9; to be submitted on October 28, 2021, with the application number 202111265530.5, the entire contents of which are by reference incorporated in this application.

technical field

The present application relates to the field of display devices, and in particular to a display device, a communication terminal, and a method for dynamically displaying screen projection images.

Background technique

A display device refers to a terminal device capable of outputting specific display images, which may be terminal devices such as smart TVs, communication terminals, smart advertising screens, and projectors. Taking smart TV as an example, smart TV is based on Internet application technology, has an open operating system and chip, has an open application platform, can realize two-way human-computer interaction, and integrates multiple functions such as audio-visual, entertainment, and data. Products are used to meet the diverse and individual needs of users.

The display device can also establish a projection connection relationship with the communication terminal through wireless LAN and other communication methods, that is, the display device can receive the projection data shared by the communication terminal, and display the received projection data in real time, that is, display the current screen of the communication terminal or The media asset screen played on the communication terminal.

Contents of the invention

An embodiment of the present application provides a display device, including: a display, a communicator, and a controller. Wherein, the display is configured to display a screen projection interface; the communicator is configured to establish a screen projection connection with a communication terminal; the controller is configured to execute the following procedural steps: detecting the number of connections of a communication terminal; according to the The number of connections plans the layout of the projection screen to obtain the display parameters of the projection screen; a notification message is sent to the communication terminal corresponding to the projection screen, the notification message includes the display parameters, and the notification message uses The method is to make the communication terminal generate screen projection data according to the display parameters; receive the screen projection data fed back by the communication terminal, and display the screen projection data according to the layout.

An embodiment of the present application provides a display device, including:

The display is used to display the screen projection interface; the communicator is used to connect with the terminal device for screen projection; the controller is configured to execute: receiving the screen projection request of at least M terminal devices, and receiving the media configuration respectively from the M terminal devices Information, matching the layout between M screen projection windows adapted to the screen; wherein, M≥1, the media configuration information includes the aspect ratio of the video projected by the terminal device; the control display follows the layout in the screen projection interface Display M screen projection windows, and display the screen projection data sent by the corresponding terminal device in each screen projection window.

An embodiment of the present application provides a display device, including: a display configured to display a user interface; a controller capable of running a playback application and at least one screen projection protocol application, and the at least one screen projection protocol application can be connected to the The playback application is configured to: when receiving a first screen projection request sent by a first mobile terminal, establish a connection with the first mobile terminal through a first screen projection protocol application corresponding to the first screen projection request. The screen projection connection between the terminals, and sending the first content to be projected by the first mobile terminal through the screen projection connection to the playback application; through the playback application, create the first playback on the user interface window, the first playing window is used to play the first content to be projected.

Description of drawings

FIG. 1 is a schematic diagram of an operation scene between a display device and a control device in an embodiment of the present application;

FIG. 2 is a block diagram of a hardware configuration of a display device in an embodiment of the present application;

FIG. 3 is a schematic diagram of display device software configuration in the embodiment of the present application;

FIG. 4 is a schematic diagram of displaying an icon control interface of a display device application program in an embodiment of the present application;

Fig. 5 is a schematic structural diagram of a communication terminal according to some embodiments;

FIG. 6 is a schematic diagram of a software architecture of a communication terminal according to some embodiments;

7 is a schematic diagram of a user interface of a communication terminal according to some embodiments;

Fig. 8 is a schematic diagram of screen projection effects according to some embodiments;

Fig. 9 is a schematic diagram of a screen projection operation process according to some embodiments;

Fig. 10 is a schematic diagram of screen projection connection of multiple communication terminals according to some embodiments;

Fig. 11 is a schematic diagram of the effect of dividing two display areas according to some embodiments;

Fig. 12 is a schematic flowchart of a method for dynamically displaying a projected screen according to some embodiments;

Fig. 13 is a schematic diagram of the effect of dividing three display areas according to some embodiments;

14 is a schematic flow diagram of detecting the number of connections in response to a connection instruction according to some embodiments;

15 is a schematic flow diagram of detecting the number of connections in response to an exit command according to some embodiments;

Figure 16 is a schematic flow diagram of matching layout templates according to some embodiments;

Fig. 17 is a schematic flow diagram of sending a reminder message according to some embodiments;

Fig. 18 is a schematic flow diagram of a communication terminal generating screen projection data according to some embodiments;

Fig. 19 is a flowchart of a multi-channel projection display method according to some embodiments;

Fig. 20(a) is a schematic layout diagram of a scene where the number of projection screens is 1 and the video stream is a single vertical screen according to some embodiments;

Fig. 20(b) is a schematic layout diagram of a scene where the number of projection screens is 1 and the video stream is a single horizontal screen according to some embodiments;

Fig. 21(a) is a schematic diagram of the first layout in a scene where the number of projection screens is 2 and double horizontal screens according to some embodiments;

Fig. 21(b) is a schematic diagram of a second layout in a scene where the number of screen projections is 2 and double horizontal screens according to some embodiments;

Fig. 21(c) is a schematic diagram of the first layout in the scene where the number of projection screens is 2 and one horizontal screen and one vertical screen according to some embodiments;

Fig. 21(d) is a schematic diagram of a second layout in a scenario where the number of projection screens is 2 and one horizontal screen and one vertical screen according to some embodiments;

Fig. 21(e) is a schematic layout diagram in a scene where the number of projection screens is 2 and dual vertical screens according to some embodiments;

Fig. 22(a) is a schematic diagram of version 1 of the layout in the scenario where the number of projection screens is 2 and the dual landscape screens (with different aspect ratios) according to some embodiments;

Fig. 22(b) is a schematic diagram of version 2 of the layout in the scenario where the number of projection screens is 2 and the dual landscape screens (with different aspect ratios) according to some embodiments;

Fig. 22(c) is a schematic diagram of version 1 of the layout in a scenario where the number of projection screens is 2 and the dual vertical screens (with different aspect ratios) according to some embodiments;

Fig. 22(d) is a schematic diagram of version 2 of the layout in the scenario where the number of projection screens is 2 and the dual vertical screens (with different aspect ratios) according to some embodiments;

Fig. 23(a) is a schematic diagram of version 1 of the layout in a scenario where the number of projection screens is 3 and there is one horizontal screen and two vertical screens according to some embodiments;

Fig. 23(b) is a schematic diagram of version 2 of the layout in a scene where the number of projection screens is 3 and there is one horizontal screen and two vertical screens according to some embodiments;

Fig. 23(c) is a schematic diagram of version 3 of the layout in a scenario where the number of projection screens is 3 and there is one horizontal screen and two vertical screens according to some embodiments;

Fig. 24 is a flowchart of interaction between a terminal device and a display device according to some embodiments;

FIG. 25 is a display diagram of an icon control interface of an application program in the display device 200 according to some embodiments;

Fig. 26 is a configuration diagram of a controller in a display device according to some embodiments;

Fig. 27 is a schematic flowchart of a screen projection method performed by a controller in a display device according to some embodiments;

Fig. 28 is another schematic flowchart of a screen projection method performed by a controller in a display device according to some embodiments;

Fig. 29 is a schematic diagram of a playback window on a display in a display device according to some embodiments;

Fig. 30 is a schematic structural diagram of screen projection interaction between multiple mobile terminals and a display device according to some embodiments;

Fig. 31 is a sequence diagram of screen projection interaction between a first mobile terminal and a display device according to some embodiments.

Detailed ways

In order to make the purpose and implementation of the application clearer, the following will clearly and completely describe the exemplary implementation of the application in conjunction with the accompanying drawings in the exemplary embodiment of the application. Obviously, the described exemplary embodiment is only the present application. Claim some of the examples, not all of them.

It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementations described below, and is not intended to limit the implementations of this application. These terms are to be understood according to their ordinary and usual meaning unless otherwise stated.

Fig. 1 is a schematic diagram of an operation scene between a display device and a control device according to an embodiment. As shown in FIG. 1 , the user can operate the display device 200 through the smart device 300 or the control device 100 .

In some embodiments, the control device 100 may be a remote control, and the communication between the remote control and the display device includes infrared protocol communication, bluetooth protocol communication, and other short-distance communication methods, and the display device 200 is controlled wirelessly or wiredly. The user can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, and the like.

In some embodiments, the smart device 300 (such as a communication terminal, a tablet computer, a computer, a notebook computer, etc.) can also be used to control the display device 200 . For example, the display device 200 is controlled using an application program running on the smart device.

In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the module for obtaining voice commands configured inside the display device 200 can directly receive the user's voice command control , the user's voice command control can also be received through the voice control device installed outside the display device 200.

In some embodiments, the display device 200 also performs data communication with the server 400 . The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200 .

Fig. 2 is a configuration block diagram of the control device 100 according to some embodiments. As shown in FIG. 2 , the control device 100 includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply. The control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, and acts as an intermediary for the interaction between the user and the display device 200 .

FIG. 3 shows a block diagram of a hardware configuration of a display device 200 according to an exemplary embodiment.

In some embodiments, the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface. at least one.

In some embodiments the controller includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, a first interface to an nth interface for input/output.

In some embodiments, the communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module and other network communication protocol chips or near field communication protocol chips, and an infrared receiver.

In some embodiments, the user input interface can be used to receive a control signal of the control device 100 (such as an infrared remote controller, etc.).

In some embodiments, the detector 230 may include a light receiver, an image collector, a sound collector, and the like.

In some embodiments, the external device interface 240 may include, but is not limited to, the following: high-definition multimedia interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), Any one or more interfaces such as RGB ports.

In some embodiments, the system of the display device may include a kernel (Kernel), a command parser (shell), a file system and application programs.

Referring to Fig. 4, in some embodiments, the system is divided into four layers, from top to bottom are applications (Applications) layer (application layer), application framework (Application Framework) layer (framework layer), Android runtime (Android runtime) and system library layer (system runtime library layer), as well as the kernel layer.

FIG. 5 shows a schematic structural diagram of a communication terminal 300 .

The embodiment will be specifically described below by taking the communication terminal 300 as an example. It should be understood that the communication terminal 300 shown in FIG. 5 is only an example, and that the communication terminal 300 may have more or fewer components than those shown in FIG. 5, two or more components may be combined, or Different component configurations are possible. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.

As shown in Figure 5, the communication terminal 300 includes: a radio frequency (radio frequency, RF) circuit 310, a memory 320, a display unit 330, a camera 340, a sensor 350, an audio circuit 360, and a wireless fidelity (Wireless Fidelity, Wi-Fi) circuit 370, processor 380, Bluetooth circuit 381, and power supply 390 and other components.

FIG. 6 is a block diagram of the software structure of the communication terminal 300 according to the embodiment of the present invention.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, which are respectively the application program layer, the application program framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer from top to bottom.

The workflow of the software and hardware of the communication terminal 300 will be described below in conjunction with capturing and photographing scenes.

When the touch screen 331 receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into original input events (including touch coordinates, time stamps of touch operations, and other information). Raw input events are stored at the kernel level. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Take the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon as an example. The camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer. Camera 340 captures still images or video.

As shown in FIG. 7 , the user can open a corresponding application program by touching an application icon on the user interface, or open a corresponding folder by touching a folder icon on the user interface.

Based on the above display device 200 and communication terminal 300 , by establishing a screen projection connection, the display device 200 can play and display the content in the communication terminal 300 , as shown in FIG. 8 . In some embodiments, as shown in FIG. 9 , in order to establish a screen projection connection relationship, the user may send a screen projection connection request through the communication terminal 300 , and send the screen projection connection request to the display device 200 through the WiFi network. The display device 200 completes the transmission protocol configuration according to the screen projection connection request, so as to establish a screen projection data transmission channel with the communication terminal 300 .

The display device 200 can also realize the screen projection connection with the communication terminal 300 through different connection methods. For example, when the display device 200 and the communication terminal 300 are connected to the same wireless local area network, a screen projection connection can be established based on the WiFi network. For another example, when both the display device 200 and the communication terminal 300 are provided with NFC (Near Field Communication, near field communication) components, the screen projection connection relationship can be established through the NFC components. Obviously, other wired or wireless connection methods can also be used between the display device 200 and the communication terminal 300 to establish a screen projection connection relationship, such as RF radio frequency connection, infrared connection, cellular network, and connections provided by those skilled in the art in the above embodiments Other connection methods that can be thought of on the basis of the method.

After the screen projection connection is established, the display device 200 may receive relevant data of the communication terminal 300 through the screen projection data channel, so as to play relevant content on the display device 200 . Under different screen projection data transmission protocols, the screen projection data acquired by the display device 200 has different forms. For example, under the DLNA (Digital Living Network Alliance, Digital Living Network Alliance) protocol, the display device 200 can obtain the link address of a certain resource in the communication terminal 300 through the screen projection data channel, and obtain media asset content data by accessing the link address , so as to play the media asset data. At the same time, the display device 200 can also obtain control instructions sent by the communication terminal 300 through the screen projection data channel when playing media asset data, such as fast forward, pause, and stop projection. For example, after a connection is established through a communication protocol, the communication terminal 300 such as a mobile phone and the display device 200 will negotiate device functions and network conditions to select a suitable video and audio transmission format. After establishing the session of the video stream, a series of real-time streaming protocol (RTSP) control commands are used to control the playback and termination of the video stream.

Under other screen projection connection protocols such as Miracast screen projection, Airplay screen projection, NFC mirror projection, and Music Broadcast projection, the display device 200 can directly obtain video data streams from the communication terminal 300 . The video data stream may be the video data stream corresponding to the final display screen of the display unit 310 of the communication terminal 300, that is, the video data stream includes the content screen of the media assets in the communication terminal 300, the UI interface, and other information that can be displayed on the display unit 310. content of the video screen. The video data stream may also be only the video data that can be acquired in the communication terminal 300, such as the video data of the media assets being played. At this time, the communication terminal 300 can be used as a signal source of the display device 200 , and the display device 200 can continuously obtain video data through the projection data channel, and display it on the display 275 in real time, so as to realize the synchronous display effect with the communication terminal 300 .

The communication terminal 300 can acquire different screen projection data according to different screen projection protocols. For example, for system application or screen projection directly at the system level, the communication terminal 300 may adopt a Miracast screen projection, Airplay screen projection or WiDi screen projection protocol to establish a screen projection data channel.

Miracast screen projection is applicable to Android devices, that is, Android devices can share video images through Miracast screen projection. Generally, smart devices above Android 5.0 have built-in Miracast protocol, so they can directly perform wireless screen projection through the WiFi P2P function between the screen projection device and the WiFi network card of the receiver device without installing any software. In addition, Miracast also supports simultaneous transmission of audio and video. However, due to too many Android devices, Miracast is not fully compatible with all Android devices, and Miracast-based screencasting does not have a screencasting code function, which cannot prevent malicious or accidental screencasting.

Airplay screen projection is a wireless playback technology introduced by Apple, which is suitable for wireless screen projection of iOS and Mac systems. Apple series devices have a built-in Airplay screen projection function, which can be used with a screen projector that supports AirPlay server service, and the effect is clear and stable. The protocol comes with a screen projection code function. However, it is necessary to ensure that the Apple device and the screen projector are used in the same LAN (same network segment), and cannot be used across network segments or VLANs.

WiDi screen projection is a wireless playback technology led by Intel, and it is also based on WiFi Direct direct connection technology. Windows 8/10 laptops natively support WiDi screen projection, which can be directly screen projection without installing any software, and supports USB reverse control function.

The above three screen projection protocols can not only obtain the video data stream in the communication terminal 300, but also can obtain the audio data stream in the communication terminal 300, so that the video data and audio data can be synchronously sent to the display device 200, so that the display device 200 can Play video and audio synchronously. Since the above-mentioned Miracast screen projection protocol and Airplay screen projection protocol require system-level application permissions, the Miracast screen projection protocol and Airplay screen projection protocol are usually only applicable to system applications, or the display device 200 and communication terminal 300 of the same manufacturer pass the built-in rules of the operating system. Directly establish a screencast connection.

The display device 200 may establish a screen projection connection with the communication terminal 300 based on any one of the screen projection methods provided in the foregoing embodiments. In some embodiments, the display device 200 may also establish a screen projection connection relationship with multiple communication terminals through one or more of the above screen projection methods. For example, the display device 200 may establish a screen projection connection with two communication terminals 300 at the same time, and divide a display area in the display screen for respectively displaying the screen projection screen shared by the two communication terminals 300 . That is, as shown in FIG. 10 , the screen projection image shared by terminal A may be displayed on the left side of the user interface displayed on the display device 200 , and the screen projection image shared by terminal B may be displayed on the right side of the user interface.

In order to be able to display the screen projection screen shared by multiple communication terminals 300 on the display device 200, in some embodiments, when the display device 200 establishes a screen projection connection with multiple communication terminals 300, the display device 200 can The displayed user interface is divided into areas, such as bisecting, trisecting, and quartering. After the multiple display areas are obtained through area division, the display device 200 displays the projected screen image in each divided area.

In addition, the communication terminal 300 may be placed in different states according to different ways of use, for example, for a communication terminal 300 such as a smart phone, it may be in a portrait state or a landscape state during use. Wherein, the vertical screen state refers to a state in which the width of the displayed picture is smaller than the height, and the horizontal screen state refers to the state in which the width of the displayed picture is greater than the height. In different placement states, the communication terminal 300 may present different screen layout modes. Based on this, when the display device 200 displays the projected screen images of multiple communication terminals 300 by area division, each display area will present different screen ratios due to the different placement states of the communication terminals 300 .

For example, as shown in FIG. 11 , for a display device 200 with a display resolution of 4096×2160, in the process of vertically bisecting the center position of the user interface to obtain two display regions, two regions with a width of 2048, A display area with a height of 2160 is to obtain a vertical shape area whose width is smaller than the height, and this type of display area is more suitable for displaying the projection screen in the portrait state. Therefore, when the communication terminal 300 corresponding to this area is in a landscape orientation state, most of the display area cannot be filled with the projected screen image, resulting in a waste of the display area.

In addition, when the placement states of the multiple communication terminals 300 are different, the display device 200 usually stretches and fills each screen projection image to fill the display area. This display mode will cause a large deformation of the projected screen, which will affect the user's viewing experience.

In order to make full use of the display area and reduce stretching and deformation of the screen, some embodiments of the present application also provide a method for dynamically displaying the screen projection screen, which is applied to the display device 200 and used to improve the user's viewing experience. Wherein, the display device 200 capable of applying the method for dynamically displaying the projected screen includes: a display 260 , a communicator 220 and a controller 250 . The display 260 is configured to display a screen projection interface. The communicator 220 is configured to establish a screen projection connection with the communication terminal 300 . The controller 250 is configured to execute the above method for dynamically displaying the projection screen, as shown in FIG. 12 , which specifically includes the following content:

The number of connections of communication terminal 300 is detected. The display device 200 can detect the running state in real time during the running process, and detect the number of communication terminals 300 that have established a screen projection connection with the display device 200 during the process of determining that the screen projection screen needs to be triggered to be displayed. The number of connections of the communication terminal 300 can be obtained by detecting and judging each communication terminal in the wireless local area network where the display device 200 is located. For example, when it is necessary to detect the number of connections of communication terminals 300, the controller 250 of the display device 200 can use the communicator 220 to call all the communication terminals 300 connected to the current wireless network, and then capture the transmission data of each communication terminal , and identify the screen projection data flag therefrom, so as to detect whether the current communication terminal 300 establishes a communication connection with the display device 200 .

The number of connections can also be obtained by detecting the data table maintained by the display device 200 in real time. For example, in the database of the display device 200, a connected device list can be maintained in real time along with the screen projection connection process, and each communication terminal 300 in the connected device list can set different status values according to whether it is online or not and establishes a screen projection connection relationship , that is, when the communication terminal 300 is online and the connection is available, the status value is "devices online=1"; and when the communication terminal 300 is offline or the connection is unavailable, the status value is "devices online=0". Based on this, the display device 200 may traverse the number of communication terminals 300 with the status value "devices online=1" when the number of connections needs to be detected.

The display device 200 may perform detection of the number of communication terminals 300 when judging that the layout of the current screen projection interface needs to be changed. For example, the display device 200 can monitor the access status of each communication terminal 300 in the screen projection connection in real time, and when it detects that any communication terminal 300 is newly connected or exits the connection, the display device 200 can perform a connection detection function of the communication terminal 300. number of steps.

In some embodiments, the display device 200 may also periodically detect the number of connections of the communication terminal 300 . For example, the display device 200 may detect the number of connections of the communication terminals 300 through heartbeat commands of each communication terminal 300 after the screen projection connection is established. By detecting the number of connections of the communication terminal 300, the display device 200 can detect the connection status of the communication terminal 300 for screen projection in time, so that when the communication terminal is offline due to network range or other hardware failures, the screen projection interface can be adjusted. .

In some embodiments, the display device 200 may also detect the number of connections of the communication terminal 300 by detecting the user's control instructions and responding to some of the control instructions. For example, when the display device 200 detects that the user has input a control command for adjusting the page layout, the display device 200 may perform the step of detecting the number of connections of the communication terminal 300, so as to trigger the display device 200 to subsequently acquire the projection data. way of adjustment.

After detecting the number of connections of the communication terminal 300, the display device 200 may plan the layout of the projected screen according to the detected number of connections, so as to obtain display parameters of the projected screen. The display device 200 may have multiple layout templates built in, and after obtaining the number of connections, the display device 200 may match the corresponding template according to the number of connections.

For different forms of the display device 200 and different screen projection connection users, the built-in layout templates may also be different. For example, for the screen projection connection for sharing the demonstration operation of the main interface of the mobile phone, since the main interface of the mobile phone is mostly in the vertical screen state, the layout mode template can be a layout template obtained by dividing equidistantly in the vertical direction, that is, on the display device 200. The screen is divided into three vertical areas whose width is smaller than the height; and for the screencasting connection for sharing the game interface operation, since the game interface is mostly in the horizontal screen state, the layout template can be obtained by dividing equidistantly in the horizontal direction. That is, the screen of the display device 200 is divided into three horizontal regions whose width is greater than the height, for displaying the game screen shared by the mobile phone.

In some embodiments, when the display device 200 does not explicitly set a screen projection connection channel, in order to make full use of the display area, the display device 200 may plan the screen layout according to the principle of maximizing display. For example, when displaying 3 projection screen areas, one vertical area and two horizontal areas may be maintained. In this way, each display area can maintain a shape of 16:9 or 9:16, so as to adapt to the aspect ratio of communication terminal 300 .

In some embodiments, after detecting the number of connections of the communication terminal 300, the display device 200 may add a display window to the user interface to plan a layout based on the number of connections. For example, when it is detected that the number of communication terminals 300 with which the current display device 200 has established a screen projection connection is 2, two display windows may be added to the user interface for respectively displaying the screen projection images shared by each communication terminal 300 . Each display window may have a fixed shape and ratio, and the display device 200 may be required to provide a display image according to the ratio.

In some embodiments, when multiple display windows are used to display multiple screen projection images, if the number of communication terminals 300 that establish screen projection connections with the display device 200 is large, the multiple display windows can also be kept at a specific hierarchical relationship. For example, the hierarchical relationship of the display windows may be set according to the order in which the screen projection connection is established with the display device 200 , that is, the screen projection screen shared by the communication terminal 300 that establishes the screen projection connection last is displayed on the top layer.

After the display device 200 plans the layout of the projection screen, the display device 200 may also determine display parameters of the projection screen according to the planned layout. The display parameters are used to indicate the screen ratio and size of each display area in the layout mode planned by the display device 200, and the projection data requirements calculated according to the ratio and size of each display area, such as code rate, resolution etc.

According to the determined display parameters, the display device 200 may send a notification message to the communication terminal 300 corresponding to the screen projection screen. Wherein, the notification message includes a display parameter, and the notification message is used to enable the communication terminal 300 to generate screen projection data according to the display parameter. For example, as shown in FIG. 13 , when the display device 200 detects that the current number of connections to the communication terminal 300 is three, it can determine the display areas of the three projection screens by planning the layout, that is, the first area is the resolution “1080 ×1920” vertical area; area two and area three are horizontal areas with a resolution of “1280×720”. Based on the above display parameters, the display device 200 can generate a notification message, and send the generated notification message to the communication terminal 300, so that the communication terminal 300 can generate screen projection data according to the above resolution.

In order to generate screen projection data, the communication terminal 300 may have a built-in screen recording program, and after receiving the notification message sent by the display device 200, start the screen recording program to analyze the notification message and extract the display parameters. Then set the screen parameters in the screen recording program according to the display parameters. For example, when the display parameter is vertical and 1080×1920, the communication terminal 300 can control the screen recording process of the screen recording program so that the resolution of the generated screen recording video image is 1080×1920.

It should be noted that, if the display resolution of the communication terminal 300 does not match the resolution set in the notification message, the communication terminal 300 can process the screen by padding or cropping so that the generated screen projection data conforms to the resolution set in the notification message. Displays the form specified in the argument. For example, the screen ratio of each display area set in the display device 200 is 9:16, but when the screen ratio of the communication terminal 300 is 9:18, in order to be able to display completely, the communication terminal 300 can perform a reduction process on the display screen, so that The projection screen can be fully displayed in height. At this time, in the screen corresponding to the generated projection data, black areas will be filled on both sides to maintain the overall screen ratio of 9:16.

After the communication terminal 300 generates the projection data, it can feed back the generated projection data to the display device 200, and the display device 200 can display the received projection data, that is, display the images in the projection data according to the layout. For example, when the display device 200 respectively sends notification messages containing display parameters "vertical, 1080×1920", "horizontal, 1280×720", and "horizontal, 1280×720" to three communication terminals, the three communication terminals 300 can respectively feed back three projection data streams with resolutions of 1080×1920, 1280×720, and 1280×720, and display device 200 then parses and renders the received three projection data streams to finally form Cast screen.

It should be noted that, in order to simultaneously display multiple screen projection images on the display device 200, the display device 200 may create a data buffer area for each communication terminal 300 when receiving multiple screen projection data streams for Store the screen projection data sent by them, and each display control on the display device 200 obtains video data from the corresponding data buffer area for display. When a communication terminal 300 is disconnected, the display device 200 can also correspond to it. The data cache area is destroyed and released.

The method for dynamically displaying the projected screen provided by the above embodiment can plan the layout of multiple projected screens in the screen projecting interface by detecting the number of connections of the communication terminal 300, and send a message containing the display parameters of each area to each communication terminal 300 The notification message enables the communication terminal 300 to generate screen projection data in a manner matching the layout manner. Such a display mode can ensure that the picture presented by the final display device 200 can adapt to the layout requirements, alleviate the deformation of the picture, and improve the viewing experience of the user.

In some embodiments, the display device 200 may start detecting the number of connections of the communication terminal 300 when the communication terminal 300 is connected. That is, as shown in FIG. 14 , the display device 200 may acquire a connection instruction input by the user for establishing a screen projection connection, and initiate a screen projection connection to the communication terminal 300 in response to the connection instruction.

Wherein, the connection instruction can be input in different ways. For example, for the communication terminal 300 that establishes a screen projection connection with the display device 200 for the first time. The user can first connect to the wireless network where the display device 200 is located through network settings, and then perform interactive operations on the communication terminal 300, such as clicking the "screen projection" button to initiate a screen projection connection. The display device 200 can display the screen projection connection pairing interface after receiving the screen projection connection. At this time, the user needs to perform interactive operations again based on the screen projection connection pairing interface, such as selecting the corresponding communication in the screen projection device list. The terminal 300 enables the display device 200 and the communication terminal 300 to establish a screen projection connection. It can be seen that, for the communication terminal 300 connecting for the first time, the connection instruction is a combination of a series of interactive actions based on the display device 200 and the communication terminal 300 .

As another example, for the communication terminal 300 that is not the first to establish a screen projection connection with the display device 200, since the display device 200 and the communication terminal 300 can store each other's device information, the user can directly click "Screencast" on the communication terminal 300 The button initiates a screen projection operation. At this time, the display device 200 can also receive a screen projection connection instruction, and automatically responds to the connection instruction to implement a mutual handshake with the communication terminal 300 to complete the screen projection connection.

After the configuration of the screen projection connection is completed, the display device 200 may also traverse the number of connections of the communication terminal 300 . For example, after a communication terminal 300 initiates to establish a screen projection connection, the display device 200 may first wait for the configuration process of the screen projection connection. After the configuration of the screen projection connection is completed, the total number of communication terminals 300 currently connected is traversed. That is, the detected number of connections of the communication terminals 300 includes the communication terminals 300 that have newly established a screen projection connection.

In some embodiments, the display device 200 may also detect the number of connections when it detects that the communication terminal 300 stops the screen projection connection. As shown in FIG. , obtain the exit command input by the user for disconnecting the screen projection connection, and in response to the exit instruction, disconnect the screen projection connection of the communication terminal 300, and traverse the connections of the remaining communication terminals 300 after disconnecting the screen projection connection quantity.

For example, when three communication terminals 300 establish a screen projection connection with the display device 200 at the same time, the user can click the "stop screen projection" button on one of the communication terminals 300 to control the communication terminal 300 to no longer execute the screen recording program. At this time, the display device 200 may automatically disconnect the screen projection connection established with the communication terminal 300 when detecting that the communication terminal 300 no longer sends screen projection data. And after the screen projection connection is disconnected, it is detected that the number of connections of the remaining communication terminals 300 is 2. Therefore, according to the usage status of the two communication terminals 300, the layout mode in the screen projection interface is re-planned.

It can be seen that, in the above-mentioned embodiment, when a new communication terminal 300 establishes a screen projection connection, or when any communication terminal 300 quits screen projection, the number of screen projection images in the screen projection interface presented by the display device 200 changes, correspondingly The layout of the will also change. Therefore, the above method for detecting the number of connections can automatically adapt to the layout of the screen on the display device 200, adjust the displayed content in time, and improve user experience.

In some embodiments, after the display device 200 establishes a screen projection connection with the communication terminal 300, the display device 200 can also establish a screen projection data channel when initiating a screen projection connection to the communication terminal 300, and the screen projection data channel can be established successfully After that, create an indicator to display the projection data received through the projection data channel through the indicator.

Wherein, the display control is a display control program built in the display device 200, such as surface View or Texture View. Indicators can be added to display in different ways. For example, when the display device 200 includes one display control, the display control may be displayed in a full-screen manner. When the display device 200 includes two display controls, the two display controls are in the form of two vertically arranged windows on the left and right.

As shown in FIG. 16 , in some embodiments, in order to determine the display parameters, the display device 200 can first detect the screen size information of the display 260 when planning the layout of the projected screen according to the number of connections, and then use the number of connections and screen The size information is an index to match the layout template in the database, so as to extract the display parameters in the layout template.

For different types of display devices 200 , the screen size information of the display 260 is different, and thus the layout methods conforming to the screen specifications are also different. For example, for smart TVs with conventional display ratios, in order to make full use of the display space, when displaying three projection screens, one vertical and two horizontal display area layouts can be adapted; while for ultra-widescreen movie playback devices , when displaying 3 projection screens, the layout of the display area can be vertically divided into thirds.

In addition, in the display device 200 or the server 400, a database may be operated and maintained, and layout templates corresponding to various screen sizes and connection quantities may be stored in the database. Therefore, the display device 200 can determine the best layout mode suitable for the current display state according to the screen size information and the number of connections.

In some embodiments, the display device 200 may also determine the final layout mode according to the matching result of the layout template. That is, in the step of matching the layout template in the database, if no layout template is matched in the database, the layout method can be determined directly by using the matched layout template; and if no layout template is matched in the database, the display area can be Perform zoning to determine the appropriate layout. That is, the display device 200 can acquire the device parameters of the communication terminal 300, divide the display area into sub-regions with the same number of connections according to the device parameters, and then traverse the graphic parameters of each sub-region to generate display parameters.

For example, when the layout template is not matched in the database, the display device 200 may first detect the device parameters of the communication terminal 300 that has currently established a screen projection connection, such as model, screen size and specification. The display device 200 then determines the corresponding display area shape according to the screen size specifications of all the connected communication terminals 300 . Obviously, in order to obtain a better display effect, the shape of the display area may be similar to the shape of the screen of the communication terminal 300 . The display device 200 then divides the display area according to the determined shape of the display area, so as to maximize the utilization of the display space under the premise of ensuring the integrity of the display content.

In some embodiments, the display device 200 may also extract display parameters and generate a notification message according to the display parameters in the step of sending a notification message to the communication terminal 300 corresponding to the projected screen. Wherein, the display parameters include the display direction of each display window. The display direction is used to instruct the communication terminal to generate screen projection data according to a set direction. In order to adapt to the placement state of the communication terminal 300, the display device 200 may preferentially match the corresponding communication terminal 300 according to the shape of each display area after determining the layout mode. For example, when the display direction of the display window is vertical, the projection screen of the communication terminal 300 in the vertical screen state can be preferentially displayed in this window; when the display direction of the display window is horizontal, it can be preferentially displayed in this window. A projected screen image of the communication terminal 300 in a horizontal screen state.

Since the larger the number of projection screens contained in the user interface, the smaller the area occupied by each display area, the worse the final image effect presented in each display area, and when the display area is small, the display Areas cannot obtain corresponding effects for high-quality display content. Therefore, in order to reduce the total data transmission volume, after detecting the number of connections of the communication terminal 300, the display device 200 can also determine the image quality of the projection data according to the number of connections. That is, the display parameters also include the code rate and resolution of each display window, so that the communication terminal 300 can generate screen projection data with reference to the set code rate and resolution.

Obviously, when the number of communication terminals 300 that establish screen projection connections with the display device 200 is larger, the range of each corresponding display area is smaller, and the code rate and resolution can be set to be smaller, so that the transmission data of the projection data The smaller the amount, and vice versa. Wherein, there may be a one-to-one correspondence or a many-to-one correspondence between the code rate and the number of connections. For example, when the display device 200 is connected to 1 communication terminal 300 or 2 communication terminals 300 at the same time, the transmission code rate can be set to 5Mb/s; if the display device 200 is connected to 3 communication terminals 300 or 4 communication terminals 300 at the same time, it can be set The transmission code rate is 3Mb/s.

The resolution may be the same as the number of pixels occupied by each display area in the width and height directions after the display area is divided. For example, when the number of pixels occupied by the first display area is 1080×1920, the resolution of the corresponding projection data may also be 1080×1920. When the number of pixels occupied by the second display region and the third display region is 1280×720, the resolution of the corresponding projection data may also be 1280×720.

After the display parameters are determined, the notification message generated according to the display parameters also needs to include the above parameter content, that is, the notification message includes the display direction, code rate and resolution, so that the communication terminal 300 can, after receiving the notification message, The display direction, bit rate and resolution can be analyzed from the notification message, and the projection data can be generated according to the above parameters to ensure the consistency between the projection screen and each display area.

It should be noted that due to the limitation of the layout mode, even when the placement state of the communication terminal 300 does not match the display direction corresponding to the display area during the process of generating the projection data, the communication terminal 300 still generates the projection according to the shape of the display area. data. For example, when the display area is in a vertical shape and the corresponding communication terminal 300 is in a horizontal screen state, the communication terminal 300 still generates a screen projection image according to the vertical shape, but in the process of generating screen projection data, it can be displayed at the top of the display screen. Black areas are added to the bottom and bottom respectively, so that the overall screen appears as a vertical screen with a width smaller than the height, so that the screen projection screen can be displayed in the vertical shape display area without stretching and deforming.

However, when the display area does not match the placement state, it is easy to cause most of the screen in the display area to be displayed as a background filled with a solid color. Therefore, in some embodiments, when the display device 200 detects that the display area does not match the placement state, The user is prompted to adjust the placement state of the communication terminal 300 through the prompt information. As shown in FIG. 17, the display device 200 may perform the following steps:

Step S171, detecting the effective frame range of each screen-casting screen in the screen-casting interface;

Step S172, calculating the display ratio of the effective picture area to the total area of the display window;

Step S173, if the display ratio is less than or equal to the ratio threshold, that is, it can be determined that the display area does not match the placement state, then execute step S174, otherwise execute step S175;

Step S174, sending a prompt message to the communication terminal 300 to prompt the user to rotate the direction manually or automatically through the internal mechanism of the communication terminal 300;

Step S175, end the process.

For example, when the display area is in a vertical shape and the corresponding communication terminal 300 is in a horizontal screen state, the display device 200 can detect that the display ratio is less than or equal to the ratio threshold, so it can send a prompt message to the communication terminal 300, such as "the current display The orientation does not match, please turn the mobile phone to the vertical screen state", prompting the user to adjust the communication terminal 300 to the tree screen state, and increase the display ratio of the effective screen in the projection data.

Based on the display device 200 above, a communication terminal 300 is also provided in some embodiments of the present application, including: a display unit 330 , a communication unit, and a processor 380 . Wherein, the display unit 330 is configured to display a terminal interface, and the communication unit is configured to establish a screen projection connection with the display device 200; as shown in FIG. 18 , the processor 380 is configured to perform the following steps:

Step 181: Receive a notification message sent by the display device 200, the notification message includes the display parameters of the screen projection screen, and the display parameters are when the display device 200 plans the layout mode of the screen projection screen according to the current number of communication terminals 300 connections get;

Step 182, generating projection data according to the display parameters;

Step 183 , feeding back the screen projection data to the display device 200 .

Based on the above-mentioned display device 200 and communication terminal 300, some embodiments of the present application also provide a method for dynamically displaying a projection screen, including the following steps:

The display device 200 detects the number of connections of the communication terminal 300, and plans the layout of the projected screen according to the number of connections, so as to obtain the display parameters of the projected screen;

The display device 200 sends a notification message to the communication terminal 300 corresponding to the projection screen, and the notification message includes the display parameter;

The communication terminal 300 receives the notification message sent by the display device 200, and generates screen projection data according to the display parameters;

The communication terminal 300 feeds back the screen projection data to the display device 200;

The display device 200 displays the screen projection data according to the layout.

The method for dynamically displaying the projected screen provided by the above embodiment can plan the layout of the projected screen by detecting the number of connected communication terminals 300 when the user triggers a change in the number of communication terminals 300 connected to the display device 200 . Then send a notification message including the display parameters of the screen projection to the communication terminal 300 , so that the communication terminal 300 can generate screen projection data conforming to the layout according to the display parameters, and display it on the display device 200 . The method can enable the display device 200 to support multi-channel screen projection, dynamically adjust the display layout of the screen projection screen, and notify the communication terminal 300 to adjust the screen recording, so as to make the layout of the screen projection screen displayed more reasonable.

The above embodiments introduce the hardware/software architecture and function realization of the display device. In some application scenarios, the terminal device can push media data to the display device. For example, the user can project the video of interest to the display device for large-screen playback through some application programs with screen projection function; or the terminal device and the display device Establish a screen projection connection through certain protocols, such as DLNA (Digital Living Network Alliance, Digital Living Network Alliance) protocol, etc. After the terminal device and the display device are successfully connected for screen projection, the terminal device can project the screen projection data to the display device, and the display device will display the projection data on the screen projection interface. Wherein, the terminal device may be a mobile terminal such as a smart phone, a tablet computer, or a notebook computer, or may be a desktop computer or other small-screen display devices.

In some embodiments, when a terminal device is playing media data, the user can select a device from the device list for playing the media data on the screen, and the display device that supports the screen projection function will be identified and automatically added to the device list In , after the user selects the display device from the device list, the media data can be projected from the terminal device to the display device for playback. When using the projection method to push media data, the terminal device and the display device can be connected to the same Wifi network, and the method for the terminal to push media data to the display device is not limited. Of course, in other technologies, the screen projection method can also be performed across networks. In this application, the specific implementation manner of screen projection is not limited.

In some embodiments, the terminal device A sends a screen projection request to the display device, and the display device can ask the user whether to agree to the screen projection of the terminal device A through a UI inquiry pop-up window, etc., and can provide several option controls, such as "this consent", "disagree" and "permanent consent". The user chooses "Permanent Agree", which is equivalent to permanently authorizing terminal device A. When terminal device A initiates a screen projection request at will, there is no need to display the UI query pop-up window, but directly establishes a projection between the display device and terminal device A. Screen connection; if the user selects "Agree this time", it is equivalent to authorizing terminal device A once, and the display device and terminal device A will establish a screen projection connection, but the screen projection permission of terminal device A is only valid for that time, subsequent terminals When device A initiates a screen projection request, it still needs to display the UI query pop-up window again; if the user chooses "disagree", it means that terminal device A is not authorized at that time, and the display device and terminal device A do not establish a screen projection connection. When terminal device A initiates a screen projection request, it still needs to display the UI query pop-up window again.

In some embodiments, the display device can maintain a list of trusted devices, and supports users to add, delete and modify device information with screen projection permission in the list of trusted devices. Device information includes but not limited to device ID, MAC Address, etc., that is, the user can customize the authorization list on the display device side. When the display device receives the screen projection request sent by the terminal device B, it queries whether the terminal device B is included in the trusted device list according to the device information carried in the screen projection request. If terminal device B is included in the list of trusted devices, a screen projection connection between terminal device B and the display device is established; if terminal device B is not included in the list of trusted devices, the display device does not respond to the screen projection request, that is, no connection with The screen projection connection of terminal device B. It should be noted that the authorization and connection mode of screen projection by the terminal device are not limited to the examples in this application.

In the current application scenarios, the one-to-one projection mode is mostly used, that is, the display device is only connected to one terminal device for projection, and the projection data can be exclusively displayed on the screen. However, in other application scenarios, for example, when the user wants to watch the tutorial or commentary of the game synchronously while playing a screen-casting game on terminal device C, the terminal device D can initiate a screen-casting request for the game tutorial/commentary video to the display device , after the screen projection connection between the display device and the terminal device D is successful, it is equivalent to accessing two channels of projection data; for another example, a family includes member 1, member 2 and member 3, and the three members can use their respective terminals to send The display device initiates a screen projection request, and after the display device responds to three screen projection requests, it displays the screens of each terminal on the screen projection interface, and the three members can interact and communicate, etc. At this time, the display device accesses three channels of screen projection data.

For the application scenario where the display device is connected to multiple screen projections, multiple screen projection windows can be displayed on the screen interface. Each screen projection window corresponds to a terminal device that is successfully connected to the screen projection, and displays the screen projection data sent by the terminal device. However, the display effect of multiple screen projection windows on the screen projection interface affects the user's perception, and there are many display problems: for example, the arrangement of multiple screen projection windows is not compact enough, resulting in a lot of blank space in the screen projection interface; The distribution of the screen projection windows is chaotic, and the overall layout is not regular; some screen projection windows are small in size, so that users cannot clearly see the details of the video in the window; If it is changed to a vertical screen display, the video display uniformity between the terminal device and the display device is poor, and so on. In this regard, the present application aims to plan and lay out multiple screen projection windows to improve the display effect of the screen projection interface.

Referring to FIG. 19 , an embodiment of the present application provides a multi-channel projection display method, the method is configured to be executed by the controller 250 in the display device, that is, the controller 250 is the execution subject of the method, and the method includes the following steps :

Step S01, according to the currently counted number of M screen projection channels and the media configuration information respectively received from the M terminal devices, match the layout among the M screen projection windows suitable for the screen;

Step S02, display M screen projection windows according to the layout in the screen projection interface, and display the screen projection data sent by the corresponding terminal device in each screen projection window.

Among them, M is used to indicate the number of screen projection channels, that is, the number of terminal devices that have established screen projection connections with the display device. The number of screen projection channels will change dynamically with the connection and disconnection of screen projection connections. The numbers are counted in order to accurately plan the typesetting and layout of the screencasting interface. When M is 1, single-channel projection is performed; when M is greater than or equal to 2, multiple-channel projection is performed; M can be zero, indicating that the display device has not received the projection data sent by the connected device, and the projection will not be displayed The interface is also equivalent to an empty layout on the screen projection interface.

In some embodiments, each time the display device receives a screen projection request from a target terminal device, if it creates a new screen projection connection with the target terminal device, it will cumulatively add one to the count value of the number of screen projection channels; in addition, the terminal device may Due to factors such as disconnection of the network or the terminal user’s intentional exit from projection, the display device is disconnected from the projection connection, resulting in the display device being unable to receive the projection data sent by the terminal device, and the display device needs to cut off the projection channel. When it is detected that the screen projection connection with a terminal device is disconnected, the count value of the number of screen projection channels is decremented by one. Through this embodiment, the number of screen projection channels can be automatically counted according to the screen projection connection dynamics of the display device.

In some embodiments, each time the display device receives the screen projection data of a target terminal device, the count value of the number of screen projection ways is cumulatively increased by one.

In this application, in addition to receiving the projection data from the M terminal devices, the display device side will also synchronously receive the media configuration information sent by the M terminal devices respectively.

In some embodiments, the screen projection data is related to the media category. For example, if the media is a pure video category, the screen projection data includes the video stream to be screened; if the media is an audio and video category, the screen projection data includes video streams and Audio stream; in addition, the projection data can also be in the form of images (static images or moving pictures, etc.), text or pure audio streams, etc. When the screencasting data is an audio stream, the display interface can display playback content information such as audio.

In some embodiments, the screen projection data is related to the screen recording mode of the terminal device. For example, the screen recording method can be recorded according to a fixed resolution, such as 1280*720, 1920*1080, etc., and can also be recorded according to the size of the terminal screen, such as 16:9 size, 17:9 size, 21:9 size, etc.

In some embodiments, the media configuration information includes the aspect ratio of the video projected by the terminal device, that is, the ratio of the horizontal width to the vertical height when displaying the video. The aspect ratio can be used to identify the display orientation. When the aspect ratio is greater than When or is equal to 1, that is, the width is greater than or equal to the height, the display mode is displayed in landscape mode; when the aspect ratio is less than 1, that is, the width is smaller than the height, the display mode is displayed in portrait mode. The creation of the projection window can be constrained according to the aspect ratio and screen orientation indicated in the media configuration information, so that the projection window and the corresponding terminal device can maintain the same aspect ratio and display orientation, which can avoid media correlation after projection. Misconfiguration problem.

In some embodiments, the media configuration information may also include a screen aspect ratio of the terminal device. When the aspect ratio is greater than or equal to 1, that is, the width is greater than or equal to the height, the display mode is in landscape mode; when the aspect ratio is less than 1, that is, the width is smaller than the height, the display mode is in portrait mode.

In some embodiments, the media configuration information may further include a media category. After the display device reads the media configuration information, it may know the media category to be screen-casted, so as to display screen-casting windows with different effects according to the media category. For example, for pure video, audio and video, and image media, the projection window can display visualized projection data, and the projection window should match the aspect ratio and screen orientation of the video/image; for text-based media, the projection window can Visually display readable text information, but the aspect ratio and screen orientation of the projection window are not limited; for audio-only media, the projection window cannot display visual images, and voice playback controls can be set in the projection window (such as small Speaker), when playing the audio stream of the projection screen, the voice playback control can be presented as a dynamic special effect of "playing". Similarly, the aspect ratio and screen orientation of the projection window corresponding to the pure audio stream are not limited.

Optionally, for a projection window with an unlimited aspect ratio and screen orientation under a special media category, a preset window layout can be used. The preset window layout refers to a preset window layout with a fixed aspect ratio (or a fixed Width and height) and screen orientation, after identifying the media to be screened as text or pure audio, directly call the preset window layout, and then load the screen projection data into the window layout; Instead of setting the window layout, priority is given to creating screen-adapting screen-casting windows for media with limited aspect ratios and screen orientations such as pure video, audio, video, and images, and then creating and displaying appropriate screen-casting windows in the blank area of the screen-casting interface. screen projection window to display text and audio-only projection data. This method can preferentially match the limited projection window, and then maximize the screen display area, making multiple projection windows more compact and reducing retention time. white area. It should be noted that the creation of screen projection windows and UI display settings under different media types are not limited to the embodiments of this application.

In step S01, M sets of size information can be calculated according to the screen resolution of the display device or the aspect ratio and screen direction of the video projected by M terminal devices, and the number of screen projection channels, and the size information includes the width windowWidthi And the height windowHeighti, that is to construct windowWidthi*windowHeighti, where i is used to identify the size information/serial number of the projection window, 1≤i≤M, and there is a mapping relationship between the value of i and the device information of the terminal device. For example, M=3, serial number 1 corresponds to terminal device A, serial number 2 corresponds to terminal device B, and serial number 3 corresponds to terminal device C, then when i is equal to 1, it indicates that screen projection window 1 created according to the size information of the first group is used for The screen projection data of the terminal device A is displayed, so that the screen projection data from which terminal device should be kept displayed in each screen projection window can be accurately identified.

In some examples, the sequence numbers may be allocated according to the time sequence of the screen projection connection, for example, the earlier the screen projection is connected, the smaller the sequence number. For example, using a pre-typesetting method, the screen projection screen is pre-typeset at the moment of screen projection connection, and after receiving the screen projection data, the screen is displayed according to the pre-typesetting method.

In another example, the sequence numbers may be allocated according to the order in which the screen projection data is received, for example, typesetting is performed according to the data stream received in real time.

In the above exemplary implementation manner, the media configuration information of the display device may be to record video at a fixed resolution, such as 1920*1080, 1280*720, and so on.

In the exemplary implementation described above, the media configuration information of the display device may record video according to the ratio of the screen, and the ratio of the current screen is not the ratio of 16:9/9:16 of the large-screen terminal, but other ratios such as 17:9, 21:16, etc. In an exemplary implementation manner, M screen projection windows are constructed according to M sets of size information, and the screen projection windows and their corresponding terminal devices maintain the same aspect ratio and display orientation. Since each set of size information defines a specific width and height of a screen projection window, a screen projection window matching the size information can be constructed. After the sizes of the M screen projection windows are all determined, it is also necessary to plan a specific display position of each screen projection window on the screen interface.

In some embodiments, the position distribution information is calculated according to the media configuration information received from the terminal device, the position distribution information includes the coordinates (xi, yi) of a preset reference point Pointi on the M screen projection windows, and the preset It is assumed that the coordinates (xi, yi) of the reference point Pointi are used to locate the position of the screen projection window on the screen. Since the screen projection window is generally a rectangle, some key points in the rectangle can be selected by the preset reference point. Optionally, the preset reference point is any one of the four vertices of the rectangle or the center point of the rectangle, etc. The selection principle is to Subject to calculation and positioning. Finally, according to the calculated location distribution information, the created M screen projection windows are positioned and arranged, so that the sizes and positions of the M screen projection windows are determined and the layout is formed.

The following will describe with a specific UI example. Figure 20(a) exemplarily shows the screen projection interface in the scenario where the number of screen projection channels is 1 and the video stream is a single vertical screen. The screen resolution of the display device is 1920*1080 (that is, the default aspect ratio of the display device screen is 16:9), and the media configuration file sent by terminal device A indicates that the aspect ratio is 9:16 (vertical screen) as an example, then The screen projection window 1 to be created is also a vertical screen, and the height of the screen projection window 1 is adapted to the height of the screen, so the width of the screen projection window 1 is 1080*9/16=607.5, that is, windowWidth1*windowHeight1=607.5*1080. When determining the position of the screen projection window 1, use the top left corner of the screen as the origin, the positive direction of the x-axis to the right, and the positive direction of the y-axis to construct a coordinate system, and take the top left corner of the screen projection window 1 as the For the preset reference point Point1, considering that the display mode with a single window centered in vertical screen is better in viewing effect, the coordinates (x1, y1)=(656.25, 0) of the preset reference point Point1 are calculated. Since the coordinate position of the preset reference point Point1 and the size of the screen projection window 1 have been determined, the position of the screen projection window 1 in the overall screen projection interface is uniquely determined, and the resulting layout is shown in Figure 20(a) Show.

Figure 20(b) exemplarily shows the screen projection interface in the scenario where the number of screen projection channels is 1 and the video stream is a single horizontal screen. The screen resolution of the display device is 1920*1080 (that is, the default aspect ratio of the display device screen is 16:9), and the media configuration file sent by terminal device A indicates that the aspect ratio is 16:9 (horizontal screen), for example, then The projection window 1 to be created is also a horizontal screen. Since it is a single-window display mode, and the aspect ratio of the projection window 1 and the screen is fully adapted, the projection window 1 can be displayed in full screen, that is, windowWidth1*windowHeight1 =1920*1080, taking the upper left corner vertex of the screen projection window 1 as the preset reference point Point1, then the coordinates of the preset reference point Point1 (x1, y1)=(0, 0), the resulting layout is shown in Figure 20 ( b) as shown.

Fig. 21(a) and Fig. 21(b) exemplarily show two kinds of screen-casting interfaces in the scene where the number of screen-casting channels is 2 and the scene is double landscape. Fig. 21(a) is the screen-casting window of two landscape Arrange horizontally side by side. Figure 21(b) is to arrange two horizontal screen projection windows vertically. At this time, the display device has been connected to the two-way projection screen initiated by terminal device A and terminal device B to display The screen resolution of the device is 1920*1080, and the media configuration files sent by terminal device A and terminal device B indicate that the aspect ratio is 16:9 (horizontal screen) as an example, then the created projection window 1 (corresponding to the A) and screen projection window 2 (corresponding to terminal device B) are also horizontal screens.

For the example in Figure 21(a), since they are arranged side by side horizontally, in order to maximize the screen area occupied, the sum of the widths of the screen projection window 1 and the screen projection window 2 is equal to the screen width, that is, windowWidth1+windowWidth2=1920, in addition In order to make the size of the dual screen projection window more balanced, you can set windowWidth1=windowWidth2=960, then windowHeight1=windowHeight2=960*9/16=540, that is, windowWidth1*windowHeight1=windowWidth2*windowHeight2=960*540, to screen The upper left corner of window 1 is used as the preset reference point Point1, and the upper left corner of the screen projection window 2 is used as the preset reference point Point2. Considering that the display mode of dual windows arranged in a horizontal screen and vertically centered is better in terms of viewing effect, Thus, the coordinates (x1, y1)=(0, 270) of the preset reference point Point1 are calculated, and the coordinates (x2, y2)=(960, 270) of the preset reference point Point2 are calculated. The resulting layout is shown in Figure 21 (a) shown.

For Figure 21(b), since they are vertically arranged side by side, in order to maximize the screen area occupied, the sum of the heights of the screen projection window 1 and the screen projection window 2 is equal to the screen height, that is, windowHeight1+windowHeight2=1080. The size of the dual projection screen window is more balanced. You can set windowHeight1=windowHeight2=540, then windowWidth1=windowWidth2=540*16/9=960, that is, windowWidth1*windowHeight1=windowWidth2*windowHeight2=960*540, using the projection window 1 The upper left corner of the screen is used as the default reference point Point1, and the upper left corner of the screen projection window 2 is used as the preset reference point Point2. Considering that the display mode of the dual windows side by side and horizontally centered in the horizontal screen is better in viewing effect, thus Calculate the coordinates (x1, y1)=(480,0) of the preset reference point Point1, and the coordinates (x2, y2)=(480,540) of the preset reference point Point2, the resulting layout is shown in Figure 21(b ) shown.

In some embodiments, after matching and generating the layout, during the process of displaying multiple screen projection windows according to the layout on the screen projection interface and loading and displaying the screen projection data of each channel, there may be new screen projection channels accessed at any time, or there may be At least one terminal device is disconnected from the screen projection connection, that is, the number of screen projection channels may change during this period, and the layout needs to be updated according to the changed screen projection channels to realize adaptive adjustment of the layout following the screen projection channels. Here The adjustment involves the number of screen projection windows in the layout, the size of each screen projection window, and the relative position distribution of each screen projection window.

For example, in the scenario shown in Figure 20(b), the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 16:9 (horizontal screen), the controller 250 detects that the number of projection screens is changed from 1 to 2, then the layout can be updated to the version shown in Figure 21(a) or Figure 21(b); for example, in Figure 21(a) or Figure 21(b) In the scenario, terminal device B suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1, and can update the layout to the version shown in Figure 20(b).

Fig. 21(c) and Fig. 21(d) are two kinds of screen-casting interfaces in the scene where the number of screen-casting channels is 2 and one horizontal screen and one vertical screen according to some embodiments, and Fig. 21(c) shows the horizontal screen The screen projection window is displayed on the left, and the screen projection window for the vertical screen is displayed on the right; Figure 21(d) shows the screen projection window for the vertical screen on the left, and the screen projection window for the landscape screen is displayed on the right. At this time, the display device has connected to the two-way screen projection initiated by terminal device A and terminal device B. The screen resolution of the display device is 1920*1080, and the media configuration file sent by terminal device A indicates that the aspect ratio is 16:9 ( Horizontal screen), the media configuration file sent by terminal device B indicates that the aspect ratio is 9:16 (vertical screen) as an example, then the created projection window 1 (corresponding to terminal device A) is a horizontal Corresponding to terminal device B) is a vertical screen.

For Figure 21(c), since two windows are arranged side by side, one horizontal and one vertical, in order to maximize the screen area occupied, the sum of the widths of screen projection window 1 and screen projection window 2 is equal to the screen width, namely windowWidth1+windowWidth2 =1920, make the height of the projection window 2 of the vertical screen fit the height of the screen, that is, windowHeight2=1080, then windowWidth2=1080*9/16=607.5, windowWidth1=1920-607.5=1312.5, windowHeight1=1312.5*9/16= 738.3, then windowWidth1*windowHeight1＝1312.5*738.3, windowWidth2*windowHeight2＝607.5*1080, take the upper left corner of the screen projection window 1 as the preset reference point Point1, and use the upper left corner of the screen projection window 2 as the preset reference point Point2 , considering that the viewing effect of the projection window 1 is better in the vertically centered display mode, the coordinates of the preset reference point Point1 are calculated (x1, y1)=(0, 170.9), and the coordinates of the preset reference point Point2 are calculated (x2, y2) = (1312.5, 0), the resulting layout is shown in Figure 21(c).

The difference between Figure 21(d) and Figure 21(c) is that the horizontal screen and the vertical screen are distributed on different sides in the horizontal direction, so the size of the projection window 1 and the projection window 2 are the same, and the coordinates of Point1 and Point2 different. That is, windowWidth1*windowHeight1＝1312.5*738.3, windowWidth2*window Height2＝607.5*1080, the upper left corner of the screen projection window 1 is used as the preset reference point Point1, and the upper left corner of the screen projection window 2 is used as the preset reference point Point2, Considering that the viewing effect of the projection screen window 1 is better in the vertically centered display mode, the coordinates (x2, y2)=(0, 0) of the preset reference point Point2 are calculated, and the coordinates of the preset reference point Point1 ( x1, y1) = (607.5, 170.9), and the resulting layout is shown in Fig. 21(d).

In some examples, for example, in the single-way vertical screen scenario in FIG. 20(a), the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 16: 9 (horizontal screen), the controller 250 detects that the number of projection screens has changed from 1 to 2, then the layout can be updated to a dual-channel-horizontal-vertical screen similar to that shown in Figure 21(c) or Figure 21(d) screen version.

In some examples, for example, in the single-channel landscape scenario of FIG. 20(b), the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 9: 16 (vertical screen), the controller 250 detects that the number of projection screens has been changed from 1 to 2, and the layout can be updated to a dual-channel-horizontal-vertical screen as shown in Figure 21(c) or Figure 21(d) Version.

In some examples, for example, in the scenario shown in Figure 21(c) or Figure 21(d), terminal device B suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1 , the layout can be updated to the version of the single horizontal screen window (screen projection window 1) full-screen exclusive screen shown in FIG. 20(b).

In some examples, for example, in the scenario shown in Figure 21(c) or Figure 21(d), terminal device A suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1 , the layout can be updated to a version similar to that shown in Figure 20(a) in which the single vertical screen window (screen projection window 2) monopolizes the screen.

Fig. 21(e) exemplarily shows the screen-casting interface in a dual-vertical-screen scene with two screen-casting channels. The screen resolution of the display device is 1920*1080, and the media configuration files sent by terminal device A and terminal device B indicate that the aspect ratio is 9:16 (vertical screen) as an example, then the created screen projection window 1 (corresponding to terminal Device A) and screen projection window 2 (corresponding to terminal device B) are also vertical screens.

For Figure 21(e), since the screen projection window 1 and the screen projection window 2 are both vertical screens, the heights of the two screen projection windows are adapted to the height of the screen, that is, windowHeight1=windowHeight2=1080, and windowWidth1 can be calculated from this =windowWidth2=1080*9/16=607.5, windowWidth1*windowHeight1=windowWidth2*windowHeight2=607.5*1080. The upper left corner of the screen projection window 1 is used as the preset reference point Point1, and the upper left corner of the screen projection window 2 is used as the preset reference point Point2. Considering that the two vertical screen windows are evenly spaced along the horizontal direction, the viewing effect is better. Good, then the distance value dx calculated is (1920-2*607.5)/3=235, then the coordinates (x1, y1)=(235, 0) of the preset reference point Point1, the coordinates of the preset reference point Point2 (x2 , y2)=(2*dx+607.5,0)=(1077.5,0), and the resulting layout is shown in Figure 21(e).

In some examples, for example, in the single vertical screen scenario in FIG. 20(a), the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 9: 16 (vertical screen), the controller 250 detects that the number of projection screens has been changed from 1 to 2, and the layout can be updated to the version with two channels and two vertical screens as shown in FIG. 21(e).

In some examples, for example, in the dual-channel dual-vertical screen scenario of 21(e), terminal device A or terminal device B disconnects the screen projection connection, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1, then it can The layout is updated to a version similar to that shown in Figure 20(a) where a single vertical window occupies the screen.

In some examples, in any of the scenarios in Figures 20(a) to 20(b) and 21(a) to 21(e), if all screen projection connections are disconnected, the controller detects that When the number of screen projection channels becomes 0, the screen projection interface will be exited, and the UI display interface will return to that before screen projection.

The above takes single-channel and dual-channel projection as examples to explain the layout construction and planning process in detail. When the number of projection channels M is greater than or equal to 3, the adaptability of the layout matching mechanism can refer to the previous example. The layout matching can be It has the characteristics of adaptability, flexibility and diversity, etc., and this application will not list and repeat them one by one. When the number of screen projection channels, screen orientation and width-to-height ratio of each screen projection media are the same, the layout forms of each screen projection window are different, resulting in a variety of layouts. 21(a) and 21(b), the aforementioned two-way-horizontal-vertical screen scenario includes at least the two versions of FIG. 21(c) and 21(d).

In some embodiments, when multiple versions of the layout are matched, the present application performs optimal screening based on the principle of maximum area, that is, calculates the sum of the areas of all projection windows in each version of the layout, and selects the target with the largest sum of areas Layout, that is, Smax=Max{k=1,2...,Q∣Sk}, where k is used to represent the version number of the layout, Q is the number of versions of the matched layout, 1≤k≤Q,

In the formula, si is the area of the i-th screen projection window in the layout of the k-th version, and M is the total number of screen projection windows in all versions of the layout. In this way, the maximum area sum Smax is calculated, and then the version corresponding to Smax is used as the selected target layout, and the target layout is displayed on the screen projection interface, so that each projection window can occupy the screen more fully and to the maximum extent, and each projection screen The layout between the windows is compact, and the blank area is reduced as much as possible to improve the display effect of the screen projection interface. Since the screen projection window is rectangular, the area of a single screen projection window is equal to the product of its width and height.

As an example, in a dual-channel dual-horizontal screen scenario, the screen resolution of the display device is 1920*1080, the media configuration file sent by terminal device A indicates that the aspect ratio is 17:9 (horizontal screen), and terminal device B sends As an example, the media configuration file indicated that the ratio of width to height is 16:9 (horizontal screen), that is, the media of the two screens have different ratios of width to height, and two screen projection windows are constructed. Screen projection window 1 corresponds to terminal device A. Screen projection window 2 corresponds to terminal device B.

As shown in Figure 22(a), version 1 of the layout is arranged side by side in a horizontal direction. In order to maximize the occupation of the screen area, the sum of the widths of the screen projection window 1 and the screen projection window 2 is equal to the screen width, that is, windowWidth1+windowWidth2 =1920, in addition, in order to make the size of the dual projection screen window more balanced, you can set windowWidth1=windowWidth2=960, then windowHeight1=960*9/17=508, windowHeight2=960*9/16=540, windowWidth1*windowHeight1=960 *508, windowWidth2*windowHeight2=960*540, then for version 1, the sum of the areas of the screen projection windows S1=960*508+960*540=1006080.

Figure 22(b) shows version 2 of the layout. Since they are arranged side by side vertically, in order to maximize the screen area occupied, the sum of the heights of screen projection window 1 and screen projection window 2 is equal to the height of the screen, that is, windowHeight1+windowHeight2= 1080, and to make the size of the dual projection screen more balanced, you can set windowHeight1=windowHeight2=540, windowWidth1=540*17/9=1020, windowWidth2=540*16/9=960, you can get windowWidth1*windowHeight1= 1020*540, windowWidth2*windowHeight2=960*540, then for version 2, the sum of the areas of the screen projection windows S2=1020*540+960*540=1069200. It is obvious by comparison that S2 is larger than S1, so version 2 is taken as the target layout for the final display.

Figure 22(c) exemplarily shows a screen-casting interface in a dual-vertical-screen scene with two screen-casting channels. In Figure 22(c), the screen-casting windows of two vertical screens are arranged side by side in the horizontal direction , and the screen recording resolutions of the two terminal devices are different. For example, the screen recording resolution of terminal device A is 1920*1080, and the resolution of terminal device B’s screen recording is 2040*1080. The media sent by terminal device A and terminal device B The configuration file indicates that the aspect ratio is 9:16 (vertical screen) and 9:17 (vertical screen), then the created screen projection window 1 (corresponding to terminal device A) and screen projection window 2 (corresponding to terminal device B) Both are vertical screens, but the horizontal display sizes of the two projection windows are different.

For Figure 22(c), since the screen projection window 1 and the screen projection window 2 are both vertical screens, the heights of the two screen projection windows are adapted to the height of the screen, that is, windowHeight1=windowHeight2=1080, and windowWidth1 can be calculated from this ＝1080*9/16＝607.5, windowWidth2＝1080*9/17＝571.7, the upper left corner of the screen projection window 1 is used as the default reference point Point1, and the upper left corner of the screen projection window 2 is used as the preset reference point Point2, Considering that the viewing effect is better when the two vertical screen windows are evenly spaced along the horizontal direction, the calculated spacing value dx is (1920-607.5-571.7)/3=247, and the coordinates of the preset reference point Point1 (x1, y1)=(247,0), the coordinates (x2,y2)=(2*dx+607.5,0)=(1101.5,0) of the preset reference point Point2, the resulting layout is shown in Figure 22(c) Show.

Figure 22(d) exemplarily shows another screen-casting interface in a dual-vertical-screen scene with the number of screen-casting channels being 2. In Figure 22(d), the screen-casting windows of two vertical screens are also arranged side by side in the horizontal direction Arrangement, the screen recording resolution of terminal device A is 2040*1080, and the screen recording resolution of terminal device B is 1920*1080. According to the calculation method shown in Figure 22(c), the result is shown in Figure 22(d). Layout.

In some embodiments, if there are Q versions of the layout, and each version has the same sum of the area of the projection window, that is, S1=S2=...=SQ, any version can be randomly selected as the target layout.

In some embodiments, the screen projection windows in the layout may or may not be sorted according to the time when the screen projection is accessed. Taking three-way screen projection as an example, assuming that the display device is successively connected to terminal device A, terminal device C, and terminal device B for screen projection, the media configuration information sent by terminal device A indicates that its aspect ratio is 16:9 (horizontal screen) , the media configuration information sent by terminal device C indicates that its aspect ratio is 9:16 (vertical screen), and the media configuration information sent by terminal device B indicates that its aspect ratio is 9:16 (vertical screen), which corresponds to three-way A scene with one horizontal screen and two vertical screens. In this scenario, three versions of the layout can be matched. Figure 23(a) illustrates version 1, Figure 23(b) illustrates version 2, and Figure 23(c) illustrates version 3.

Among them, version 1 of the example in Figure 23(a) supports arranging screen projection window 1 (corresponding to terminal device A), screen projection window 2 (corresponding to terminal device C) and screen projection window in sequence in the horizontal direction according to the sequence of access projection screen 3 (corresponding to terminal device B), presenting a layout of horizontal screen-vertical screen-vertical screen, and setting the only horizontal screen window as the first display; Figure 23(b) presents a layout of vertical screen-horizontal screen-vertical screen , that is, set the only horizontal screen window to be displayed in the horizontal center, and the double vertical screen windows are symmetrically arranged on both sides of the horizontal screen window; Figure 23(c) presents a layout of vertical screen-vertical screen-horizontal screen, and the only The horizontal screen window is set as the last display. The versions shown in Fig. 23(b) and Fig. 23(c) do not support sorting according to the chronological order of accessing the projected screens.

Since the layouts of the three versions in the above scenarios all comply with the principle of maximum area, any version can be output as the target layout. In practical applications, you can also refer to the visual viewing effect to choose a layout of a certain version, such as Figure 23(b ) shows the version 2, which has the best layout symmetry, so in the scenario of three channels, one horizontal screen and two vertical screens, it is preferable to use the layout of version 2 for projection display.

In some embodiments, referring to FIG. 24 , this embodiment of the present application also provides an interaction process between a terminal device and a display device, including:

Step S201, the terminal device side: when receiving the user's screen projection operation on the media, send a screen projection request to the display device. Wherein, the screen projection request includes at least device information of the terminal device;

Step S202, the display device side: upon receiving the screen projection request, establish a screen projection connection with the terminal device, and feed back authorized information to the terminal device;

Step S203, on the display device side: add one to the cumulative number of projection channels;

Step S204, the terminal device side: when receiving the authorized information, send the projection data and media configuration information to the display device;

According to some embodiments, after receiving the screen projection request, the display device may inquire whether the terminal device has screen projection permission. If the terminal device has the permission to cast the screen, the display device will feed back the authorized information to the terminal device; if the terminal device does not have the permission to cast the screen, the display device will feed back unauthorized information to the terminal device, so that the unauthorized information received by the terminal device After the information, the projection data and media configuration information will no longer be sent. Among them, the way for the display device to inquire about the screen projection permission of the terminal device is not limited to asking the user whether to authorize the terminal device through a UI pop-up window, or querying the list of trusted devices maintained by the device.

Step S205, the display device side: receiving the screen projection data and media configuration information sent by the terminal device, and querying the currently counted number M of screen projection channels. M is greater than or equal to 1;

Step S206, the display device side: obtain the media configuration data correspondingly sent by the M terminal devices currently connected by screen projection. The media configuration data should at least include the aspect ratio of the media video, and the screen orientation information can be obtained through the aspect ratio;

Step S207, the display device side: calculate M groups of size information according to the screen resolution or the display direction and width-to-height ratio of the M terminal device projection media, and the number M of projection channels;

Step S208, display device side: construct M screen projection windows according to M groups of size information. The screen orientation and aspect ratio of the screen projection window and the displayed media are consistent;

Step S209, the display device side: calculate the position distribution information according to the screen resolution and the size information of the M groups. The position distribution information includes coordinates of preset reference points on the M projection screen windows, and the coordinates of the preset reference points are used to locate the position of the screen projection window on the screen;

Step S210, the display device side: according to the location distribution information, locate and arrange the M screen projection windows to obtain the layout. Optionally, if multiple versions of the layout are obtained, each version can be screened based on the principle of maximum area, and the target layout with the largest sum of the areas of the projection windows can be output;

Step S211, display device side: display the layout in the screen projection interface, and display the screen projection data sent by the corresponding terminal device in each screen projection window in the layout. Optionally, there is a one-to-one correspondence between the screen projection window and the device ID of the terminal device, so that the display device can accurately project the screen projection data to the designated screen projection window for display;

Step S212, the display device side: monitor the number of screen projection channels in real time during the process of displaying the screen projection interface;

Step S213, on the display device side: when it is detected that the number of screen projection channels has changed, update the layout;

Step S214, display device side: exit the screen projection interface when it is detected that the number of screen projection channels has decreased to zero.

Based on the configuration information such as the number of projection channels and the width-to-height ratio and screen orientation of multi-channel projection media, this application realizes dynamic adaptive layout matching on the projection interface, so that the typesetting and layout of M projection windows are more suitable for the screen , the multi-channel projection windows are displayed in the largest range on the screen, and the distribution of the projection windows is more compact, reducing the blank space of the projection interface. In addition, the aesthetics of the screen projection interface can also be considered, such as the uniformity and symmetry of the distribution of the screen projection windows, the relative balance of the size of the windows, etc., and the size and position distribution of each screen projection window can be set. The obtained layout significantly improves the display effect of the screen projection interface, thereby enhancing the user's viewing experience of the screen projection.

Fig. 25 shows a display diagram of an icon control interface of an application program in the display device 200 according to some embodiments. In some embodiments, after the display device is started, it can directly enter the interface of the preset video-on-demand program. The interface of the video-on-demand program can be as shown in FIG. , the content displayed in the content display area will change with the selected control in the navigation bar. The program in the application layer can be integrated in the video-on-demand program and displayed through a control in the navigation bar, or can be further displayed after the application control in the navigation bar is selected.

In some embodiments, after the display device is started, it can directly enter the display interface of the signal source selected last time, or the signal source selection interface, wherein the signal source can be a preset video-on-demand program, or an HDMI interface, a live TV interface At least one of the above, after the user selects different signal sources, the display can display the content obtained from different signal sources.

With the development of mobile Internet technology, mobile terminals such as mobile phones and tablet computers have more and more functions, and more and more users use mobile terminals to watch videos and browse files. However, due to the small screen size of mobile terminals such as mobile phones and tablet computers, and the large screen size of display devices such as televisions and desktop monitors, many users still like to watch videos and browse files on such display devices.

Taking the TV as an example, when the user wants to watch the video played by the mobile terminal on the TV, he can use the screen projection function of the mobile terminal to establish a screen projection connection with the TV, so as to watch the video on the mobile terminal and the TV synchronously . In some embodiments, when performing screen projection, the screen projection application in the mobile terminal is first started, and after the screen projection application in the mobile terminal is started, it automatically searches for a display device that can perform screen projection in the local area network currently connected to the mobile terminal , and then the display interface of the mobile terminal displays the device names of each display device that can cast a screen in a list. After the user selects and clicks the device name of the display device that needs to be projected in the display interface of the mobile terminal according to the actual screen projection needs, the mobile terminal can establish a screen projection connection with the display device selected by the user.

After the mobile terminal is connected to the display device for screen projection, the mobile terminal can send the data displayed on the mobile terminal to the display device, so as to watch the video in the mobile terminal on the display device. However, the screen projection method in the related art cannot simultaneously perform screen projection and playback on one display device for content sent through multiple screen projection protocols.

In order to solve this technical problem, an embodiment of the present application provides a display device, including: a display and a controller, wherein the display is configured to display a user interface. A playback application and at least one screen projection protocol application can be run in the controller, and the at least one screen projection protocol application can be connected to the playback application, and each screen projection protocol application can independently perform data communication with the playback application.

Each screen projection protocol application adopts a different screen projection protocol. As an example, the existing screen projection protocols include: DLNA projection protocol, Miracast projection protocol, Airplay projection protocol and other projection protocols. Based on the two types of content to be projected on the mobile terminal (one is the media data of third-party media resources on the mobile terminal, and the other is the real-time screen of the mobile terminal), the DLNA screen projection protocol can support third-party media on the mobile terminal The push and projection of the media data of resources; the Miracast projection protocol can support the mirror projection of the real-time screen of the mobile terminal screen; the Airplay projection protocol can not only support the push projection of the media data of the third-party media resources on the mobile terminal, but also can It supports the mirror projection of the real-time screen of the screen of the mobile terminal; other screen projection protocols can be the protocols developed by each display device manufacturer.

In some embodiments, for each existing screen projection protocol, a screen projection protocol application can be independently developed in the controller. Exemplarily, referring to FIG. 26, for the DLNA screen projection protocol, a DLNA projection screen protocol application is developed; for the Miracast screen projection protocol, a Miracast screen projection protocol application is developed; for the Airplay screen projection protocol, an Airplay projection screen protocol application is developed; For other screen projection protocols, develop corresponding other screen projection protocol applications.

Each screencasting protocol application exists in the form of a background service, and can independently perform data communication with the playback application. The playback application is an independent player application, which is not responsible for the protocol interaction with the mobile terminal, but only responsible for receiving and playing media data of various screen projection protocols.

Referring to the workflow schematic diagram shown in Figure 27, in order to solve the problem in the related art that the projection method cannot simultaneously project and play the display data on multiple mobile terminals on one display device, the controller in this embodiment is configured To perform the following steps:

Step 701: When receiving a first screen projection request sent by a first mobile terminal, establish a projection with the first mobile terminal through a first screen projection protocol application corresponding to the first screen projection request. screen connection, and sending the first screen-to-be-cast content transmitted by the first mobile terminal through the screen-casting connection to the playback application;

Step 702: Create a first playback window on the user interface through the playback application, and the first playback window is used to play the first content to be projected.

The first mobile terminal may be the only mobile terminal that performs screen projection connection with the display device, or may be any one of multiple mobile terminals that perform screen projection connection with the display device. The first screen projection request may be a screen projection request sent by the first mobile terminal to the display device through any screen projection protocol. The first screen-casting protocol application is a screen-casting protocol application that adopts the same screen-casting protocol as the first screen-casting request, and the first play window may be the only play window created by the play application, or multiple play windows created by the play application. Play any of the windows.

When the first mobile terminal sends a first screen projection request to the display device, the controller receives the screen projection request. Through the application of the first screen projection protocol, respond to and process the screen projection request, and establish a screen projection connection. The screen protocol application creates a playback window for playback, but sends the first content to be projected to the playback application through the first screen projection protocol application, and the playback application creates a playback window (ie, the first playback window) for playback. When there is a screen projection request sent by another screen projection protocol, the corresponding screen projection protocol application responds to the screen projection request, and the playback application will continue to run at this time, so the first playback window will not be affected, and will Continue to play the first content to be projected.

The screen projection protocol application is only responsible for the protocol interaction with the mobile terminal, not for the playback of the screen projection content. The playback application is only responsible for creating a corresponding playback window to play the received projection content, and is not responsible for the protocol interaction with the mobile terminal. When there are multiple mobile terminals sending multiple screen projection requests to the display device through multiple screen projection protocols, the controller can independently perform protocol interaction with multiple mobile terminals through multiple screen projection protocol applications, establish a screen projection connection, and then Forward the content to be projected to the playback application. As long as the playback application receives one content to be projected, it will create a playback window to play. If it receives multiple contents to be projected at the same time or successively, it will create multiple independent playback windows accordingly. In this way, it is possible to simultaneously play the content to be screened sent through multiple screen projection protocols on the same display device.

With reference to FIG. 28 , how to simultaneously play content sent through multiple screen projection protocols on one display device is further explained. When the display device is playing the screen-casting content of the first mobile terminal, if a screen-casting request is received again, in order to prevent the playback of the current screen-casting content from being terminated, the controller is further configured to perform the following steps:

Step 801: When receiving a second screen projection request sent by a second mobile terminal, establish a projection with the second mobile terminal through a second screen projection protocol application corresponding to the second screen projection request. screen connection, and sending the second screen content to be projected transmitted by the second mobile terminal through the screen projection connection to the playback application;

It should be noted that the second mobile terminal and the first mobile terminal may be the same mobile terminal, or two different mobile terminals, and the second screen projection protocol is applied to use the same screen projection as the second screen projection request. The screencasting protocol application of the protocol, the first screencasting request and the second screencasting request are two different screencasting requests sent by the user based on two different content to be screencasted, and the two screencasting requests can use the same projection The screen protocol can be used to send out, or different screen projection protocols can be used to send out. If the two screencasting requests are sent using the same screencasting protocol, then the first screencasting protocol application and the second screencasting protocol application are the same screencasting protocol application, if the two screencasting requests use different screencasting protocols sent, then the first screen projection protocol application and the second screen projection protocol application are two different screen projection protocol applications.

Step 802: Create a second playback window on the user interface through the playback application, and the second playback window is used to play the second content to be projected.

Referring to FIG. 29 , the second playback window and the first playback window are independent playback windows that can exist on the display at the same time, and the two playback windows may overlap partially or not at all.

In some embodiments, the playback application can adaptively set the size and position of the first playback window and the second playback window, so that the two playback windows can be completely displayed on the display. Or, the playback application can adjust the size and position of the first playback window and the second playback window according to the user's preferences.

Based on the above process, those skilled in the art can undoubtedly know how to simultaneously play multiple screen-casting contents when the display device receives more screen-casting requests, which will not be repeated here.

The content to be projected on the mobile terminal is generally divided into two types: one is the media data of the third-party media resources on the mobile terminal, and the other is the real-time screen of the mobile terminal.

Based on these two types, the screen projection protocol application in the controller needs to use different methods to forward the content to be screen projection to the playback application, which will be described in detail next. For ease of understanding, the following content is only described for the first mobile terminal, but there is no doubt that the same principle can be used for any mobile terminal that sends a screen projection request.

In some embodiments, before sending the first screen-to-be-cast content transmitted by the first mobile terminal through the screen-casting connection to the playback application, the controller is further configured to:

Obtain the first screen-to-be-cast content through the first screen-casting protocol application; wherein, if the first screen-to-be-cast content belongs to media data of a third-party media resource, then the first screen-to-be-cast content is The link address of the target web page, if the first screen-to-be-projected content belongs to the real-time screen, then the first screen-to-be-projected content is the screen video stream of the first mobile terminal.

When receiving the first screen projection request, the first screen projection protocol application responds, and negotiates a protocol command with the screen projection application that sent the first screen projection request. After the negotiation is completed, the first screen projection protocol application acquires the first mobile terminal The first content to be screened sent.

For the media data of the third-party media resource, what the first mobile terminal sends to the first screen projection protocol application is a webpage link address of a third-party media resource, and the webpage link address is the URL (Uniform Resource Locator, Uniform Resource Locator) address, this embodiment is called the link address of the target web page, usually an http type film source address. For the real-time screen image, what the first mobile terminal sends to the first screen projection protocol application is a video stream of the real-time screen image.

In some embodiments, if the first screen-to-be-projected content received by the first screen-casting protocol application is the link address of the target webpage, the link address of the target webpage may be sent to the player via the broadcast method of Android or other inter-process communication methods. application.

After receiving the link address of the target web page, the playing application creates a first play window, connects to the target link address, and then plays the video content obtained from the target link address in the first play window.

As an example, if the screen projection application in the first mobile terminal sends the first screen projection request to the display device through the DLNA screen projection protocol, the DLNA screen projection protocol application in the display device controller responds, and the screen projection application performs an agreement command Negotiation. After the negotiation is completed, the projection connection is established. If the content to be projected sent by the projection application through the projection connection is the URL address of a source in a third-party video software, the DLNA projection protocol application will receive the The URL address of the URL is notified to the playback application through the broadcast of android or other inter-process communication methods. After receiving the URL address, the playback application creates a playback window (or playback control) to play the film source corresponding to the URL address.

In some embodiments, if the first screen-to-be-cast content received by the first screen-casting protocol application is the screen video stream of the first mobile terminal, the first screen-to-be-cast content can be sent to the playback application by sharing memory. . Specifically, the controller is configured as:

Through the application of the first screen projection protocol, a first shared memory is created, and the first shared memory has a unique index name.

Through the first screen projection protocol application, the screen video stream of the first mobile terminal is stored in the first shared memory in real time, and a first read instruction is sent to the playback application, and the first read The instruction fetch includes the index name of the first shared memory.

When receiving the first read instruction, based on the unique index name, the playback application can read the screen video stream from the first shared memory in real time, and play the video stream from the first shared memory in the first playback window. Screen video stream read in real time.

As an example, if the screen-casting application in the first mobile terminal sends the first screen-casting request to the display device through the Miracast screen-casting protocol, the Miracast screen-casting protocol application in the display device controller responds, and the command of the protocol with the screen-casting application Negotiation, after the negotiation is completed, the Miracast screen projection protocol application creates a shared memory, that is, the first shared memory, and names the first shared memory, and then passes the name of the first shared memory through android broadcast or other communication methods between processes , notify the playback application, and at the same time, the Miracast screen projection protocol application receives the screen video stream transmitted in real time from the first mobile terminal through other data transmission protocols, writes it into the first shared memory, and the playback application creates a playback window, from The real-time screen video stream is read from the first shared memory for playback.

In some embodiments, if the first screen-to-be-cast content received by the first screen-casting protocol application is the screen video stream of the first mobile terminal, in addition to sharing memory, other inter-process communication methods can also be used to upload the second Once the content to be cast is sent to the playback application. The inter-process communication mode may be a communication mode such as a named pipe or a socket socket.

When the screen projection application in the first mobile terminal stops the screen projection operation, it notifies the first screen projection protocol application in the display device controller through the screen projection protocol, and the first screen projection protocol application notifies the playback application to stop playing.

In some embodiments, if the first screen-to-be-cast content is a real-time screen of the first mobile terminal, after the playback stops, the first screen-casting protocol application can delete the created first shared memory.

The screen projection method provided by this application will be described below with reference to specific examples. Assume that there are four mobile terminals that have sent screen projection requests to the display device. Suppose that the first mobile terminal pushes a source of video software A to the display device through the DLNA projection protocol, and the second mobile terminal pushes a source of video software A to the display device through the DLNA projection protocol. The display device pushes a video source of video software B, the third mobile terminal pushes its own screen image to the display device through the miracast screen projection protocol, and the fourth mobile terminal pushes its own screen image to the display device through the airplay screen projection protocol. Each protocol application in the display device controller: DLNA protocol application, miracast protocol application, and airplay protocol application respectively respond to the four screen projection requests. As shown in Figure 30, the interaction process between the four mobile terminals and the display device is as follows:

(1) The first mobile terminal pushes a video source of video software A to the display device through the DLNA projection protocol, and the DLNA projection protocol application receives the projection request and interacts with the first mobile terminal using the standard DLNA protocol. After the interaction is completed, the URL address of a video source of video software A is received, and the DLNA projection protocol application starts the playback application, and transmits the URL address to the playback application through process communication, and the playback application creates a playback window player 1 (on the monitor The display form on the user interface is a playback window), and the URL address is connected to play the video.

(2) The second mobile terminal pushes a source of video software B to the display device through the DLNA projection protocol, and the DLNA projection protocol application receives the projection request and interacts with the second mobile terminal using the standard DLNA protocol. After the interaction is completed, the URL address of a video source of video software B is received, the DLNA projection protocol application starts the playback application, and transmits the URL address to the playback application through process communication, and the playback application creates a playback window player 2 (on the monitor The form on the user interface is the second playback window), and connect to the URL address to play the video.

(2) The third mobile terminal pushes its own screen image to the display device through the miracast screen projection protocol. The miracast screen projection protocol application receives the screen projection request, conducts miracast connection command negotiation, and transmits the screen image data after the negotiation is completed , the miracast screencasting protocol application creates a shared memory named miracast_sharememry1, and at the same time notifies the playback application to read the video stream from the shared memory miracast_sharememry1 in real time, and at the same time the playback application creates a playback window player3 at a suitable position on the display interface (on the display user interface The form of expression is the third play window) to play the video stream.

(4) The fourth mobile terminal pushes its own screen image to the display device through the airplay screen projection protocol, and the airplay screen projection protocol application receives the screen projection request, conducts airplay connection command negotiation, and transmits the screen image data after the negotiation is completed , the airplay projection protocol application creates a shared memory named airplay_sharememry1, and at the same time notifies the playback application to read the video stream from the shared memory airplay_sharememry1 in real time, and at the same time, the playback application creates a playback window player4 at a suitable position on the display interface (on the display user interface The form of expression is the fourth playback window) to play the video stream.

For the display device provided by the embodiment of the present application, a playback application is independently developed in the controller. When there are multiple mobile terminals sending multiple screen projection requests to the display device through multiple screen projection protocols, the controller can use multiple screen projection The protocol application independently interacts with multiple mobile terminals, establishes a projection connection, and then forwards the content to be projected to the playback application. As long as the playback application receives a content to be projected, it will create a playback window for playback. After receiving multiple content to be projected one after another, multiple independent playback windows will be created accordingly to play. In this way, the content to be projected through multiple screen projection protocols can be played on the same display device at the same time.

In some embodiments, the present application discloses a screen projection method. FIG. 31 is a flow chart of multi-terminal interaction between a mobile terminal and a display device. With reference to FIG. 31 , the screen projection method includes:

Step 3101, the first mobile terminal sends a first screen projection request;

Step 3102, the first screen projection protocol application responds to the first screen projection request, and establishes a screen projection connection with the first mobile terminal, and the first screen projection protocol application uses Protocol application of the same projection protocol;

Step 3103, the first mobile terminal sends the first screen-to-be-cast content to the first screen-casting protocol application through the screen-casting connection;

Step 3104, the first screen-casting protocol application sends the first content to be screen-casted to the playback application;

Step 3105, the playing application creates a first playing window on the user interface displayed on the display, and the first playing window is used to play the first content to be mirrored.

In the above screen projection method, when the first mobile terminal sends a first screen projection request to the display device, the controller receives the screen projection request. Through the application of the first screen projection protocol, respond to and process the screen projection request, and establish a screen projection connection. The screen protocol application creates a playback window for playback, but sends the first content to be projected to the playback application through the first screen projection protocol application, and the playback application creates a playback window (ie, the first playback window) for playback. When there is a screen projection request sent by another screen projection protocol, the corresponding screen projection protocol application responds to the screen projection request, and the playback application will continue to run at this time, so the first playback window will not be affected, and will Continue to play the first content to be projected. In this way, it is possible to simultaneously play on the display device the content to be projected sent through multiple screen projection protocols.

For convenience of explanation, the above description has been made in conjunction with specific implementation manners. However, the above exemplary discussion is not intended to be exhaustive or to limit the implementations to the precise forms disclosed above. Many modifications and variations are possible in light of the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various modified embodiments suitable for specific use considerations.

Claims

A display device comprising:

a display configured to display a screen projection interface;

The communicator is configured to establish a screen projection connection with the communication terminal;

Controller, configured as:

Detect the number of connections of communication terminals;

planning the layout of the projection screen according to the number of connections, so as to obtain the display parameters of the projection screen;

Sending a notification message to the communication terminal corresponding to the screen projection screen, the notification message including the display parameters, and the notification message is used to enable the communication terminal to generate screen projection data according to the display parameters;

receiving the screen projection data fed back by the communication terminal, and displaying the screen projection data according to the layout.
The display device according to claim 1, the controller is further configured to:

Obtain the connection instruction input by the user for establishing the screen projection connection;

Initiating a screen projection connection to the communication terminal in response to the connection instruction;

After the screen projection connection is completed, the number of connections of the communication terminal is traversed.
The display device according to claim 2, the controller is further configured to:

Establish a projection data channel;

After the screen projection data channel is successfully established, a display control is created, and the display control is used to display the screen projection data received through the screen projection data channel.
The display device according to claim 1, the controller is further configured to:

Obtain the exit command input by the user for disconnecting the screen projection connection;

In response to the exit instruction, disconnect the screen projection connection corresponding to the communication terminal;

Traverse the number of connections of the remaining communication terminals.
The display device according to claim 1, the controller is further configured to:

Detecting screen size information of the display;

matching the layout template in a database by using the number of connections and the screen size information as an index;

The display parameters are extracted in the layout template.
The display device according to claim 5, the controller is further configured to:

If the layout template is not matched in the database, acquire device parameters of the communication terminal;

According to the device parameters, divide the display area into sub-areas equal to the number of connections;

Iterating over the graphic parameters of each sub-region to generate the display parameters.
The display device according to claim 1, the controller is further configured to:

Extracting the display parameters, the display parameters including the display direction, code rate and resolution of each display window, the display direction is used to instruct the communication terminal to generate screen projection data according to the set direction;

A notification message is generated according to the display parameters, and the notification message includes the display direction, code rate and resolution.
The display device according to claim 1, the controller is further configured to:

Detect the effective screen range of each screen-casting screen in the screen-casting interface;

Calculating the display ratio of the effective picture range to the total range of the display window;

If the display ratio is less than or equal to the ratio threshold, a prompt message is sent to the communication terminal, where the prompt message is used to remind the communication terminal of a rotation direction.
A communication terminal, comprising:

a display unit configured to display a terminal interface;

a communication unit configured to establish a screen projection connection with the display device;

Processor, configured as:

Receiving a notification message sent by the display device, the notification message including display parameters of the projection screen, the display parameters being obtained when the display device plans the layout of the projection screen according to the number of current communication terminal connections;

Generate screen projection data according to the display parameters;

The screen projection data is fed back to the display device.
A method for dynamically displaying a projected screen image, comprising:

The display device detects the number of connections of the communication terminal, and plans the layout of the projected screen according to the number of connections, so as to obtain the display parameters of the projected screen;

The display device sends a notification message to the communication terminal corresponding to the projection screen, and the notification message includes the display parameter;

The communication terminal receives the notification message sent by the display device, and generates projection data according to the display parameters;

The communication terminal feeds back the projection data to the display device;

The display device displays the projection data according to the layout.