WO2023011058A1 - Display device, communication terminal, and projected-screen image dynamic display method - Google Patents
Display device, communication terminal, and projected-screen image dynamic display method Download PDFInfo
- Publication number
- WO2023011058A1 WO2023011058A1 PCT/CN2022/102439 CN2022102439W WO2023011058A1 WO 2023011058 A1 WO2023011058 A1 WO 2023011058A1 CN 2022102439 W CN2022102439 W CN 2022102439W WO 2023011058 A1 WO2023011058 A1 WO 2023011058A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- screen
- display
- projection
- display device
- screen projection
- Prior art date
Links
- 238000004891 communication Methods 0.000 title claims abstract description 235
- 238000000034 method Methods 0.000 title claims abstract description 79
- 238000005266 casting Methods 0.000 claims description 47
- 230000004044 response Effects 0.000 claims description 6
- 238000013439 planning Methods 0.000 claims description 4
- 230000000977 initiatory effect Effects 0.000 claims description 2
- 230000008859 change Effects 0.000 abstract description 6
- 230000000875 corresponding effect Effects 0.000 description 51
- 238000010586 diagram Methods 0.000 description 43
- 230000008569 process Effects 0.000 description 23
- 230000000694 effects Effects 0.000 description 21
- 230000003993 interaction Effects 0.000 description 15
- 230000006870 function Effects 0.000 description 13
- 230000005540 biological transmission Effects 0.000 description 11
- 230000009977 dual effect Effects 0.000 description 11
- 238000005516 engineering process Methods 0.000 description 6
- 230000002452 interceptive effect Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000013475 authorization Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 230000000739 chaotic effect Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
- 238000013316 zoning Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
Definitions
- the present application relates to the field of display devices, and in particular to a display device, a communication terminal, and a method for dynamically displaying screen projection images.
- a display device refers to a terminal device capable of outputting specific display images, which may be terminal devices such as smart TVs, communication terminals, smart advertising screens, and projectors.
- terminal devices such as smart TVs, communication terminals, smart advertising screens, and projectors.
- smart TV is based on Internet application technology, has an open operating system and chip, has an open application platform, can realize two-way human-computer interaction, and integrates multiple functions such as audio-visual, entertainment, and data. Products are used to meet the diverse and individual needs of users.
- An embodiment of the present application provides a display device, including: a display, a communicator, and a controller.
- the display is configured to display a screen projection interface
- the communicator is configured to establish a screen projection connection with a communication terminal
- the controller is configured to execute the following procedural steps: detecting the number of connections of a communication terminal; according to the The number of connections plans the layout of the projection screen to obtain the display parameters of the projection screen; a notification message is sent to the communication terminal corresponding to the projection screen, the notification message includes the display parameters, and the notification message uses The method is to make the communication terminal generate screen projection data according to the display parameters; receive the screen projection data fed back by the communication terminal, and display the screen projection data according to the layout.
- An embodiment of the present application provides a display device, including:
- the display is used to display the screen projection interface; the communicator is used to connect with the terminal device for screen projection; the controller is configured to execute: receiving the screen projection request of at least M terminal devices, and receiving the media configuration respectively from the M terminal devices Information, matching the layout between M screen projection windows adapted to the screen; wherein, M ⁇ 1, the media configuration information includes the aspect ratio of the video projected by the terminal device; the control display follows the layout in the screen projection interface Display M screen projection windows, and display the screen projection data sent by the corresponding terminal device in each screen projection window.
- An embodiment of the present application provides a display device, including: a display configured to display a user interface; a controller capable of running a playback application and at least one screen projection protocol application, and the at least one screen projection protocol application can be connected to the
- the playback application is configured to: when receiving a first screen projection request sent by a first mobile terminal, establish a connection with the first mobile terminal through a first screen projection protocol application corresponding to the first screen projection request.
- FIG. 1 is a schematic diagram of an operation scene between a display device and a control device in an embodiment of the present application
- FIG. 2 is a block diagram of a hardware configuration of a display device in an embodiment of the present application
- FIG. 3 is a schematic diagram of display device software configuration in the embodiment of the present application.
- FIG. 4 is a schematic diagram of displaying an icon control interface of a display device application program in an embodiment of the present application
- FIG. 6 is a schematic diagram of a software architecture of a communication terminal according to some embodiments.
- FIG. 7 is a schematic diagram of a user interface of a communication terminal according to some embodiments.
- Fig. 8 is a schematic diagram of screen projection effects according to some embodiments.
- Fig. 11 is a schematic diagram of the effect of dividing two display areas according to some embodiments.
- Fig. 12 is a schematic flowchart of a method for dynamically displaying a projected screen according to some embodiments
- Fig. 13 is a schematic diagram of the effect of dividing three display areas according to some embodiments.
- connection instruction 14 is a schematic flow diagram of detecting the number of connections in response to a connection instruction according to some embodiments
- 15 is a schematic flow diagram of detecting the number of connections in response to an exit command according to some embodiments.
- Figure 16 is a schematic flow diagram of matching layout templates according to some embodiments.
- Fig. 18 is a schematic flow diagram of a communication terminal generating screen projection data according to some embodiments.
- Fig. 19 is a flowchart of a multi-channel projection display method according to some embodiments.
- Fig. 20(a) is a schematic layout diagram of a scene where the number of projection screens is 1 and the video stream is a single vertical screen according to some embodiments;
- Fig. 20(b) is a schematic layout diagram of a scene where the number of projection screens is 1 and the video stream is a single horizontal screen according to some embodiments;
- Fig. 21(a) is a schematic diagram of the first layout in a scene where the number of projection screens is 2 and double horizontal screens according to some embodiments;
- Fig. 21(b) is a schematic diagram of a second layout in a scene where the number of screen projections is 2 and double horizontal screens according to some embodiments;
- Fig. 21(d) is a schematic diagram of a second layout in a scenario where the number of projection screens is 2 and one horizontal screen and one vertical screen according to some embodiments;
- Fig. 21(e) is a schematic layout diagram in a scene where the number of projection screens is 2 and dual vertical screens according to some embodiments;
- Fig. 22(a) is a schematic diagram of version 1 of the layout in the scenario where the number of projection screens is 2 and the dual landscape screens (with different aspect ratios) according to some embodiments;
- Fig. 22(b) is a schematic diagram of version 2 of the layout in the scenario where the number of projection screens is 2 and the dual landscape screens (with different aspect ratios) according to some embodiments;
- Fig. 22(c) is a schematic diagram of version 1 of the layout in a scenario where the number of projection screens is 2 and the dual vertical screens (with different aspect ratios) according to some embodiments;
- Fig. 22(d) is a schematic diagram of version 2 of the layout in the scenario where the number of projection screens is 2 and the dual vertical screens (with different aspect ratios) according to some embodiments;
- Fig. 23(a) is a schematic diagram of version 1 of the layout in a scenario where the number of projection screens is 3 and there is one horizontal screen and two vertical screens according to some embodiments;
- Fig. 23(c) is a schematic diagram of version 3 of the layout in a scenario where the number of projection screens is 3 and there is one horizontal screen and two vertical screens according to some embodiments;
- FIG. 25 is a display diagram of an icon control interface of an application program in the display device 200 according to some embodiments.
- Fig. 26 is a configuration diagram of a controller in a display device according to some embodiments.
- Fig. 27 is a schematic flowchart of a screen projection method performed by a controller in a display device according to some embodiments
- Fig. 29 is a schematic diagram of a playback window on a display in a display device according to some embodiments.
- Fig. 30 is a schematic structural diagram of screen projection interaction between multiple mobile terminals and a display device according to some embodiments.
- Fig. 31 is a sequence diagram of screen projection interaction between a first mobile terminal and a display device according to some embodiments.
- control device 100 may be a remote control, and the communication between the remote control and the display device includes infrared protocol communication, bluetooth protocol communication, and other short-distance communication methods, and the display device 200 is controlled wirelessly or wiredly.
- the user can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, and the like.
- Fig. 2 is a configuration block diagram of the control device 100 according to some embodiments.
- the control device 100 includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply.
- the control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, and acts as an intermediary for the interaction between the user and the display device 200 .
- FIG. 3 shows a block diagram of a hardware configuration of a display device 200 according to an exemplary embodiment.
- the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface. at least one.
- the controller includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, a first interface to an nth interface for input/output.
- the user input interface can be used to receive a control signal of the control device 100 (such as an infrared remote controller, etc.).
- the external device interface 240 may include, but is not limited to, the following: high-definition multimedia interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), Any one or more interfaces such as RGB ports.
- HDMI high-definition multimedia interface
- component analog or data high-definition component input interface
- CVBS composite video input interface
- USB USB input interface
- the system of the display device may include a kernel (Kernel), a command parser (shell), a file system and application programs.
- Kernel kernel
- shell command parser
- file system file system
- application programs application programs
- the system is divided into four layers, from top to bottom are applications (Applications) layer (application layer), application framework (Application Framework) layer (framework layer), Android runtime (Android runtime) and system library layer (system runtime library layer), as well as the kernel layer.
- applications Applications
- Application Framework application framework
- Android runtime Android runtime
- system library layer system runtime library layer
- the embodiment will be specifically described below by taking the communication terminal 300 as an example. It should be understood that the communication terminal 300 shown in FIG. 5 is only an example, and that the communication terminal 300 may have more or fewer components than those shown in FIG. 5, two or more components may be combined, or Different component configurations are possible.
- the various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.
- the communication terminal 300 includes: a radio frequency (radio frequency, RF) circuit 310, a memory 320, a display unit 330, a camera 340, a sensor 350, an audio circuit 360, and a wireless fidelity (Wireless Fidelity, Wi-Fi) circuit 370, processor 380, Bluetooth circuit 381, and power supply 390 and other components.
- RF radio frequency
- FIG. 6 is a block diagram of the software structure of the communication terminal 300 according to the embodiment of the present invention.
- the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces.
- the Android system is divided into four layers, which are respectively the application program layer, the application program framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer from top to bottom.
- a corresponding hardware interrupt is sent to the kernel layer.
- the kernel layer processes touch operations into original input events (including touch coordinates, time stamps of touch operations, and other information). Raw input events are stored at the kernel level.
- the application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Take the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon as an example.
- the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer.
- Camera 340 captures still images or video.
- the display device 200 can play and display the content in the communication terminal 300 , as shown in FIG. 8 .
- the user may send a screen projection connection request through the communication terminal 300 , and send the screen projection connection request to the display device 200 through the WiFi network.
- the display device 200 completes the transmission protocol configuration according to the screen projection connection request, so as to establish a screen projection data transmission channel with the communication terminal 300 .
- the display device 200 can also realize the screen projection connection with the communication terminal 300 through different connection methods. For example, when the display device 200 and the communication terminal 300 are connected to the same wireless local area network, a screen projection connection can be established based on the WiFi network. For another example, when both the display device 200 and the communication terminal 300 are provided with NFC (Near Field Communication, near field communication) components, the screen projection connection relationship can be established through the NFC components. Obviously, other wired or wireless connection methods can also be used between the display device 200 and the communication terminal 300 to establish a screen projection connection relationship, such as RF radio frequency connection, infrared connection, cellular network, and connections provided by those skilled in the art in the above embodiments Other connection methods that can be thought of on the basis of the method.
- NFC Near Field Communication, near field communication
- the display device 200 may receive relevant data of the communication terminal 300 through the screen projection data channel, so as to play relevant content on the display device 200 .
- the screen projection data acquired by the display device 200 has different forms.
- the display device 200 can obtain the link address of a certain resource in the communication terminal 300 through the screen projection data channel, and obtain media asset content data by accessing the link address , so as to play the media asset data.
- the display device 200 can also obtain control instructions sent by the communication terminal 300 through the screen projection data channel when playing media asset data, such as fast forward, pause, and stop projection.
- the communication terminal 300 such as a mobile phone and the display device 200 will negotiate device functions and network conditions to select a suitable video and audio transmission format.
- a series of real-time streaming protocol (RTSP) control commands are used to control the playback and termination of the video stream.
- RTSP real-time streaming protocol
- the communication terminal 300 can be used as a signal source of the display device 200 , and the display device 200 can continuously obtain video data through the projection data channel, and display it on the display 275 in real time, so as to realize the synchronous display effect with the communication terminal 300 .
- Miracast screen projection is applicable to Android devices, that is, Android devices can share video images through Miracast screen projection.
- Android devices can share video images through Miracast screen projection.
- smart devices above Android 5.0 have built-in Miracast protocol, so they can directly perform wireless screen projection through the WiFi P2P function between the screen projection device and the WiFi network card of the receiver device without installing any software.
- Miracast also supports simultaneous transmission of audio and video.
- Miracast is not fully compatible with all Android devices, and Miracast-based screencasting does not have a screencasting code function, which cannot prevent malicious or accidental screencasting.
- Airplay screen projection is a wireless playback technology introduced by Apple, which is suitable for wireless screen projection of iOS and Mac systems.
- Apple series devices have a built-in Airplay screen projection function, which can be used with a screen projector that supports AirPlay server service, and the effect is clear and stable.
- the protocol comes with a screen projection code function.
- it is necessary to ensure that the Apple device and the screen projector are used in the same LAN (same network segment), and cannot be used across network segments or VLANs.
- WiDi screen projection is a wireless playback technology led by Intel, and it is also based on WiFi Direct direct connection technology. Windows 8/10 laptops natively support WiDi screen projection, which can be directly screen projection without installing any software, and supports USB reverse control function.
- the display device 200 can The displayed user interface is divided into areas, such as bisecting, trisecting, and quartering. After the multiple display areas are obtained through area division, the display device 200 displays the projected screen image in each divided area.
- the communication terminal 300 may be placed in different states according to different ways of use, for example, for a communication terminal 300 such as a smart phone, it may be in a portrait state or a landscape state during use.
- the vertical screen state refers to a state in which the width of the displayed picture is smaller than the height
- the horizontal screen state refers to the state in which the width of the displayed picture is greater than the height.
- the communication terminal 300 may present different screen layout modes. Based on this, when the display device 200 displays the projected screen images of multiple communication terminals 300 by area division, each display area will present different screen ratios due to the different placement states of the communication terminals 300 .
- the display device 200 when the placement states of the multiple communication terminals 300 are different, the display device 200 usually stretches and fills each screen projection image to fill the display area. This display mode will cause a large deformation of the projected screen, which will affect the user's viewing experience.
- the display device 200 capable of applying the method for dynamically displaying the projected screen includes: a display 260 , a communicator 220 and a controller 250 .
- the display 260 is configured to display a screen projection interface.
- the communicator 220 is configured to establish a screen projection connection with the communication terminal 300 .
- the controller 250 is configured to execute the above method for dynamically displaying the projection screen, as shown in FIG. 12 , which specifically includes the following content:
- the number of connections can also be obtained by detecting the data table maintained by the display device 200 in real time.
- the display device 200 may perform detection of the number of communication terminals 300 when judging that the layout of the current screen projection interface needs to be changed. For example, the display device 200 can monitor the access status of each communication terminal 300 in the screen projection connection in real time, and when it detects that any communication terminal 300 is newly connected or exits the connection, the display device 200 can perform a connection detection function of the communication terminal 300. number of steps.
- the display device 200 may also periodically detect the number of connections of the communication terminal 300 .
- the display device 200 may detect the number of connections of the communication terminals 300 through heartbeat commands of each communication terminal 300 after the screen projection connection is established. By detecting the number of connections of the communication terminal 300, the display device 200 can detect the connection status of the communication terminal 300 for screen projection in time, so that when the communication terminal is offline due to network range or other hardware failures, the screen projection interface can be adjusted. .
- the display device 200 may also detect the number of connections of the communication terminal 300 by detecting the user's control instructions and responding to some of the control instructions. For example, when the display device 200 detects that the user has input a control command for adjusting the page layout, the display device 200 may perform the step of detecting the number of connections of the communication terminal 300, so as to trigger the display device 200 to subsequently acquire the projection data. way of adjustment.
- the built-in layout templates may also be different.
- the layout mode template can be a layout template obtained by dividing equidistantly in the vertical direction, that is, on the display device 200.
- the screen is divided into three vertical areas whose width is smaller than the height; and for the screencasting connection for sharing the game interface operation, since the game interface is mostly in the horizontal screen state, the layout template can be obtained by dividing equidistantly in the horizontal direction. That is, the screen of the display device 200 is divided into three horizontal regions whose width is greater than the height, for displaying the game screen shared by the mobile phone.
- the display device 200 may plan the screen layout according to the principle of maximizing display. For example, when displaying 3 projection screen areas, one vertical area and two horizontal areas may be maintained. In this way, each display area can maintain a shape of 16:9 or 9:16, so as to adapt to the aspect ratio of communication terminal 300 .
- the display device 200 may add a display window to the user interface to plan a layout based on the number of connections. For example, when it is detected that the number of communication terminals 300 with which the current display device 200 has established a screen projection connection is 2, two display windows may be added to the user interface for respectively displaying the screen projection images shared by each communication terminal 300 . Each display window may have a fixed shape and ratio, and the display device 200 may be required to provide a display image according to the ratio.
- the multiple display windows when multiple display windows are used to display multiple screen projection images, if the number of communication terminals 300 that establish screen projection connections with the display device 200 is large, the multiple display windows can also be kept at a specific hierarchical relationship.
- the hierarchical relationship of the display windows may be set according to the order in which the screen projection connection is established with the display device 200 , that is, the screen projection screen shared by the communication terminal 300 that establishes the screen projection connection last is displayed on the top layer.
- the display device 200 may also determine display parameters of the projection screen according to the planned layout.
- the display parameters are used to indicate the screen ratio and size of each display area in the layout mode planned by the display device 200, and the projection data requirements calculated according to the ratio and size of each display area, such as code rate, resolution etc.
- the display device 200 may send a notification message to the communication terminal 300 corresponding to the screen projection screen.
- the notification message includes a display parameter
- the notification message is used to enable the communication terminal 300 to generate screen projection data according to the display parameter.
- the display device 200 when the display device 200 detects that the current number of connections to the communication terminal 300 is three, it can determine the display areas of the three projection screens by planning the layout, that is, the first area is the resolution “1080 ⁇ 1920” vertical area; area two and area three are horizontal areas with a resolution of “1280 ⁇ 720”. Based on the above display parameters, the display device 200 can generate a notification message, and send the generated notification message to the communication terminal 300, so that the communication terminal 300 can generate screen projection data according to the above resolution.
- the communication terminal 300 may have a built-in screen recording program, and after receiving the notification message sent by the display device 200, start the screen recording program to analyze the notification message and extract the display parameters. Then set the screen parameters in the screen recording program according to the display parameters. For example, when the display parameter is vertical and 1080 ⁇ 1920, the communication terminal 300 can control the screen recording process of the screen recording program so that the resolution of the generated screen recording video image is 1080 ⁇ 1920.
- the communication terminal 300 can process the screen by padding or cropping so that the generated screen projection data conforms to the resolution set in the notification message. Displays the form specified in the argument. For example, the screen ratio of each display area set in the display device 200 is 9:16, but when the screen ratio of the communication terminal 300 is 9:18, in order to be able to display completely, the communication terminal 300 can perform a reduction process on the display screen, so that The projection screen can be fully displayed in height. At this time, in the screen corresponding to the generated projection data, black areas will be filled on both sides to maintain the overall screen ratio of 9:16.
- the communication terminal 300 After the communication terminal 300 generates the projection data, it can feed back the generated projection data to the display device 200, and the display device 200 can display the received projection data, that is, display the images in the projection data according to the layout.
- the display device 200 respectively sends notification messages containing display parameters "vertical, 1080 ⁇ 1920", “horizontal, 1280 ⁇ 720", and “horizontal, 1280 ⁇ 720" to three communication terminals
- the three communication terminals 300 can respectively feed back three projection data streams with resolutions of 1080 ⁇ 1920, 1280 ⁇ 720, and 1280 ⁇ 720
- display device 200 then parses and renders the received three projection data streams to finally form Cast screen.
- the display device 200 may create a data buffer area for each communication terminal 300 when receiving multiple screen projection data streams for Store the screen projection data sent by them, and each display control on the display device 200 obtains video data from the corresponding data buffer area for display.
- the display device 200 can also correspond to it. The data cache area is destroyed and released.
- the method for dynamically displaying the projected screen provided by the above embodiment can plan the layout of multiple projected screens in the screen projecting interface by detecting the number of connections of the communication terminal 300, and send a message containing the display parameters of each area to each communication terminal 300
- the notification message enables the communication terminal 300 to generate screen projection data in a manner matching the layout manner.
- Such a display mode can ensure that the picture presented by the final display device 200 can adapt to the layout requirements, alleviate the deformation of the picture, and improve the viewing experience of the user.
- the display device 200 may start detecting the number of connections of the communication terminal 300 when the communication terminal 300 is connected. That is, as shown in FIG. 14 , the display device 200 may acquire a connection instruction input by the user for establishing a screen projection connection, and initiate a screen projection connection to the communication terminal 300 in response to the connection instruction.
- the connection instruction can be input in different ways.
- the communication terminal 300 that establishes a screen projection connection with the display device 200 for the first time.
- the user can first connect to the wireless network where the display device 200 is located through network settings, and then perform interactive operations on the communication terminal 300, such as clicking the "screen projection" button to initiate a screen projection connection.
- the display device 200 can display the screen projection connection pairing interface after receiving the screen projection connection.
- the user needs to perform interactive operations again based on the screen projection connection pairing interface, such as selecting the corresponding communication in the screen projection device list.
- the terminal 300 enables the display device 200 and the communication terminal 300 to establish a screen projection connection. It can be seen that, for the communication terminal 300 connecting for the first time, the connection instruction is a combination of a series of interactive actions based on the display device 200 and the communication terminal 300 .
- the user can directly click "Screencast" on the communication terminal 300
- the button initiates a screen projection operation.
- the display device 200 can also receive a screen projection connection instruction, and automatically responds to the connection instruction to implement a mutual handshake with the communication terminal 300 to complete the screen projection connection.
- the display device 200 may also traverse the number of connections of the communication terminal 300 . For example, after a communication terminal 300 initiates to establish a screen projection connection, the display device 200 may first wait for the configuration process of the screen projection connection. After the configuration of the screen projection connection is completed, the total number of communication terminals 300 currently connected is traversed. That is, the detected number of connections of the communication terminals 300 includes the communication terminals 300 that have newly established a screen projection connection.
- the display device 200 may also detect the number of connections when it detects that the communication terminal 300 stops the screen projection connection. As shown in FIG. , obtain the exit command input by the user for disconnecting the screen projection connection, and in response to the exit instruction, disconnect the screen projection connection of the communication terminal 300, and traverse the connections of the remaining communication terminals 300 after disconnecting the screen projection connection quantity.
- the user can click the "stop screen projection" button on one of the communication terminals 300 to control the communication terminal 300 to no longer execute the screen recording program.
- the display device 200 may automatically disconnect the screen projection connection established with the communication terminal 300 when detecting that the communication terminal 300 no longer sends screen projection data. And after the screen projection connection is disconnected, it is detected that the number of connections of the remaining communication terminals 300 is 2. Therefore, according to the usage status of the two communication terminals 300, the layout mode in the screen projection interface is re-planned.
- the above method for detecting the number of connections can automatically adapt to the layout of the screen on the display device 200, adjust the displayed content in time, and improve user experience.
- the display device 200 can also establish a screen projection data channel when initiating a screen projection connection to the communication terminal 300, and the screen projection data channel can be established successfully After that, create an indicator to display the projection data received through the projection data channel through the indicator.
- the display control is a display control program built in the display device 200, such as surface View or Texture View. Indicators can be added to display in different ways. For example, when the display device 200 includes one display control, the display control may be displayed in a full-screen manner. When the display device 200 includes two display controls, the two display controls are in the form of two vertically arranged windows on the left and right.
- the display device 200 in order to determine the display parameters, can first detect the screen size information of the display 260 when planning the layout of the projected screen according to the number of connections, and then use the number of connections and screen
- the size information is an index to match the layout template in the database, so as to extract the display parameters in the layout template.
- a database may be operated and maintained, and layout templates corresponding to various screen sizes and connection quantities may be stored in the database. Therefore, the display device 200 can determine the best layout mode suitable for the current display state according to the screen size information and the number of connections.
- the display device 200 may also determine the final layout mode according to the matching result of the layout template. That is, in the step of matching the layout template in the database, if no layout template is matched in the database, the layout method can be determined directly by using the matched layout template; and if no layout template is matched in the database, the display area can be Perform zoning to determine the appropriate layout. That is, the display device 200 can acquire the device parameters of the communication terminal 300, divide the display area into sub-regions with the same number of connections according to the device parameters, and then traverse the graphic parameters of each sub-region to generate display parameters.
- the display device 200 may also extract display parameters and generate a notification message according to the display parameters in the step of sending a notification message to the communication terminal 300 corresponding to the projected screen.
- the display parameters include the display direction of each display window.
- the display direction is used to instruct the communication terminal to generate screen projection data according to a set direction.
- the display device 200 may preferentially match the corresponding communication terminal 300 according to the shape of each display area after determining the layout mode. For example, when the display direction of the display window is vertical, the projection screen of the communication terminal 300 in the vertical screen state can be preferentially displayed in this window; when the display direction of the display window is horizontal, it can be preferentially displayed in this window. A projected screen image of the communication terminal 300 in a horizontal screen state.
- the display device 200 can also determine the image quality of the projection data according to the number of connections. That is, the display parameters also include the code rate and resolution of each display window, so that the communication terminal 300 can generate screen projection data with reference to the set code rate and resolution.
- the code rate and resolution can be set to be smaller, so that the transmission data of the projection data The smaller the amount, and vice versa.
- the transmission code rate can be set to 5Mb/s; if the display device 200 is connected to 3 communication terminals 300 or 4 communication terminals 300 at the same time, it can be set The transmission code rate is 3Mb/s.
- the resolution may be the same as the number of pixels occupied by each display area in the width and height directions after the display area is divided. For example, when the number of pixels occupied by the first display area is 1080 ⁇ 1920, the resolution of the corresponding projection data may also be 1080 ⁇ 1920. When the number of pixels occupied by the second display region and the third display region is 1280 ⁇ 720, the resolution of the corresponding projection data may also be 1280 ⁇ 720.
- the display device 200 may perform the following steps:
- Step S171 detecting the effective frame range of each screen-casting screen in the screen-casting interface
- Step S172 calculating the display ratio of the effective picture area to the total area of the display window
- Step S174 sending a prompt message to the communication terminal 300 to prompt the user to rotate the direction manually or automatically through the internal mechanism of the communication terminal 300;
- the display device 200 can detect that the display ratio is less than or equal to the ratio threshold, so it can send a prompt message to the communication terminal 300, such as "the current display The orientation does not match, please turn the mobile phone to the vertical screen state", prompting the user to adjust the communication terminal 300 to the tree screen state, and increase the display ratio of the effective screen in the projection data.
- a communication terminal 300 is also provided in some embodiments of the present application, including: a display unit 330 , a communication unit, and a processor 380 .
- the display unit 330 is configured to display a terminal interface
- the communication unit is configured to establish a screen projection connection with the display device 200; as shown in FIG. 18 , the processor 380 is configured to perform the following steps:
- Step 181 Receive a notification message sent by the display device 200, the notification message includes the display parameters of the screen projection screen, and the display parameters are when the display device 200 plans the layout mode of the screen projection screen according to the current number of communication terminals 300 connections get;
- Step 183 feeding back the screen projection data to the display device 200 .
- some embodiments of the present application also provide a method for dynamically displaying a projection screen, including the following steps:
- the display device 200 detects the number of connections of the communication terminal 300, and plans the layout of the projected screen according to the number of connections, so as to obtain the display parameters of the projected screen;
- the display device 200 sends a notification message to the communication terminal 300 corresponding to the projection screen, and the notification message includes the display parameter;
- the communication terminal 300 receives the notification message sent by the display device 200, and generates screen projection data according to the display parameters;
- the communication terminal 300 feeds back the screen projection data to the display device 200;
- the display device 200 displays the screen projection data according to the layout.
- the method for dynamically displaying the projected screen provided by the above embodiment can plan the layout of the projected screen by detecting the number of connected communication terminals 300 when the user triggers a change in the number of communication terminals 300 connected to the display device 200 . Then send a notification message including the display parameters of the screen projection to the communication terminal 300 , so that the communication terminal 300 can generate screen projection data conforming to the layout according to the display parameters, and display it on the display device 200 .
- the method can enable the display device 200 to support multi-channel screen projection, dynamically adjust the display layout of the screen projection screen, and notify the communication terminal 300 to adjust the screen recording, so as to make the layout of the screen projection screen displayed more reasonable.
- the terminal device can push media data to the display device.
- the user can project the video of interest to the display device for large-screen playback through some application programs with screen projection function; or the terminal device and the display device Establish a screen projection connection through certain protocols, such as DLNA (Digital Living Network Alliance, Digital Living Network Alliance) protocol, etc.
- DLNA Digital Living Network Alliance, Digital Living Network Alliance
- the terminal device can project the screen projection data to the display device, and the display device will display the projection data on the screen projection interface.
- the terminal device may be a mobile terminal such as a smart phone, a tablet computer, or a notebook computer, or may be a desktop computer or other small-screen display devices.
- the user when a terminal device is playing media data, the user can select a device from the device list for playing the media data on the screen, and the display device that supports the screen projection function will be identified and automatically added to the device list
- the media data can be projected from the terminal device to the display device for playback.
- the terminal device and the display device can be connected to the same Wifi network, and the method for the terminal to push media data to the display device is not limited.
- the screen projection method can also be performed across networks. In this application, the specific implementation manner of screen projection is not limited.
- the terminal device A sends a screen projection request to the display device, and the display device can ask the user whether to agree to the screen projection of the terminal device A through a UI inquiry pop-up window, etc., and can provide several option controls, such as "this consent”, “disagree” and “permanent consent”.
- the user chooses "Permanent Agree", which is equivalent to permanently authorizing terminal device A.
- the display device can maintain a list of trusted devices, and supports users to add, delete and modify device information with screen projection permission in the list of trusted devices.
- Device information includes but not limited to device ID, MAC Address, etc., that is, the user can customize the authorization list on the display device side.
- the display device receives the screen projection request sent by the terminal device B, it queries whether the terminal device B is included in the trusted device list according to the device information carried in the screen projection request. If terminal device B is included in the list of trusted devices, a screen projection connection between terminal device B and the display device is established; if terminal device B is not included in the list of trusted devices, the display device does not respond to the screen projection request, that is, no connection with The screen projection connection of terminal device B.
- the authorization and connection mode of screen projection by the terminal device are not limited to the examples in this application.
- each screen projection window corresponds to a terminal device that is successfully connected to the screen projection, and displays the screen projection data sent by the terminal device.
- the display effect of multiple screen projection windows on the screen projection interface affects the user's perception, and there are many display problems: for example, the arrangement of multiple screen projection windows is not compact enough, resulting in a lot of blank space in the screen projection interface; The distribution of the screen projection windows is chaotic, and the overall layout is not regular; some screen projection windows are small in size, so that users cannot clearly see the details of the video in the window; If it is changed to a vertical screen display, the video display uniformity between the terminal device and the display device is poor, and so on.
- the present application aims to plan and lay out multiple screen projection windows to improve the display effect of the screen projection interface.
- an embodiment of the present application provides a multi-channel projection display method, the method is configured to be executed by the controller 250 in the display device, that is, the controller 250 is the execution subject of the method, and the method includes the following steps :
- Step S01 according to the currently counted number of M screen projection channels and the media configuration information respectively received from the M terminal devices, match the layout among the M screen projection windows suitable for the screen;
- M is used to indicate the number of screen projection channels, that is, the number of terminal devices that have established screen projection connections with the display device.
- the number of screen projection channels will change dynamically with the connection and disconnection of screen projection connections.
- the numbers are counted in order to accurately plan the typesetting and layout of the screencasting interface.
- M is 1, single-channel projection is performed; when M is greater than or equal to 2, multiple-channel projection is performed; M can be zero, indicating that the display device has not received the projection data sent by the connected device, and the projection will not be displayed
- the interface is also equivalent to an empty layout on the screen projection interface.
- the media configuration information includes the aspect ratio of the video projected by the terminal device, that is, the ratio of the horizontal width to the vertical height when displaying the video.
- the aspect ratio can be used to identify the display orientation.
- the aspect ratio is greater than When or is equal to 1, that is, the width is greater than or equal to the height, the display mode is displayed in landscape mode; when the aspect ratio is less than 1, that is, the width is smaller than the height, the display mode is displayed in portrait mode.
- the creation of the projection window can be constrained according to the aspect ratio and screen orientation indicated in the media configuration information, so that the projection window and the corresponding terminal device can maintain the same aspect ratio and display orientation, which can avoid media correlation after projection. Misconfiguration problem.
- the media configuration information may also include a screen aspect ratio of the terminal device.
- the aspect ratio is greater than or equal to 1, that is, the width is greater than or equal to the height, the display mode is in landscape mode; when the aspect ratio is less than 1, that is, the width is smaller than the height, the display mode is in portrait mode.
- the media configuration information may further include a media category.
- the display device After the display device reads the media configuration information, it may know the media category to be screen-casted, so as to display screen-casting windows with different effects according to the media category.
- the projection window can display visualized projection data, and the projection window should match the aspect ratio and screen orientation of the video/image; for text-based media, the projection window can Visually display readable text information, but the aspect ratio and screen orientation of the projection window are not limited; for audio-only media, the projection window cannot display visual images, and voice playback controls can be set in the projection window (such as small Speaker), when playing the audio stream of the projection screen, the voice playback control can be presented as a dynamic special effect of "playing".
- the aspect ratio and screen orientation of the projection window corresponding to the pure audio stream are not limited.
- the sequence numbers may be allocated according to the time sequence of the screen projection connection, for example, the earlier the screen projection is connected, the smaller the sequence number.
- the screen projection screen is pre-typeset at the moment of screen projection connection, and after receiving the screen projection data, the screen is displayed according to the pre-typesetting method.
- the media configuration information of the display device may be to record video at a fixed resolution, such as 1920*1080, 1280*720, and so on.
- the media configuration information of the display device may record video according to the ratio of the screen, and the ratio of the current screen is not the ratio of 16:9/9:16 of the large-screen terminal, but other ratios such as 17:9, 21:16, etc.
- M screen projection windows are constructed according to M sets of size information, and the screen projection windows and their corresponding terminal devices maintain the same aspect ratio and display orientation. Since each set of size information defines a specific width and height of a screen projection window, a screen projection window matching the size information can be constructed. After the sizes of the M screen projection windows are all determined, it is also necessary to plan a specific display position of each screen projection window on the screen interface.
- the position distribution information is calculated according to the media configuration information received from the terminal device, the position distribution information includes the coordinates (xi, yi) of a preset reference point Pointi on the M screen projection windows, and the preset It is assumed that the coordinates (xi, yi) of the reference point Pointi are used to locate the position of the screen projection window on the screen. Since the screen projection window is generally a rectangle, some key points in the rectangle can be selected by the preset reference point. Optionally, the preset reference point is any one of the four vertices of the rectangle or the center point of the rectangle, etc. The selection principle is to Subject to calculation and positioning. Finally, according to the calculated location distribution information, the created M screen projection windows are positioned and arranged, so that the sizes and positions of the M screen projection windows are determined and the layout is formed.
- Figure 20(a) exemplarily shows the screen projection interface in the scenario where the number of screen projection channels is 1 and the video stream is a single vertical screen.
- the screen resolution of the display device is 1920*1080 (that is, the default aspect ratio of the display device screen is 16:9), and the media configuration file sent by terminal device A indicates that the aspect ratio is 9:16 (vertical screen) as an example, then
- Figure 20(b) exemplarily shows the screen projection interface in the scenario where the number of screen projection channels is 1 and the video stream is a single horizontal screen.
- the screen resolution of the display device is 1920*1080 (that is, the default aspect ratio of the display device screen is 16:9), and the media configuration file sent by terminal device A indicates that the aspect ratio is 16:9 (horizontal screen), for example, then
- the projection window 1 to be created is also a horizontal screen.
- the upper left corner of the screen is used as the default reference point Point1
- the upper left corner of the screen projection window 2 is used as the preset reference point Point2.
- the layout after matching and generating the layout, during the process of displaying multiple screen projection windows according to the layout on the screen projection interface and loading and displaying the screen projection data of each channel, there may be new screen projection channels accessed at any time, or there may be At least one terminal device is disconnected from the screen projection connection, that is, the number of screen projection channels may change during this period, and the layout needs to be updated according to the changed screen projection channels to realize adaptive adjustment of the layout following the screen projection channels.
- the adjustment involves the number of screen projection windows in the layout, the size of each screen projection window, and the relative position distribution of each screen projection window.
- the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 16:9 (horizontal screen), the controller 250 detects that the number of projection screens is changed from 1 to 2, then the layout can be updated to the version shown in Figure 21(a) or Figure 21(b); for example, in Figure 21(a) or Figure 21(b)
- terminal device B suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1, and can update the layout to the version shown in Figure 20(b).
- Fig. 21(c) and Fig. 21(d) are two kinds of screen-casting interfaces in the scene where the number of screen-casting channels is 2 and one horizontal screen and one vertical screen according to some embodiments, and Fig. 21(c) shows the horizontal screen The screen projection window is displayed on the left, and the screen projection window for the vertical screen is displayed on the right; Figure 21(d) shows the screen projection window for the vertical screen on the left, and the screen projection window for the landscape screen is displayed on the right.
- the display device has connected to the two-way screen projection initiated by terminal device A and terminal device B.
- the screen resolution of the display device is 1920*1080, and the media configuration file sent by terminal device A indicates that the aspect ratio is 16:9 ( Horizontal screen), the media configuration file sent by terminal device B indicates that the aspect ratio is 9:16 (vertical screen) as an example, then the created projection window 1 (corresponding to terminal device A) is a horizontal Corresponding to terminal device B) is a vertical screen.
- terminal device B suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1 , the layout can be updated to the version of the single horizontal screen window (screen projection window 1) full-screen exclusive screen shown in FIG. 20(b).
- terminal device A suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1 , the layout can be updated to a version similar to that shown in Figure 20(a) in which the single vertical screen window (screen projection window 2) monopolizes the screen.
- the upper left corner of the screen projection window 1 is used as the preset reference point Point1
- the upper left corner of the screen projection window 2 is used as the preset reference point Point2. Considering that the two vertical screen windows are evenly spaced along the horizontal direction, the viewing effect is better.
- the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 9: 16 (vertical screen), the controller 250 detects that the number of projection screens has been changed from 1 to 2, and the layout can be updated to the version with two channels and two vertical screens as shown in FIG. 21(e).
- terminal device A or terminal device B disconnects the screen projection connection, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1, then it can The layout is updated to a version similar to that shown in Figure 20(a) where a single vertical window occupies the screen.
- the controller detects that When the number of screen projection channels becomes 0, the screen projection interface will be exited, and the UI display interface will return to that before screen projection.
- the adaptability of the layout matching mechanism can refer to the previous example.
- the layout matching can be It has the characteristics of adaptability, flexibility and diversity, etc., and this application will not list and repeat them one by one.
- the layout forms of each screen projection window are different, resulting in a variety of layouts. 21(a) and 21(b)
- the aforementioned two-way-horizontal-vertical screen scenario includes at least the two versions of FIG. 21(c) and 21(d).
- the maximum area sum Smax is calculated, and then the version corresponding to Smax is used as the selected target layout, and the target layout is displayed on the screen projection interface, so that each projection window can occupy the screen more fully and to the maximum extent, and each projection screen
- the layout between the windows is compact, and the blank area is reduced as much as possible to improve the display effect of the screen projection interface. Since the screen projection window is rectangular, the area of a single screen projection window is equal to the product of its width and height.
- the screen resolution of the display device is 1920*1080
- the media configuration file sent by terminal device A indicates that the aspect ratio is 17:9 (horizontal screen)
- terminal device B sends
- the media configuration file indicated that the ratio of width to height is 16:9 (horizontal screen), that is, the media of the two screens have different ratios of width to height
- two screen projection windows are constructed.
- Screen projection window 1 corresponds to terminal device A.
- Screen projection window 2 corresponds to terminal device B.
- version 1 of the layout is arranged side by side in a horizontal direction.
- windowWidth1*windowHeight1 960 *508
- Figure 22(c) exemplarily shows a screen-casting interface in a dual-vertical-screen scene with two screen-casting channels.
- the screen-casting windows of two vertical screens are arranged side by side in the horizontal direction , and the screen recording resolutions of the two terminal devices are different.
- the screen recording resolution of terminal device A is 1920*1080
- the resolution of terminal device B’s screen recording is 2040*1080.
- the media sent by terminal device A and terminal device B The configuration file indicates that the aspect ratio is 9:16 (vertical screen) and 9:17 (vertical screen), then the created screen projection window 1 (corresponding to terminal device A) and screen projection window 2 (corresponding to terminal device B) Both are vertical screens, but the horizontal display sizes of the two projection windows are different.
- Figure 22(d) exemplarily shows another screen-casting interface in a dual-vertical-screen scene with the number of screen-casting channels being 2.
- the screen-casting windows of two vertical screens are also arranged side by side in the horizontal direction Arrangement, the screen recording resolution of terminal device A is 2040*1080, and the screen recording resolution of terminal device B is 1920*1080.
- the result is shown in Figure 22(d). Layout.
- any version can be randomly selected as the target layout.
- the screen projection windows in the layout may or may not be sorted according to the time when the screen projection is accessed.
- the media configuration information sent by terminal device A indicates that its aspect ratio is 16:9 (horizontal screen)
- the media configuration information sent by terminal device C indicates that its aspect ratio is 9:16 (vertical screen)
- the media configuration information sent by terminal device B indicates that its aspect ratio is 9:16 (vertical screen), which corresponds to three-way A scene with one horizontal screen and two vertical screens.
- three versions of the layout can be matched.
- Figure 23(a) illustrates version 1
- Figure 23(b) illustrates version 2
- Figure 23(c) illustrates version 3.
- any version can be output as the target layout.
- Step S201 the terminal device side: when receiving the user's screen projection operation on the media, send a screen projection request to the display device.
- the screen projection request includes at least device information of the terminal device;
- Step S202 the display device side: upon receiving the screen projection request, establish a screen projection connection with the terminal device, and feed back authorized information to the terminal device;
- Step S203 on the display device side: add one to the cumulative number of projection channels;
- Step S204 the terminal device side: when receiving the authorized information, send the projection data and media configuration information to the display device;
- the display device may inquire whether the terminal device has screen projection permission. If the terminal device has the permission to cast the screen, the display device will feed back the authorized information to the terminal device; if the terminal device does not have the permission to cast the screen, the display device will feed back unauthorized information to the terminal device, so that the unauthorized information received by the terminal device After the information, the projection data and media configuration information will no longer be sent.
- the way for the display device to inquire about the screen projection permission of the terminal device is not limited to asking the user whether to authorize the terminal device through a UI pop-up window, or querying the list of trusted devices maintained by the device.
- Step S205 the display device side: receiving the screen projection data and media configuration information sent by the terminal device, and querying the currently counted number M of screen projection channels.
- M is greater than or equal to 1;
- Step S209 the display device side: calculate the position distribution information according to the screen resolution and the size information of the M groups.
- the position distribution information includes coordinates of preset reference points on the M projection screen windows, and the coordinates of the preset reference points are used to locate the position of the screen projection window on the screen;
- Step S210 the display device side: according to the location distribution information, locate and arrange the M screen projection windows to obtain the layout.
- each version can be screened based on the principle of maximum area, and the target layout with the largest sum of the areas of the projection windows can be output;
- Step S211 display device side: display the layout in the screen projection interface, and display the screen projection data sent by the corresponding terminal device in each screen projection window in the layout.
- the display device can accurately project the screen projection data to the designated screen projection window for display;
- Step S212 the display device side: monitor the number of screen projection channels in real time during the process of displaying the screen projection interface;
- Step S214 display device side: exit the screen projection interface when it is detected that the number of screen projection channels has decreased to zero.
- this application realizes dynamic adaptive layout matching on the projection interface, so that the typesetting and layout of M projection windows are more suitable for the screen , the multi-channel projection windows are displayed in the largest range on the screen, and the distribution of the projection windows is more compact, reducing the blank space of the projection interface.
- the aesthetics of the screen projection interface can also be considered, such as the uniformity and symmetry of the distribution of the screen projection windows, the relative balance of the size of the windows, etc., and the size and position distribution of each screen projection window can be set.
- the obtained layout significantly improves the display effect of the screen projection interface, thereby enhancing the user's viewing experience of the screen projection.
- the display device after the display device is started, it can directly enter the display interface of the signal source selected last time, or the signal source selection interface, wherein the signal source can be a preset video-on-demand program, or an HDMI interface, a live TV interface
- the display can display the content obtained from different signal sources.
- mobile terminals such as mobile phones and tablet computers have more and more functions, and more and more users use mobile terminals to watch videos and browse files.
- mobile terminals such as mobile phones and tablet computers
- display devices such as televisions and desktop monitors
- the screen projection application in the mobile terminal when performing screen projection, the screen projection application in the mobile terminal is first started, and after the screen projection application in the mobile terminal is started, it automatically searches for a display device that can perform screen projection in the local area network currently connected to the mobile terminal , and then the display interface of the mobile terminal displays the device names of each display device that can cast a screen in a list. After the user selects and clicks the device name of the display device that needs to be projected in the display interface of the mobile terminal according to the actual screen projection needs, the mobile terminal can establish a screen projection connection with the display device selected by the user.
- the mobile terminal After the mobile terminal is connected to the display device for screen projection, the mobile terminal can send the data displayed on the mobile terminal to the display device, so as to watch the video in the mobile terminal on the display device.
- the screen projection method in the related art cannot simultaneously perform screen projection and playback on one display device for content sent through multiple screen projection protocols.
- an embodiment of the present application provides a display device, including: a display and a controller, wherein the display is configured to display a user interface.
- a playback application and at least one screen projection protocol application can be run in the controller, and the at least one screen projection protocol application can be connected to the playback application, and each screen projection protocol application can independently perform data communication with the playback application.
- Each screen projection protocol application adopts a different screen projection protocol.
- the existing screen projection protocols include: DLNA projection protocol, Miracast projection protocol, Airplay projection protocol and other projection protocols.
- the DLNA screen projection protocol can support third-party media on the mobile terminal The push and projection of the media data of resources;
- the Miracast projection protocol can support the mirror projection of the real-time screen of the mobile terminal screen;
- the Airplay projection protocol can not only support the push projection of the media data of the third-party media resources on the mobile terminal, but also can It supports the mirror projection of the real-time screen of the screen of the mobile terminal;
- other screen projection protocols can be the protocols developed by each display device manufacturer.
- the controller in this embodiment is configured To perform the following steps:
- Step 702 Create a first playback window on the user interface through the playback application, and the first playback window is used to play the first content to be projected.
- the first mobile terminal may be the only mobile terminal that performs screen projection connection with the display device, or may be any one of multiple mobile terminals that perform screen projection connection with the display device.
- the first screen projection request may be a screen projection request sent by the first mobile terminal to the display device through any screen projection protocol.
- the first screen-casting protocol application is a screen-casting protocol application that adopts the same screen-casting protocol as the first screen-casting request, and the first play window may be the only play window created by the play application, or multiple play windows created by the play application. Play any of the windows.
- the controller receives the screen projection request.
- the screen protocol application creates a playback window for playback, but sends the first content to be projected to the playback application through the first screen projection protocol application, and the playback application creates a playback window (ie, the first playback window) for playback.
- the corresponding screen projection protocol application responds to the screen projection request, and the playback application will continue to run at this time, so the first playback window will not be affected, and will Continue to play the first content to be projected.
- the controller is further configured to perform the following steps:
- the second mobile terminal and the first mobile terminal may be the same mobile terminal, or two different mobile terminals, and the second screen projection protocol is applied to use the same screen projection as the second screen projection request.
- the screencasting protocol application of the protocol, the first screencasting request and the second screencasting request are two different screencasting requests sent by the user based on two different content to be screencasted, and the two screencasting requests can use the same projection
- the screen protocol can be used to send out, or different screen projection protocols can be used to send out.
- the first screencasting protocol application and the second screencasting protocol application are the same screencasting protocol application, if the two screencasting requests use different screencasting protocols sent, then the first screen projection protocol application and the second screen projection protocol application are two different screen projection protocol applications.
- the second playback window and the first playback window are independent playback windows that can exist on the display at the same time, and the two playback windows may overlap partially or not at all.
- the playback application can adaptively set the size and position of the first playback window and the second playback window, so that the two playback windows can be completely displayed on the display. Or, the playback application can adjust the size and position of the first playback window and the second playback window according to the user's preferences.
- the content to be projected on the mobile terminal is generally divided into two types: one is the media data of the third-party media resources on the mobile terminal, and the other is the real-time screen of the mobile terminal.
- the controller before sending the first screen-to-be-cast content transmitted by the first mobile terminal through the screen-casting connection to the playback application, the controller is further configured to:
- the first screen-to-be-cast content through the first screen-casting protocol application; wherein, if the first screen-to-be-cast content belongs to media data of a third-party media resource, then the first screen-to-be-cast content is The link address of the target web page, if the first screen-to-be-projected content belongs to the real-time screen, then the first screen-to-be-projected content is the screen video stream of the first mobile terminal.
- the first screen projection protocol application When receiving the first screen projection request, the first screen projection protocol application responds, and negotiates a protocol command with the screen projection application that sent the first screen projection request. After the negotiation is completed, the first screen projection protocol application acquires the first mobile terminal The first content to be screened sent.
- the link address of the target webpage may be sent to the player via the broadcast method of Android or other inter-process communication methods. application.
- the playing application After receiving the link address of the target web page, the playing application creates a first play window, connects to the target link address, and then plays the video content obtained from the target link address in the first play window.
- the screen projection application in the first mobile terminal sends the first screen projection request to the display device through the DLNA screen projection protocol
- the DLNA screen projection protocol application in the display device controller responds, and the screen projection application performs an agreement command Negotiation. After the negotiation is completed, the projection connection is established.
- the content to be projected sent by the projection application through the projection connection is the URL address of a source in a third-party video software
- the DLNA projection protocol application will receive the The URL address of the URL is notified to the playback application through the broadcast of android or other inter-process communication methods.
- the playback application creates a playback window (or playback control) to play the film source corresponding to the URL address.
- the first screen-to-be-cast content received by the first screen-casting protocol application is the screen video stream of the first mobile terminal
- the first screen-to-be-cast content can be sent to the playback application by sharing memory.
- the controller is configured as:
- a first shared memory is created, and the first shared memory has a unique index name.
- the screen video stream of the first mobile terminal is stored in the first shared memory in real time, and a first read instruction is sent to the playback application, and the first read
- the instruction fetch includes the index name of the first shared memory.
- the playback application can read the screen video stream from the first shared memory in real time, and play the video stream from the first shared memory in the first playback window. Screen video stream read in real time.
- the Miracast screen-casting protocol application in the display device controller responds, and the command of the protocol with the screen-casting application Negotiation, after the negotiation is completed, the Miracast screen projection protocol application creates a shared memory, that is, the first shared memory, and names the first shared memory, and then passes the name of the first shared memory through android broadcast or other communication methods between processes , notify the playback application, and at the same time, the Miracast screen projection protocol application receives the screen video stream transmitted in real time from the first mobile terminal through other data transmission protocols, writes it into the first shared memory, and the playback application creates a playback window, from The real-time screen video stream is read from the first shared memory for playback.
- the inter-process communication mode may be a communication mode such as a named pipe or a socket socket.
- the screen projection application in the first mobile terminal stops the screen projection operation, it notifies the first screen projection protocol application in the display device controller through the screen projection protocol, and the first screen projection protocol application notifies the playback application to stop playing.
- the first screen-casting protocol application can delete the created first shared memory.
- the screen projection method provided by this application will be described below with reference to specific examples. Assume that there are four mobile terminals that have sent screen projection requests to the display device. Suppose that the first mobile terminal pushes a source of video software A to the display device through the DLNA projection protocol, and the second mobile terminal pushes a source of video software A to the display device through the DLNA projection protocol. The display device pushes a video source of video software B, the third mobile terminal pushes its own screen image to the display device through the miracast screen projection protocol, and the fourth mobile terminal pushes its own screen image to the display device through the airplay screen projection protocol.
- Each protocol application in the display device controller: DLNA protocol application, miracast protocol application, and airplay protocol application respectively respond to the four screen projection requests. As shown in Figure 30, the interaction process between the four mobile terminals and the display device is as follows:
- the first mobile terminal pushes a video source of video software A to the display device through the DLNA projection protocol, and the DLNA projection protocol application receives the projection request and interacts with the first mobile terminal using the standard DLNA protocol.
- the URL address of a video source of video software A is received, and the DLNA projection protocol application starts the playback application, and transmits the URL address to the playback application through process communication, and the playback application creates a playback window player 1 (on the monitor
- the display form on the user interface is a playback window), and the URL address is connected to play the video.
- the second mobile terminal pushes a source of video software B to the display device through the DLNA projection protocol, and the DLNA projection protocol application receives the projection request and interacts with the second mobile terminal using the standard DLNA protocol.
- the URL address of a video source of video software B is received, the DLNA projection protocol application starts the playback application, and transmits the URL address to the playback application through process communication, and the playback application creates a playback window player 2 (on the monitor The form on the user interface is the second playback window), and connect to the URL address to play the video.
- the third mobile terminal pushes its own screen image to the display device through the miracast screen projection protocol.
- the miracast screen projection protocol application receives the screen projection request, conducts miracast connection command negotiation, and transmits the screen image data after the negotiation is completed , the miracast screencasting protocol application creates a shared memory named miracast_sharememry1, and at the same time notifies the playback application to read the video stream from the shared memory miracast_sharememry1 in real time, and at the same time the playback application creates a playback window player3 at a suitable position on the display interface (on the display user interface The form of expression is the third play window) to play the video stream.
- the fourth mobile terminal pushes its own screen image to the display device through the airplay screen projection protocol, and the airplay screen projection protocol application receives the screen projection request, conducts airplay connection command negotiation, and transmits the screen image data after the negotiation is completed , the airplay projection protocol application creates a shared memory named airplay_sharememry1, and at the same time notifies the playback application to read the video stream from the shared memory airplay_sharememry1 in real time, and at the same time, the playback application creates a playback window player4 at a suitable position on the display interface (on the display user interface The form of expression is the fourth playback window) to play the video stream.
- a playback application is independently developed in the controller.
- the controller can use multiple screen projection
- the protocol application independently interacts with multiple mobile terminals, establishes a projection connection, and then forwards the content to be projected to the playback application.
- the playback application receives a content to be projected, it will create a playback window for playback.
- multiple independent playback windows will be created accordingly to play. In this way, the content to be projected through multiple screen projection protocols can be played on the same display device at the same time.
- Step 3102 the first screen projection protocol application responds to the first screen projection request, and establishes a screen projection connection with the first mobile terminal, and the first screen projection protocol application uses Protocol application of the same projection protocol;
- Step 3103 the first mobile terminal sends the first screen-to-be-cast content to the first screen-casting protocol application through the screen-casting connection;
- Step 3104 the first screen-casting protocol application sends the first content to be screen-casted to the playback application;
- Step 3105 the playing application creates a first playing window on the user interface displayed on the display, and the first playing window is used to play the first content to be mirrored.
- the controller receives the screen projection request.
- the screen protocol application creates a playback window for playback, but sends the first content to be projected to the playback application through the first screen projection protocol application, and the playback application creates a playback window (ie, the first playback window) for playback.
- the corresponding screen projection protocol application responds to the screen projection request, and the playback application will continue to run at this time, so the first playback window will not be affected, and will Continue to play the first content to be projected. In this way, it is possible to simultaneously play on the display device the content to be projected sent through multiple screen projection protocols.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
The present application provides a display device (200), a communication terminal (300), and a projected-screen image dynamic display method. The method comprises: when a user triggers a change in the number of communication terminals (300) connected to a display device (200), measuring the number of connections of the communication terminals (300) to plan a layout mode of a projected-screen image; and then, sending to the communication terminals (300) a notification message including a projected-screen image display parameter, such that the communication terminals (300) can generate, according to the display parameter, screen projection data conforming to the layout mode, and display the screen projection data in the display device (200).
Description
相关申请的交叉引用Cross References to Related Applications
本申请要求在2021年08月02日提交、申请号为202110881496.8;在2021年08月02日提交、申请号为202110881335.9;在2021年10月28日提交、申请号为202111265530.5,其全部内容通过引用结合在本申请中。This application is required to be submitted on August 2, 2021, with the application number 202110881496.8; to be submitted on August 2, 2021, with the application number 202110881335.9; to be submitted on October 28, 2021, with the application number 202111265530.5, the entire contents of which are by reference incorporated in this application.
本申请涉及显示设备领域,尤其涉及一种显示设备、通信终端及投屏画面动态显示方法。The present application relates to the field of display devices, and in particular to a display device, a communication terminal, and a method for dynamically displaying screen projection images.
显示设备是指能够输出具体显示画面的终端设备,可以是智能电视、通信终端、智能广告屏、投影仪等终端设备。以智能电视为例,智能电视是基于Internet应用技术,具备开放式操作系统与芯片,拥有开放式应用平台,可实现双向人机交互功能,集影音、娱乐、数据等多种功能于一体的电视产品,用于满足用户多样化和个性化需求。A display device refers to a terminal device capable of outputting specific display images, which may be terminal devices such as smart TVs, communication terminals, smart advertising screens, and projectors. Taking smart TV as an example, smart TV is based on Internet application technology, has an open operating system and chip, has an open application platform, can realize two-way human-computer interaction, and integrates multiple functions such as audio-visual, entertainment, and data. Products are used to meet the diverse and individual needs of users.
显示设备还可以通过无线局域网等通信方式与通信终端建立投屏连接关系,即显示设备可以接收通信终端共享的投屏数据,并实时显示接收到的投屏数据,即显示通信终端的当前画面或者在通信终端上播放的媒资画面。The display device can also establish a projection connection relationship with the communication terminal through wireless LAN and other communication methods, that is, the display device can receive the projection data shared by the communication terminal, and display the received projection data in real time, that is, display the current screen of the communication terminal or The media asset screen played on the communication terminal.
发明内容Contents of the invention
本申请实施例提供一种显示设备,包括:显示器、通信器和控制器。其中,所述显示器被配置为显示投屏界面;所述通信器被配置为与通信终端建立投屏连接;所述控制器被配置为执行以下程序步骤:检测通信终端的连接数量;根据所述连接数量规划投屏画面的布局方式,以获得所述投屏画面的显示参数;向所述投屏画面对应的通信终端发送通知消息,所述通知消息包括所述显示参数,所述通知消息用于使所述通信终端按照所述显示参数生成投屏数据;接收所述通信终端反馈的所述投屏数据,以及按照所述布局方式显示所述投屏数据。An embodiment of the present application provides a display device, including: a display, a communicator, and a controller. Wherein, the display is configured to display a screen projection interface; the communicator is configured to establish a screen projection connection with a communication terminal; the controller is configured to execute the following procedural steps: detecting the number of connections of a communication terminal; according to the The number of connections plans the layout of the projection screen to obtain the display parameters of the projection screen; a notification message is sent to the communication terminal corresponding to the projection screen, the notification message includes the display parameters, and the notification message uses The method is to make the communication terminal generate screen projection data according to the display parameters; receive the screen projection data fed back by the communication terminal, and display the screen projection data according to the layout.
本申请实施例提供一种显示设备,包括:An embodiment of the present application provides a display device, including:
显示器,用于显示投屏界面;通信器,用于与终端设备投屏连接;控制器,被配置为执行:接收至少M个终端设备的投屏请求,从M个终端设备分别接收的媒体配置信息,匹配适应于屏幕的M个投屏窗口间的布局;其中,M≥1,所述媒体配置信息包括终端设备投屏的视频的宽高比例;控制显示器在投屏界面中按照所述布局显示M个投屏窗口,以及在每个投屏窗口内显示对应的终端设备发送的投屏数据。The display is used to display the screen projection interface; the communicator is used to connect with the terminal device for screen projection; the controller is configured to execute: receiving the screen projection request of at least M terminal devices, and receiving the media configuration respectively from the M terminal devices Information, matching the layout between M screen projection windows adapted to the screen; wherein, M≥1, the media configuration information includes the aspect ratio of the video projected by the terminal device; the control display follows the layout in the screen projection interface Display M screen projection windows, and display the screen projection data sent by the corresponding terminal device in each screen projection window.
本申请实施例提供一种显示设备,包括:显示器,被配置为显示用户界面;控制器,可运行一个播放应用和至少一个投屏协议应用,所述至少一个投屏协议应用均可连接至所述播放应用,被配置为:在接收到第一移动终端发送的第一投屏请求时,通过与所述第一 投屏请求相对应的第一投屏协议应用,建立与所述第一移动终端之间的投屏连接,以及将所述第一移动终端通过投屏连接传输过来的第一待投屏内容发送至播放应用;通过所述播放应用,在所述用户界面上创建第一播放窗口,所述第一播放窗口用于播放所述第一待投屏内容。An embodiment of the present application provides a display device, including: a display configured to display a user interface; a controller capable of running a playback application and at least one screen projection protocol application, and the at least one screen projection protocol application can be connected to the The playback application is configured to: when receiving a first screen projection request sent by a first mobile terminal, establish a connection with the first mobile terminal through a first screen projection protocol application corresponding to the first screen projection request. The screen projection connection between the terminals, and sending the first content to be projected by the first mobile terminal through the screen projection connection to the playback application; through the playback application, create the first playback on the user interface window, the first playing window is used to play the first content to be projected.
图1为本申请实施例中显示设备与控制装置之间操作场景的示意图;FIG. 1 is a schematic diagram of an operation scene between a display device and a control device in an embodiment of the present application;
图2为本申请实施例中显示设备的硬件配置框图;FIG. 2 is a block diagram of a hardware configuration of a display device in an embodiment of the present application;
图3为本申请实施例中显示设备软件配置示意图;FIG. 3 is a schematic diagram of display device software configuration in the embodiment of the present application;
图4为本申请实施例中显示设备应用程序的图标控件界面显示示意图;FIG. 4 is a schematic diagram of displaying an icon control interface of a display device application program in an embodiment of the present application;
图5为根据一些实施例的通信终端的结构示意图;Fig. 5 is a schematic structural diagram of a communication terminal according to some embodiments;
图6为根据一些实施例的通信终端的软件架构示意图;FIG. 6 is a schematic diagram of a software architecture of a communication terminal according to some embodiments;
图7为根据一些实施例的通信终端的用户界面示意图;7 is a schematic diagram of a user interface of a communication terminal according to some embodiments;
图8为根据一些实施例的投屏效果示意图;Fig. 8 is a schematic diagram of screen projection effects according to some embodiments;
图9为根据一些实施例的投屏操作流程示意图;Fig. 9 is a schematic diagram of a screen projection operation process according to some embodiments;
图10为根据一些实施例的多通信终端投屏连接示意图;Fig. 10 is a schematic diagram of screen projection connection of multiple communication terminals according to some embodiments;
图11为根据一些实施例的划分两个显示区域的效果示意图;Fig. 11 is a schematic diagram of the effect of dividing two display areas according to some embodiments;
图12为根据一些实施例的投屏画面动态显示方法流程示意图;Fig. 12 is a schematic flowchart of a method for dynamically displaying a projected screen according to some embodiments;
图13为根据一些实施例的划分三个显示区域的效果示意图;Fig. 13 is a schematic diagram of the effect of dividing three display areas according to some embodiments;
图14为根据一些实施例的响应连接指令检测连接数量的流程示意图;14 is a schematic flow diagram of detecting the number of connections in response to a connection instruction according to some embodiments;
图15为根据一些实施例的响应退出指令检测连接数量的流程示意图;15 is a schematic flow diagram of detecting the number of connections in response to an exit command according to some embodiments;
图16为根据一些实施例的匹配布局模板的流程示意图;Figure 16 is a schematic flow diagram of matching layout templates according to some embodiments;
图17为根据一些实施例的发送提示消息的流程示意图;Fig. 17 is a schematic flow diagram of sending a reminder message according to some embodiments;
图18为根据一些实施例的通信终端生成投屏数据流程示意图;Fig. 18 is a schematic flow diagram of a communication terminal generating screen projection data according to some embodiments;
图19为根据一些实施例的多路投屏显示方法流程图;Fig. 19 is a flowchart of a multi-channel projection display method according to some embodiments;
图20(a)为根据一些实施例的投屏路数为1且视频流为单竖屏场景下的布局示意图;Fig. 20(a) is a schematic layout diagram of a scene where the number of projection screens is 1 and the video stream is a single vertical screen according to some embodiments;
图20(b)为根据一些实施例的投屏路数为1且视频流为单横屏场景下的布局示意图;Fig. 20(b) is a schematic layout diagram of a scene where the number of projection screens is 1 and the video stream is a single horizontal screen according to some embodiments;
图21(a)为根据一些实施例的投屏路数为2且双横屏场景下的第一种布局示意图;Fig. 21(a) is a schematic diagram of the first layout in a scene where the number of projection screens is 2 and double horizontal screens according to some embodiments;
图21(b)为根据一些实施例的投屏路数为2且双横屏场景下的第二种布局示意图;Fig. 21(b) is a schematic diagram of a second layout in a scene where the number of screen projections is 2 and double horizontal screens according to some embodiments;
图21(c)为根据一些实施例的投屏路数为2且一横屏、一竖屏场景下的第一种布局示意图;Fig. 21(c) is a schematic diagram of the first layout in the scene where the number of projection screens is 2 and one horizontal screen and one vertical screen according to some embodiments;
图21(d)为根据一些实施例的投屏路数为2且一横屏、一竖屏场景下的第二种布局示意图;Fig. 21(d) is a schematic diagram of a second layout in a scenario where the number of projection screens is 2 and one horizontal screen and one vertical screen according to some embodiments;
图21(e)为根据一些实施例的投屏路数为2且双竖屏场景下的布局示意图;Fig. 21(e) is a schematic layout diagram in a scene where the number of projection screens is 2 and dual vertical screens according to some embodiments;
图22(a)为根据一些实施例的投屏路数为2且双横屏(宽高比例不同)场景下的布局的版本1的示意图;Fig. 22(a) is a schematic diagram of version 1 of the layout in the scenario where the number of projection screens is 2 and the dual landscape screens (with different aspect ratios) according to some embodiments;
图22(b)为根据一些实施例的投屏路数为2且双横屏(宽高比例不同)场景下的布局的版本2的示意图;Fig. 22(b) is a schematic diagram of version 2 of the layout in the scenario where the number of projection screens is 2 and the dual landscape screens (with different aspect ratios) according to some embodiments;
图22(c)为根据一些实施例的投屏路数为2且双竖屏(宽高比例不同)场景下的布局 的版本1的示意图;Fig. 22(c) is a schematic diagram of version 1 of the layout in a scenario where the number of projection screens is 2 and the dual vertical screens (with different aspect ratios) according to some embodiments;
图22(d)为根据一些实施例的投屏路数为2且双竖屏(宽高比例不同)场景下的布局的版本2的示意图;Fig. 22(d) is a schematic diagram of version 2 of the layout in the scenario where the number of projection screens is 2 and the dual vertical screens (with different aspect ratios) according to some embodiments;
图23(a)为根据一些实施例的投屏路数为3且一横屏、两竖屏场景下的布局的版本1的示意图;Fig. 23(a) is a schematic diagram of version 1 of the layout in a scenario where the number of projection screens is 3 and there is one horizontal screen and two vertical screens according to some embodiments;
图23(b)为根据一些实施例的投屏路数为3且一横屏、两竖屏场景下的布局的版本2的示意图;Fig. 23(b) is a schematic diagram of version 2 of the layout in a scene where the number of projection screens is 3 and there is one horizontal screen and two vertical screens according to some embodiments;
图23(c)为根据一些实施例的投屏路数为3且一横屏、两竖屏场景下的布局的版本3的示意图;Fig. 23(c) is a schematic diagram of version 3 of the layout in a scenario where the number of projection screens is 3 and there is one horizontal screen and two vertical screens according to some embodiments;
图24为根据一些实施例的终端设备与显示设备的交互流程图;Fig. 24 is a flowchart of interaction between a terminal device and a display device according to some embodiments;
图25为根据一些实施例的显示设备200中应用程序的图标控件界面显示图;FIG. 25 is a display diagram of an icon control interface of an application program in the display device 200 according to some embodiments;
图26为根据一些实施例的显示设备中控制器的配置图;Fig. 26 is a configuration diagram of a controller in a display device according to some embodiments;
图27为根据一些实施例的显示设备中控制器执行投屏方法的一种流程示意图;Fig. 27 is a schematic flowchart of a screen projection method performed by a controller in a display device according to some embodiments;
图28为根据一些实施例的显示设备中控制器执行投屏方法的另一种流程示意图;Fig. 28 is another schematic flowchart of a screen projection method performed by a controller in a display device according to some embodiments;
图29为根据一些实施例的显示设备中显示器上播放窗口的示意图;Fig. 29 is a schematic diagram of a playback window on a display in a display device according to some embodiments;
图30为根据一些实施例的多个移动终端与显示设备之间投屏交互的结构示意图;Fig. 30 is a schematic structural diagram of screen projection interaction between multiple mobile terminals and a display device according to some embodiments;
图31为根据一些实施例的第一移动终端与显示设备之间投屏交互的时序图。Fig. 31 is a sequence diagram of screen projection interaction between a first mobile terminal and a display device according to some embodiments.
为使本申请的目的和实施方式更加清楚,下面将结合本申请示例性实施例中的附图,对本申请示例性实施方式进行清楚、完整地描述,显然,描述的示例性实施例仅是本申请一部分实施例,而不是全部的实施例。In order to make the purpose and implementation of the application clearer, the following will clearly and completely describe the exemplary implementation of the application in conjunction with the accompanying drawings in the exemplary embodiment of the application. Obviously, the described exemplary embodiment is only the present application. Claim some of the examples, not all of them.
需要说明的是,本申请中对于术语的简要说明,仅是为了方便理解接下来描述的实施方式,而不是意图限定本申请的实施方式。除非另有说明,这些术语应当按照其普通和通常的含义理解。It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementations described below, and is not intended to limit the implementations of this application. These terms are to be understood according to their ordinary and usual meaning unless otherwise stated.
图1为根据实施例中显示设备与控制装置之间操作场景的示意图。如图1所示,用户可通过智能设备300或控制装置100操作显示设备200。Fig. 1 is a schematic diagram of an operation scene between a display device and a control device according to an embodiment. As shown in FIG. 1 , the user can operate the display device 200 through the smart device 300 or the control device 100 .
在一些实施例中,控制装置100可以是遥控器,遥控器和显示设备的通信包括红外协议通信或蓝牙协议通信,及其他短距离通信方式,通过无线或有线方式来控制显示设备200。用户可以通过遥控器上按键、语音输入、控制面板输入等输入用户指令,来控制显示设备200。In some embodiments, the control device 100 may be a remote control, and the communication between the remote control and the display device includes infrared protocol communication, bluetooth protocol communication, and other short-distance communication methods, and the display device 200 is controlled wirelessly or wiredly. The user can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, and the like.
在一些实施例中,也可以使用智能设备300(如通信终端、平板电脑、计算机、笔记本电脑等)以控制显示设备200。例如,使用在智能设备上运行的应用程序控制显示设备200。In some embodiments, the smart device 300 (such as a communication terminal, a tablet computer, a computer, a notebook computer, etc.) can also be used to control the display device 200 . For example, the display device 200 is controlled using an application program running on the smart device.
在一些实施例中,显示设备200还可以采用除了控制装置100和智能设备300之外的方式进行控制,例如,可以通过显示设备200设备内部配置的获取语音指令的模块直接接收用户的语音指令控制,也可以通过显示设备200设备外部设置的语音控制设备来接收用户的语音指令控制。In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the module for obtaining voice commands configured inside the display device 200 can directly receive the user's voice command control , the user's voice command control can also be received through the voice control device installed outside the display device 200.
在一些实施例中,显示设备200还与服务器400进行数据通信。可允许显示设备200通 过局域网(LAN)、无线局域网(WLAN)和其他网络进行通信连接。服务器400可以向显示设备200提供各种内容和互动。In some embodiments, the display device 200 also performs data communication with the server 400 . The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200 .
图2为根据一些实施例中控制装置100的配置框图。如图2所示,控制装置100包括控制器110、通信接口130、用户输入/输出接口140、存储器、供电电源。控制装置100可接收用户的输入操作指令,且将操作指令转换为显示设备200可识别和响应的指令,起用户与显示设备200之间进行交互的中介作用。Fig. 2 is a configuration block diagram of the control device 100 according to some embodiments. As shown in FIG. 2 , the control device 100 includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply. The control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, and acts as an intermediary for the interaction between the user and the display device 200 .
图3示出了根据示例性实施例中显示设备200的硬件配置框图。FIG. 3 shows a block diagram of a hardware configuration of a display device 200 according to an exemplary embodiment.
在一些实施例中,显示设备200包括调谐解调器210、通信器220、检测器230、外部装置接口240、控制器250、显示器260、音频输出接口270、存储器、供电电源、用户接口中的至少一种。In some embodiments, the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface. at least one.
在一些实施例中控制器包括处理器,视频处理器,音频处理器,图形处理器,RAM,ROM,用于输入/输出的第一接口至第n接口。In some embodiments the controller includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, a first interface to an nth interface for input/output.
在一些实施例中,通信器220是用于根据各种通信协议类型与外部设备或服务器进行通信的组件。例如:通信器可以包括Wifi模块,蓝牙模块,有线以太网模块等其他网络通信协议芯片或近场通信协议芯片,以及红外接收器中的至少一种。In some embodiments, the communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module and other network communication protocol chips or near field communication protocol chips, and an infrared receiver.
在一些实施例中,用户输入接口,可用于接收控制装置100(如:红外遥控器等)的控制信号。In some embodiments, the user input interface can be used to receive a control signal of the control device 100 (such as an infrared remote controller, etc.).
在一些实施例中,检测器230可以包括光接收器、图像采集器和声音采集器等。In some embodiments, the detector 230 may include a light receiver, an image collector, a sound collector, and the like.
在一些实施例中,外部装置接口240可以包括但不限于如下:高清多媒体接口(HDMI)、模拟或数据高清分量输入接口(分量)、复合视频输入接口(CVBS)、USB输入接口(USB)、RGB端口等任一个或多个接口。In some embodiments, the external device interface 240 may include, but is not limited to, the following: high-definition multimedia interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), Any one or more interfaces such as RGB ports.
在一些实施例中,显示设备的系统可以包括内核(Kernel)、命令解析器(shell)、文件系统和应用程序。In some embodiments, the system of the display device may include a kernel (Kernel), a command parser (shell), a file system and application programs.
参见图4,在一些实施例中,将系统分为四层,从上至下分别为应用程序(Applications)层(应用层),应用程序框架(Application Framework)层(框架层),安卓运行时(Android runtime)和系统库层(系统运行库层),以及内核层。Referring to Fig. 4, in some embodiments, the system is divided into four layers, from top to bottom are applications (Applications) layer (application layer), application framework (Application Framework) layer (framework layer), Android runtime (Android runtime) and system library layer (system runtime library layer), as well as the kernel layer.
图5示出了通信终端300的结构示意图。FIG. 5 shows a schematic structural diagram of a communication terminal 300 .
下面以通信终端300为例对实施例进行具体说明。应该理解的是,图5所示通信终端300仅是一个范例,并且通信终端300可以具有比图5中所示的更多的或者更少的部件,可以组合两个或多个的部件,或者可以具有不同的部件配置。图中所示出的各种部件可以在包括一个或多个信号处理和/或专用集成电路在内的硬件、软件、或硬件和软件的组合中实现。The embodiment will be specifically described below by taking the communication terminal 300 as an example. It should be understood that the communication terminal 300 shown in FIG. 5 is only an example, and that the communication terminal 300 may have more or fewer components than those shown in FIG. 5, two or more components may be combined, or Different component configurations are possible. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.
如图5所示,通信终端300包括:射频(radio frequency,RF)电路310、存储器320、显示单元330、摄像头340、传感器350、音频电路360、无线保真(Wireless Fidelity,Wi-Fi)电路370、处理器380、蓝牙电路381、以及电源390等部件。As shown in Figure 5, the communication terminal 300 includes: a radio frequency (radio frequency, RF) circuit 310, a memory 320, a display unit 330, a camera 340, a sensor 350, an audio circuit 360, and a wireless fidelity (Wireless Fidelity, Wi-Fi) circuit 370, processor 380, Bluetooth circuit 381, and power supply 390 and other components.
图6是本发明实施例的通信终端300的软件结构框图。FIG. 6 is a block diagram of the software structure of the communication terminal 300 according to the embodiment of the present invention.
分层架构将软件分成若干个层,每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中,将Android系统分为四层,从上至下分别为应用程序层,应用程序框架层,安卓运行时(Android runtime)和系统库,以及内核层。The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, which are respectively the application program layer, the application program framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer from top to bottom.
下面结合捕获拍照场景,说明通信终端300软件以及硬件的工作流程。The workflow of the software and hardware of the communication terminal 300 will be described below in conjunction with capturing and photographing scenes.
当触摸屏331接收到触摸操作,相应的硬件中断被发给内核层。内核层将触摸操作加 工成原始输入事件(包括触摸坐标,触摸操作的时间戳等信息)。原始输入事件被存储在内核层。应用程序框架层从内核层获取原始输入事件,识别该输入事件所对应的控件。以该触摸操作是触摸单击操作,该单击操作所对应的控件为相机应用图标的控件为例,相机应用调用应用框架层的接口,启动相机应用,进而通过调用内核层启动摄像头驱动,通过摄像头340捕获静态图像或视频。When the touch screen 331 receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into original input events (including touch coordinates, time stamps of touch operations, and other information). Raw input events are stored at the kernel level. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Take the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon as an example. The camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer. Camera 340 captures still images or video.
如图7所示,用户通过触摸用户界面上的应用图标可以打开相应的应用程序,或者通过触摸用户界面上的文件夹图标可以打开相应的文件夹。As shown in FIG. 7 , the user can open a corresponding application program by touching an application icon on the user interface, or open a corresponding folder by touching a folder icon on the user interface.
基于上述显示设备200和通信终端300,可以通过建立投屏连接,使显示设备200播放显示通信终端300中的内容,如图8所示。在一些实施例中,如图9所示,为了建立投屏连接关系,用户可以通过通信终端300发出投屏连接请求,并通过WiFi网络将投屏连接请求发送给显示设备200。显示设备200再根据投屏连接请求完成传输协议配置,从而与通信终端300之间建立投屏数据的传输通道。Based on the above display device 200 and communication terminal 300 , by establishing a screen projection connection, the display device 200 can play and display the content in the communication terminal 300 , as shown in FIG. 8 . In some embodiments, as shown in FIG. 9 , in order to establish a screen projection connection relationship, the user may send a screen projection connection request through the communication terminal 300 , and send the screen projection connection request to the display device 200 through the WiFi network. The display device 200 completes the transmission protocol configuration according to the screen projection connection request, so as to establish a screen projection data transmission channel with the communication terminal 300 .
显示设备200还可以通过不同连接方式实现与通信终端300之间的投屏连接。例如,在显示设备200和通信终端300接入同一个无线局域网时,可以基于WiFi网络建立投屏连接。又例如,在显示设备200和通信终端300上均设有NFC(Near Field Communication,近场通信)组件时,可以通过NFC组件建立投屏连接关系。显然,显示设备200与通信终端300之间还可以采用其他有线或无线连接方式,从而建立投屏连接关系,例如RF射频连接、红外连接、蜂窝网络以及本领域技术人员在上述实施例提供的连接方式基础上所能够联想到的其他连接方式。The display device 200 can also realize the screen projection connection with the communication terminal 300 through different connection methods. For example, when the display device 200 and the communication terminal 300 are connected to the same wireless local area network, a screen projection connection can be established based on the WiFi network. For another example, when both the display device 200 and the communication terminal 300 are provided with NFC (Near Field Communication, near field communication) components, the screen projection connection relationship can be established through the NFC components. Obviously, other wired or wireless connection methods can also be used between the display device 200 and the communication terminal 300 to establish a screen projection connection relationship, such as RF radio frequency connection, infrared connection, cellular network, and connections provided by those skilled in the art in the above embodiments Other connection methods that can be thought of on the basis of the method.
在建立投屏连接后,显示设备200可以通过投屏数据通道接收通信终端300的相关数据,从而在显示设备200中播放相关内容。在不同的投屏数据传输协议下,显示设备200获取的投屏数据形式不同。例如,在DLNA(Digital Living Network Alliance,数字生活网络联盟)协议下,显示设备200可以通过投屏数据通道获取通信终端300中某一资源的链接地址,并通过访问该链接地址获取媒资内容数据,从而播放该媒资数据。同时显示设备200还可以在播放媒资数据时,通过投屏数据通道获取通信终端300发送的控制指令,如快进、暂停、停止投屏等。例如,通过通讯协议建立好连接后,手机等通信终端300与显示设备200会进行设备功能与网络条件的协商,以选择适合的影音传输格式。建立影音串流的联机协议(Session)后,再通过一连串的实时串流协议(RTSP)控制命令,以控制影音串流的播放及终止。After the screen projection connection is established, the display device 200 may receive relevant data of the communication terminal 300 through the screen projection data channel, so as to play relevant content on the display device 200 . Under different screen projection data transmission protocols, the screen projection data acquired by the display device 200 has different forms. For example, under the DLNA (Digital Living Network Alliance, Digital Living Network Alliance) protocol, the display device 200 can obtain the link address of a certain resource in the communication terminal 300 through the screen projection data channel, and obtain media asset content data by accessing the link address , so as to play the media asset data. At the same time, the display device 200 can also obtain control instructions sent by the communication terminal 300 through the screen projection data channel when playing media asset data, such as fast forward, pause, and stop projection. For example, after a connection is established through a communication protocol, the communication terminal 300 such as a mobile phone and the display device 200 will negotiate device functions and network conditions to select a suitable video and audio transmission format. After establishing the session of the video stream, a series of real-time streaming protocol (RTSP) control commands are used to control the playback and termination of the video stream.
而在Miracast投屏、Airplay投屏、NFC镜像投屏、乐播投屏等其他投屏连接协议下,显示设备200可以直接从通信终端300获取视频数据流。该视频数据流可以是通信终端300的显示单元310最终显示画面对应的视频数据流,即该视频数据流中包括通信终端300中的媒资内容画面、UI界面以及其他可在显示单元310中显示的视频画面内容。视频数据流也可以仅为通信终端300中可被获取视频数据,如正在播放的媒资视频数据等。此时,通信终端300可以作为显示设备200的一个信号源,显示设备200可以通过投屏数据通道不断获取视频数据,并实时显示在显示器275中,实现与通信终端300同步显示的效果。Under other screen projection connection protocols such as Miracast screen projection, Airplay screen projection, NFC mirror projection, and Music Broadcast projection, the display device 200 can directly obtain video data streams from the communication terminal 300 . The video data stream may be the video data stream corresponding to the final display screen of the display unit 310 of the communication terminal 300, that is, the video data stream includes the content screen of the media assets in the communication terminal 300, the UI interface, and other information that can be displayed on the display unit 310. content of the video screen. The video data stream may also be only the video data that can be acquired in the communication terminal 300, such as the video data of the media assets being played. At this time, the communication terminal 300 can be used as a signal source of the display device 200 , and the display device 200 can continuously obtain video data through the projection data channel, and display it on the display 275 in real time, so as to realize the synchronous display effect with the communication terminal 300 .
通信终端300根据不同的投屏协议能够获取不同的投屏数据。例如,对于系统应用或直接在系统层级进行投屏,通信终端300可以采用Miracast投屏、Airplay投屏或WiDi投屏协议建立投屏数据通道。The communication terminal 300 can acquire different screen projection data according to different screen projection protocols. For example, for system application or screen projection directly at the system level, the communication terminal 300 may adopt a Miracast screen projection, Airplay screen projection or WiDi screen projection protocol to establish a screen projection data channel.
Miracast投屏可适用于Android设备,即Android设备可通过Miracast投屏的方式分享视 频画面。通常,Android5.0以上的智能设备内置Miracast协议,因此可以在无需安装任何软件的情况下直接通过投屏设备与接收端设备WiFi网卡间的WiFiP2P功能进行直接无线投屏。此外,Miracast投屏还支持音视频同步传输。但是,由于Android设备太多太杂,Miracast不能完全兼容所有Android设备,并且基于Miracast的投屏没有投屏码功能,不能阻止恶意或者误投屏。Miracast screen projection is applicable to Android devices, that is, Android devices can share video images through Miracast screen projection. Generally, smart devices above Android 5.0 have built-in Miracast protocol, so they can directly perform wireless screen projection through the WiFi P2P function between the screen projection device and the WiFi network card of the receiver device without installing any software. In addition, Miracast also supports simultaneous transmission of audio and video. However, due to too many Android devices, Miracast is not fully compatible with all Android devices, and Miracast-based screencasting does not have a screencasting code function, which cannot prevent malicious or accidental screencasting.
Airplay投屏是由苹果公司推出的一种无线播放技术,适用于iOS和Mac系统的无线投屏。苹果系列设备内置Airplay投屏功能,配合支持AirPlay server服务的投屏器使用,效果清晰稳定,协议自带投屏码功能。但是,需保证苹果设备与投屏器在同一个局域网下(同网段)使用,不能跨网段、跨VLAN使用。Airplay screen projection is a wireless playback technology introduced by Apple, which is suitable for wireless screen projection of iOS and Mac systems. Apple series devices have a built-in Airplay screen projection function, which can be used with a screen projector that supports AirPlay server service, and the effect is clear and stable. The protocol comes with a screen projection code function. However, it is necessary to ensure that the Apple device and the screen projector are used in the same LAN (same network segment), and cannot be used across network segments or VLANs.
WiDi投屏是Intel主导的一种无线播放技术,也基于WiFi Direct直连技术。Windows8/10笔记本电脑原生支持WiDi投屏,可以无需安装任何软件直接投屏,并且支持USB反向控制功能。WiDi screen projection is a wireless playback technology led by Intel, and it is also based on WiFi Direct direct connection technology. Windows 8/10 laptops natively support WiDi screen projection, which can be directly screen projection without installing any software, and supports USB reverse control function.
上述三种投屏协议不仅能够获取通信终端300中的视频数据流,而且能够获取通信终端300中的音频数据流,从而可以将视频数据和音频数据同步发送给显示设备200,使显示设备200能够同步播放视频和音频。由于上述Miracast投屏协议和Airplay投屏协议需要系统级应用权限,因此Miracast投屏协议和Airplay投屏协议通常仅适用于系统应用,或者同一厂商的显示设备200和通信终端300通过操作系统内置规则直接建立投屏连接。The above three screen projection protocols can not only obtain the video data stream in the communication terminal 300, but also can obtain the audio data stream in the communication terminal 300, so that the video data and audio data can be synchronously sent to the display device 200, so that the display device 200 can Play video and audio synchronously. Since the above-mentioned Miracast screen projection protocol and Airplay screen projection protocol require system-level application permissions, the Miracast screen projection protocol and Airplay screen projection protocol are usually only applicable to system applications, or the display device 200 and communication terminal 300 of the same manufacturer pass the built-in rules of the operating system. Directly establish a screencast connection.
显示设备200可以基于上述实施例中提供的任一种投屏方式,与通信终端300建立投屏连接。在一些实施例中,显示设备200还可以通过上述投屏方式中的一种或多种,与多个通信终端建立投屏连接关系。例如,显示设备200可以同时与两个通信终端300建立投屏连接,并在显示画面中划分出显示区域,用于分别显示两个通信终端300共享的投屏画面。即如图10所示,可以在显示设备200所显示用户界面的左侧显示终端A共享的投屏画面,在用户界面的右侧显示终端B共享的投屏画面。The display device 200 may establish a screen projection connection with the communication terminal 300 based on any one of the screen projection methods provided in the foregoing embodiments. In some embodiments, the display device 200 may also establish a screen projection connection relationship with multiple communication terminals through one or more of the above screen projection methods. For example, the display device 200 may establish a screen projection connection with two communication terminals 300 at the same time, and divide a display area in the display screen for respectively displaying the screen projection screen shared by the two communication terminals 300 . That is, as shown in FIG. 10 , the screen projection image shared by terminal A may be displayed on the left side of the user interface displayed on the display device 200 , and the screen projection image shared by terminal B may be displayed on the right side of the user interface.
为了能够在显示设备200上显示多个通信终端300共享的投屏画面,在一些实施例中,当显示设备200与多个通信终端300建立投屏连接时,显示设备200可以对用于投屏显示的用户界面进行区域划分,如二等分、三等分、四等分等。通过区域划分所获得多个显示区域后,显示设备200再分别将投屏画面显示在每个划分区域中。In order to be able to display the screen projection screen shared by multiple communication terminals 300 on the display device 200, in some embodiments, when the display device 200 establishes a screen projection connection with multiple communication terminals 300, the display device 200 can The displayed user interface is divided into areas, such as bisecting, trisecting, and quartering. After the multiple display areas are obtained through area division, the display device 200 displays the projected screen image in each divided area.
另外,通信终端300可以按照使用方式的不同,可能处于不同的放置状态,如对于智能手机等通信终端300,其在使用中可能处于竖屏状态和横屏状态。其中,竖屏状态是指所显示画面的宽度小于高度的状态,横屏状态是指所显示画面的宽度大于高度的状态。在不同的放置状态下,通信终端300可以呈现不同的画面布局方式。基于此,当显示设备200通过区域划分方式显示多个通信终端300的投屏画面时,各显示区域会因为通信终端300所处放置状态的不同,而呈现为不同的画面比例。In addition, the communication terminal 300 may be placed in different states according to different ways of use, for example, for a communication terminal 300 such as a smart phone, it may be in a portrait state or a landscape state during use. Wherein, the vertical screen state refers to a state in which the width of the displayed picture is smaller than the height, and the horizontal screen state refers to the state in which the width of the displayed picture is greater than the height. In different placement states, the communication terminal 300 may present different screen layout modes. Based on this, when the display device 200 displays the projected screen images of multiple communication terminals 300 by area division, each display area will present different screen ratios due to the different placement states of the communication terminals 300 .
例如,如图11所示,对于显示分辨率为4096×2160的显示设备200,在用户界面中心位置竖向二等分获得两个显示区域的过程中,区域划分可以获得两个宽度为2048、高度为2160的显示区域,即获得宽度小于高度的竖向形状区域,这种形式的显示区域更适合显示竖屏状态的投屏画面。因此,当该区域对应的通信终端300处于横屏状态时,会导致投屏画面不能充满大部分显示区域,造成显示区域的浪费。For example, as shown in FIG. 11 , for a display device 200 with a display resolution of 4096×2160, in the process of vertically bisecting the center position of the user interface to obtain two display regions, two regions with a width of 2048, A display area with a height of 2160 is to obtain a vertical shape area whose width is smaller than the height, and this type of display area is more suitable for displaying the projection screen in the portrait state. Therefore, when the communication terminal 300 corresponding to this area is in a landscape orientation state, most of the display area cannot be filled with the projected screen image, resulting in a waste of the display area.
另外,在多个通信终端300所处的放置状态不相同时,显示设备200往往通过拉伸填充的方式,使每个投屏画面都能够充满显示区域。这种显示方式会使投屏画面发生较大变形, 影响用户的观影体验。In addition, when the placement states of the multiple communication terminals 300 are different, the display device 200 usually stretches and fills each screen projection image to fill the display area. This display mode will cause a large deformation of the projected screen, which will affect the user's viewing experience.
为了充分利用显示区域、减小画面拉伸变形情况,本申请的部分实施例中还提供一种投屏画面动态显示方法,应用于显示设备200,用于提高用户的观影体验。其中,能够应用所述投屏画面动态显示方法的显示设备200包括:显示器260、通信器220以及控制器250。所述显示器260被配置为显示投屏界面。所述通信器220被配置为与通信终端300建立投屏连接。所述控制器250则被配置为执行上述投屏画面动态显示方法,如图12所示,具体包括以下内容:In order to make full use of the display area and reduce stretching and deformation of the screen, some embodiments of the present application also provide a method for dynamically displaying the screen projection screen, which is applied to the display device 200 and used to improve the user's viewing experience. Wherein, the display device 200 capable of applying the method for dynamically displaying the projected screen includes: a display 260 , a communicator 220 and a controller 250 . The display 260 is configured to display a screen projection interface. The communicator 220 is configured to establish a screen projection connection with the communication terminal 300 . The controller 250 is configured to execute the above method for dynamically displaying the projection screen, as shown in FIG. 12 , which specifically includes the following content:
检测通信终端300的连接数量。显示设备200可以在运行过程中实时检测运行状态,并在确定需要触发显示投屏画面的过程中,检测与显示设备200建立投屏连接的通信终端300的数量。通信终端300的连接数量,可以通过对显示设备200所在无线局域网络中各通信终端进行分别检测判断获得。例如,在需要检测通信终端300的连接数量时,显示设备200的控制器250可以通过通信器220,调用当前无线网络中接入的所有通信终端300,再通过抓取每个通信终端的传输数据,并从中识别投屏数据标志,以检测当前通信终端300是否与显示设备200建立通信连接。The number of connections of communication terminal 300 is detected. The display device 200 can detect the running state in real time during the running process, and detect the number of communication terminals 300 that have established a screen projection connection with the display device 200 during the process of determining that the screen projection screen needs to be triggered to be displayed. The number of connections of the communication terminal 300 can be obtained by detecting and judging each communication terminal in the wireless local area network where the display device 200 is located. For example, when it is necessary to detect the number of connections of communication terminals 300, the controller 250 of the display device 200 can use the communicator 220 to call all the communication terminals 300 connected to the current wireless network, and then capture the transmission data of each communication terminal , and identify the screen projection data flag therefrom, so as to detect whether the current communication terminal 300 establishes a communication connection with the display device 200 .
连接数量还可以通过显示设备200实时维护的数据表进行检测获得。例如,在显示设备200的数据库中,可以随着投屏连接过程实时维护一个连接设备列表,在连接设备列表中的各通信终端300可以根据是否在线并建立投屏连接关系,设置不同的状态值,即通信终端300在线且连接可用时,状态值“devices online=1”;而通信终端300离线或者连接不可用时,状态值“devices online=0”。基于此,显示设备200可以在需要对连接数量进行检测时,遍历状态值“devices online=1”的通信终端300的数量。The number of connections can also be obtained by detecting the data table maintained by the display device 200 in real time. For example, in the database of the display device 200, a connected device list can be maintained in real time along with the screen projection connection process, and each communication terminal 300 in the connected device list can set different status values according to whether it is online or not and establishes a screen projection connection relationship , that is, when the communication terminal 300 is online and the connection is available, the status value is "devices online=1"; and when the communication terminal 300 is offline or the connection is unavailable, the status value is "devices online=0". Based on this, the display device 200 may traverse the number of communication terminals 300 with the status value "devices online=1" when the number of connections needs to be detected.
显示设备200可以在判断当前投屏界面布局需要改变时执行对通信终端300数量的检测。例如,显示设备200可以实时监听每个通信终端300在投屏连接中的接入状态,当检测到任一通信终端300新接入或退出连接时,显示设备200可以执行检测通信终端300的连接数量的步骤。The display device 200 may perform detection of the number of communication terminals 300 when judging that the layout of the current screen projection interface needs to be changed. For example, the display device 200 can monitor the access status of each communication terminal 300 in the screen projection connection in real time, and when it detects that any communication terminal 300 is newly connected or exits the connection, the display device 200 can perform a connection detection function of the communication terminal 300. number of steps.
在一些实施例中,显示设备200还可以周期性检测通信终端300的连接数量。例如,显示设备200可以在开始建立投屏连接后,通过各通信终端300的心跳命令对通信终端300连接数量进行检测。通过检测通信终端300的连接数量,显示设备200可以及时对通信终端300进行投屏连接状态进行检测,从而在通信终端因超出网络范围或其他硬件故障原因离线时,可以对投屏界面的进行调整。In some embodiments, the display device 200 may also periodically detect the number of connections of the communication terminal 300 . For example, the display device 200 may detect the number of connections of the communication terminals 300 through heartbeat commands of each communication terminal 300 after the screen projection connection is established. By detecting the number of connections of the communication terminal 300, the display device 200 can detect the connection status of the communication terminal 300 for screen projection in time, so that when the communication terminal is offline due to network range or other hardware failures, the screen projection interface can be adjusted. .
在一些实施例中,显示设备200还可以通过对用户的控制指令进行检测,并针对部分控制指令做出响应,检测通信终端300的连接数量。例如,当显示设备200检测到用户输入了用于对页面布局方式进行调整的控制指令时,显示设备200可以执行检测通信终端300的连接数量的步骤,以触发显示设备200后续对投屏数据获取方式的调整。In some embodiments, the display device 200 may also detect the number of connections of the communication terminal 300 by detecting the user's control instructions and responding to some of the control instructions. For example, when the display device 200 detects that the user has input a control command for adjusting the page layout, the display device 200 may perform the step of detecting the number of connections of the communication terminal 300, so as to trigger the display device 200 to subsequently acquire the projection data. way of adjustment.
在检测到通信终端300的连接数量后,显示设备200可以根据检测到的连接数量规划投屏画面的布局方式,以获得投屏画面的显示参数。显示设备200中可以内置多种布局方式模板,在获取连接数量后,显示设备200可以根据连接数量匹配响应的模板。After detecting the number of connections of the communication terminal 300, the display device 200 may plan the layout of the projected screen according to the detected number of connections, so as to obtain display parameters of the projected screen. The display device 200 may have multiple layout templates built in, and after obtaining the number of connections, the display device 200 may match the corresponding template according to the number of connections.
对于不同的显示设备200形式,以及不同的投屏连接用户,内置的布局方式模板也可以不同。例如,对于分享手机主界面演示操作的投屏连接,由于手机主界面操作多处于竖屏状态,因此布局方式模板可以是通过竖直方向等距离划分方式获得的布局模板,即在显示设备200的屏幕上划分三个宽度小于高度的竖向区域;而对于分享游戏界面操作的投屏 连接,由于游戏界面多处于横屏状态,因此布局方式模板可以是通过水平方向等距离划分方式获得布局模板,即在显示设备200的屏幕上划分三个宽度大于高度的横向区域,用于显示手机共享的游戏画面。For different forms of the display device 200 and different screen projection connection users, the built-in layout templates may also be different. For example, for the screen projection connection for sharing the demonstration operation of the main interface of the mobile phone, since the main interface of the mobile phone is mostly in the vertical screen state, the layout mode template can be a layout template obtained by dividing equidistantly in the vertical direction, that is, on the display device 200. The screen is divided into three vertical areas whose width is smaller than the height; and for the screencasting connection for sharing the game interface operation, since the game interface is mostly in the horizontal screen state, the layout template can be obtained by dividing equidistantly in the horizontal direction. That is, the screen of the display device 200 is divided into three horizontal regions whose width is greater than the height, for displaying the game screen shared by the mobile phone.
在一些实施例中,当显示设备200没有明确设置投屏连接通道时,为了能够充分利用显示区域,显示设备200可以按照最大化显示的原则规划画面布局。例如,在显示3个投屏画面区域时,可以保持一个竖向区域和两个横向区域。如此,可以每个显示区域都保持为16:9或9:16的形状,以适应通信终端300的画面比例。In some embodiments, when the display device 200 does not explicitly set a screen projection connection channel, in order to make full use of the display area, the display device 200 may plan the screen layout according to the principle of maximizing display. For example, when displaying 3 projection screen areas, one vertical area and two horizontal areas may be maintained. In this way, each display area can maintain a shape of 16:9 or 9:16, so as to adapt to the aspect ratio of communication terminal 300 .
在一些实施例中,显示设备200还可以在检测到通信终端300的连接数量后,可以通过向用户界面中添加显示窗口的方式,以基于连接数量规划布局方式。例如,当检测到当前显示设备200建立投屏连接的通信终端300数量为2个时,可以向用户界面添加2个显示窗口,用于分别显示每个通信终端300共享的投屏画面。每个显示窗口可以具有固定的形状和比例,并且可以要求显示设备200按照该比例提供显示画面。In some embodiments, after detecting the number of connections of the communication terminal 300, the display device 200 may add a display window to the user interface to plan a layout based on the number of connections. For example, when it is detected that the number of communication terminals 300 with which the current display device 200 has established a screen projection connection is 2, two display windows may be added to the user interface for respectively displaying the screen projection images shared by each communication terminal 300 . Each display window may have a fixed shape and ratio, and the display device 200 may be required to provide a display image according to the ratio.
在一些实施例中,当使用多个显示窗口显示多个投屏画面时,如果与显示设备200建立投屏连接的通信终端300的数量较多时,多个显示窗口之间还可以保持在特定的层级关系。例如,可以按照与显示设备200建立投屏连接的顺序,设置显示窗口的层级关系,即最后建立投屏连接的通信终端300所共享的投屏画面显示在最上层。In some embodiments, when multiple display windows are used to display multiple screen projection images, if the number of communication terminals 300 that establish screen projection connections with the display device 200 is large, the multiple display windows can also be kept at a specific hierarchical relationship. For example, the hierarchical relationship of the display windows may be set according to the order in which the screen projection connection is established with the display device 200 , that is, the screen projection screen shared by the communication terminal 300 that establishes the screen projection connection last is displayed on the top layer.
在显示设备200规划投屏画面的布局方式后,显示设备200还可以根据规划的布局方式确定投屏画面的显示参数。显示参数用于表示显示设备200所规划的布局方式中,每个显示区域的画面比例、尺寸等内容,以及根据每个显示区域的比例、尺寸所计算获得的投屏数据要求,如码率、分辨率等。After the display device 200 plans the layout of the projection screen, the display device 200 may also determine display parameters of the projection screen according to the planned layout. The display parameters are used to indicate the screen ratio and size of each display area in the layout mode planned by the display device 200, and the projection data requirements calculated according to the ratio and size of each display area, such as code rate, resolution etc.
根据确定的显示参数,显示设备200可以向投屏画面对应的通信终端300发送通知消息。其中,所述通知消息包括显示参数,所述通知消息用于使通信终端300按照显示参数生成投屏数据。例如,如图13所示,显示设备200在检测到当前通信终端300的连接数量为3个时,可以通过规划布局方式确定三个投屏画面的显示区域,即:区域一为分辨率“1080×1920”的竖向区域;区域二和区域三为分辨率“1280×720”的横向区域。基于上述显示参数,显示设备200可以生成通知消息,并将生成的通知消息发送给通信终端300,以使通信终端300可以按照上述分辨率生成投屏数据。According to the determined display parameters, the display device 200 may send a notification message to the communication terminal 300 corresponding to the screen projection screen. Wherein, the notification message includes a display parameter, and the notification message is used to enable the communication terminal 300 to generate screen projection data according to the display parameter. For example, as shown in FIG. 13 , when the display device 200 detects that the current number of connections to the communication terminal 300 is three, it can determine the display areas of the three projection screens by planning the layout, that is, the first area is the resolution “1080 ×1920” vertical area; area two and area three are horizontal areas with a resolution of “1280×720”. Based on the above display parameters, the display device 200 can generate a notification message, and send the generated notification message to the communication terminal 300, so that the communication terminal 300 can generate screen projection data according to the above resolution.
为了生成投屏数据,通信终端300可以内置录屏程序,并在收到显示设备200发送的通知消息后,启动录屏程序对通知消息进行解析,提取出显示参数。再根据显示参数设置录屏程序中的画面参数。如显示参数为竖向、1080×1920时,通信终端300可以控制录屏程序的录屏过程,使生成的录屏视频画面的分辨率为1080×1920。In order to generate screen projection data, the communication terminal 300 may have a built-in screen recording program, and after receiving the notification message sent by the display device 200, start the screen recording program to analyze the notification message and extract the display parameters. Then set the screen parameters in the screen recording program according to the display parameters. For example, when the display parameter is vertical and 1080×1920, the communication terminal 300 can control the screen recording process of the screen recording program so that the resolution of the generated screen recording video image is 1080×1920.
需要说明的是,如果通信终端300的显示分辨率与通知消息中设定的分辨率不相符时,通信终端300可以通过填充或裁切的方式,对画面进行处理以使生成的投屏数据符合显示参数中规定的形式。例如,显示设备200中设置的每个显示区域的画面比例为9:16,但通信终端300的屏幕比例为9:18时,为了能够显示完全,通信终端300可以对显示画面执行缩小处理,使投屏画面可以在高度上显示完全,此时,生成的投屏数据所对应画面中,将会在两侧填充黑色区域,以使整体画面维持在9:16的画面比例。It should be noted that, if the display resolution of the communication terminal 300 does not match the resolution set in the notification message, the communication terminal 300 can process the screen by padding or cropping so that the generated screen projection data conforms to the resolution set in the notification message. Displays the form specified in the argument. For example, the screen ratio of each display area set in the display device 200 is 9:16, but when the screen ratio of the communication terminal 300 is 9:18, in order to be able to display completely, the communication terminal 300 can perform a reduction process on the display screen, so that The projection screen can be fully displayed in height. At this time, in the screen corresponding to the generated projection data, black areas will be filled on both sides to maintain the overall screen ratio of 9:16.
在通信终端300生成投屏数据以后,可以将生成的投屏数据反馈给显示设备200,显示设备200则可以对接收到的投屏数据进行显示,即按照布局方式显示投屏数据中的画面。例如,在显示设备200分别将包含显示参数“竖向,1080×1920”、“横向,1280×720”、“横向, 1280×720”的通知消息发送给三个通信终端时,三个通信终端300可以分别反馈分辨率为1080×1920、1280×720和1280×720的三个投屏数据流,显示设备200再分别对接收到的三个投屏数据流,进行解析、渲染,以最终形成投屏画面。After the communication terminal 300 generates the projection data, it can feed back the generated projection data to the display device 200, and the display device 200 can display the received projection data, that is, display the images in the projection data according to the layout. For example, when the display device 200 respectively sends notification messages containing display parameters "vertical, 1080×1920", "horizontal, 1280×720", and "horizontal, 1280×720" to three communication terminals, the three communication terminals 300 can respectively feed back three projection data streams with resolutions of 1080×1920, 1280×720, and 1280×720, and display device 200 then parses and renders the received three projection data streams to finally form Cast screen.
需要说明的是,为了能够在显示设备200上同时显示多个投屏画面,显示设备200可以在接收到多个投屏数据流时,对每个通信终端300各创建一个数据缓存区,用于存放其各自发送过来的投屏数据,显示设备200上布局的各显示控件从对应的数据缓存区获取视频数据进行显示,当某个通信终端300断开连接时,显示设备200还可以对其对应的数据缓存区进行销毁释放。It should be noted that, in order to simultaneously display multiple screen projection images on the display device 200, the display device 200 may create a data buffer area for each communication terminal 300 when receiving multiple screen projection data streams for Store the screen projection data sent by them, and each display control on the display device 200 obtains video data from the corresponding data buffer area for display. When a communication terminal 300 is disconnected, the display device 200 can also correspond to it. The data cache area is destroyed and released.
上述实施例提供的投屏画面动态显示方法可以通过检测通信终端300的连接数量规划多个投屏画面在投屏界面中的布局方式,并向每个通信终端300发送包含每个区域显示参数的通知消息,使通信终端300可以按照与布局方式相匹配的方式生成投屏数据。这样的显示方式可以保证最终显示设备200所呈现的画面能够适应布局要求,缓解画面的变形,提高用户观影体验。The method for dynamically displaying the projected screen provided by the above embodiment can plan the layout of multiple projected screens in the screen projecting interface by detecting the number of connections of the communication terminal 300, and send a message containing the display parameters of each area to each communication terminal 300 The notification message enables the communication terminal 300 to generate screen projection data in a manner matching the layout manner. Such a display mode can ensure that the picture presented by the final display device 200 can adapt to the layout requirements, alleviate the deformation of the picture, and improve the viewing experience of the user.
在一些实施例中,显示设备200可以在通信终端300接入时,启动检测通信终端300的连接数量。即如图14所示,显示设备200可以获取用户输入的用于建立投屏连接的连接指令,并响应于所述连接指令,向所述通信终端300发起投屏连接。In some embodiments, the display device 200 may start detecting the number of connections of the communication terminal 300 when the communication terminal 300 is connected. That is, as shown in FIG. 14 , the display device 200 may acquire a connection instruction input by the user for establishing a screen projection connection, and initiate a screen projection connection to the communication terminal 300 in response to the connection instruction.
其中,连接指令可以通过不同的方式完成输入。例如,对于首次与显示设备200建立投屏连接的通信终端300。用户可以先通过网络设置连接显示设备200所在的无线网络,再通过通信终端300上执行交互操作,如点击“投屏”按钮,发起投屏连接。显示设备200则可以在接收到投屏连接后,显示投屏连接配对界面,此时用户需要再基于该投屏连接的配对界面,再次执行交互操作,如在投屏设备列表中选中对应的通信终端300,使显示设备200与通信终端300可以建立投屏连接。可见,对于首次连接的通信终端300,所述连接指令为上述基于显示设备200和通信终端300的一系列交互动作组合。Wherein, the connection instruction can be input in different ways. For example, for the communication terminal 300 that establishes a screen projection connection with the display device 200 for the first time. The user can first connect to the wireless network where the display device 200 is located through network settings, and then perform interactive operations on the communication terminal 300, such as clicking the "screen projection" button to initiate a screen projection connection. The display device 200 can display the screen projection connection pairing interface after receiving the screen projection connection. At this time, the user needs to perform interactive operations again based on the screen projection connection pairing interface, such as selecting the corresponding communication in the screen projection device list. The terminal 300 enables the display device 200 and the communication terminal 300 to establish a screen projection connection. It can be seen that, for the communication terminal 300 connecting for the first time, the connection instruction is a combination of a series of interactive actions based on the display device 200 and the communication terminal 300 .
又例如,对于不是首次与显示设备200建立投屏连接的通信终端300,由于显示设备200和通信终端300中可以相互存储对方的设备信息,因此用户可以直接在通信终端300上点击“投屏”按钮发起投屏操作,此时显示设备200同样可以接收到投屏连接指令,并自动响应于该连接指令实现与通信终端300的相互握手,以完成投屏连接。As another example, for the communication terminal 300 that is not the first to establish a screen projection connection with the display device 200, since the display device 200 and the communication terminal 300 can store each other's device information, the user can directly click "Screencast" on the communication terminal 300 The button initiates a screen projection operation. At this time, the display device 200 can also receive a screen projection connection instruction, and automatically responds to the connection instruction to implement a mutual handshake with the communication terminal 300 to complete the screen projection connection.
在投屏连接配置完成后,显示设备200还可以遍历通信终端300的连接数量。例如,在一个通信终端300发起建立投屏连接后,显示设备200可以先等待投屏连接的配置过程。当投屏连接的配置完成后,再遍历当前连接的通信终端300总数量。即检测的通信终端300的连接数量中,包括新建立投屏连接的通信终端300。After the configuration of the screen projection connection is completed, the display device 200 may also traverse the number of connections of the communication terminal 300 . For example, after a communication terminal 300 initiates to establish a screen projection connection, the display device 200 may first wait for the configuration process of the screen projection connection. After the configuration of the screen projection connection is completed, the total number of communication terminals 300 currently connected is traversed. That is, the detected number of connections of the communication terminals 300 includes the communication terminals 300 that have newly established a screen projection connection.
在一些实施例中,显示设备200还可以在检测到通信终端300停止投屏连接时,执行对连接数量的检测,如图15所示,显示设备200可以在检测通信终端300的连接数量的步骤中,获取用户输入的用于断开投屏连接的退出指令,并响应于该退出指令,断开通信终端300的投屏连接,并在断开投屏连接后,遍历剩余通信终端300的连接数量。In some embodiments, the display device 200 may also detect the number of connections when it detects that the communication terminal 300 stops the screen projection connection. As shown in FIG. , obtain the exit command input by the user for disconnecting the screen projection connection, and in response to the exit instruction, disconnect the screen projection connection of the communication terminal 300, and traverse the connections of the remaining communication terminals 300 after disconnecting the screen projection connection quantity.
例如,当3个通信终端300同时与显示设备200建立投屏连接时,用户可以在其中一个通信终端300上点击“停止投屏”按钮,控制通信终端300不再执行录屏程序。此时,显示设备200可以在检测到通信终端300不再发送投屏数据时,自动断开与通信终端300建立的投屏连接。并在断开投屏连接后,检测剩余通信终端300的连接数量为2个。从而按照2个通信终端300的使用状态,重新规划投屏界面中的布局方式。For example, when three communication terminals 300 establish a screen projection connection with the display device 200 at the same time, the user can click the "stop screen projection" button on one of the communication terminals 300 to control the communication terminal 300 to no longer execute the screen recording program. At this time, the display device 200 may automatically disconnect the screen projection connection established with the communication terminal 300 when detecting that the communication terminal 300 no longer sends screen projection data. And after the screen projection connection is disconnected, it is detected that the number of connections of the remaining communication terminals 300 is 2. Therefore, according to the usage status of the two communication terminals 300, the layout mode in the screen projection interface is re-planned.
可见,在上述实施例中,由于在新的通信终端300建立投屏连接,或者任一通信终端300退出投屏时,显示设备200所呈现的投屏界面中投屏画面的数量发生变化,相应的布局方式也将发生变化。因此,上述连接数量检测方式可以自动适应显示设备200上画面的布局方式,及时调整显示内容,提高用户体验。It can be seen that, in the above-mentioned embodiment, when a new communication terminal 300 establishes a screen projection connection, or when any communication terminal 300 quits screen projection, the number of screen projection images in the screen projection interface presented by the display device 200 changes, correspondingly The layout of the will also change. Therefore, the above method for detecting the number of connections can automatically adapt to the layout of the screen on the display device 200, adjust the displayed content in time, and improve user experience.
在一些实施例中,当显示设备200与通信终端300建立投屏连接后,显示设备200还可以在向通信终端300发起投屏连接时,建立投屏数据通道,并在投屏数据通道建立成功后,创建显示控件,以通过显示控件显示通过投屏数据通道接收的投屏数据。In some embodiments, after the display device 200 establishes a screen projection connection with the communication terminal 300, the display device 200 can also establish a screen projection data channel when initiating a screen projection connection to the communication terminal 300, and the screen projection data channel can be established successfully After that, create an indicator to display the projection data received through the projection data channel through the indicator.
其中,显示控件是一种内置在显示设备200中的显示控制程序,如surface View或Texture View等。显示控件可以添加数量,呈现为不同的显示方式。例如,当显示设备200中包含1个显示控件时,显示控件可以按照全屏的方式进行显示。当显示设备200包含2个显示控件时,两个显示控件为左右两个竖式布局的窗口形式。Wherein, the display control is a display control program built in the display device 200, such as surface View or Texture View. Indicators can be added to display in different ways. For example, when the display device 200 includes one display control, the display control may be displayed in a full-screen manner. When the display device 200 includes two display controls, the two display controls are in the form of two vertically arranged windows on the left and right.
如图16所示,在一些实施例中,为了可以确定显示参数,显示设备200可以在根据连接数量规划投屏画面的布局方式时,先检测显示器260的屏幕尺寸信息,再以连接数量和屏幕尺寸信息为索引,在数据库中匹配布局模板,从而在布局模板中提取显示参数。As shown in FIG. 16 , in some embodiments, in order to determine the display parameters, the display device 200 can first detect the screen size information of the display 260 when planning the layout of the projected screen according to the number of connections, and then use the number of connections and screen The size information is an index to match the layout template in the database, so as to extract the display parameters in the layout template.
对于不同类型的显示设备200,其显示器260的屏幕尺寸信息不同,因此符合其屏幕规格的布局方式也不同。例如,对于常规显示比例的智能电视,为了充分利用显示空间,在显示3个投屏画面时,可以适配一个竖向、两个横向的显示区域布局方式;而对于超宽屏的电影类播放设备,其在显示3个投屏画面时,可以采用竖向三等分的显示区域布局方式。For different types of display devices 200 , the screen size information of the display 260 is different, and thus the layout methods conforming to the screen specifications are also different. For example, for smart TVs with conventional display ratios, in order to make full use of the display space, when displaying three projection screens, one vertical and two horizontal display area layouts can be adapted; while for ultra-widescreen movie playback devices , when displaying 3 projection screens, the layout of the display area can be vertically divided into thirds.
此外,在显示设备200或者服务器400中,可以运行维护一个数据库,数据库中可以存储各种屏幕尺寸规格和连接数量对应的布局模板。因此,显示设备200可以根据屏幕尺寸信息和连接数量,确定与当前显示状态相适应的最佳布局方式。In addition, in the display device 200 or the server 400, a database may be operated and maintained, and layout templates corresponding to various screen sizes and connection quantities may be stored in the database. Therefore, the display device 200 can determine the best layout mode suitable for the current display state according to the screen size information and the number of connections.
在一些实施例中,显示设备200还可以根据布局模板的匹配结果,确定最终采用的布局方式。即在数据库中匹配布局模板的步骤中,如果在数据库中未匹配到布局模板,则可以直接利用匹配获得的布局模板确定布局方式;而如果在数据库中未匹配到布局模板,则可以对显示区域执行区域划分,确定合适的布局方式。即显示设备200可以获取通信终端300的设备参数,并根据设备参数,在显示区域中划分与连接数量相同的子区域,再通过遍历每个子区域的图形参数,以生成显示参数。In some embodiments, the display device 200 may also determine the final layout mode according to the matching result of the layout template. That is, in the step of matching the layout template in the database, if no layout template is matched in the database, the layout method can be determined directly by using the matched layout template; and if no layout template is matched in the database, the display area can be Perform zoning to determine the appropriate layout. That is, the display device 200 can acquire the device parameters of the communication terminal 300, divide the display area into sub-regions with the same number of connections according to the device parameters, and then traverse the graphic parameters of each sub-region to generate display parameters.
例如,在数据库中未匹配到布局模板时,显示设备200可以先检测当前已建立投屏连接的通信终端300的设备参数,如型号、屏幕尺寸规格等。显示设备200再根据所连接的所有通信终端300的屏幕尺寸规格确定相对应的显示区域形状。显然,为了获得更好的显示效果,显示区域形状可以与通信终端300的屏幕形状相似。显示设备200再根据确定的显示区域形状执行显示区域的划分,从而在保证显示内容完整的前提下,最大化利用显示空间。For example, when the layout template is not matched in the database, the display device 200 may first detect the device parameters of the communication terminal 300 that has currently established a screen projection connection, such as model, screen size and specification. The display device 200 then determines the corresponding display area shape according to the screen size specifications of all the connected communication terminals 300 . Obviously, in order to obtain a better display effect, the shape of the display area may be similar to the shape of the screen of the communication terminal 300 . The display device 200 then divides the display area according to the determined shape of the display area, so as to maximize the utilization of the display space under the premise of ensuring the integrity of the display content.
在一些实施例中,显示设备200还可以在向投屏画面对应的通信终端300发送通知消息的步骤中,提取显示参数,并根据显示参数生成通知消息。其中,所述显示参数包括每个显示窗口的显示方向。所述显示方向用于指示所述通信终端按照设定的方向生成投屏数据。为了适应通信终端300的放置状态,显示设备200在确定布局方式后,可以优先按照每个显示区域的形状匹配相适应的通信终端300。例如,当显示窗口的显示方向为竖向时,可以在此窗口中优先显示处于竖屏状态的通信终端300的投屏画面;当显示窗口的显示方向为横向时,可以在此窗口中优先显示处于横屏状态的通信终端300的投屏画面。In some embodiments, the display device 200 may also extract display parameters and generate a notification message according to the display parameters in the step of sending a notification message to the communication terminal 300 corresponding to the projected screen. Wherein, the display parameters include the display direction of each display window. The display direction is used to instruct the communication terminal to generate screen projection data according to a set direction. In order to adapt to the placement state of the communication terminal 300, the display device 200 may preferentially match the corresponding communication terminal 300 according to the shape of each display area after determining the layout mode. For example, when the display direction of the display window is vertical, the projection screen of the communication terminal 300 in the vertical screen state can be preferentially displayed in this window; when the display direction of the display window is horizontal, it can be preferentially displayed in this window. A projected screen image of the communication terminal 300 in a horizontal screen state.
由于在用户界面中所包含的投屏画面数量越多,则每个显示区域所占有的面积越小, 在每个显示区域中所呈现的最终画面效果越差,而当显示区域较小时,显示区域无法针对高画质的显示内容获得相应的效果,因此为了减少总的数据传输量,在检测到通信终端300的连接数量后,显示设备200还可以根据连接数量确定投屏数据的画面质量。即所述显示参数还包括每个显示窗口的码率和分辨率,以使通信终端300可以参照设置的码率和分辨率生成投屏数据。Since the larger the number of projection screens contained in the user interface, the smaller the area occupied by each display area, the worse the final image effect presented in each display area, and when the display area is small, the display Areas cannot obtain corresponding effects for high-quality display content. Therefore, in order to reduce the total data transmission volume, after detecting the number of connections of the communication terminal 300, the display device 200 can also determine the image quality of the projection data according to the number of connections. That is, the display parameters also include the code rate and resolution of each display window, so that the communication terminal 300 can generate screen projection data with reference to the set code rate and resolution.
显然,当与显示设备200建立投屏连接的通信终端300的数量越多时,相应的每个显示区域的范围越小,则可以设置码率和分辨率越小,以使投屏数据的传输数据量越小,反之亦然。其中,码率可以与连接数量之间保持一一对应,或者多对一的对应关系。如显示设备200同时连接1个通信终端300或2个通信终端300时,可以设置传输码率为5Mb/s;如显示设备200同时连接3个通信终端300或4个通信终端300时,可以设置传输码率为3Mb/s。Obviously, when the number of communication terminals 300 that establish screen projection connections with the display device 200 is larger, the range of each corresponding display area is smaller, and the code rate and resolution can be set to be smaller, so that the transmission data of the projection data The smaller the amount, and vice versa. Wherein, there may be a one-to-one correspondence or a many-to-one correspondence between the code rate and the number of connections. For example, when the display device 200 is connected to 1 communication terminal 300 or 2 communication terminals 300 at the same time, the transmission code rate can be set to 5Mb/s; if the display device 200 is connected to 3 communication terminals 300 or 4 communication terminals 300 at the same time, it can be set The transmission code rate is 3Mb/s.
分辨率可以与划分显示区域后,每个显示区域在宽度和高度方向所占的像素数量相同。例如,第一显示区域所占像素数量为1080×1920时,对应投屏数据的分辨率也可以为1080×1920。第二显示区域和第三显示区域所占像素数量为1280×720时,对应投屏数据的分辨率也可以为1280×720。The resolution may be the same as the number of pixels occupied by each display area in the width and height directions after the display area is divided. For example, when the number of pixels occupied by the first display area is 1080×1920, the resolution of the corresponding projection data may also be 1080×1920. When the number of pixels occupied by the second display region and the third display region is 1280×720, the resolution of the corresponding projection data may also be 1280×720.
在确定显示参数后,根据显示参数生成的通知消息中也需要包括上述参数内容,即通知消息中包括所述显示方向、码率和分辨率,以使通信终端300可以在接收到通知消息后,可以从通知消息中解析出显示方向、码率和分辨率,并按照上述参数生成投屏数据,保证投屏画面与各显示区域的一致性。After the display parameters are determined, the notification message generated according to the display parameters also needs to include the above parameter content, that is, the notification message includes the display direction, code rate and resolution, so that the communication terminal 300 can, after receiving the notification message, The display direction, bit rate and resolution can be analyzed from the notification message, and the projection data can be generated according to the above parameters to ensure the consistency between the projection screen and each display area.
需要说明的是,由于布局方式的限制,即使在生成投屏数据的过程中,通信终端300的放置状态与显示区域对应的显示方向不匹配时,通信终端300仍按照显示区域的形状生成投屏数据。例如,当显示区域为竖向形状,而对应通信终端300为横屏状态时,通信终端300依然按照竖向形状生成投屏画面,但在生成投屏数据的过程中,可以在显示画面的顶部和底部分别添加黑色区域,使整体画面呈现为宽度小于高度的竖向画面,以使投屏画面可以在不拉伸变形的状态下显示在竖向形状的显示区域内。It should be noted that due to the limitation of the layout mode, even when the placement state of the communication terminal 300 does not match the display direction corresponding to the display area during the process of generating the projection data, the communication terminal 300 still generates the projection according to the shape of the display area. data. For example, when the display area is in a vertical shape and the corresponding communication terminal 300 is in a horizontal screen state, the communication terminal 300 still generates a screen projection image according to the vertical shape, but in the process of generating screen projection data, it can be displayed at the top of the display screen. Black areas are added to the bottom and bottom respectively, so that the overall screen appears as a vertical screen with a width smaller than the height, so that the screen projection screen can be displayed in the vertical shape display area without stretching and deforming.
但由于显示区域与放置状态不匹配时,容易造成显示区域的大部分画面显示为纯色填充的背景,因此,在一些实施例中,显示设备200可以在检测到显示区域与放置状态不匹配时,通过提示信息提示用户调整通信终端300的放置状态。如图17所示,显示设备200可以执行以下步骤:However, when the display area does not match the placement state, it is easy to cause most of the screen in the display area to be displayed as a background filled with a solid color. Therefore, in some embodiments, when the display device 200 detects that the display area does not match the placement state, The user is prompted to adjust the placement state of the communication terminal 300 through the prompt information. As shown in FIG. 17, the display device 200 may perform the following steps:
步骤S171,检测投屏界面中每个投屏画面的有效画面范围;Step S171, detecting the effective frame range of each screen-casting screen in the screen-casting interface;
步骤S172,计算有效画面范围占显示窗口总范围的显示比例;Step S172, calculating the display ratio of the effective picture area to the total area of the display window;
步骤S173,如果显示比例小于或等于比例阈值,即可以确定显示区域与放置状态不匹配,则执行步骤S174,否则执行步骤S175;Step S173, if the display ratio is less than or equal to the ratio threshold, that is, it can be determined that the display area does not match the placement state, then execute step S174, otherwise execute step S175;
步骤S174,向通信终端300发送提示消息,以提示用户手动或通过通信终端300内部机构自动旋转方向;Step S174, sending a prompt message to the communication terminal 300 to prompt the user to rotate the direction manually or automatically through the internal mechanism of the communication terminal 300;
步骤S175,结束该过程。Step S175, end the process.
例如,在当显示区域为竖向形状,而对应通信终端300为横屏状态时,显示设备200可以检测到显示比例小于或等于比例阈值,因此可以向通信终端300发送提示消息,如“当前显示方向不匹配,请转动手机至竖屏状态”,提示用户将通信终端300调整至树屏状态,增大投屏数据中有效画面的显示比例。For example, when the display area is in a vertical shape and the corresponding communication terminal 300 is in a horizontal screen state, the display device 200 can detect that the display ratio is less than or equal to the ratio threshold, so it can send a prompt message to the communication terminal 300, such as "the current display The orientation does not match, please turn the mobile phone to the vertical screen state", prompting the user to adjust the communication terminal 300 to the tree screen state, and increase the display ratio of the effective screen in the projection data.
基于上述显示设备200,在本申请的部分实施例中还提供一种通信终端300,包括:显 示单元330、通信单元以及处理器380。其中,所述显示单元330被配置为显示终端界面,所述通信单元被配置为与显示设备200建立投屏连接;如图18所示,所述处理器380被配置为执行以下步骤:Based on the display device 200 above, a communication terminal 300 is also provided in some embodiments of the present application, including: a display unit 330 , a communication unit, and a processor 380 . Wherein, the display unit 330 is configured to display a terminal interface, and the communication unit is configured to establish a screen projection connection with the display device 200; as shown in FIG. 18 , the processor 380 is configured to perform the following steps:
步骤181,接收所述显示设备200发送的通知消息,所述通知消息包括投屏画面的显示参数,所述显示参数为所述显示设备200根据当前通信终端300连接数量规划投屏画面布局方式时获得;Step 181: Receive a notification message sent by the display device 200, the notification message includes the display parameters of the screen projection screen, and the display parameters are when the display device 200 plans the layout mode of the screen projection screen according to the current number of communication terminals 300 connections get;
步骤182,按照所述显示参数生成投屏数据;Step 182, generating projection data according to the display parameters;
步骤183,向所述显示设备200反馈所述投屏数据。Step 183 , feeding back the screen projection data to the display device 200 .
基于上述显示设备200和通信终端300,本申请的部分实施例中还提供一种投屏画面动态显示方法,包括以下步骤:Based on the above-mentioned display device 200 and communication terminal 300, some embodiments of the present application also provide a method for dynamically displaying a projection screen, including the following steps:
显示设备200检测通信终端300的连接数量,并根据所述连接数量规划投屏画面的布局方式,以获得所述投屏画面的显示参数;The display device 200 detects the number of connections of the communication terminal 300, and plans the layout of the projected screen according to the number of connections, so as to obtain the display parameters of the projected screen;
所述显示设备200向所述投屏画面对应的通信终端300发送通知消息,所述通知消息包括所述显示参数;The display device 200 sends a notification message to the communication terminal 300 corresponding to the projection screen, and the notification message includes the display parameter;
通信终端300接收所述显示设备200发送的通知消息,并按照所述显示参数生成投屏数据;The communication terminal 300 receives the notification message sent by the display device 200, and generates screen projection data according to the display parameters;
所述通信终端300向所述显示设备200反馈所述投屏数据;The communication terminal 300 feeds back the screen projection data to the display device 200;
所述显示设备200按照所述布局方式显示所述投屏数据。The display device 200 displays the screen projection data according to the layout.
上述实施例提供的投屏画面动态显示方法可以在用户触发连接显示设备200的通信终端300数量改变时,通过检测通信终端300的连接数量,规划投屏画面的布局方式。再向通信终端300发送包含投屏画面显示参数的通知消息,以使通信终端300可以按照显示参数生成符合布局方式的投屏数据,在显示设备200中进行显示。所述方法可以使显示设备200支持多路投屏,并且动态的调整投屏画面的显示布局,并通知通信终端300进行录屏调整,使显示的投屏画面布局更加合理。The method for dynamically displaying the projected screen provided by the above embodiment can plan the layout of the projected screen by detecting the number of connected communication terminals 300 when the user triggers a change in the number of communication terminals 300 connected to the display device 200 . Then send a notification message including the display parameters of the screen projection to the communication terminal 300 , so that the communication terminal 300 can generate screen projection data conforming to the layout according to the display parameters, and display it on the display device 200 . The method can enable the display device 200 to support multi-channel screen projection, dynamically adjust the display layout of the screen projection screen, and notify the communication terminal 300 to adjust the screen recording, so as to make the layout of the screen projection screen displayed more reasonable.
以上实施例介绍了显示设备的硬件/软件架构以及功能实现等内容。在一些应用场景中,终端设备可以向显示设备推送媒体数据,例如用户通过某些具备投屏功能的应用程序,将感兴趣的视频投射到显示设备上进行大屏播放;或者终端设备与显示设备通过某些协议建立投屏连接,比如DLNA(Digital Living Network Alliance,数字生活网络联盟)协议等。在终端设备与显示设备投屏连接成功之后,终端设备才能将投屏数据投射到显示设备,并由显示设备在投屏界面上显示投屏数据。其中,所述终端设备可以是智能手机、平板电脑、笔记本电脑等移动终端,也可是台式电脑或其他小屏显示设备等。The above embodiments introduce the hardware/software architecture and function realization of the display device. In some application scenarios, the terminal device can push media data to the display device. For example, the user can project the video of interest to the display device for large-screen playback through some application programs with screen projection function; or the terminal device and the display device Establish a screen projection connection through certain protocols, such as DLNA (Digital Living Network Alliance, Digital Living Network Alliance) protocol, etc. After the terminal device and the display device are successfully connected for screen projection, the terminal device can project the screen projection data to the display device, and the display device will display the projection data on the screen projection interface. Wherein, the terminal device may be a mobile terminal such as a smart phone, a tablet computer, or a notebook computer, or may be a desktop computer or other small-screen display devices.
在一些实施例中,终端设备在播放媒体数据时,用户可以从设备列表中选择用于投屏播放该媒体数据的设备,而支持投屏功能的显示设备会被识别并自动填加到设备列表中,用户从设备列表中选择显示设备后,即可将媒体数据由终端设备投射到显示设备端进行播放。采用投屏方式推送媒体数据时,终端设备与显示设备可接入同一个Wifi网络,终端向显示设备推送媒体数据的方式不限定。当然,在另一些技术中,投屏方式还可以跨网进行。本申请中对投屏的具体实现方式不作限定。In some embodiments, when a terminal device is playing media data, the user can select a device from the device list for playing the media data on the screen, and the display device that supports the screen projection function will be identified and automatically added to the device list In , after the user selects the display device from the device list, the media data can be projected from the terminal device to the display device for playback. When using the projection method to push media data, the terminal device and the display device can be connected to the same Wifi network, and the method for the terminal to push media data to the display device is not limited. Of course, in other technologies, the screen projection method can also be performed across networks. In this application, the specific implementation manner of screen projection is not limited.
在一些实施例中,终端设备A向显示设备发送投屏请求,显示设备可通过如UI询问弹窗等形式,询问用户是否同意终端设备A的投屏,并可提供若干选项控件,例如“本次同意”、“不同意”和“永久同意”。用户选择“永久同意”,即相当于对终端设备A进行永久授权,后续 终端设备A任意发起投屏请求时,都无需再显示UI询问弹窗,而是直接建立显示设备与终端设备A的投屏连接;若用户选择“本次同意”,即相当于对终端设备A进行一次授权,显示设备与终端设备A会建立投屏连接,但终端设备A的投屏权限仅当次有效,后续终端设备A发起投屏请求时,仍需要再次显示UI询问弹窗;若用户选择“不同意”,即相当于对终端设备A当次未被授权,显示设备与终端设备A不建立投屏连接,后续终端设备A发起投屏请求时,仍需要再次显示UI询问弹窗。In some embodiments, the terminal device A sends a screen projection request to the display device, and the display device can ask the user whether to agree to the screen projection of the terminal device A through a UI inquiry pop-up window, etc., and can provide several option controls, such as "this consent", "disagree" and "permanent consent". The user chooses "Permanent Agree", which is equivalent to permanently authorizing terminal device A. When terminal device A initiates a screen projection request at will, there is no need to display the UI query pop-up window, but directly establishes a projection between the display device and terminal device A. Screen connection; if the user selects "Agree this time", it is equivalent to authorizing terminal device A once, and the display device and terminal device A will establish a screen projection connection, but the screen projection permission of terminal device A is only valid for that time, subsequent terminals When device A initiates a screen projection request, it still needs to display the UI query pop-up window again; if the user chooses "disagree", it means that terminal device A is not authorized at that time, and the display device and terminal device A do not establish a screen projection connection. When terminal device A initiates a screen projection request, it still needs to display the UI query pop-up window again.
在一些实施例中,显示设备可以维护一个可信任设备列表,并支持用户在可信任设备列表中新增、删减和修改具备投屏权限的设备信息,设备信息包括但不限于设备ID、MAC地址等,即用户可以在显示设备端自定义授权名单。当显示设备接收到终端设备B发送的投屏请求时,根据投屏请求中携带的设备信息,查询终端设备B是否包含于可信任设备列表。若终端设备B包含于可信任设备列表中,则建立终端设备B与显示设备的投屏连接;若可信任设备列表中不包括终端设备B,则显示设备不响应投屏请求,即不建立与终端设备B的投屏连接。需要说明的是,终端设备投屏的授权和连接方式不限于本申请的示例。In some embodiments, the display device can maintain a list of trusted devices, and supports users to add, delete and modify device information with screen projection permission in the list of trusted devices. Device information includes but not limited to device ID, MAC Address, etc., that is, the user can customize the authorization list on the display device side. When the display device receives the screen projection request sent by the terminal device B, it queries whether the terminal device B is included in the trusted device list according to the device information carried in the screen projection request. If terminal device B is included in the list of trusted devices, a screen projection connection between terminal device B and the display device is established; if terminal device B is not included in the list of trusted devices, the display device does not respond to the screen projection request, that is, no connection with The screen projection connection of terminal device B. It should be noted that the authorization and connection mode of screen projection by the terminal device are not limited to the examples in this application.
在目前应用场景中,较多是采用一对一投屏模式,即显示设备仅与一个终端设备投屏连接,投屏数据可独占屏幕显示。然而在其他应用场景中,例如用户在玩终端设备C的投屏游戏时,想要同步观看该游戏的教程或解说,则可通过终端设备D向显示设备发起游戏教程/解说视频的投屏请求,显示设备与终端设备D投屏连接成功后,则相当于接入两路投屏数据;又例如,家庭中包括成员1、成员2和成员3,三个成员可通过各自使用的终端分别向显示设备发起投屏请求,显示设备响应三个投屏请求后,将各终端画面展示于投屏界面上,三个成员可以进行互动交流等,这时显示设备则接入三路投屏数据。In the current application scenarios, the one-to-one projection mode is mostly used, that is, the display device is only connected to one terminal device for projection, and the projection data can be exclusively displayed on the screen. However, in other application scenarios, for example, when the user wants to watch the tutorial or commentary of the game synchronously while playing a screen-casting game on terminal device C, the terminal device D can initiate a screen-casting request for the game tutorial/commentary video to the display device , after the screen projection connection between the display device and the terminal device D is successful, it is equivalent to accessing two channels of projection data; for another example, a family includes member 1, member 2 and member 3, and the three members can use their respective terminals to send The display device initiates a screen projection request, and after the display device responds to three screen projection requests, it displays the screens of each terminal on the screen projection interface, and the three members can interact and communicate, etc. At this time, the display device accesses three channels of screen projection data.
对于显示设备接入多路投屏的应用场景,屏幕界面上可以显示多个投屏窗口,每个投屏窗口对应于一个投屏连接成功的终端设备,并显示终端设备发送的投屏数据,但是多个投屏窗口在投屏界面上的展示效果影响着用户观感,存在着诸多显示效果的问题:例如多个投屏窗口间排布不够紧凑,导致投屏界面留白较多;多个投屏窗口分布混乱,整体布局呈现不够规整;部分投屏窗口尺寸较小,导致用户无法看清楚窗口中视频的细节;视频在终端设备为横屏显示,投射到显示设备后屏向发生错乱,变为竖屏显示,则终端设备与显示设备对视频展示统一性差,等等。对此,本申请旨在对多路投屏窗口进行规划和布局,以提升投屏界面的展示效果。For the application scenario where the display device is connected to multiple screen projections, multiple screen projection windows can be displayed on the screen interface. Each screen projection window corresponds to a terminal device that is successfully connected to the screen projection, and displays the screen projection data sent by the terminal device. However, the display effect of multiple screen projection windows on the screen projection interface affects the user's perception, and there are many display problems: for example, the arrangement of multiple screen projection windows is not compact enough, resulting in a lot of blank space in the screen projection interface; The distribution of the screen projection windows is chaotic, and the overall layout is not regular; some screen projection windows are small in size, so that users cannot clearly see the details of the video in the window; If it is changed to a vertical screen display, the video display uniformity between the terminal device and the display device is poor, and so on. In this regard, the present application aims to plan and lay out multiple screen projection windows to improve the display effect of the screen projection interface.
参照图19,本申请实施例提供一种多路投屏显示方法,所述方法被配置为由显示设备中的控制器250执行,即控制器250为方法的执行主体,所述方法包括如下步骤:Referring to FIG. 19 , an embodiment of the present application provides a multi-channel projection display method, the method is configured to be executed by the controller 250 in the display device, that is, the controller 250 is the execution subject of the method, and the method includes the following steps :
步骤S01,根据当前计数的M个投屏路数,以及从M个终端设备分别接收的媒体配置信息,匹配适应于屏幕的M个投屏窗口间的布局;Step S01, according to the currently counted number of M screen projection channels and the media configuration information respectively received from the M terminal devices, match the layout among the M screen projection windows suitable for the screen;
步骤S02,在投屏界面中按照所述布局显示M个投屏窗口,以及在每个投屏窗口内显示对应的终端设备发送的投屏数据。Step S02, display M screen projection windows according to the layout in the screen projection interface, and display the screen projection data sent by the corresponding terminal device in each screen projection window.
其中,M用于表示投屏路数,即与显示设备建立投屏连接的终端设备数量,投屏路数会随着投屏连接的接入和断开而动态变化,因此需要对投屏路数进行计数,以便准确规划投屏界面的排版和布局。M为1时,即执行单路投屏;当M大于或等于2时,即多路投屏;M可以为零,表明显示设备未接收到连接设备发送的投屏数据,则不显示投屏界面,也相当于投屏界面上布局为空。Among them, M is used to indicate the number of screen projection channels, that is, the number of terminal devices that have established screen projection connections with the display device. The number of screen projection channels will change dynamically with the connection and disconnection of screen projection connections. The numbers are counted in order to accurately plan the typesetting and layout of the screencasting interface. When M is 1, single-channel projection is performed; when M is greater than or equal to 2, multiple-channel projection is performed; M can be zero, indicating that the display device has not received the projection data sent by the connected device, and the projection will not be displayed The interface is also equivalent to an empty layout on the screen projection interface.
在一些实施例中,显示设备每接收到一个目标终端设备的投屏请求,若新建与目标终 端设备的投屏连接,则将投屏路数的计数值累积加一;此外,终端设备可能因断网或者终端用户人为退出投屏等因素,而与显示设备断开投屏连接,导致显示设备无法接收到该终端设备发送的投屏数据,则显示设备端需要切断该路投屏,因此每检测到与一个终端设备的投屏连接断开,则将投屏路数的计数值减一。通过该实施例,即可实现根据显示设备的投屏连接动态,自动计数投屏路数。In some embodiments, each time the display device receives a screen projection request from a target terminal device, if it creates a new screen projection connection with the target terminal device, it will cumulatively add one to the count value of the number of screen projection channels; in addition, the terminal device may Due to factors such as disconnection of the network or the terminal user’s intentional exit from projection, the display device is disconnected from the projection connection, resulting in the display device being unable to receive the projection data sent by the terminal device, and the display device needs to cut off the projection channel. When it is detected that the screen projection connection with a terminal device is disconnected, the count value of the number of screen projection channels is decremented by one. Through this embodiment, the number of screen projection channels can be automatically counted according to the screen projection connection dynamics of the display device.
在一些实施例中,显示设备每接收一个目标终端设备的投屏数据,则将投屏路数的计数值累积加一。In some embodiments, each time the display device receives the screen projection data of a target terminal device, the count value of the number of screen projection ways is cumulatively increased by one.
本申请中显示设备端除了接收来自M个终端设备的投屏数据,还会同步接收M个终端设备分别发送的媒体配置信息。In this application, in addition to receiving the projection data from the M terminal devices, the display device side will also synchronously receive the media configuration information sent by the M terminal devices respectively.
在一些实施例中,所述投屏数据与媒体类别有关,例如媒体为纯视频类别,则投屏数据包括被投屏的视频流;若媒体为音视频类别,则投屏数据包括视频流和音频流;此外,投屏数据还可以是图像(静态图像或动图等)、文本或纯音频流等形式。当投屏数据为音频流时,显示界面上可展示诸如音频的播放内容信息。In some embodiments, the screen projection data is related to the media category. For example, if the media is a pure video category, the screen projection data includes the video stream to be screened; if the media is an audio and video category, the screen projection data includes video streams and Audio stream; in addition, the projection data can also be in the form of images (static images or moving pictures, etc.), text or pure audio streams, etc. When the screencasting data is an audio stream, the display interface can display playback content information such as audio.
在一些实施例中,投屏数据与终端设备的录屏方式相关。例如,录屏方式可按照固定分辨率录制,如1280*720、1920*1080等,还可以按照终端屏幕的尺寸进行录制,如16:9尺寸、17:9尺寸、21:9尺寸等。In some embodiments, the screen projection data is related to the screen recording mode of the terminal device. For example, the screen recording method can be recorded according to a fixed resolution, such as 1280*720, 1920*1080, etc., and can also be recorded according to the size of the terminal screen, such as 16:9 size, 17:9 size, 21:9 size, etc.
在一些实施例中,所述媒体配置信息包括终端设备投屏的视频的宽高比例,即显示视频时横向宽度与纵向高度的比值,通过宽高比例可以识别显示屏向,当宽高比例大于或等于1时,即宽度大于或等于高度,则呈现为横屏显示模式;当宽高比例小于1时,即宽度小于高度时,则呈现为竖屏显示模式。可按照媒体配置信息中指示的宽高比例和屏向,来约束投屏窗口的创建,使投屏窗口与对应的终端设备保持一致的宽高比例和显示屏向,可以避免投屏后媒体相关配置的错乱问题。In some embodiments, the media configuration information includes the aspect ratio of the video projected by the terminal device, that is, the ratio of the horizontal width to the vertical height when displaying the video. The aspect ratio can be used to identify the display orientation. When the aspect ratio is greater than When or is equal to 1, that is, the width is greater than or equal to the height, the display mode is displayed in landscape mode; when the aspect ratio is less than 1, that is, the width is smaller than the height, the display mode is displayed in portrait mode. The creation of the projection window can be constrained according to the aspect ratio and screen orientation indicated in the media configuration information, so that the projection window and the corresponding terminal device can maintain the same aspect ratio and display orientation, which can avoid media correlation after projection. Misconfiguration problem.
在一些实施例中,所述媒体配置信息还可以包括终端设备的屏幕宽高比例。当宽高比例大于或等于1时,即宽度大于或等于高度,则呈现为横屏显示模式;当宽高比例小于1时,即宽度小于高度时,则呈现为竖屏显示模式。In some embodiments, the media configuration information may also include a screen aspect ratio of the terminal device. When the aspect ratio is greater than or equal to 1, that is, the width is greater than or equal to the height, the display mode is in landscape mode; when the aspect ratio is less than 1, that is, the width is smaller than the height, the display mode is in portrait mode.
在一些实施例中,所述媒体配置信息中还可包括媒体类别,显示设备读取媒体配置信息后,可以获知即将投屏的媒体类别,从而根据媒体类别展示不同效果的投屏窗口。例如,对于纯视频、音视频和图像型媒体,投屏窗口可以展示可视化的投屏数据,并且投屏窗口应匹配视频/图像的宽高比例和屏向;对于文本型媒体,投屏窗口可以可视化地展示可读的文本信息,但是投屏窗口的宽高比例和屏向不限定;对于纯音频型媒体,投屏窗口无法显示可视化画面,可在投屏窗口中设置语音播放控件(例如小喇叭),在播放该路投屏的音频流时,语音播放控件可呈现为“播放中”的动态特效,同样地,纯音频流对应的投屏窗口的宽高比例和屏向也不限定。In some embodiments, the media configuration information may further include a media category. After the display device reads the media configuration information, it may know the media category to be screen-casted, so as to display screen-casting windows with different effects according to the media category. For example, for pure video, audio and video, and image media, the projection window can display visualized projection data, and the projection window should match the aspect ratio and screen orientation of the video/image; for text-based media, the projection window can Visually display readable text information, but the aspect ratio and screen orientation of the projection window are not limited; for audio-only media, the projection window cannot display visual images, and voice playback controls can be set in the projection window (such as small Speaker), when playing the audio stream of the projection screen, the voice playback control can be presented as a dynamic special effect of "playing". Similarly, the aspect ratio and screen orientation of the projection window corresponding to the pure audio stream are not limited.
可选地,对于特殊媒体类别下宽高比例和屏向不限定的投屏窗口,可采用预设窗口布局,所述预设窗口布局是指预先设置好的具有固定宽高比例(或固定的宽度和高度)和屏向的投屏窗口,在识别出即将投屏的媒体为文本或纯音频等类别后,直接调用预设窗口布局,后续将投屏数据加载到窗口布局中;也可不预设窗口布局,而是优先对纯视频、音视频和图像等具有限定宽高比例和屏向的媒体创建适配的投屏窗口,然后在投屏界面中的留白区域,创建并显示相适应的投屏窗口,以展示文本和纯音频类别的投屏数据,这种方式可优先匹配限定的投屏窗口,之后最大化地占用屏幕显示区域,使多个投屏窗口间更加紧 凑,降低留白区域。需要说明的是,不同媒体类别下投屏窗口的创建与UI显示设置不限于本申请实施例。Optionally, for a projection window with an unlimited aspect ratio and screen orientation under a special media category, a preset window layout can be used. The preset window layout refers to a preset window layout with a fixed aspect ratio (or a fixed Width and height) and screen orientation, after identifying the media to be screened as text or pure audio, directly call the preset window layout, and then load the screen projection data into the window layout; Instead of setting the window layout, priority is given to creating screen-adapting screen-casting windows for media with limited aspect ratios and screen orientations such as pure video, audio, video, and images, and then creating and displaying appropriate screen-casting windows in the blank area of the screen-casting interface. screen projection window to display text and audio-only projection data. This method can preferentially match the limited projection window, and then maximize the screen display area, making multiple projection windows more compact and reducing retention time. white area. It should be noted that the creation of screen projection windows and UI display settings under different media types are not limited to the embodiments of this application.
在步骤S01中,可以根据显示设备的屏幕分辨率或M个终端设备投屏的视频的宽高比例和屏向,及投屏路数,计算出M组尺寸信息,所述尺寸信息包括宽度windowWidthi和高度windowHeighti,即构建windowWidthi*windowHeighti,其中i用于标识尺寸信息/投屏窗口的序号,1≤i≤M,i值与终端设备的设备信息存在映射关系。例如M=3,序号1对应于终端设备A,序号2对应于终端设备B,序号3对应于终端设备C,则i等于1时,表明根据第1组尺寸信息创建的投屏窗口1用于显示终端设备A的投屏数据,这样即可准确识别每个投屏窗口应该保持投放来自哪一终端设备的投屏数据。In step S01, M sets of size information can be calculated according to the screen resolution of the display device or the aspect ratio and screen direction of the video projected by M terminal devices, and the number of screen projection channels, and the size information includes the width windowWidthi And the height windowHeighti, that is to construct windowWidthi*windowHeighti, where i is used to identify the size information/serial number of the projection window, 1≤i≤M, and there is a mapping relationship between the value of i and the device information of the terminal device. For example, M=3, serial number 1 corresponds to terminal device A, serial number 2 corresponds to terminal device B, and serial number 3 corresponds to terminal device C, then when i is equal to 1, it indicates that screen projection window 1 created according to the size information of the first group is used for The screen projection data of the terminal device A is displayed, so that the screen projection data from which terminal device should be kept displayed in each screen projection window can be accurately identified.
在一些示例中,可以按照投屏连接的时间顺序分配所述序号,比如越早接入投屏则序号越小。例如,采用预排版的方式,在投屏连接时刻,就预先对投屏画面进行排版,在接收到投屏数据后,就按照所述预排版方式显示画面。In some examples, the sequence numbers may be allocated according to the time sequence of the screen projection connection, for example, the earlier the screen projection is connected, the smaller the sequence number. For example, using a pre-typesetting method, the screen projection screen is pre-typeset at the moment of screen projection connection, and after receiving the screen projection data, the screen is displayed according to the pre-typesetting method.
在另一个示例中,可以按照接收投屏数据的先后顺序分配所述序号,比如,按照实时接收到的数据流进行排版。In another example, the sequence numbers may be allocated according to the order in which the screen projection data is received, for example, typesetting is performed according to the data stream received in real time.
在上述所述示例性的实现方式中,显示设备媒体配置信息可以是按照固定分辨率进行录制视频,如1920*1080、1280*720等。In the above exemplary implementation manner, the media configuration information of the display device may be to record video at a fixed resolution, such as 1920*1080, 1280*720, and so on.
在上述所述示例性的实现方式中,显示设备媒体配置信息可以是按照屏幕的比例进行录制视频,而当前屏幕的比例非大屏终端的比例16:9/9:16,而是其他比例如17:9、21:16等。在一种示例性的实现方式中,根据M组尺寸信息,构建M个投屏窗口,投屏窗口与其对应的终端设备保持一致的宽高比例和显示屏向。由于每组尺寸信息都限定了一个投屏窗口具体的宽度和高度,因此可以构建与尺寸信息匹配的投屏窗口。M个投屏窗口的大小都确定后,还需要规划每个投屏窗口在屏幕界面中具体的显示位置。In the exemplary implementation described above, the media configuration information of the display device may record video according to the ratio of the screen, and the ratio of the current screen is not the ratio of 16:9/9:16 of the large-screen terminal, but other ratios such as 17:9, 21:16, etc. In an exemplary implementation manner, M screen projection windows are constructed according to M sets of size information, and the screen projection windows and their corresponding terminal devices maintain the same aspect ratio and display orientation. Since each set of size information defines a specific width and height of a screen projection window, a screen projection window matching the size information can be constructed. After the sizes of the M screen projection windows are all determined, it is also necessary to plan a specific display position of each screen projection window on the screen interface.
在一些实施例中,根据从终端设备接收到的媒体配置信息计算位置分布信息,所述位置分布信息包括M个投屏窗口上的预设基准点Pointi的坐标(xi,yi),所述预设基准点Pointi的坐标(xi,yi)用于定位投屏窗口在屏幕上的位置。由于投屏窗口一般为矩形,因此预设基准点可以选取矩形中的某些关键点,可选地,预设基准点为矩形四个顶点中的任意一个或者矩形的中心点等,选取原则以便于计算和定位为准。最后,根据计算得到的位置分布信息,对已创建的M个投屏窗口进行定位和排布,这样M个投屏窗口的尺寸大小和位置均已确定,布局成型。In some embodiments, the position distribution information is calculated according to the media configuration information received from the terminal device, the position distribution information includes the coordinates (xi, yi) of a preset reference point Pointi on the M screen projection windows, and the preset It is assumed that the coordinates (xi, yi) of the reference point Pointi are used to locate the position of the screen projection window on the screen. Since the screen projection window is generally a rectangle, some key points in the rectangle can be selected by the preset reference point. Optionally, the preset reference point is any one of the four vertices of the rectangle or the center point of the rectangle, etc. The selection principle is to Subject to calculation and positioning. Finally, according to the calculated location distribution information, the created M screen projection windows are positioned and arranged, so that the sizes and positions of the M screen projection windows are determined and the layout is formed.
下面将结合具体的UI示例进行说明。图20(a)示例性示出了投屏路数为1且视频流为单竖屏场景下的投屏界面,此时显示设备仅接入终端设备A最先发起的单路投屏,以显示设备的屏幕分辨率为1920*1080(即显示设备屏幕默认的宽高比例为16:9),终端设备A发送的媒体配置文件指示宽高比例为9:16(竖屏)为例,则即将创建的投屏窗口1也为竖屏,使投屏窗口1的高度适配屏幕高度,则投屏窗口1的宽度为1080*9/16=607.5,即windowWidth1*windowHeight1=607.5*1080。在确定投屏窗口1的位置时,以屏幕的左上角顶点为原点,向右为x轴正向,竖直向下为y轴正向来构建坐标系,以投屏窗口1的左上角顶点作为预设基准点Point1,考虑到单窗口以竖屏居中的展示模式在观看效果上更佳,由此计算出预设基准点Point1的坐标(x1,y1)=(656.25,0)。由于预设基准点Point1的坐标位置以及投屏窗口1的尺寸都已确定,因此投屏窗口1在整体投屏界面中的位置是唯一确定的,由此得到的布局如图20(a)所示。The following will describe with a specific UI example. Figure 20(a) exemplarily shows the screen projection interface in the scenario where the number of screen projection channels is 1 and the video stream is a single vertical screen. The screen resolution of the display device is 1920*1080 (that is, the default aspect ratio of the display device screen is 16:9), and the media configuration file sent by terminal device A indicates that the aspect ratio is 9:16 (vertical screen) as an example, then The screen projection window 1 to be created is also a vertical screen, and the height of the screen projection window 1 is adapted to the height of the screen, so the width of the screen projection window 1 is 1080*9/16=607.5, that is, windowWidth1*windowHeight1=607.5*1080. When determining the position of the screen projection window 1, use the top left corner of the screen as the origin, the positive direction of the x-axis to the right, and the positive direction of the y-axis to construct a coordinate system, and take the top left corner of the screen projection window 1 as the For the preset reference point Point1, considering that the display mode with a single window centered in vertical screen is better in viewing effect, the coordinates (x1, y1)=(656.25, 0) of the preset reference point Point1 are calculated. Since the coordinate position of the preset reference point Point1 and the size of the screen projection window 1 have been determined, the position of the screen projection window 1 in the overall screen projection interface is uniquely determined, and the resulting layout is shown in Figure 20(a) Show.
图20(b)示例性示出了投屏路数为1且视频流为单横屏场景下的投屏界面,此时显示设备仅接入终端设备A最先发起的单路投屏,以显示设备的屏幕分辨率为1920*1080(即显示设备屏幕默认的宽高比例为16:9),终端设备A发送的媒体配置文件指示宽高比例为16:9(横屏)为例,则即将创建的投屏窗口1也为横屏,由于为单窗口显示模式,并且投屏窗口1与屏幕在宽高比例上完全适配,因此可以将投屏窗口1进行全屏展示,即windowWidth1*windowHeight1=1920*1080,以投屏窗口1的左上角顶点作为预设基准点Point1,则预设基准点Point1的坐标(x1,y1)=(0,0),由此得到的布局如图20(b)所示。Figure 20(b) exemplarily shows the screen projection interface in the scenario where the number of screen projection channels is 1 and the video stream is a single horizontal screen. The screen resolution of the display device is 1920*1080 (that is, the default aspect ratio of the display device screen is 16:9), and the media configuration file sent by terminal device A indicates that the aspect ratio is 16:9 (horizontal screen), for example, then The projection window 1 to be created is also a horizontal screen. Since it is a single-window display mode, and the aspect ratio of the projection window 1 and the screen is fully adapted, the projection window 1 can be displayed in full screen, that is, windowWidth1*windowHeight1 =1920*1080, taking the upper left corner vertex of the screen projection window 1 as the preset reference point Point1, then the coordinates of the preset reference point Point1 (x1, y1)=(0, 0), the resulting layout is shown in Figure 20 ( b) as shown.
图21(a)和图21(b)示例性示出了投屏路数为2且双横屏场景下的两种投屏界面,图21(a)是将两个横屏的投屏窗口进行横向并排排布,图21(b)是将两个横屏的投屏窗口进行纵向并列排布,此时显示设备已接入终端设备A和终端设备B发起的双路投屏,以显示设备的屏幕分辨率为1920*1080,终端设备A和终端设备B发送的媒体配置文件指示宽高比例均为16:9(横屏)为例,则创建的投屏窗口1(对应于终端设备A)和投屏窗口2(对应于终端设备B)也均为横屏。Fig. 21(a) and Fig. 21(b) exemplarily show two kinds of screen-casting interfaces in the scene where the number of screen-casting channels is 2 and the scene is double landscape. Fig. 21(a) is the screen-casting window of two landscape Arrange horizontally side by side. Figure 21(b) is to arrange two horizontal screen projection windows vertically. At this time, the display device has been connected to the two-way projection screen initiated by terminal device A and terminal device B to display The screen resolution of the device is 1920*1080, and the media configuration files sent by terminal device A and terminal device B indicate that the aspect ratio is 16:9 (horizontal screen) as an example, then the created projection window 1 (corresponding to the A) and screen projection window 2 (corresponding to terminal device B) are also horizontal screens.
对于图21(a)的示例,由于为横向并排排布,为最大化地占用屏幕区域,使投屏窗口1和投屏窗口2的宽度之和等于屏幕宽度,即windowWidth1+windowWidth2=1920,此外为使双投屏窗口的尺寸大小更加均衡,可设置windowWidth1=windowWidth2=960,则windowHeight1=windowHeight2=960*9/16=540,即得到windowWidth1*windowHeight1=windowWidth2*windowHeight2=960*540,以投屏窗口1的左上角顶点作为预设基准点Point1,以投屏窗口2的左上角顶点作为预设基准点Point2,考虑到双窗口以横屏排列、纵向居中的展示模式在观看效果上更佳,由此计算出预设基准点Point1的坐标(x1,y1)=(0,270),预设基准点Point2的坐标(x2,y2)=(960,270),由此得到的布局如图21(a)所示。For the example in Figure 21(a), since they are arranged side by side horizontally, in order to maximize the screen area occupied, the sum of the widths of the screen projection window 1 and the screen projection window 2 is equal to the screen width, that is, windowWidth1+windowWidth2=1920, in addition In order to make the size of the dual screen projection window more balanced, you can set windowWidth1=windowWidth2=960, then windowHeight1=windowHeight2=960*9/16=540, that is, windowWidth1*windowHeight1=windowWidth2*windowHeight2=960*540, to screen The upper left corner of window 1 is used as the preset reference point Point1, and the upper left corner of the screen projection window 2 is used as the preset reference point Point2. Considering that the display mode of dual windows arranged in a horizontal screen and vertically centered is better in terms of viewing effect, Thus, the coordinates (x1, y1)=(0, 270) of the preset reference point Point1 are calculated, and the coordinates (x2, y2)=(960, 270) of the preset reference point Point2 are calculated. The resulting layout is shown in Figure 21 (a) shown.
对于图21(b),由于为纵向并列排布,为最大化地占用屏幕区域,使投屏窗口1和投屏窗口2的高度之和等于屏幕高度,即windowHeight1+windowHeight2=1080,此外为使双投屏窗口的尺寸大小更加均衡,可设置windowHeight1=windowHeight2=540,则windowWidth1=windowWidth2=540*16/9=960,即得到windowWidth1*windowHeight1=windowWidth2*windowHeight2=960*540,以投屏窗口1的左上角顶点作为预设基准点Point1,以投屏窗口2的左上角顶点作为预设基准点Point2,考虑到双窗口以横屏并列、横向居中的展示模式在观看效果上更佳,由此计算出预设基准点Point1的坐标(x1,y1)=(480,0),预设基准点Point2的坐标(x2,y2)=(480,540),由此得到的布局如图21(b)所示。For Figure 21(b), since they are vertically arranged side by side, in order to maximize the screen area occupied, the sum of the heights of the screen projection window 1 and the screen projection window 2 is equal to the screen height, that is, windowHeight1+windowHeight2=1080. The size of the dual projection screen window is more balanced. You can set windowHeight1=windowHeight2=540, then windowWidth1=windowWidth2=540*16/9=960, that is, windowWidth1*windowHeight1=windowWidth2*windowHeight2=960*540, using the projection window 1 The upper left corner of the screen is used as the default reference point Point1, and the upper left corner of the screen projection window 2 is used as the preset reference point Point2. Considering that the display mode of the dual windows side by side and horizontally centered in the horizontal screen is better in viewing effect, thus Calculate the coordinates (x1, y1)=(480,0) of the preset reference point Point1, and the coordinates (x2, y2)=(480,540) of the preset reference point Point2, the resulting layout is shown in Figure 21(b ) shown.
在一些实施例中,匹配生成布局后,在投屏界面上按照布局显示多个投屏窗口以及加载显示各路投屏数据的过程中,可能随时有新接入的投屏通路,也可能有至少一个终端设备断开了投屏连接,即在此期间投屏路数可能发生变更,则需要根据变更后的投屏路数去更新布局,实现布局跟随投屏路数进行自适应调整,这里所述的调整涉及布局中投屏窗口的数量、每个投屏窗口的尺寸大小以及每个投屏窗口的相对位置分布。In some embodiments, after matching and generating the layout, during the process of displaying multiple screen projection windows according to the layout on the screen projection interface and loading and displaying the screen projection data of each channel, there may be new screen projection channels accessed at any time, or there may be At least one terminal device is disconnected from the screen projection connection, that is, the number of screen projection channels may change during this period, and the layout needs to be updated according to the changed screen projection channels to realize adaptive adjustment of the layout following the screen projection channels. Here The adjustment involves the number of screen projection windows in the layout, the size of each screen projection window, and the relative position distribution of each screen projection window.
例如,在图20(b)的场景下,显示设备又与终端设备B建立起投屏连接,并且终端设备B发送的媒体配置信息中指示宽高比例为16:9(横屏),控制器250检测到投屏路数由1变更为2,则可将布局更新为图21(a)或图21(b)的所示版本;又例如,在图21(a)或图21(b)场景下,终端设备B突然断开了与显示设备的投屏连接,控制器250检测到投屏路数由2变更为1,则可将布局更新为图20(b)所示的版本。For example, in the scenario shown in Figure 20(b), the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 16:9 (horizontal screen), the controller 250 detects that the number of projection screens is changed from 1 to 2, then the layout can be updated to the version shown in Figure 21(a) or Figure 21(b); for example, in Figure 21(a) or Figure 21(b) In the scenario, terminal device B suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1, and can update the layout to the version shown in Figure 20(b).
图21(c)和图21(d)为根据一些实施例的投屏路数为2且一横屏、一竖屏场景下的两种投屏界面,图21(c)是将横屏的投屏窗口显示于左侧,将竖屏的投屏窗口显示于右侧;图21(d)是将竖屏的投屏窗口显示于左侧,将横屏的投屏窗口显示于右侧。此时显示设备已接入终端设备A和终端设备B发起的双路投屏,以显示设备的屏幕分辨率为1920*1080,终端设备A发送的媒体配置文件指示宽高比例为16:9(横屏),终端设备B发送的媒体配置文件指示宽高比例为9:16(竖屏)为例,则创建的投屏窗口1(对应于终端设备A)为横屏,投屏窗口2(对应于终端设备B)为竖屏。Fig. 21(c) and Fig. 21(d) are two kinds of screen-casting interfaces in the scene where the number of screen-casting channels is 2 and one horizontal screen and one vertical screen according to some embodiments, and Fig. 21(c) shows the horizontal screen The screen projection window is displayed on the left, and the screen projection window for the vertical screen is displayed on the right; Figure 21(d) shows the screen projection window for the vertical screen on the left, and the screen projection window for the landscape screen is displayed on the right. At this time, the display device has connected to the two-way screen projection initiated by terminal device A and terminal device B. The screen resolution of the display device is 1920*1080, and the media configuration file sent by terminal device A indicates that the aspect ratio is 16:9 ( Horizontal screen), the media configuration file sent by terminal device B indicates that the aspect ratio is 9:16 (vertical screen) as an example, then the created projection window 1 (corresponding to terminal device A) is a horizontal Corresponding to terminal device B) is a vertical screen.
对于图21(c),由于为一横一竖两个窗口并排排布,为最大化地占用屏幕区域,使投屏窗口1和投屏窗口2的宽度之和等于屏幕宽度,即windowWidth1+windowWidth2=1920,使竖屏的投屏窗口2的高度适配屏幕高度,即windowHeight2=1080,则windowWidth2=1080*9/16=607.5,windowWidth1=1920-607.5=1312.5,windowHeight1=1312.5*9/16=738.3,则windowWidth1*windowHeight1=1312.5*738.3,windowWidth2*windowHeight2=607.5*1080,以投屏窗口1的左上角顶点作为预设基准点Point1,以投屏窗口2的左上角顶点作为预设基准点Point2,考虑到投屏窗口1以纵向居中的展示模式在观看效果上更佳,由此计算出预设基准点Point1的坐标(x1,y1)=(0,170.9),预设基准点Point2的坐标(x2,y2)=(1312.5,0),由此得到的布局如图21(c)所示。For Figure 21(c), since two windows are arranged side by side, one horizontal and one vertical, in order to maximize the screen area occupied, the sum of the widths of screen projection window 1 and screen projection window 2 is equal to the screen width, namely windowWidth1+windowWidth2 =1920, make the height of the projection window 2 of the vertical screen fit the height of the screen, that is, windowHeight2=1080, then windowWidth2=1080*9/16=607.5, windowWidth1=1920-607.5=1312.5, windowHeight1=1312.5*9/16= 738.3, then windowWidth1*windowHeight1=1312.5*738.3, windowWidth2*windowHeight2=607.5*1080, take the upper left corner of the screen projection window 1 as the preset reference point Point1, and use the upper left corner of the screen projection window 2 as the preset reference point Point2 , considering that the viewing effect of the projection window 1 is better in the vertically centered display mode, the coordinates of the preset reference point Point1 are calculated (x1, y1)=(0, 170.9), and the coordinates of the preset reference point Point2 are calculated (x2, y2) = (1312.5, 0), the resulting layout is shown in Figure 21(c).
图21(d)与图21(c)区别仅在于横屏和竖屏在横向上分布在不同侧,因此投屏窗口1和投屏窗口2的尺寸大小是相同的,而Point1和Point2的坐标不同。即windowWidth1*windowHeight1=1312.5*738.3,windowWidth2*window Height2=607.5*1080,以投屏窗口1的左上角顶点作为预设基准点Point1,以投屏窗口2的左上角顶点作为预设基准点Point2,考虑到投屏窗口1以纵向居中的展示模式在观看效果上更佳,由此计算出预设基准点Point2的坐标(x2,y2)=(0,0),预设基准点Point1的坐标(x1,y1)=(607.5,170.9),由此得到的布局如图21(d)所示。The difference between Figure 21(d) and Figure 21(c) is that the horizontal screen and the vertical screen are distributed on different sides in the horizontal direction, so the size of the projection window 1 and the projection window 2 are the same, and the coordinates of Point1 and Point2 different. That is, windowWidth1*windowHeight1=1312.5*738.3, windowWidth2*window Height2=607.5*1080, the upper left corner of the screen projection window 1 is used as the preset reference point Point1, and the upper left corner of the screen projection window 2 is used as the preset reference point Point2, Considering that the viewing effect of the projection screen window 1 is better in the vertically centered display mode, the coordinates (x2, y2)=(0, 0) of the preset reference point Point2 are calculated, and the coordinates of the preset reference point Point1 ( x1, y1) = (607.5, 170.9), and the resulting layout is shown in Fig. 21(d).
在一些示例中,例如在图20(a)的单路竖屏场景下,显示设备又与终端设备B建立起投屏连接,并且终端设备B发送的媒体配置信息中指示宽高比例为16:9(横屏),控制器250检测到投屏路数由1变更为2,则可将布局更新为类似于图21(c)或图21(d)所示的双路一横屏一竖屏的版本。In some examples, for example, in the single-way vertical screen scenario in FIG. 20(a), the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 16: 9 (horizontal screen), the controller 250 detects that the number of projection screens has changed from 1 to 2, then the layout can be updated to a dual-channel-horizontal-vertical screen similar to that shown in Figure 21(c) or Figure 21(d) screen version.
在一些示例中,例如在图20(b)的单路横屏场景下,显示设备又与终端设备B建立起投屏连接,并且终端设备B发送的媒体配置信息中指示宽高比例为9:16(竖屏),控制器250检测到投屏路数由1变更为2,则可将布局更新为图21(c)或图21(d)所示的双路一横屏一竖屏的版本。In some examples, for example, in the single-channel landscape scenario of FIG. 20(b), the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 9: 16 (vertical screen), the controller 250 detects that the number of projection screens has been changed from 1 to 2, and the layout can be updated to a dual-channel-horizontal-vertical screen as shown in Figure 21(c) or Figure 21(d) Version.
在一些示例中,例如在图21(c)或图21(d)场景下,终端设备B突然断开了与显示设备的投屏连接,控制器250检测到投屏路数由2变更为1,则可将布局更新为图20(b)所示的单横屏窗口(投屏窗口1)全屏独占屏幕的版本。In some examples, for example, in the scenario shown in Figure 21(c) or Figure 21(d), terminal device B suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1 , the layout can be updated to the version of the single horizontal screen window (screen projection window 1) full-screen exclusive screen shown in FIG. 20(b).
在一些示例中,例如在图21(c)或图21(d)场景下,终端设备A突然断开了与显示设备的投屏连接,控制器250检测到投屏路数由2变更为1,则可将布局更新为类似于图20(a)所示的单竖屏窗口(投屏窗口2)独占屏幕的版本。In some examples, for example, in the scenario shown in Figure 21(c) or Figure 21(d), terminal device A suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1 , the layout can be updated to a version similar to that shown in Figure 20(a) in which the single vertical screen window (screen projection window 2) monopolizes the screen.
图21(e)示例性示出了投屏路数为2且双竖屏场景下的投屏界面,图21(e)中是将两个竖屏的投屏窗口沿横向并排排布,以显示设备的屏幕分辨率为1920*1080,终端设备A和终端设备B发送的媒体配置文件指示宽高比例均为9:16(竖屏)为例,则创建的投屏窗口1 (对应于终端设备A)和投屏窗口2(对应于终端设备B)也均为竖屏。Fig. 21(e) exemplarily shows the screen-casting interface in a dual-vertical-screen scene with two screen-casting channels. The screen resolution of the display device is 1920*1080, and the media configuration files sent by terminal device A and terminal device B indicate that the aspect ratio is 9:16 (vertical screen) as an example, then the created screen projection window 1 (corresponding to terminal Device A) and screen projection window 2 (corresponding to terminal device B) are also vertical screens.
对于图21(e),由于投屏窗口1和投屏窗口2均为竖屏,因此使两个投屏窗口的高度都适配屏幕高度,即windowHeight1=windowHeight2=1080,由此可计算出windowWidth1=windowWidth2=1080*9/16=607.5,windowWidth1*windowHeight1=windowWidth2*windowHeight2=607.5*1080。以投屏窗口1的左上角顶点作为预设基准点Point1,以投屏窗口2的左上角顶点作为预设基准点Point2,考虑到两个竖屏窗口沿横向等间距均匀排布时观看效果更佳,则计算出间距值dx为(1920-2*607.5)/3=235,则预设基准点Point1的坐标(x1,y1)=(235,0),预设基准点Point2的坐标(x2,y2)=(2*dx+607.5,0)=(1077.5,0),由此得到的布局如图21(e)所示。For Figure 21(e), since the screen projection window 1 and the screen projection window 2 are both vertical screens, the heights of the two screen projection windows are adapted to the height of the screen, that is, windowHeight1=windowHeight2=1080, and windowWidth1 can be calculated from this =windowWidth2=1080*9/16=607.5, windowWidth1*windowHeight1=windowWidth2*windowHeight2=607.5*1080. The upper left corner of the screen projection window 1 is used as the preset reference point Point1, and the upper left corner of the screen projection window 2 is used as the preset reference point Point2. Considering that the two vertical screen windows are evenly spaced along the horizontal direction, the viewing effect is better. Good, then the distance value dx calculated is (1920-2*607.5)/3=235, then the coordinates (x1, y1)=(235, 0) of the preset reference point Point1, the coordinates of the preset reference point Point2 (x2 , y2)=(2*dx+607.5,0)=(1077.5,0), and the resulting layout is shown in Figure 21(e).
在一些示例中,例如在图20(a)的单路竖屏场景下,显示设备又与终端设备B建立起投屏连接,并且终端设备B发送的媒体配置信息中指示宽高比例为9:16(竖屏),控制器250检测到投屏路数由1变更为2,则可将布局更新为图21(e)所示的双路双竖屏的版本。In some examples, for example, in the single vertical screen scenario in FIG. 20(a), the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 9: 16 (vertical screen), the controller 250 detects that the number of projection screens has been changed from 1 to 2, and the layout can be updated to the version with two channels and two vertical screens as shown in FIG. 21(e).
在一些示例中,例如在21(e)的双路双竖屏场景下,终端设备A或终端设备B断开投屏连接,控制器250检测到投屏路数由2变更为1,则可将布局更新为类似于图20(a)所示的单竖屏窗口独占屏幕的版本。In some examples, for example, in the dual-channel dual-vertical screen scenario of 21(e), terminal device A or terminal device B disconnects the screen projection connection, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1, then it can The layout is updated to a version similar to that shown in Figure 20(a) where a single vertical window occupies the screen.
在一些示例中,在图20(a)~图20(b)以及图21(a)~图21(e)中任意一个场景下,如果全部投屏连接均已断开,即控制器检测到投屏路数变为0,则退出投屏界面,使UI显示界面恢复至投屏前。In some examples, in any of the scenarios in Figures 20(a) to 20(b) and 21(a) to 21(e), if all screen projection connections are disconnected, the controller detects that When the number of screen projection channels becomes 0, the screen projection interface will be exited, and the UI display interface will return to that before screen projection.
上文以单路和双路投屏为例,详细说明了布局的构建与规划流程,当投屏路数M大于或等于3时,布局的匹配机制可适应性参照前述示例,布局的匹配可具有自适应性、灵活性和多样性等特点,本申请不再一一列举和赘述。在投屏路数、各路投屏媒体的屏向和宽高比例都相同时,由于各投屏窗口间排版形式不同,导致布局存在多样性,例如前述双路双横屏场景下至少包括图21(a)和图21(b)两种版本,前述双路一横屏一竖屏场景下至少包括图21(c)和图21(d)两种版本。The above takes single-channel and dual-channel projection as examples to explain the layout construction and planning process in detail. When the number of projection channels M is greater than or equal to 3, the adaptability of the layout matching mechanism can refer to the previous example. The layout matching can be It has the characteristics of adaptability, flexibility and diversity, etc., and this application will not list and repeat them one by one. When the number of screen projection channels, screen orientation and width-to-height ratio of each screen projection media are the same, the layout forms of each screen projection window are different, resulting in a variety of layouts. 21(a) and 21(b), the aforementioned two-way-horizontal-vertical screen scenario includes at least the two versions of FIG. 21(c) and 21(d).
在一些实施例中,在匹配到多个版本的布局时,本申请基于最大面积原则进行最优筛选,即计算每个版本的布局中全部投屏窗口的面积总和,筛选出面积总和最大的目标布局,即Smax=Max{k=1,2…,Q∣Sk},其中k用于表示布局的版本序号,Q为匹配出的布局的版本数量,1≤k≤Q,
公式中si为第k个版本的布局中第i个投屏窗口的面积,M为所有版本的布局中投屏窗口的总数。通过这种方式计算出最大面积总和Smax,然后将Smax对应的版本作为筛选出的目标布局,并在投屏界面显示目标布局,使得各投屏窗口更充分并且最大化地占用屏幕,各投屏窗口间排版紧凑,尽可能地降低留白区域,提升投屏界面的展示效果。由于投屏窗口为矩形,因此单独一个投屏窗口的面积等于其宽度和高度的乘积。
In some embodiments, when multiple versions of the layout are matched, the present application performs optimal screening based on the principle of maximum area, that is, calculates the sum of the areas of all projection windows in each version of the layout, and selects the target with the largest sum of areas Layout, that is, Smax=Max{k=1,2...,Q∣Sk}, where k is used to represent the version number of the layout, Q is the number of versions of the matched layout, 1≤k≤Q, In the formula, si is the area of the i-th screen projection window in the layout of the k-th version, and M is the total number of screen projection windows in all versions of the layout. In this way, the maximum area sum Smax is calculated, and then the version corresponding to Smax is used as the selected target layout, and the target layout is displayed on the screen projection interface, so that each projection window can occupy the screen more fully and to the maximum extent, and each projection screen The layout between the windows is compact, and the blank area is reduced as much as possible to improve the display effect of the screen projection interface. Since the screen projection window is rectangular, the area of a single screen projection window is equal to the product of its width and height.
作为一种示例,以双路双横屏场景下,显示设备的屏幕分辨率为1920*1080,终端设备A发送的媒体配置文件指示宽高比例为17:9(横屏),终端设备B发送的媒体配置文件指示宽高比例为16:9(横屏)为例,即两路投屏的媒体具备不同的宽高比例,构建两个投屏窗口,投屏窗口1对应于终端设备A,投屏窗口2对应于终端设备B。As an example, in a dual-channel dual-horizontal screen scenario, the screen resolution of the display device is 1920*1080, the media configuration file sent by terminal device A indicates that the aspect ratio is 17:9 (horizontal screen), and terminal device B sends As an example, the media configuration file indicated that the ratio of width to height is 16:9 (horizontal screen), that is, the media of the two screens have different ratios of width to height, and two screen projection windows are constructed. Screen projection window 1 corresponds to terminal device A. Screen projection window 2 corresponds to terminal device B.
如图22(a)示出布局的版本1,由于为横向并排排布,为最大化地占用屏幕区域,使投屏窗口1和投屏窗口2的宽度之和等于屏幕宽度,即windowWidth1+windowWidth2=1920,此外为使双投屏窗口的尺寸大小更加均衡,可设置windowWidth1=windowWidth2=960,则 windowHeight1=960*9/17=508,windowHeight2=960*9/16=540,windowWidth1*windowHeight1=960*508,windowWidth2*windowHeight2=960*540,则对于版本1,其投屏窗口的面积总和S1=960*508+960*540=1006080。As shown in Figure 22(a), version 1 of the layout is arranged side by side in a horizontal direction. In order to maximize the occupation of the screen area, the sum of the widths of the screen projection window 1 and the screen projection window 2 is equal to the screen width, that is, windowWidth1+windowWidth2 =1920, in addition, in order to make the size of the dual projection screen window more balanced, you can set windowWidth1=windowWidth2=960, then windowHeight1=960*9/17=508, windowHeight2=960*9/16=540, windowWidth1*windowHeight1=960 *508, windowWidth2*windowHeight2=960*540, then for version 1, the sum of the areas of the screen projection windows S1=960*508+960*540=1006080.
图22(b)示出布局的版本2,由于为纵向并列排布,为最大化地占用屏幕区域,使投屏窗口1和投屏窗口2的高度之和等于屏幕高度,即windowHeight1+windowHeight2=1080,此外为使双投屏窗口的尺寸大小更加均衡,可设置windowHeight1=windowHeight2=540,windowWidth1=540*17/9=1020,windowWidth2=540*16/9=960,即可得到windowWidth1*windowHeight1=1020*540,windowWidth2*windowHeight2=960*540,则对于版本2,其投屏窗口的面积总和S2=1020*540+960*540=1069200。通过比较显而易见,S2大于S1,因此将版本2作为最终显示的目标布局。Figure 22(b) shows version 2 of the layout. Since they are arranged side by side vertically, in order to maximize the screen area occupied, the sum of the heights of screen projection window 1 and screen projection window 2 is equal to the height of the screen, that is, windowHeight1+windowHeight2= 1080, and to make the size of the dual projection screen more balanced, you can set windowHeight1=windowHeight2=540, windowWidth1=540*17/9=1020, windowWidth2=540*16/9=960, you can get windowWidth1*windowHeight1= 1020*540, windowWidth2*windowHeight2=960*540, then for version 2, the sum of the areas of the screen projection windows S2=1020*540+960*540=1069200. It is obvious by comparison that S2 is larger than S1, so version 2 is taken as the target layout for the final display.
图22(c)示例性示出了投屏路数为2且双竖屏场景下的一种投屏界面,图22(c)中是将两个竖屏的投屏窗口沿横向并排排布,且两个终端设备录屏的分辨率不同,如终端设备A录屏的分辨率是1920*1080,终端设备B录屏的分辨率是2040*1080,终端设备A和终端设备B发送的媒体配置文件指示宽高比例分别为9:16(竖屏)和9:17(竖屏),则创建的投屏窗口1(对应于终端设备A)和投屏窗口2(对应于终端设备B)也均为竖屏,但是两个投屏窗口的横向显示尺寸不同。Figure 22(c) exemplarily shows a screen-casting interface in a dual-vertical-screen scene with two screen-casting channels. In Figure 22(c), the screen-casting windows of two vertical screens are arranged side by side in the horizontal direction , and the screen recording resolutions of the two terminal devices are different. For example, the screen recording resolution of terminal device A is 1920*1080, and the resolution of terminal device B’s screen recording is 2040*1080. The media sent by terminal device A and terminal device B The configuration file indicates that the aspect ratio is 9:16 (vertical screen) and 9:17 (vertical screen), then the created screen projection window 1 (corresponding to terminal device A) and screen projection window 2 (corresponding to terminal device B) Both are vertical screens, but the horizontal display sizes of the two projection windows are different.
对于图22(c),由于投屏窗口1和投屏窗口2均为竖屏,因此使两个投屏窗口的高度都适配屏幕高度,即windowHeight1=windowHeight2=1080,由此可计算出windowWidth1=1080*9/16=607.5,windowWidth2=1080*9/17=571.7,以投屏窗口1的左上角顶点作为预设基准点Point1,以投屏窗口2的左上角顶点作为预设基准Point2,考虑到两个竖屏窗口沿横向等间距均匀排布时观看效果更佳,则计算出间距值dx为(1920-607.5-571.7)/3=247,则预设基准点Point1的坐标(x1,y1)=(247,0),预设基准点Point2的坐标(x2,y2)=(2*dx+607.5,0)=(1101.5,0),由此得到的布局如图22(c)所示。For Figure 22(c), since the screen projection window 1 and the screen projection window 2 are both vertical screens, the heights of the two screen projection windows are adapted to the height of the screen, that is, windowHeight1=windowHeight2=1080, and windowWidth1 can be calculated from this =1080*9/16=607.5, windowWidth2=1080*9/17=571.7, the upper left corner of the screen projection window 1 is used as the default reference point Point1, and the upper left corner of the screen projection window 2 is used as the preset reference point Point2, Considering that the viewing effect is better when the two vertical screen windows are evenly spaced along the horizontal direction, the calculated spacing value dx is (1920-607.5-571.7)/3=247, and the coordinates of the preset reference point Point1 (x1, y1)=(247,0), the coordinates (x2,y2)=(2*dx+607.5,0)=(1101.5,0) of the preset reference point Point2, the resulting layout is shown in Figure 22(c) Show.
图22(d)示例性示出了投屏路数为2且双竖屏场景下的另一种投屏界面,图22(d)中同样是将两个竖屏的投屏窗口沿横向并排排布,终端设备A录屏的分辨率是2040*1080,终端设备B录屏的分辨率是1920*1080,按照图22(c)所示的计算方式,得到如图22(d)所示的布局。Figure 22(d) exemplarily shows another screen-casting interface in a dual-vertical-screen scene with the number of screen-casting channels being 2. In Figure 22(d), the screen-casting windows of two vertical screens are also arranged side by side in the horizontal direction Arrangement, the screen recording resolution of terminal device A is 2040*1080, and the screen recording resolution of terminal device B is 1920*1080. According to the calculation method shown in Figure 22(c), the result is shown in Figure 22(d). Layout.
在一些实施例中,若存在Q个版本的布局,并且每个版本具有相同的投屏窗口的面积总和,即S1=S2=…=SQ,则可以随机选取任意一个版本作为目标布局。In some embodiments, if there are Q versions of the layout, and each version has the same sum of the area of the projection window, that is, S1=S2=...=SQ, any version can be randomly selected as the target layout.
在一些实施例中,布局中各投屏窗口可能是按照接入投屏的时间进行排序,也可能不是。以三路投屏为例,假设显示设备先后依次与终端设备A、终端设备C和终端设备B投屏连接,终端设备A发送的媒体配置信息指示其宽高比例为16:9(横屏),终端设备C发送的媒体配置信息指示其宽高比例为9:16(竖屏),终端设备B发送的媒体配置信息指示其宽高比例为9:16(竖屏),即对应于三路一横屏两竖屏的场景。该场景下可以匹配出三个版本的布局,图23(a)示例了版本1,图23(b)示例了版本2,图23(c)示例了版本3。In some embodiments, the screen projection windows in the layout may or may not be sorted according to the time when the screen projection is accessed. Taking three-way screen projection as an example, assuming that the display device is successively connected to terminal device A, terminal device C, and terminal device B for screen projection, the media configuration information sent by terminal device A indicates that its aspect ratio is 16:9 (horizontal screen) , the media configuration information sent by terminal device C indicates that its aspect ratio is 9:16 (vertical screen), and the media configuration information sent by terminal device B indicates that its aspect ratio is 9:16 (vertical screen), which corresponds to three-way A scene with one horizontal screen and two vertical screens. In this scenario, three versions of the layout can be matched. Figure 23(a) illustrates version 1, Figure 23(b) illustrates version 2, and Figure 23(c) illustrates version 3.
其中,图23(a)示例的版本1支持按照接入投屏的先后顺序沿横向依次排布投屏窗口1(对应终端设备A)、投屏窗口2(对应终端设备C)和投屏窗口3(对应终端设备B),呈现为横屏-竖屏-竖屏的排版,将唯一的横屏窗口设置为首位展示;图23(b)则呈现为竖屏-横屏-竖屏的排版,即将唯一的横屏窗口设置为横向居中展示,双竖屏窗口对称分列在横屏窗口的两侧;图23(c)则呈现为竖屏-竖屏-横屏的排版,即将唯一的横屏窗口设置为末 位展示。图23(b)和图23(c)示例的版本则不支持按照接入投屏的时间顺序排序。Among them, version 1 of the example in Figure 23(a) supports arranging screen projection window 1 (corresponding to terminal device A), screen projection window 2 (corresponding to terminal device C) and screen projection window in sequence in the horizontal direction according to the sequence of access projection screen 3 (corresponding to terminal device B), presenting a layout of horizontal screen-vertical screen-vertical screen, and setting the only horizontal screen window as the first display; Figure 23(b) presents a layout of vertical screen-horizontal screen-vertical screen , that is, set the only horizontal screen window to be displayed in the horizontal center, and the double vertical screen windows are symmetrically arranged on both sides of the horizontal screen window; Figure 23(c) presents a layout of vertical screen-vertical screen-horizontal screen, and the only The horizontal screen window is set as the last display. The versions shown in Fig. 23(b) and Fig. 23(c) do not support sorting according to the chronological order of accessing the projected screens.
由于上述场景下三个版本的布局均符合最大面积原则,因此可以输出任意一个版本作为目标布局,在实际应用中还可参考视觉观看效果,择一选取某一版本的布局,比如图23(b)示出的版本2,其布局的对称性最佳,因此在三路一横屏两竖屏的场景下,可优选按照版本2的布局进行投屏显示。Since the layouts of the three versions in the above scenarios all comply with the principle of maximum area, any version can be output as the target layout. In practical applications, you can also refer to the visual viewing effect to choose a layout of a certain version, such as Figure 23(b ) shows the version 2, which has the best layout symmetry, so in the scenario of three channels, one horizontal screen and two vertical screens, it is preferable to use the layout of version 2 for projection display.
在一些实施例中,参照图24,本申请实施例还提供一种终端设备与显示设备间的交互流程,包括:In some embodiments, referring to FIG. 24 , this embodiment of the present application also provides an interaction process between a terminal device and a display device, including:
步骤S201,终端设备端:在接收到用户对媒体的投屏操作时,向显示设备发送投屏请求。其中,所述投屏请求中至少包括终端设备的设备信息;Step S201, the terminal device side: when receiving the user's screen projection operation on the media, send a screen projection request to the display device. Wherein, the screen projection request includes at least device information of the terminal device;
步骤S202,显示设备端:在接收到投屏请求时,建立与终端设备的投屏连接,并向终端设备反馈已授权信息;Step S202, the display device side: upon receiving the screen projection request, establish a screen projection connection with the terminal device, and feed back authorized information to the terminal device;
步骤S203,显示设备端:将投屏路数累积加一;Step S203, on the display device side: add one to the cumulative number of projection channels;
步骤S204,终端设备端:在接收到已授权信息时,将投屏数据和媒体配置信息发送给显示设备;Step S204, the terminal device side: when receiving the authorized information, send the projection data and media configuration information to the display device;
在根据一些实施例中,显示设备接收到投屏请求后,可以查询终端设备是否具备投屏权限。若终端设备具备投屏权限,则显示设备向终端设备反馈所述已授权信息;若终端设备不具备投屏权限,则显示设备向终端设备反馈未授权信息,以使终端设备接收到的未授权信息之后,不再发送投屏数据和媒体配置信息。其中,显示设备查询终端设备投屏权限的方式不限于通过UI弹窗来询问用户是否为终端设备授权,或者查询本机维护的可信任设备列表等。According to some embodiments, after receiving the screen projection request, the display device may inquire whether the terminal device has screen projection permission. If the terminal device has the permission to cast the screen, the display device will feed back the authorized information to the terminal device; if the terminal device does not have the permission to cast the screen, the display device will feed back unauthorized information to the terminal device, so that the unauthorized information received by the terminal device After the information, the projection data and media configuration information will no longer be sent. Among them, the way for the display device to inquire about the screen projection permission of the terminal device is not limited to asking the user whether to authorize the terminal device through a UI pop-up window, or querying the list of trusted devices maintained by the device.
步骤S205,显示设备端:接收终端设备发送的投屏数据和媒体配置信息,查询当前计数的投屏路数M。M大于或等于1;Step S205, the display device side: receiving the screen projection data and media configuration information sent by the terminal device, and querying the currently counted number M of screen projection channels. M is greater than or equal to 1;
步骤S206,显示设备端:获取当前已投屏接入的M个终端设备对应发送的媒体配置数据。媒体配置数据中至少应包括媒体视频的宽高比例,通过宽高比例可以获取到屏向信息;Step S206, the display device side: obtain the media configuration data correspondingly sent by the M terminal devices currently connected by screen projection. The media configuration data should at least include the aspect ratio of the media video, and the screen orientation information can be obtained through the aspect ratio;
步骤S207,显示设备端:根据屏幕分辨率或M个终端设备投屏媒体的显示屏向和宽高比例,及投屏路数M,计算出M组尺寸信息;Step S207, the display device side: calculate M groups of size information according to the screen resolution or the display direction and width-to-height ratio of the M terminal device projection media, and the number M of projection channels;
步骤S208,显示设备端:根据M组尺寸信息,构建M个投屏窗口。投屏窗口与其显示的媒体在屏向和宽高比例上保持一致;Step S208, display device side: construct M screen projection windows according to M groups of size information. The screen orientation and aspect ratio of the screen projection window and the displayed media are consistent;
步骤S209,显示设备端:根据屏幕分辨率和M组尺寸信息,计算位置分布信息。所述位置分布信息包括M个投屏窗口上的预设基准点的坐标,所述预设基准点的坐标用于定位投屏窗口在屏幕上的位置;Step S209, the display device side: calculate the position distribution information according to the screen resolution and the size information of the M groups. The position distribution information includes coordinates of preset reference points on the M projection screen windows, and the coordinates of the preset reference points are used to locate the position of the screen projection window on the screen;
步骤S210,显示设备端:根据所述位置分布信息,对M个投屏窗口进行定位和排布,得到布局。可选地,若得到多个版本的布局,可以基于最大面积原则对各个版本进行筛选,并输出投屏窗口的面积之和最大的目标布局;Step S210, the display device side: according to the location distribution information, locate and arrange the M screen projection windows to obtain the layout. Optionally, if multiple versions of the layout are obtained, each version can be screened based on the principle of maximum area, and the target layout with the largest sum of the areas of the projection windows can be output;
步骤S211,显示设备端:在投屏界面中显示所述布局,以及在所述布局中的每个投屏窗口内显示对应的终端设备发送的投屏数据。可选地,投屏窗口与终端设备的设备ID具有一一对应关系,以便显示设备将投屏数据准确投放到指定的投屏窗口进行显示;Step S211, display device side: display the layout in the screen projection interface, and display the screen projection data sent by the corresponding terminal device in each screen projection window in the layout. Optionally, there is a one-to-one correspondence between the screen projection window and the device ID of the terminal device, so that the display device can accurately project the screen projection data to the designated screen projection window for display;
步骤S212,显示设备端:在显示投屏界面过程中实时监控投屏路数;Step S212, the display device side: monitor the number of screen projection channels in real time during the process of displaying the screen projection interface;
步骤S213,显示设备端:在检测到投屏路数发生变更时,对所述布局进行更新;Step S213, on the display device side: when it is detected that the number of screen projection channels has changed, update the layout;
步骤S214,显示设备端:在检测到投屏路数减为零时,退出投屏界面。Step S214, display device side: exit the screen projection interface when it is detected that the number of screen projection channels has decreased to zero.
本申请基于投屏路数以及多路投屏媒体的宽高比例和屏向等配置信息,实现对投屏界面进行动态自适应布局匹配,使得M个投屏窗口的排版和布局更适配屏幕,多路投屏窗口在屏幕内最大范围的展示,投屏窗口分布更紧凑,减少了投屏界面的留白。此外,还可考虑投屏界面展示的美观性,例如投屏窗口间分布的均匀性、对称性、窗口间尺寸大小的相对均衡性等方面,设置每个投屏窗口的尺寸和位置分布,由此得到的布局显著提升了投屏界面的展示效果,进而提升用户观看投屏的体验。Based on the configuration information such as the number of projection channels and the width-to-height ratio and screen orientation of multi-channel projection media, this application realizes dynamic adaptive layout matching on the projection interface, so that the typesetting and layout of M projection windows are more suitable for the screen , the multi-channel projection windows are displayed in the largest range on the screen, and the distribution of the projection windows is more compact, reducing the blank space of the projection interface. In addition, the aesthetics of the screen projection interface can also be considered, such as the uniformity and symmetry of the distribution of the screen projection windows, the relative balance of the size of the windows, etc., and the size and position distribution of each screen projection window can be set. The obtained layout significantly improves the display effect of the screen projection interface, thereby enhancing the user's viewing experience of the screen projection.
图25示出了根据一些实施例的显示设备200中应用程序的图标控件界面显示图。在一些实施例中,显示设备启动后可以直接进入预置的视频点播程序的界面,视频点播程序的界面可以如图25中所示,至少包括导航栏510和位于导航栏510下方的内容显示区,内容显示区中显示的内容会随导航栏中被选中控件的变化而变化。应用程序层中的程序可以被集成在视频点播程序中通过导航栏的一个控件进行展示,也可以在导航栏中的应用控件被选中后进行进一步显示。Fig. 25 shows a display diagram of an icon control interface of an application program in the display device 200 according to some embodiments. In some embodiments, after the display device is started, it can directly enter the interface of the preset video-on-demand program. The interface of the video-on-demand program can be as shown in FIG. , the content displayed in the content display area will change with the selected control in the navigation bar. The program in the application layer can be integrated in the video-on-demand program and displayed through a control in the navigation bar, or can be further displayed after the application control in the navigation bar is selected.
在一些实施例中,显示设备启动后可以直接进入上次选择的信号源的显示界面,或者信号源选择界面,其中信号源可以是预置的视频点播程序,还可以是HDMI接口,直播电视接口等中的至少一种,用户选择不同的信号源后,显示器可以显示从不同信号源获得的内容。In some embodiments, after the display device is started, it can directly enter the display interface of the signal source selected last time, or the signal source selection interface, wherein the signal source can be a preset video-on-demand program, or an HDMI interface, a live TV interface At least one of the above, after the user selects different signal sources, the display can display the content obtained from different signal sources.
随着移动互联网技术的发展,手机、平板电脑等移动终端的功能越来越丰富,越来越多的用户使用移动终端观看视频、浏览文件等。但是,由于手机、平板电脑等移动终端的屏幕尺寸较小,而电视机、台式显示器等显示设备的屏幕尺寸较大,很多用户还是喜欢在这类显示设备上观看视频、浏览文件。With the development of mobile Internet technology, mobile terminals such as mobile phones and tablet computers have more and more functions, and more and more users use mobile terminals to watch videos and browse files. However, due to the small screen size of mobile terminals such as mobile phones and tablet computers, and the large screen size of display devices such as televisions and desktop monitors, many users still like to watch videos and browse files on such display devices.
以电视机为例,当用户希望在电视机上观看移动终端中播放的视频时,可以使用移动终端的投屏功能,与电视机建立投屏连接,从而实现在移动终端和电视机上同步观看该视频。一些实施例中,在进行投屏时,先开启移动终端中的投屏应用,移动终端中的投屏应用在启动之后,在移动终端当前接入的局域网内自动搜索可以进行投屏的显示设备,然后在移动终端的显示界面以列表的方式展示各个可以投屏的显示设备的设备名称。用户根据实际投屏需要,在移动终端的显示界面中选择点击需要投屏的显示设备的设备名称后,移动终端即可与用户点击选择的显示设备建立投屏连接。Taking the TV as an example, when the user wants to watch the video played by the mobile terminal on the TV, he can use the screen projection function of the mobile terminal to establish a screen projection connection with the TV, so as to watch the video on the mobile terminal and the TV synchronously . In some embodiments, when performing screen projection, the screen projection application in the mobile terminal is first started, and after the screen projection application in the mobile terminal is started, it automatically searches for a display device that can perform screen projection in the local area network currently connected to the mobile terminal , and then the display interface of the mobile terminal displays the device names of each display device that can cast a screen in a list. After the user selects and clicks the device name of the display device that needs to be projected in the display interface of the mobile terminal according to the actual screen projection needs, the mobile terminal can establish a screen projection connection with the display device selected by the user.
在移动终端与显示设备连接投屏连接之后,移动终端便可以将移动终端上显示的数据发送到该显示设备上,实现在该显示设备上观看移动终端中的视频。但是,相关技术中的投屏方法,无法在一个显示设备上同时对通过多个投屏协议发送过来的内容进行投屏播放。After the mobile terminal is connected to the display device for screen projection, the mobile terminal can send the data displayed on the mobile terminal to the display device, so as to watch the video in the mobile terminal on the display device. However, the screen projection method in the related art cannot simultaneously perform screen projection and playback on one display device for content sent through multiple screen projection protocols.
为了解决这一技术问题,本申请实施例提供的一种显示设备,包括:显示器及控制器,其中,显示器被配置为显示用户界面。控制器中可运行一个播放应用和至少一个投屏协议应用,所述至少一个投屏协议应用均可连接至所述播放应用,每个投屏协议应用均能够独立与播放应用进行数据通信。In order to solve this technical problem, an embodiment of the present application provides a display device, including: a display and a controller, wherein the display is configured to display a user interface. A playback application and at least one screen projection protocol application can be run in the controller, and the at least one screen projection protocol application can be connected to the playback application, and each screen projection protocol application can independently perform data communication with the playback application.
每个投屏协议应用均采用不同的投屏协议,作为示例,现有的投屏协议包括:DLNA投屏协议、Miracast投屏协议、Airplay投屏协议以及其它投屏协议。基于移动终端待投屏内容的两种类型(一类是移动终端上第三方媒体资源的媒体数据,另一类是移动终端的屏幕实时画面),DLNA投屏协议能够支持移动终端上第三方媒体资源的媒体数据的推送投屏;Miracast投屏协议能够支持移动终端的屏幕实时画面的镜像投屏;Airplay投屏协议既能够支持移动终端上第三方媒体资源的媒体数据的推送投屏,也能够支持移动终端的屏 幕实时画面的镜像投屏;其它投屏协议可以是各显示设备生产厂家自开发的协议。Each screen projection protocol application adopts a different screen projection protocol. As an example, the existing screen projection protocols include: DLNA projection protocol, Miracast projection protocol, Airplay projection protocol and other projection protocols. Based on the two types of content to be projected on the mobile terminal (one is the media data of third-party media resources on the mobile terminal, and the other is the real-time screen of the mobile terminal), the DLNA screen projection protocol can support third-party media on the mobile terminal The push and projection of the media data of resources; the Miracast projection protocol can support the mirror projection of the real-time screen of the mobile terminal screen; the Airplay projection protocol can not only support the push projection of the media data of the third-party media resources on the mobile terminal, but also can It supports the mirror projection of the real-time screen of the screen of the mobile terminal; other screen projection protocols can be the protocols developed by each display device manufacturer.
一些实施例中,可以针对现有的每一种投屏协议,在控制器中独立开发一个投屏协议应用。示例性的,参见图26,针对DLNA投屏协议,开发一个DLNA投屏协议应用;针对Miracast投屏协议,开发一个Miracast投屏协议应用;针对Airplay投屏协议,开发一个Airplay投屏协议应用;针对其它投屏协议,开发相应的其它投屏协议应用。In some embodiments, for each existing screen projection protocol, a screen projection protocol application can be independently developed in the controller. Exemplarily, referring to FIG. 26, for the DLNA screen projection protocol, a DLNA projection screen protocol application is developed; for the Miracast screen projection protocol, a Miracast screen projection protocol application is developed; for the Airplay screen projection protocol, an Airplay projection screen protocol application is developed; For other screen projection protocols, develop corresponding other screen projection protocol applications.
每种投屏协议应用均以后台服务形式存在,且都可以独立与播放应用之间进行数据通信。播放应用为一个独立的播放器应用,不负责与移动终端之间的协议交互,只负责接收并播放各种投屏协议的媒体数据。Each screencasting protocol application exists in the form of a background service, and can independently perform data communication with the playback application. The playback application is an independent player application, which is not responsible for the protocol interaction with the mobile terminal, but only responsible for receiving and playing media data of various screen projection protocols.
参见图27所示的工作流程示意图,为了解决相关技术中投屏方法无法在一个显示设备上同时对多个移动终端上的显示数据进行投屏播放的问题,本实施例中的控制器被配置为执行下述步骤:Referring to the workflow schematic diagram shown in Figure 27, in order to solve the problem in the related art that the projection method cannot simultaneously project and play the display data on multiple mobile terminals on one display device, the controller in this embodiment is configured To perform the following steps:
步骤701,在接收到第一移动终端发送的第一投屏请求时,通过与所述第一投屏请求相对应的第一投屏协议应用,建立与所述第一移动终端之间的投屏连接,以及将所述第一移动终端通过投屏连接传输过来的第一待投屏内容发送至播放应用;Step 701: When receiving a first screen projection request sent by a first mobile terminal, establish a projection with the first mobile terminal through a first screen projection protocol application corresponding to the first screen projection request. screen connection, and sending the first screen-to-be-cast content transmitted by the first mobile terminal through the screen-casting connection to the playback application;
步骤702,通过所述播放应用,在所述用户界面上创建第一播放窗口,所述第一播放窗口用于播放所述第一待投屏内容。Step 702: Create a first playback window on the user interface through the playback application, and the first playback window is used to play the first content to be projected.
第一移动终端可以是唯一一个与显示设备进行投屏连接的移动终端,也可以是多个与显示设备进行投屏连接的移动终端中的任一个。第一投屏请求可以是第一移动终端通过任意投屏协议发送至显示设备的投屏请求。第一投屏协议应用为与所述第一投屏请求采用相同投屏协议的投屏协议应用,第一播放窗口可以是播放应用创建的唯一一个播放窗口,也可以是播放应用创建的多个播放窗口中的任一个。The first mobile terminal may be the only mobile terminal that performs screen projection connection with the display device, or may be any one of multiple mobile terminals that perform screen projection connection with the display device. The first screen projection request may be a screen projection request sent by the first mobile terminal to the display device through any screen projection protocol. The first screen-casting protocol application is a screen-casting protocol application that adopts the same screen-casting protocol as the first screen-casting request, and the first play window may be the only play window created by the play application, or multiple play windows created by the play application. Play any of the windows.
当第一移动终端向显示设备发送第一投屏请求时,由控制器接收该投屏请求。通过第一投屏协议应用,对该投屏请求进行响应及处理,并建立投屏连接,在接收到第一移动终端通过投屏连接传输过来的第一待投屏内容,不通过第一投屏协议应用创建播放窗口进行播放,而是通过第一投屏协议应用将第一待投屏内容发送给播放应用,由播放应用创建一个播放窗口(即第一播放窗口)进行播放。当存在通过另一投屏协议发送过来的投屏请求时,对应的投屏协议应用对该投屏请求进行响应,此时播放应用仍会继续运行,因而第一播放窗口不会受到影响,将继续对第一待投屏内容进行播放。When the first mobile terminal sends a first screen projection request to the display device, the controller receives the screen projection request. Through the application of the first screen projection protocol, respond to and process the screen projection request, and establish a screen projection connection. The screen protocol application creates a playback window for playback, but sends the first content to be projected to the playback application through the first screen projection protocol application, and the playback application creates a playback window (ie, the first playback window) for playback. When there is a screen projection request sent by another screen projection protocol, the corresponding screen projection protocol application responds to the screen projection request, and the playback application will continue to run at this time, so the first playback window will not be affected, and will Continue to play the first content to be projected.
投屏协议应用只负责与移动终端的协议交互,不负责投屏内容的播放,播放应用只负责创建相应的播放窗口对接收到的投屏内容进行播放,不负责与移动终端的协议交互。当存在多个移动终端通过多种投屏协议向显示设备发送多个投屏请求时,控制器可以通过多个投屏协议应用各自独立与多个移动终端进行协议交互,建立投屏连接,然后向播放应用转发待投屏内容,播放应用只要接收到一个待投屏内容,便创建一个播放窗口进行播放,若同时或陆续接收到多个待投屏内容,便相应创建多个独立的播放窗口进行播放,如此,便可实现在同一显示设备上同时对通过多种投屏协议发送过来的待投屏内容进行播放。The screen projection protocol application is only responsible for the protocol interaction with the mobile terminal, not for the playback of the screen projection content. The playback application is only responsible for creating a corresponding playback window to play the received projection content, and is not responsible for the protocol interaction with the mobile terminal. When there are multiple mobile terminals sending multiple screen projection requests to the display device through multiple screen projection protocols, the controller can independently perform protocol interaction with multiple mobile terminals through multiple screen projection protocol applications, establish a screen projection connection, and then Forward the content to be projected to the playback application. As long as the playback application receives one content to be projected, it will create a playback window to play. If it receives multiple contents to be projected at the same time or successively, it will create multiple independent playback windows accordingly. In this way, it is possible to simultaneously play the content to be screened sent through multiple screen projection protocols on the same display device.
结合图28,进一步对如何在一个显示设备上同时播放通过多个投屏协议发送过来的内容进行解释说明。当显示设备正在播放第一移动终端的投屏内容时,若又接收到一个投屏请求,为了防止当前投屏内容的播放被终止,控制器被进一步配置为执行下述步骤:With reference to FIG. 28 , how to simultaneously play content sent through multiple screen projection protocols on one display device is further explained. When the display device is playing the screen-casting content of the first mobile terminal, if a screen-casting request is received again, in order to prevent the playback of the current screen-casting content from being terminated, the controller is further configured to perform the following steps:
步骤801,在接收到第二移动终端发送的第二投屏请求时,通过与所述第二投屏请求相对应的第二投屏协议应用,建立与所述第二移动终端之间的投屏连接,以及将所述第二 移动终端通过投屏连接传输过来的第二待投屏内容发送至播放应用;Step 801: When receiving a second screen projection request sent by a second mobile terminal, establish a projection with the second mobile terminal through a second screen projection protocol application corresponding to the second screen projection request. screen connection, and sending the second screen content to be projected transmitted by the second mobile terminal through the screen projection connection to the playback application;
需要说明的是,第二移动终端与第一移动终端可以为同一个移动终端,也可以为两个不同的移动终端,第二投屏协议应用为与所述第二投屏请求采用相同投屏协议的投屏协议应用,第一投屏请求和第二投屏请求是用户基于两种不同的待投屏内容所发出的两个不同投屏请求,这两个投屏请求可以采用同一种投屏协议发出,也可以采用不同的投屏协议发出。如果这两个投屏请求采用同一种投屏协议发出,那么第一投屏协议应用和第二投屏协议应用为同一个投屏协议应用,如果这两个投屏请求采用不同的投屏协议发出,那么第一投屏协议应用和第二投屏协议应用为两个不同的投屏协议应用。It should be noted that the second mobile terminal and the first mobile terminal may be the same mobile terminal, or two different mobile terminals, and the second screen projection protocol is applied to use the same screen projection as the second screen projection request. The screencasting protocol application of the protocol, the first screencasting request and the second screencasting request are two different screencasting requests sent by the user based on two different content to be screencasted, and the two screencasting requests can use the same projection The screen protocol can be used to send out, or different screen projection protocols can be used to send out. If the two screencasting requests are sent using the same screencasting protocol, then the first screencasting protocol application and the second screencasting protocol application are the same screencasting protocol application, if the two screencasting requests use different screencasting protocols sent, then the first screen projection protocol application and the second screen projection protocol application are two different screen projection protocol applications.
步骤802,通过所述播放应用,在所述用户界面上创建第二播放窗口,所述第二播放窗口用于播放所述第二待投屏内容。Step 802: Create a second playback window on the user interface through the playback application, and the second playback window is used to play the second content to be projected.
参见图29,所述第二播放窗口与所述第一播放窗口为互相独立的播放窗口,能够同时存在于显示器上,且这两个播放窗口可以存在部分重叠,或者不存在任何重叠。Referring to FIG. 29 , the second playback window and the first playback window are independent playback windows that can exist on the display at the same time, and the two playback windows may overlap partially or not at all.
在一些实施例中,播放应用可以适应性的设置第一播放窗口和第二播放窗口的大小和位置,使得这两个播放窗口均能完整的显示在显示器上。或者,播放应用可以根据用户的偏好设置,调整第一播放窗口和第二播放窗口的大小及位置。In some embodiments, the playback application can adaptively set the size and position of the first playback window and the second playback window, so that the two playback windows can be completely displayed on the display. Or, the playback application can adjust the size and position of the first playback window and the second playback window according to the user's preferences.
基于上述流程,本领域技术人员可以毫无疑义的获知,当显示设备接收到更多的投屏请求时,如何对多个待投屏内容同时进行播放,在此不再赘述。Based on the above process, those skilled in the art can undoubtedly know how to simultaneously play multiple screen-casting contents when the display device receives more screen-casting requests, which will not be repeated here.
移动终端待投屏内容通常分为两种类型:一类是移动终端上第三方媒体资源的媒体数据,另一类是移动终端的屏幕实时画面。The content to be projected on the mobile terminal is generally divided into two types: one is the media data of the third-party media resources on the mobile terminal, and the other is the real-time screen of the mobile terminal.
基于这两种类型,控制器中的投屏协议应用需要通过不同的方法,将待投屏内容转发至播放应用,接下来将详细说明。为了便于理解,下述内容仅针对第一移动终端进行说明,但毫无疑义的是,针对任何发出投屏请求的移动终端,都可以使用同样的原理实现。Based on these two types, the screen projection protocol application in the controller needs to use different methods to forward the content to be screen projection to the playback application, which will be described in detail next. For ease of understanding, the following content is only described for the first mobile terminal, but there is no doubt that the same principle can be used for any mobile terminal that sends a screen projection request.
在一些实施例中,在将所述第一移动终端通过投屏连接传输过来的第一待投屏内容发送至播放应用之前,所述控制器还被配置为:In some embodiments, before sending the first screen-to-be-cast content transmitted by the first mobile terminal through the screen-casting connection to the playback application, the controller is further configured to:
通过所述第一投屏协议应用,获取所述第一待投屏内容;其中,若所述第一待投屏内容属于第三方媒体资源的媒体数据,则所述第一待投屏内容为目标网页链接地址,若所述第一待投屏内容属于屏幕实时画面,则所述第一待投屏内容为所述第一移动终端的屏幕视频流。Obtain the first screen-to-be-cast content through the first screen-casting protocol application; wherein, if the first screen-to-be-cast content belongs to media data of a third-party media resource, then the first screen-to-be-cast content is The link address of the target web page, if the first screen-to-be-projected content belongs to the real-time screen, then the first screen-to-be-projected content is the screen video stream of the first mobile terminal.
在接收到第一投屏请求时,第一投屏协议应用响应,与发出第一投屏请求的投屏应用进行协议的命令协商,协商完成后,第一投屏协议应用获取第一移动终端发送过来的第一待投屏内容。When receiving the first screen projection request, the first screen projection protocol application responds, and negotiates a protocol command with the screen projection application that sent the first screen projection request. After the negotiation is completed, the first screen projection protocol application acquires the first mobile terminal The first content to be screened sent.
针对第三方媒体资源的媒体数据,第一移动终端发送给第一投屏协议应用的是一个第三方媒体资源的网页链接地址,该网页链接地址为第一待投屏内容对应的URL(Uniform Resource Locator,统一资源定位器)地址,本实施例称为目标网页链接地址,通常情况下是一个http类型的片源地址。针对屏幕实时画面,第一移动终端发送给第一投屏协议应用的是屏幕实时画面的视频流。For the media data of the third-party media resource, what the first mobile terminal sends to the first screen projection protocol application is a webpage link address of a third-party media resource, and the webpage link address is the URL (Uniform Resource Locator, Uniform Resource Locator) address, this embodiment is called the link address of the target web page, usually an http type film source address. For the real-time screen image, what the first mobile terminal sends to the first screen projection protocol application is a video stream of the real-time screen image.
在一些实施例中,若第一投屏协议应用接收到的第一待投屏内容为目标网页链接地址,可以通过Android的广播方式或者其余进程间的通信方式,将目标网页链接地址发送至播放应用。In some embodiments, if the first screen-to-be-projected content received by the first screen-casting protocol application is the link address of the target webpage, the link address of the target webpage may be sent to the player via the broadcast method of Android or other inter-process communication methods. application.
播放应用在接收到目标网页链接地址之后,创建第一播放窗口,并连接目标链接地址, 然后在所述第一播放窗口中播放从所述目标链接地址获取的视频内容。After receiving the link address of the target web page, the playing application creates a first play window, connects to the target link address, and then plays the video content obtained from the target link address in the first play window.
作为示例,若第一移动终端中的投屏应用通过DLNA投屏协议向显示设备发送第一投屏请求,显示设备控制器中的DLNA投屏协议应用进行响应,与投屏应用进行协议的命令协商,协商完成后建立投屏连接,若投屏应用通过投屏连接发送过来的待投屏内容为某个第三方视频软件中某个片源的URL地址,则DLNA投屏协议应用将接收到的URL地址通过android的广播或其它进程间通信方式,通知给播放应用,播放应用在接收到URL地址后,创建一个播放窗口(或播放控件)对该URL地址对应的片源进行播放。As an example, if the screen projection application in the first mobile terminal sends the first screen projection request to the display device through the DLNA screen projection protocol, the DLNA screen projection protocol application in the display device controller responds, and the screen projection application performs an agreement command Negotiation. After the negotiation is completed, the projection connection is established. If the content to be projected sent by the projection application through the projection connection is the URL address of a source in a third-party video software, the DLNA projection protocol application will receive the The URL address of the URL is notified to the playback application through the broadcast of android or other inter-process communication methods. After receiving the URL address, the playback application creates a playback window (or playback control) to play the film source corresponding to the URL address.
在一些实施例中,若第一投屏协议应用接收到的第一待投屏内容为第一移动终端的屏幕视频流,可以通过共享内存的方式,将第一待投屏内容发送至播放应用。具体的,所述控制器被配置为:In some embodiments, if the first screen-to-be-cast content received by the first screen-casting protocol application is the screen video stream of the first mobile terminal, the first screen-to-be-cast content can be sent to the playback application by sharing memory. . Specifically, the controller is configured as:
通过所述第一投屏协议应用,创建第一共享内存,所述第一共享内存具有唯一的索引名称。Through the application of the first screen projection protocol, a first shared memory is created, and the first shared memory has a unique index name.
通过所述第一投屏协议应用,将所述第一移动终端的屏幕视频流实时存入所述第一共享内存中,并向所述播放应用发出第一读取指令,所述第一读取指令包括所述第一共享内存的索引名称。Through the first screen projection protocol application, the screen video stream of the first mobile terminal is stored in the first shared memory in real time, and a first read instruction is sent to the playback application, and the first read The instruction fetch includes the index name of the first shared memory.
在接收到第一读取指令时,基于唯一的索引名称,播放应用能够从第一共享内存中实时读取屏幕视频流,并在所述第一播放窗口中播放从所述第一共享内存中实时读取的屏幕视频流。When receiving the first read instruction, based on the unique index name, the playback application can read the screen video stream from the first shared memory in real time, and play the video stream from the first shared memory in the first playback window. Screen video stream read in real time.
作为示例,若第一移动终端中的投屏应用通过Miracast投屏协议向显示设备发送第一投屏请求,显示设备控制器中的Miracast投屏协议应用进行响应,与投屏应用进行协议的命令协商,协商完成后,Miracast投屏协议应用创建一个共享内存,即第一共享内存,并为第一共享内存进行命名,然后把第一共享内存的名字通过android的广播或其它进程间的通信方式,通知给播放应用,同时,Miracast投屏协议应用接收从第一移动终端中通过其它数据传输协议实时传输过来的屏幕视频流,写入到第一共享内存中,播放应用创建一个播放窗口,从第一共享内存中读取实时的屏幕视频流进行播放。As an example, if the screen-casting application in the first mobile terminal sends the first screen-casting request to the display device through the Miracast screen-casting protocol, the Miracast screen-casting protocol application in the display device controller responds, and the command of the protocol with the screen-casting application Negotiation, after the negotiation is completed, the Miracast screen projection protocol application creates a shared memory, that is, the first shared memory, and names the first shared memory, and then passes the name of the first shared memory through android broadcast or other communication methods between processes , notify the playback application, and at the same time, the Miracast screen projection protocol application receives the screen video stream transmitted in real time from the first mobile terminal through other data transmission protocols, writes it into the first shared memory, and the playback application creates a playback window, from The real-time screen video stream is read from the first shared memory for playback.
在一些实施例中,若第一投屏协议应用接收到的第一待投屏内容为第一移动终端的屏幕视频流,除了共享内存的方式,也可以通过其余的进程间通信方式,将第一待投屏内容发送至播放应用。进程间通信方式可以为命名管道或者套接字socket等通信方式。In some embodiments, if the first screen-to-be-cast content received by the first screen-casting protocol application is the screen video stream of the first mobile terminal, in addition to sharing memory, other inter-process communication methods can also be used to upload the second Once the content to be cast is sent to the playback application. The inter-process communication mode may be a communication mode such as a named pipe or a socket socket.
当第一移动终端中的投屏应用停止投屏操作时,通过投屏协议,通知显示设备控制器中的第一投屏协议应用,第一投屏协议应用再通知播放应用停止播放。When the screen projection application in the first mobile terminal stops the screen projection operation, it notifies the first screen projection protocol application in the display device controller through the screen projection protocol, and the first screen projection protocol application notifies the playback application to stop playing.
在一些实施例中,如果第一待投屏内容是第一移动终端的屏幕实时画面,则在停止播放后,第一投屏协议应用可以删除创建的第一共享内存。In some embodiments, if the first screen-to-be-cast content is a real-time screen of the first mobile terminal, after the playback stops, the first screen-casting protocol application can delete the created first shared memory.
以下结合具体示例,对本申请提供的投屏方法进行说明。假设存在四个移动终端向显示设备发出了投屏请求,假设第一个移动终端通过DLNA投屏协议向显示设备推送视频软件A的某个片源,第二个移动终端通过DLNA投屏协议向显示设备推送视频软件B的某个片源,第三个移动终端通过miracast投屏协议向显示设备推送自身的屏幕画面,第四个移动终端通过airplay投屏协议向显示设备推送自身的屏幕画面。显示设备控制器中各协议应用:DLNA协议应用,miracast协议应用,airplay协议应用分别对这四个投屏请求进行响应。如图30所示,这四个移动终端分别与显示设备之间的交互过程见如下:The screen projection method provided by this application will be described below with reference to specific examples. Assume that there are four mobile terminals that have sent screen projection requests to the display device. Suppose that the first mobile terminal pushes a source of video software A to the display device through the DLNA projection protocol, and the second mobile terminal pushes a source of video software A to the display device through the DLNA projection protocol. The display device pushes a video source of video software B, the third mobile terminal pushes its own screen image to the display device through the miracast screen projection protocol, and the fourth mobile terminal pushes its own screen image to the display device through the airplay screen projection protocol. Each protocol application in the display device controller: DLNA protocol application, miracast protocol application, and airplay protocol application respectively respond to the four screen projection requests. As shown in Figure 30, the interaction process between the four mobile terminals and the display device is as follows:
(1)第一个移动终端通过DLNA投屏协议向显示设备推送视频软件A的某个片源, DLNA投屏协议应用接收到投屏请求,与第一个移动终端进行标准DLNA协议的交互,交互完成后,接收到视频软件A某个片源的URL地址,DLNA投屏协议应用启动播放应用,并把该URL地址通过进程通信方式传输给播放应用,播放应用创建播放窗口player 1(在显示器用户界面上的表现形式为一个播放窗口),并连接该URL地址进行视频播放。(1) The first mobile terminal pushes a video source of video software A to the display device through the DLNA projection protocol, and the DLNA projection protocol application receives the projection request and interacts with the first mobile terminal using the standard DLNA protocol. After the interaction is completed, the URL address of a video source of video software A is received, and the DLNA projection protocol application starts the playback application, and transmits the URL address to the playback application through process communication, and the playback application creates a playback window player 1 (on the monitor The display form on the user interface is a playback window), and the URL address is connected to play the video.
(2)第二个移动终端通过DLNA投屏协议向显示设备推送视频软件B的某个片源,DLNA投屏协议应用接收到投屏请求,与第二个移动终端进行标准DLNA协议的交互,交互完成后,接收到视频软件B某个片源的URL地址,DLNA投屏协议应用启动播放应用,并把该URL地址通过进程通信方式传输给播放应用,播放应用创建播放窗口player 2(在显示器用户界面上的表现形式为第二个播放窗口),并连接该URL地址进行视频播放。(2) The second mobile terminal pushes a source of video software B to the display device through the DLNA projection protocol, and the DLNA projection protocol application receives the projection request and interacts with the second mobile terminal using the standard DLNA protocol. After the interaction is completed, the URL address of a video source of video software B is received, the DLNA projection protocol application starts the playback application, and transmits the URL address to the playback application through process communication, and the playback application creates a playback window player 2 (on the monitor The form on the user interface is the second playback window), and connect to the URL address to play the video.
(2)第三个移动终端通过miracast投屏协议向显示设备推送自身的屏幕画面,miracast投屏协议应用接收到该投屏请求,进行miracast连接的命令协商,协商完成后进行屏幕画面数据的传输,miracast投屏协议应用创建一个共享内存,命名为miracast_sharememry1,同时通知播放应用从共享内存miracast_sharememry1中实时读取视频流,同时播放应用在显示器界面的合适位置创建播放窗口player3(在显示器用户界面上的表现形式为第三个播放窗口)对视频流进行播放。(2) The third mobile terminal pushes its own screen image to the display device through the miracast screen projection protocol. The miracast screen projection protocol application receives the screen projection request, conducts miracast connection command negotiation, and transmits the screen image data after the negotiation is completed , the miracast screencasting protocol application creates a shared memory named miracast_sharememry1, and at the same time notifies the playback application to read the video stream from the shared memory miracast_sharememry1 in real time, and at the same time the playback application creates a playback window player3 at a suitable position on the display interface (on the display user interface The form of expression is the third play window) to play the video stream.
(4)第四个移动终端通过airplay投屏协议向显示设备推送自身的屏幕画面,airplay投屏协议应用接收到该投屏请求,进行airplay连接的命令协商,协商完成后进行屏幕画面数据的传输,airplay投屏协议应用创建一个共享内存,命名为airplay_sharememry1,同时通知播放应用从共享内存airplay_sharememry1中实时读取视频流,同时播放应用在显示器界面的合适位置创建播放窗口player4(在显示器用户界面上的表现形式为第四个播放窗口)对视频流进行播放。(4) The fourth mobile terminal pushes its own screen image to the display device through the airplay screen projection protocol, and the airplay screen projection protocol application receives the screen projection request, conducts airplay connection command negotiation, and transmits the screen image data after the negotiation is completed , the airplay projection protocol application creates a shared memory named airplay_sharememry1, and at the same time notifies the playback application to read the video stream from the shared memory airplay_sharememry1 in real time, and at the same time, the playback application creates a playback window player4 at a suitable position on the display interface (on the display user interface The form of expression is the fourth playback window) to play the video stream.
本申请实施例提供的显示设备,控制器中独立开发了一个播放应用,当存在多个移动终端通过多种投屏协议向显示设备发送多个投屏请求时,控制器可以通过多个投屏协议应用各自独立与多个移动终端进行协议交互,建立投屏连接,然后向播放应用转发待投屏内容,播放应用只要接收到一个待投屏内容,便创建一个播放窗口进行播放,若同时或陆续接收到多个待投屏内容,便相应创建多个独立的播放窗口进行播放,如此,便可实现在同一显示设备上同时对通过多种投屏协议发送过来的待投屏内容进行播放。For the display device provided by the embodiment of the present application, a playback application is independently developed in the controller. When there are multiple mobile terminals sending multiple screen projection requests to the display device through multiple screen projection protocols, the controller can use multiple screen projection The protocol application independently interacts with multiple mobile terminals, establishes a projection connection, and then forwards the content to be projected to the playback application. As long as the playback application receives a content to be projected, it will create a playback window for playback. After receiving multiple content to be projected one after another, multiple independent playback windows will be created accordingly to play. In this way, the content to be projected through multiple screen projection protocols can be played on the same display device at the same time.
在一些实施例中,本申请公开了一种投屏方法,图31为移动终端与显示设备之间的多端交互流程图,结合图31,该投屏方法包括:In some embodiments, the present application discloses a screen projection method. FIG. 31 is a flow chart of multi-terminal interaction between a mobile terminal and a display device. With reference to FIG. 31 , the screen projection method includes:
步骤3101,第一移动终端发送第一投屏请求;Step 3101, the first mobile terminal sends a first screen projection request;
步骤3102,第一投屏协议应用响应第一投屏请求,并建立与所述第一移动终端之间的投屏连接,所述第一投屏协议应用为与所述第一投屏请求采用相同投屏协议的协议应用;Step 3102, the first screen projection protocol application responds to the first screen projection request, and establishes a screen projection connection with the first mobile terminal, and the first screen projection protocol application uses Protocol application of the same projection protocol;
步骤3103,第一移动终端通过所述投屏连接向所述第一投屏协议应用发送第一待投屏内容;Step 3103, the first mobile terminal sends the first screen-to-be-cast content to the first screen-casting protocol application through the screen-casting connection;
步骤3104,第一投屏协议应用将所述第一待投屏内容发送至播放应用;Step 3104, the first screen-casting protocol application sends the first content to be screen-casted to the playback application;
步骤3105,播放应用在显示器显示的用户界面上创建第一播放窗口,所述第一播放窗口用于播放所述第一待投屏内容。Step 3105, the playing application creates a first playing window on the user interface displayed on the display, and the first playing window is used to play the first content to be mirrored.
上述投屏方法,当第一移动终端向显示设备发送第一投屏请求时,由控制器接收该投屏请求。通过第一投屏协议应用,对该投屏请求进行响应及处理,并建立投屏连接,在接收到第一移动终端通过投屏连接传输过来的第一待投屏内容,不通过第一投屏协议应用创 建播放窗口进行播放,而是通过第一投屏协议应用将第一待投屏内容发送给播放应用,由播放应用创建一个播放窗口(即第一播放窗口)进行播放。当存在通过另一投屏协议发送过来的投屏请求时,对应的投屏协议应用对该投屏请求进行响应,此时播放应用仍会继续运行,因而第一播放窗口不会受到影响,将继续对第一待投屏内容进行播放。如此,可以在显示设备上实现同时播放通过多个投屏协议发送过来的待投屏内容。In the above screen projection method, when the first mobile terminal sends a first screen projection request to the display device, the controller receives the screen projection request. Through the application of the first screen projection protocol, respond to and process the screen projection request, and establish a screen projection connection. The screen protocol application creates a playback window for playback, but sends the first content to be projected to the playback application through the first screen projection protocol application, and the playback application creates a playback window (ie, the first playback window) for playback. When there is a screen projection request sent by another screen projection protocol, the corresponding screen projection protocol application responds to the screen projection request, and the playback application will continue to run at this time, so the first playback window will not be affected, and will Continue to play the first content to be projected. In this way, it is possible to simultaneously play on the display device the content to be projected sent through multiple screen projection protocols.
为了方便解释,已经结合具体的实施方式进行了上述说明。但是,上述示例性的讨论不是意图穷尽或者将实施方式限定到上述公开的具体形式。根据上述的教导,可以得到多种修改和变形。上述实施方式的选择和描述是为了更好的解释原理以及实际的应用,从而使得本领域技术人员更好的使用所述实施方式以及适于具体使用考虑的各种不同的变形的实施方式。For convenience of explanation, the above description has been made in conjunction with specific implementation manners. However, the above exemplary discussion is not intended to be exhaustive or to limit the implementations to the precise forms disclosed above. Many modifications and variations are possible in light of the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various modified embodiments suitable for specific use considerations.
Claims (10)
- 一种显示设备,包括:A display device comprising:显示器,被配置为显示投屏界面;a display configured to display a screen projection interface;通信器,被配置为与通信终端建立投屏连接;The communicator is configured to establish a screen projection connection with the communication terminal;控制器,被配置为:Controller, configured as:检测通信终端的连接数量;Detect the number of connections of communication terminals;根据所述连接数量规划投屏画面的布局方式,以获得所述投屏画面的显示参数;planning the layout of the projection screen according to the number of connections, so as to obtain the display parameters of the projection screen;向所述投屏画面对应的通信终端发送通知消息,所述通知消息包括所述显示参数,所述通知消息用于使所述通信终端按照所述显示参数生成投屏数据;Sending a notification message to the communication terminal corresponding to the screen projection screen, the notification message including the display parameters, and the notification message is used to enable the communication terminal to generate screen projection data according to the display parameters;接收所述通信终端反馈的所述投屏数据,以及按照所述布局方式显示所述投屏数据。receiving the screen projection data fed back by the communication terminal, and displaying the screen projection data according to the layout.
- 根据权利要求1所述的显示设备,所述控制器被进一步配置为:The display device according to claim 1, the controller is further configured to:获取用户输入的用于建立投屏连接的连接指令;Obtain the connection instruction input by the user for establishing the screen projection connection;响应于所述连接指令,向所述通信终端发起投屏连接;Initiating a screen projection connection to the communication terminal in response to the connection instruction;在完成所述投屏连接后,遍历所述通信终端的连接数量。After the screen projection connection is completed, the number of connections of the communication terminal is traversed.
- 根据权利要求2所述的显示设备,所述控制器被进一步配置为:The display device according to claim 2, the controller is further configured to:建立投屏数据通道;Establish a projection data channel;在所述投屏数据通道建立成功后,创建显示控件,所述显示控件用于显示通过所述投屏数据通道接收的投屏数据。After the screen projection data channel is successfully established, a display control is created, and the display control is used to display the screen projection data received through the screen projection data channel.
- 根据权利要求1所述的显示设备,所述控制器被进一步配置为:The display device according to claim 1, the controller is further configured to:获取用户输入的用于断开投屏连接的退出指令;Obtain the exit command input by the user for disconnecting the screen projection connection;响应于所述退出指令,断开所述通信终端对应的投屏连接;In response to the exit instruction, disconnect the screen projection connection corresponding to the communication terminal;遍历剩余通信终端的连接数量。Traverse the number of connections of the remaining communication terminals.
- 根据权利要求1所述的显示设备,所述控制器被进一步配置为:The display device according to claim 1, the controller is further configured to:检测所述显示器的屏幕尺寸信息;Detecting screen size information of the display;以所述连接数量和所述屏幕尺寸信息为索引,在数据库中匹配布局模板;matching the layout template in a database by using the number of connections and the screen size information as an index;在所述布局模板中提取所述显示参数。The display parameters are extracted in the layout template.
- 根据权利要求5所述的显示设备,所述控制器被进一步配置为:The display device according to claim 5, the controller is further configured to:如果在所述数据库中未匹配到所述布局模板,获取所述通信终端的设备参数;If the layout template is not matched in the database, acquire device parameters of the communication terminal;根据所述设备参数,在显示区域中划分与所述连接数量相同的子区域;According to the device parameters, divide the display area into sub-areas equal to the number of connections;遍历每个子区域的图形参数,以生成所述显示参数。Iterating over the graphic parameters of each sub-region to generate the display parameters.
- 根据权利要求1所述的显示设备,所述控制器被进一步配置为:The display device according to claim 1, the controller is further configured to:提取所述显示参数,所述显示参数包括每个显示窗口的显示方向、码率和分辨率,所述显示方向用于指示所述通信终端按照设定的方向生成投屏数据;Extracting the display parameters, the display parameters including the display direction, code rate and resolution of each display window, the display direction is used to instruct the communication terminal to generate screen projection data according to the set direction;根据所述显示参数生成通知消息,所述通知消息中包括所述显示方向、码率和分辨率。A notification message is generated according to the display parameters, and the notification message includes the display direction, code rate and resolution.
- 根据权利要求1所述的显示设备,所述控制器被进一步配置为:The display device according to claim 1, the controller is further configured to:检测投屏界面中每个投屏画面的有效画面范围;Detect the effective screen range of each screen-casting screen in the screen-casting interface;计算所述有效画面范围占显示窗口总范围的显示比例;Calculating the display ratio of the effective picture range to the total range of the display window;如果所述显示比例小于或等于比例阈值,向所述通信终端发送提示消息,所述提示消息用于提示所述通信终端旋转方向。If the display ratio is less than or equal to the ratio threshold, a prompt message is sent to the communication terminal, where the prompt message is used to remind the communication terminal of a rotation direction.
- 一种通信终端,包括:A communication terminal, comprising:显示单元,被配置为显示终端界面;a display unit configured to display a terminal interface;通信单元,被配置为与显示设备建立投屏连接;a communication unit configured to establish a screen projection connection with the display device;处理器,被配置为:Processor, configured as:接收所述显示设备发送的通知消息,所述通知消息包括投屏画面的显示参数,所述显示参数为所述显示设备根据当前通信终端连接数量规划投屏画面布局方式时获得;Receiving a notification message sent by the display device, the notification message including display parameters of the projection screen, the display parameters being obtained when the display device plans the layout of the projection screen according to the number of current communication terminal connections;按照所述显示参数生成投屏数据;Generate screen projection data according to the display parameters;向所述显示设备反馈所述投屏数据。The screen projection data is fed back to the display device.
- 一种投屏画面动态显示方法,包括:A method for dynamically displaying a projected screen image, comprising:显示设备检测通信终端的连接数量,并根据所述连接数量规划投屏画面的布局方式,以获得所述投屏画面的显示参数;The display device detects the number of connections of the communication terminal, and plans the layout of the projected screen according to the number of connections, so as to obtain the display parameters of the projected screen;所述显示设备向所述投屏画面对应的通信终端发送通知消息,所述通知消息包括所述显示参数;The display device sends a notification message to the communication terminal corresponding to the projection screen, and the notification message includes the display parameter;通信终端接收所述显示设备发送的通知消息,并按照所述显示参数生成投屏数据;The communication terminal receives the notification message sent by the display device, and generates projection data according to the display parameters;所述通信终端向所述显示设备反馈所述投屏数据;The communication terminal feeds back the projection data to the display device;所述显示设备按照所述布局方式显示所述投屏数据。The display device displays the projection data according to the layout.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202280053896.2A CN117837150A (en) | 2021-08-02 | 2022-06-29 | Display device, communication terminal and screen-throwing picture dynamic display method |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110881496.8A CN114286152A (en) | 2021-08-02 | 2021-08-02 | Display device, communication terminal and screen projection picture dynamic display method |
CN202110881335.9A CN113613072B (en) | 2021-08-02 | 2021-08-02 | Multi-channel screen-throwing display method and display equipment |
CN202110881496.8 | 2021-08-02 | ||
CN202110881335.9 | 2021-08-02 | ||
CN202111265530.5 | 2021-10-28 | ||
CN202111265530.5A CN113992963A (en) | 2021-10-28 | 2021-10-28 | Display device and screen projection method |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023011058A1 true WO2023011058A1 (en) | 2023-02-09 |
Family
ID=85155158
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2022/102439 WO2023011058A1 (en) | 2021-08-02 | 2022-06-29 | Display device, communication terminal, and projected-screen image dynamic display method |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN117837150A (en) |
WO (1) | WO2023011058A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116347145A (en) * | 2023-03-27 | 2023-06-27 | 广州南翼信息科技有限公司 | Multi-device screen sharing method, screen throwing device, display device and sharing system |
CN117156190A (en) * | 2023-04-21 | 2023-12-01 | 荣耀终端有限公司 | Screen projection management method and device |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150201193A1 (en) * | 2012-01-10 | 2015-07-16 | Google Inc. | Encoding and decoding techniques for remote screen sharing of media content using video source and display parameters |
WO2017042983A1 (en) * | 2015-09-07 | 2017-03-16 | 株式会社オプティム | Screen sharing device, screen sharing method, screen sharing system, and program therefor |
CN111601120A (en) * | 2020-04-21 | 2020-08-28 | 江苏大学 | Wireless screen transmission display system and display method |
CN112235626A (en) * | 2020-10-15 | 2021-01-15 | Oppo广东移动通信有限公司 | Video rendering method and device, electronic equipment and storage medium |
CN112394892A (en) * | 2019-08-15 | 2021-02-23 | 北京字节跳动网络技术有限公司 | Screen projection method, screen projection equipment, mobile terminal and storage medium |
CN112423102A (en) * | 2020-11-18 | 2021-02-26 | 努比亚技术有限公司 | Small window screen projection control method and device and computer readable storage medium |
CN112988102A (en) * | 2021-05-11 | 2021-06-18 | 荣耀终端有限公司 | Screen projection method and device |
CN113613072A (en) * | 2021-08-02 | 2021-11-05 | 海信视像科技股份有限公司 | Multi-path screen projection display method and display equipment |
CN114286152A (en) * | 2021-08-02 | 2022-04-05 | 海信视像科技股份有限公司 | Display device, communication terminal and screen projection picture dynamic display method |
-
2022
- 2022-06-29 CN CN202280053896.2A patent/CN117837150A/en active Pending
- 2022-06-29 WO PCT/CN2022/102439 patent/WO2023011058A1/en active Application Filing
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150201193A1 (en) * | 2012-01-10 | 2015-07-16 | Google Inc. | Encoding and decoding techniques for remote screen sharing of media content using video source and display parameters |
WO2017042983A1 (en) * | 2015-09-07 | 2017-03-16 | 株式会社オプティム | Screen sharing device, screen sharing method, screen sharing system, and program therefor |
CN112394892A (en) * | 2019-08-15 | 2021-02-23 | 北京字节跳动网络技术有限公司 | Screen projection method, screen projection equipment, mobile terminal and storage medium |
CN111601120A (en) * | 2020-04-21 | 2020-08-28 | 江苏大学 | Wireless screen transmission display system and display method |
CN112235626A (en) * | 2020-10-15 | 2021-01-15 | Oppo广东移动通信有限公司 | Video rendering method and device, electronic equipment and storage medium |
CN112423102A (en) * | 2020-11-18 | 2021-02-26 | 努比亚技术有限公司 | Small window screen projection control method and device and computer readable storage medium |
CN112988102A (en) * | 2021-05-11 | 2021-06-18 | 荣耀终端有限公司 | Screen projection method and device |
CN113613072A (en) * | 2021-08-02 | 2021-11-05 | 海信视像科技股份有限公司 | Multi-path screen projection display method and display equipment |
CN114286152A (en) * | 2021-08-02 | 2022-04-05 | 海信视像科技股份有限公司 | Display device, communication terminal and screen projection picture dynamic display method |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116347145A (en) * | 2023-03-27 | 2023-06-27 | 广州南翼信息科技有限公司 | Multi-device screen sharing method, screen throwing device, display device and sharing system |
CN116347145B (en) * | 2023-03-27 | 2023-10-31 | 广州南翼信息科技有限公司 | Multi-device screen sharing method, screen throwing device, display device and sharing system |
CN117156190A (en) * | 2023-04-21 | 2023-12-01 | 荣耀终端有限公司 | Screen projection management method and device |
Also Published As
Publication number | Publication date |
---|---|
CN117837150A (en) | 2024-04-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10896019B2 (en) | Screen sharing system, screen sharing method, and storage medium | |
WO2023011058A1 (en) | Display device, communication terminal, and projected-screen image dynamic display method | |
US20180375987A1 (en) | Method, apparatus and mobile terminal for device control based on a mobile terminal | |
US20240053944A1 (en) | Display apparatus and method for controlling screen projections from multiple devices to same screen | |
US20180077461A1 (en) | Electronic device, interractive mehotd therefor, user terminal and server | |
CN113613072B (en) | Multi-channel screen-throwing display method and display equipment | |
US20120081615A1 (en) | Remote control | |
CN114286165B (en) | Display equipment, mobile terminal, and screen-throwing data transmission method and system | |
CN114286152A (en) | Display device, communication terminal and screen projection picture dynamic display method | |
US20160205427A1 (en) | User terminal apparatus, system, and control method thereof | |
WO2020248714A1 (en) | Data transmission method and device | |
US11662971B2 (en) | Display apparatus and cast method | |
CN118435566A (en) | Display equipment, intelligent home system and multi-screen control method for display equipment | |
KR20200116378A (en) | Apparatus and method for displaying videoes of a plurality of broadcast channels | |
WO2024045770A1 (en) | Content control method and apparatus, storage medium and electronic device | |
CN116567333A (en) | Display equipment and multi-window image quality display method | |
WO2023024630A1 (en) | Display device, terminal device, and content display method | |
CN116980554A (en) | Display equipment and video conference interface display method | |
US10863233B2 (en) | Wireliss docking system for audio-video | |
CN114286320A (en) | Display device, mobile terminal and Bluetooth connection method | |
WO2024021036A1 (en) | Model control method, apparatus, device, system, and computer storage medium | |
CN113992963A (en) | Display device and screen projection method | |
CN115857854A (en) | Display device and screen projection picture switching method | |
CN117809592A (en) | Display device and layer refresh rate setting method | |
CN114968150A (en) | Screen projection control method and terminal equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22851773 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 202280053896.2 Country of ref document: CN |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 22851773 Country of ref document: EP Kind code of ref document: A1 |