WO2023011058A1 - Dispositif d'affichage, terminal de communication et procédé d'affichage dynamique d'image sur un écran projeté - Google Patents

Dispositif d'affichage, terminal de communication et procédé d'affichage dynamique d'image sur un écran projeté Download PDF

Info

Publication number
WO2023011058A1
WO2023011058A1 PCT/CN2022/102439 CN2022102439W WO2023011058A1 WO 2023011058 A1 WO2023011058 A1 WO 2023011058A1 CN 2022102439 W CN2022102439 W CN 2022102439W WO 2023011058 A1 WO2023011058 A1 WO 2023011058A1
Authority
WO
WIPO (PCT)
Prior art keywords
screen
display
projection
display device
screen projection
Prior art date
Application number
PCT/CN2022/102439
Other languages
English (en)
Chinese (zh)
Inventor
宋子全
马晓燕
刘美玉
庞秀娟
张娜
Original Assignee
海信视像科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202110881335.9A external-priority patent/CN113613072B/zh
Priority claimed from CN202110881496.8A external-priority patent/CN114286152A/zh
Priority claimed from CN202111265530.5A external-priority patent/CN113992963A/zh
Application filed by 海信视像科技股份有限公司 filed Critical 海信视像科技股份有限公司
Priority to CN202280053896.2A priority Critical patent/CN117837150A/zh
Publication of WO2023011058A1 publication Critical patent/WO2023011058A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering

Definitions

  • the present application relates to the field of display devices, and in particular to a display device, a communication terminal, and a method for dynamically displaying screen projection images.
  • a display device refers to a terminal device capable of outputting specific display images, which may be terminal devices such as smart TVs, communication terminals, smart advertising screens, and projectors.
  • terminal devices such as smart TVs, communication terminals, smart advertising screens, and projectors.
  • smart TV is based on Internet application technology, has an open operating system and chip, has an open application platform, can realize two-way human-computer interaction, and integrates multiple functions such as audio-visual, entertainment, and data. Products are used to meet the diverse and individual needs of users.
  • An embodiment of the present application provides a display device, including: a display, a communicator, and a controller.
  • the display is configured to display a screen projection interface
  • the communicator is configured to establish a screen projection connection with a communication terminal
  • the controller is configured to execute the following procedural steps: detecting the number of connections of a communication terminal; according to the The number of connections plans the layout of the projection screen to obtain the display parameters of the projection screen; a notification message is sent to the communication terminal corresponding to the projection screen, the notification message includes the display parameters, and the notification message uses The method is to make the communication terminal generate screen projection data according to the display parameters; receive the screen projection data fed back by the communication terminal, and display the screen projection data according to the layout.
  • An embodiment of the present application provides a display device, including:
  • the display is used to display the screen projection interface; the communicator is used to connect with the terminal device for screen projection; the controller is configured to execute: receiving the screen projection request of at least M terminal devices, and receiving the media configuration respectively from the M terminal devices Information, matching the layout between M screen projection windows adapted to the screen; wherein, M ⁇ 1, the media configuration information includes the aspect ratio of the video projected by the terminal device; the control display follows the layout in the screen projection interface Display M screen projection windows, and display the screen projection data sent by the corresponding terminal device in each screen projection window.
  • An embodiment of the present application provides a display device, including: a display configured to display a user interface; a controller capable of running a playback application and at least one screen projection protocol application, and the at least one screen projection protocol application can be connected to the
  • the playback application is configured to: when receiving a first screen projection request sent by a first mobile terminal, establish a connection with the first mobile terminal through a first screen projection protocol application corresponding to the first screen projection request.
  • FIG. 1 is a schematic diagram of an operation scene between a display device and a control device in an embodiment of the present application
  • FIG. 2 is a block diagram of a hardware configuration of a display device in an embodiment of the present application
  • FIG. 3 is a schematic diagram of display device software configuration in the embodiment of the present application.
  • FIG. 4 is a schematic diagram of displaying an icon control interface of a display device application program in an embodiment of the present application
  • FIG. 6 is a schematic diagram of a software architecture of a communication terminal according to some embodiments.
  • FIG. 7 is a schematic diagram of a user interface of a communication terminal according to some embodiments.
  • Fig. 8 is a schematic diagram of screen projection effects according to some embodiments.
  • Fig. 11 is a schematic diagram of the effect of dividing two display areas according to some embodiments.
  • Fig. 12 is a schematic flowchart of a method for dynamically displaying a projected screen according to some embodiments
  • Fig. 13 is a schematic diagram of the effect of dividing three display areas according to some embodiments.
  • connection instruction 14 is a schematic flow diagram of detecting the number of connections in response to a connection instruction according to some embodiments
  • 15 is a schematic flow diagram of detecting the number of connections in response to an exit command according to some embodiments.
  • Figure 16 is a schematic flow diagram of matching layout templates according to some embodiments.
  • Fig. 18 is a schematic flow diagram of a communication terminal generating screen projection data according to some embodiments.
  • Fig. 19 is a flowchart of a multi-channel projection display method according to some embodiments.
  • Fig. 20(a) is a schematic layout diagram of a scene where the number of projection screens is 1 and the video stream is a single vertical screen according to some embodiments;
  • Fig. 20(b) is a schematic layout diagram of a scene where the number of projection screens is 1 and the video stream is a single horizontal screen according to some embodiments;
  • Fig. 21(a) is a schematic diagram of the first layout in a scene where the number of projection screens is 2 and double horizontal screens according to some embodiments;
  • Fig. 21(b) is a schematic diagram of a second layout in a scene where the number of screen projections is 2 and double horizontal screens according to some embodiments;
  • Fig. 21(d) is a schematic diagram of a second layout in a scenario where the number of projection screens is 2 and one horizontal screen and one vertical screen according to some embodiments;
  • Fig. 21(e) is a schematic layout diagram in a scene where the number of projection screens is 2 and dual vertical screens according to some embodiments;
  • Fig. 22(a) is a schematic diagram of version 1 of the layout in the scenario where the number of projection screens is 2 and the dual landscape screens (with different aspect ratios) according to some embodiments;
  • Fig. 22(b) is a schematic diagram of version 2 of the layout in the scenario where the number of projection screens is 2 and the dual landscape screens (with different aspect ratios) according to some embodiments;
  • Fig. 22(c) is a schematic diagram of version 1 of the layout in a scenario where the number of projection screens is 2 and the dual vertical screens (with different aspect ratios) according to some embodiments;
  • Fig. 22(d) is a schematic diagram of version 2 of the layout in the scenario where the number of projection screens is 2 and the dual vertical screens (with different aspect ratios) according to some embodiments;
  • Fig. 23(a) is a schematic diagram of version 1 of the layout in a scenario where the number of projection screens is 3 and there is one horizontal screen and two vertical screens according to some embodiments;
  • Fig. 23(c) is a schematic diagram of version 3 of the layout in a scenario where the number of projection screens is 3 and there is one horizontal screen and two vertical screens according to some embodiments;
  • FIG. 25 is a display diagram of an icon control interface of an application program in the display device 200 according to some embodiments.
  • Fig. 26 is a configuration diagram of a controller in a display device according to some embodiments.
  • Fig. 27 is a schematic flowchart of a screen projection method performed by a controller in a display device according to some embodiments
  • Fig. 29 is a schematic diagram of a playback window on a display in a display device according to some embodiments.
  • Fig. 30 is a schematic structural diagram of screen projection interaction between multiple mobile terminals and a display device according to some embodiments.
  • Fig. 31 is a sequence diagram of screen projection interaction between a first mobile terminal and a display device according to some embodiments.
  • control device 100 may be a remote control, and the communication between the remote control and the display device includes infrared protocol communication, bluetooth protocol communication, and other short-distance communication methods, and the display device 200 is controlled wirelessly or wiredly.
  • the user can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, and the like.
  • Fig. 2 is a configuration block diagram of the control device 100 according to some embodiments.
  • the control device 100 includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply.
  • the control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, and acts as an intermediary for the interaction between the user and the display device 200 .
  • FIG. 3 shows a block diagram of a hardware configuration of a display device 200 according to an exemplary embodiment.
  • the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface. at least one.
  • the controller includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, a first interface to an nth interface for input/output.
  • the user input interface can be used to receive a control signal of the control device 100 (such as an infrared remote controller, etc.).
  • the external device interface 240 may include, but is not limited to, the following: high-definition multimedia interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), Any one or more interfaces such as RGB ports.
  • HDMI high-definition multimedia interface
  • component analog or data high-definition component input interface
  • CVBS composite video input interface
  • USB USB input interface
  • the system of the display device may include a kernel (Kernel), a command parser (shell), a file system and application programs.
  • Kernel kernel
  • shell command parser
  • file system file system
  • application programs application programs
  • the system is divided into four layers, from top to bottom are applications (Applications) layer (application layer), application framework (Application Framework) layer (framework layer), Android runtime (Android runtime) and system library layer (system runtime library layer), as well as the kernel layer.
  • applications Applications
  • Application Framework application framework
  • Android runtime Android runtime
  • system library layer system runtime library layer
  • the embodiment will be specifically described below by taking the communication terminal 300 as an example. It should be understood that the communication terminal 300 shown in FIG. 5 is only an example, and that the communication terminal 300 may have more or fewer components than those shown in FIG. 5, two or more components may be combined, or Different component configurations are possible.
  • the various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.
  • the communication terminal 300 includes: a radio frequency (radio frequency, RF) circuit 310, a memory 320, a display unit 330, a camera 340, a sensor 350, an audio circuit 360, and a wireless fidelity (Wireless Fidelity, Wi-Fi) circuit 370, processor 380, Bluetooth circuit 381, and power supply 390 and other components.
  • RF radio frequency
  • FIG. 6 is a block diagram of the software structure of the communication terminal 300 according to the embodiment of the present invention.
  • the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces.
  • the Android system is divided into four layers, which are respectively the application program layer, the application program framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer from top to bottom.
  • a corresponding hardware interrupt is sent to the kernel layer.
  • the kernel layer processes touch operations into original input events (including touch coordinates, time stamps of touch operations, and other information). Raw input events are stored at the kernel level.
  • the application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Take the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon as an example.
  • the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer.
  • Camera 340 captures still images or video.
  • the display device 200 can play and display the content in the communication terminal 300 , as shown in FIG. 8 .
  • the user may send a screen projection connection request through the communication terminal 300 , and send the screen projection connection request to the display device 200 through the WiFi network.
  • the display device 200 completes the transmission protocol configuration according to the screen projection connection request, so as to establish a screen projection data transmission channel with the communication terminal 300 .
  • the display device 200 can also realize the screen projection connection with the communication terminal 300 through different connection methods. For example, when the display device 200 and the communication terminal 300 are connected to the same wireless local area network, a screen projection connection can be established based on the WiFi network. For another example, when both the display device 200 and the communication terminal 300 are provided with NFC (Near Field Communication, near field communication) components, the screen projection connection relationship can be established through the NFC components. Obviously, other wired or wireless connection methods can also be used between the display device 200 and the communication terminal 300 to establish a screen projection connection relationship, such as RF radio frequency connection, infrared connection, cellular network, and connections provided by those skilled in the art in the above embodiments Other connection methods that can be thought of on the basis of the method.
  • NFC Near Field Communication, near field communication
  • the display device 200 may receive relevant data of the communication terminal 300 through the screen projection data channel, so as to play relevant content on the display device 200 .
  • the screen projection data acquired by the display device 200 has different forms.
  • the display device 200 can obtain the link address of a certain resource in the communication terminal 300 through the screen projection data channel, and obtain media asset content data by accessing the link address , so as to play the media asset data.
  • the display device 200 can also obtain control instructions sent by the communication terminal 300 through the screen projection data channel when playing media asset data, such as fast forward, pause, and stop projection.
  • the communication terminal 300 such as a mobile phone and the display device 200 will negotiate device functions and network conditions to select a suitable video and audio transmission format.
  • a series of real-time streaming protocol (RTSP) control commands are used to control the playback and termination of the video stream.
  • RTSP real-time streaming protocol
  • the communication terminal 300 can be used as a signal source of the display device 200 , and the display device 200 can continuously obtain video data through the projection data channel, and display it on the display 275 in real time, so as to realize the synchronous display effect with the communication terminal 300 .
  • Miracast screen projection is applicable to Android devices, that is, Android devices can share video images through Miracast screen projection.
  • Android devices can share video images through Miracast screen projection.
  • smart devices above Android 5.0 have built-in Miracast protocol, so they can directly perform wireless screen projection through the WiFi P2P function between the screen projection device and the WiFi network card of the receiver device without installing any software.
  • Miracast also supports simultaneous transmission of audio and video.
  • Miracast is not fully compatible with all Android devices, and Miracast-based screencasting does not have a screencasting code function, which cannot prevent malicious or accidental screencasting.
  • Airplay screen projection is a wireless playback technology introduced by Apple, which is suitable for wireless screen projection of iOS and Mac systems.
  • Apple series devices have a built-in Airplay screen projection function, which can be used with a screen projector that supports AirPlay server service, and the effect is clear and stable.
  • the protocol comes with a screen projection code function.
  • it is necessary to ensure that the Apple device and the screen projector are used in the same LAN (same network segment), and cannot be used across network segments or VLANs.
  • WiDi screen projection is a wireless playback technology led by Intel, and it is also based on WiFi Direct direct connection technology. Windows 8/10 laptops natively support WiDi screen projection, which can be directly screen projection without installing any software, and supports USB reverse control function.
  • the display device 200 can The displayed user interface is divided into areas, such as bisecting, trisecting, and quartering. After the multiple display areas are obtained through area division, the display device 200 displays the projected screen image in each divided area.
  • the communication terminal 300 may be placed in different states according to different ways of use, for example, for a communication terminal 300 such as a smart phone, it may be in a portrait state or a landscape state during use.
  • the vertical screen state refers to a state in which the width of the displayed picture is smaller than the height
  • the horizontal screen state refers to the state in which the width of the displayed picture is greater than the height.
  • the communication terminal 300 may present different screen layout modes. Based on this, when the display device 200 displays the projected screen images of multiple communication terminals 300 by area division, each display area will present different screen ratios due to the different placement states of the communication terminals 300 .
  • the display device 200 when the placement states of the multiple communication terminals 300 are different, the display device 200 usually stretches and fills each screen projection image to fill the display area. This display mode will cause a large deformation of the projected screen, which will affect the user's viewing experience.
  • the display device 200 capable of applying the method for dynamically displaying the projected screen includes: a display 260 , a communicator 220 and a controller 250 .
  • the display 260 is configured to display a screen projection interface.
  • the communicator 220 is configured to establish a screen projection connection with the communication terminal 300 .
  • the controller 250 is configured to execute the above method for dynamically displaying the projection screen, as shown in FIG. 12 , which specifically includes the following content:
  • the number of connections can also be obtained by detecting the data table maintained by the display device 200 in real time.
  • the display device 200 may perform detection of the number of communication terminals 300 when judging that the layout of the current screen projection interface needs to be changed. For example, the display device 200 can monitor the access status of each communication terminal 300 in the screen projection connection in real time, and when it detects that any communication terminal 300 is newly connected or exits the connection, the display device 200 can perform a connection detection function of the communication terminal 300. number of steps.
  • the display device 200 may also periodically detect the number of connections of the communication terminal 300 .
  • the display device 200 may detect the number of connections of the communication terminals 300 through heartbeat commands of each communication terminal 300 after the screen projection connection is established. By detecting the number of connections of the communication terminal 300, the display device 200 can detect the connection status of the communication terminal 300 for screen projection in time, so that when the communication terminal is offline due to network range or other hardware failures, the screen projection interface can be adjusted. .
  • the display device 200 may also detect the number of connections of the communication terminal 300 by detecting the user's control instructions and responding to some of the control instructions. For example, when the display device 200 detects that the user has input a control command for adjusting the page layout, the display device 200 may perform the step of detecting the number of connections of the communication terminal 300, so as to trigger the display device 200 to subsequently acquire the projection data. way of adjustment.
  • the built-in layout templates may also be different.
  • the layout mode template can be a layout template obtained by dividing equidistantly in the vertical direction, that is, on the display device 200.
  • the screen is divided into three vertical areas whose width is smaller than the height; and for the screencasting connection for sharing the game interface operation, since the game interface is mostly in the horizontal screen state, the layout template can be obtained by dividing equidistantly in the horizontal direction. That is, the screen of the display device 200 is divided into three horizontal regions whose width is greater than the height, for displaying the game screen shared by the mobile phone.
  • the display device 200 may plan the screen layout according to the principle of maximizing display. For example, when displaying 3 projection screen areas, one vertical area and two horizontal areas may be maintained. In this way, each display area can maintain a shape of 16:9 or 9:16, so as to adapt to the aspect ratio of communication terminal 300 .
  • the display device 200 may add a display window to the user interface to plan a layout based on the number of connections. For example, when it is detected that the number of communication terminals 300 with which the current display device 200 has established a screen projection connection is 2, two display windows may be added to the user interface for respectively displaying the screen projection images shared by each communication terminal 300 . Each display window may have a fixed shape and ratio, and the display device 200 may be required to provide a display image according to the ratio.
  • the multiple display windows when multiple display windows are used to display multiple screen projection images, if the number of communication terminals 300 that establish screen projection connections with the display device 200 is large, the multiple display windows can also be kept at a specific hierarchical relationship.
  • the hierarchical relationship of the display windows may be set according to the order in which the screen projection connection is established with the display device 200 , that is, the screen projection screen shared by the communication terminal 300 that establishes the screen projection connection last is displayed on the top layer.
  • the display device 200 may also determine display parameters of the projection screen according to the planned layout.
  • the display parameters are used to indicate the screen ratio and size of each display area in the layout mode planned by the display device 200, and the projection data requirements calculated according to the ratio and size of each display area, such as code rate, resolution etc.
  • the display device 200 may send a notification message to the communication terminal 300 corresponding to the screen projection screen.
  • the notification message includes a display parameter
  • the notification message is used to enable the communication terminal 300 to generate screen projection data according to the display parameter.
  • the display device 200 when the display device 200 detects that the current number of connections to the communication terminal 300 is three, it can determine the display areas of the three projection screens by planning the layout, that is, the first area is the resolution “1080 ⁇ 1920” vertical area; area two and area three are horizontal areas with a resolution of “1280 ⁇ 720”. Based on the above display parameters, the display device 200 can generate a notification message, and send the generated notification message to the communication terminal 300, so that the communication terminal 300 can generate screen projection data according to the above resolution.
  • the communication terminal 300 may have a built-in screen recording program, and after receiving the notification message sent by the display device 200, start the screen recording program to analyze the notification message and extract the display parameters. Then set the screen parameters in the screen recording program according to the display parameters. For example, when the display parameter is vertical and 1080 ⁇ 1920, the communication terminal 300 can control the screen recording process of the screen recording program so that the resolution of the generated screen recording video image is 1080 ⁇ 1920.
  • the communication terminal 300 can process the screen by padding or cropping so that the generated screen projection data conforms to the resolution set in the notification message. Displays the form specified in the argument. For example, the screen ratio of each display area set in the display device 200 is 9:16, but when the screen ratio of the communication terminal 300 is 9:18, in order to be able to display completely, the communication terminal 300 can perform a reduction process on the display screen, so that The projection screen can be fully displayed in height. At this time, in the screen corresponding to the generated projection data, black areas will be filled on both sides to maintain the overall screen ratio of 9:16.
  • the communication terminal 300 After the communication terminal 300 generates the projection data, it can feed back the generated projection data to the display device 200, and the display device 200 can display the received projection data, that is, display the images in the projection data according to the layout.
  • the display device 200 respectively sends notification messages containing display parameters "vertical, 1080 ⁇ 1920", “horizontal, 1280 ⁇ 720", and “horizontal, 1280 ⁇ 720" to three communication terminals
  • the three communication terminals 300 can respectively feed back three projection data streams with resolutions of 1080 ⁇ 1920, 1280 ⁇ 720, and 1280 ⁇ 720
  • display device 200 then parses and renders the received three projection data streams to finally form Cast screen.
  • the display device 200 may create a data buffer area for each communication terminal 300 when receiving multiple screen projection data streams for Store the screen projection data sent by them, and each display control on the display device 200 obtains video data from the corresponding data buffer area for display.
  • the display device 200 can also correspond to it. The data cache area is destroyed and released.
  • the method for dynamically displaying the projected screen provided by the above embodiment can plan the layout of multiple projected screens in the screen projecting interface by detecting the number of connections of the communication terminal 300, and send a message containing the display parameters of each area to each communication terminal 300
  • the notification message enables the communication terminal 300 to generate screen projection data in a manner matching the layout manner.
  • Such a display mode can ensure that the picture presented by the final display device 200 can adapt to the layout requirements, alleviate the deformation of the picture, and improve the viewing experience of the user.
  • the display device 200 may start detecting the number of connections of the communication terminal 300 when the communication terminal 300 is connected. That is, as shown in FIG. 14 , the display device 200 may acquire a connection instruction input by the user for establishing a screen projection connection, and initiate a screen projection connection to the communication terminal 300 in response to the connection instruction.
  • the connection instruction can be input in different ways.
  • the communication terminal 300 that establishes a screen projection connection with the display device 200 for the first time.
  • the user can first connect to the wireless network where the display device 200 is located through network settings, and then perform interactive operations on the communication terminal 300, such as clicking the "screen projection" button to initiate a screen projection connection.
  • the display device 200 can display the screen projection connection pairing interface after receiving the screen projection connection.
  • the user needs to perform interactive operations again based on the screen projection connection pairing interface, such as selecting the corresponding communication in the screen projection device list.
  • the terminal 300 enables the display device 200 and the communication terminal 300 to establish a screen projection connection. It can be seen that, for the communication terminal 300 connecting for the first time, the connection instruction is a combination of a series of interactive actions based on the display device 200 and the communication terminal 300 .
  • the user can directly click "Screencast" on the communication terminal 300
  • the button initiates a screen projection operation.
  • the display device 200 can also receive a screen projection connection instruction, and automatically responds to the connection instruction to implement a mutual handshake with the communication terminal 300 to complete the screen projection connection.
  • the display device 200 may also traverse the number of connections of the communication terminal 300 . For example, after a communication terminal 300 initiates to establish a screen projection connection, the display device 200 may first wait for the configuration process of the screen projection connection. After the configuration of the screen projection connection is completed, the total number of communication terminals 300 currently connected is traversed. That is, the detected number of connections of the communication terminals 300 includes the communication terminals 300 that have newly established a screen projection connection.
  • the display device 200 may also detect the number of connections when it detects that the communication terminal 300 stops the screen projection connection. As shown in FIG. , obtain the exit command input by the user for disconnecting the screen projection connection, and in response to the exit instruction, disconnect the screen projection connection of the communication terminal 300, and traverse the connections of the remaining communication terminals 300 after disconnecting the screen projection connection quantity.
  • the user can click the "stop screen projection" button on one of the communication terminals 300 to control the communication terminal 300 to no longer execute the screen recording program.
  • the display device 200 may automatically disconnect the screen projection connection established with the communication terminal 300 when detecting that the communication terminal 300 no longer sends screen projection data. And after the screen projection connection is disconnected, it is detected that the number of connections of the remaining communication terminals 300 is 2. Therefore, according to the usage status of the two communication terminals 300, the layout mode in the screen projection interface is re-planned.
  • the above method for detecting the number of connections can automatically adapt to the layout of the screen on the display device 200, adjust the displayed content in time, and improve user experience.
  • the display device 200 can also establish a screen projection data channel when initiating a screen projection connection to the communication terminal 300, and the screen projection data channel can be established successfully After that, create an indicator to display the projection data received through the projection data channel through the indicator.
  • the display control is a display control program built in the display device 200, such as surface View or Texture View. Indicators can be added to display in different ways. For example, when the display device 200 includes one display control, the display control may be displayed in a full-screen manner. When the display device 200 includes two display controls, the two display controls are in the form of two vertically arranged windows on the left and right.
  • the display device 200 in order to determine the display parameters, can first detect the screen size information of the display 260 when planning the layout of the projected screen according to the number of connections, and then use the number of connections and screen
  • the size information is an index to match the layout template in the database, so as to extract the display parameters in the layout template.
  • a database may be operated and maintained, and layout templates corresponding to various screen sizes and connection quantities may be stored in the database. Therefore, the display device 200 can determine the best layout mode suitable for the current display state according to the screen size information and the number of connections.
  • the display device 200 may also determine the final layout mode according to the matching result of the layout template. That is, in the step of matching the layout template in the database, if no layout template is matched in the database, the layout method can be determined directly by using the matched layout template; and if no layout template is matched in the database, the display area can be Perform zoning to determine the appropriate layout. That is, the display device 200 can acquire the device parameters of the communication terminal 300, divide the display area into sub-regions with the same number of connections according to the device parameters, and then traverse the graphic parameters of each sub-region to generate display parameters.
  • the display device 200 may also extract display parameters and generate a notification message according to the display parameters in the step of sending a notification message to the communication terminal 300 corresponding to the projected screen.
  • the display parameters include the display direction of each display window.
  • the display direction is used to instruct the communication terminal to generate screen projection data according to a set direction.
  • the display device 200 may preferentially match the corresponding communication terminal 300 according to the shape of each display area after determining the layout mode. For example, when the display direction of the display window is vertical, the projection screen of the communication terminal 300 in the vertical screen state can be preferentially displayed in this window; when the display direction of the display window is horizontal, it can be preferentially displayed in this window. A projected screen image of the communication terminal 300 in a horizontal screen state.
  • the display device 200 can also determine the image quality of the projection data according to the number of connections. That is, the display parameters also include the code rate and resolution of each display window, so that the communication terminal 300 can generate screen projection data with reference to the set code rate and resolution.
  • the code rate and resolution can be set to be smaller, so that the transmission data of the projection data The smaller the amount, and vice versa.
  • the transmission code rate can be set to 5Mb/s; if the display device 200 is connected to 3 communication terminals 300 or 4 communication terminals 300 at the same time, it can be set The transmission code rate is 3Mb/s.
  • the resolution may be the same as the number of pixels occupied by each display area in the width and height directions after the display area is divided. For example, when the number of pixels occupied by the first display area is 1080 ⁇ 1920, the resolution of the corresponding projection data may also be 1080 ⁇ 1920. When the number of pixels occupied by the second display region and the third display region is 1280 ⁇ 720, the resolution of the corresponding projection data may also be 1280 ⁇ 720.
  • the display device 200 may perform the following steps:
  • Step S171 detecting the effective frame range of each screen-casting screen in the screen-casting interface
  • Step S172 calculating the display ratio of the effective picture area to the total area of the display window
  • Step S174 sending a prompt message to the communication terminal 300 to prompt the user to rotate the direction manually or automatically through the internal mechanism of the communication terminal 300;
  • the display device 200 can detect that the display ratio is less than or equal to the ratio threshold, so it can send a prompt message to the communication terminal 300, such as "the current display The orientation does not match, please turn the mobile phone to the vertical screen state", prompting the user to adjust the communication terminal 300 to the tree screen state, and increase the display ratio of the effective screen in the projection data.
  • a communication terminal 300 is also provided in some embodiments of the present application, including: a display unit 330 , a communication unit, and a processor 380 .
  • the display unit 330 is configured to display a terminal interface
  • the communication unit is configured to establish a screen projection connection with the display device 200; as shown in FIG. 18 , the processor 380 is configured to perform the following steps:
  • Step 181 Receive a notification message sent by the display device 200, the notification message includes the display parameters of the screen projection screen, and the display parameters are when the display device 200 plans the layout mode of the screen projection screen according to the current number of communication terminals 300 connections get;
  • Step 183 feeding back the screen projection data to the display device 200 .
  • some embodiments of the present application also provide a method for dynamically displaying a projection screen, including the following steps:
  • the display device 200 detects the number of connections of the communication terminal 300, and plans the layout of the projected screen according to the number of connections, so as to obtain the display parameters of the projected screen;
  • the display device 200 sends a notification message to the communication terminal 300 corresponding to the projection screen, and the notification message includes the display parameter;
  • the communication terminal 300 receives the notification message sent by the display device 200, and generates screen projection data according to the display parameters;
  • the communication terminal 300 feeds back the screen projection data to the display device 200;
  • the display device 200 displays the screen projection data according to the layout.
  • the method for dynamically displaying the projected screen provided by the above embodiment can plan the layout of the projected screen by detecting the number of connected communication terminals 300 when the user triggers a change in the number of communication terminals 300 connected to the display device 200 . Then send a notification message including the display parameters of the screen projection to the communication terminal 300 , so that the communication terminal 300 can generate screen projection data conforming to the layout according to the display parameters, and display it on the display device 200 .
  • the method can enable the display device 200 to support multi-channel screen projection, dynamically adjust the display layout of the screen projection screen, and notify the communication terminal 300 to adjust the screen recording, so as to make the layout of the screen projection screen displayed more reasonable.
  • the terminal device can push media data to the display device.
  • the user can project the video of interest to the display device for large-screen playback through some application programs with screen projection function; or the terminal device and the display device Establish a screen projection connection through certain protocols, such as DLNA (Digital Living Network Alliance, Digital Living Network Alliance) protocol, etc.
  • DLNA Digital Living Network Alliance, Digital Living Network Alliance
  • the terminal device can project the screen projection data to the display device, and the display device will display the projection data on the screen projection interface.
  • the terminal device may be a mobile terminal such as a smart phone, a tablet computer, or a notebook computer, or may be a desktop computer or other small-screen display devices.
  • the user when a terminal device is playing media data, the user can select a device from the device list for playing the media data on the screen, and the display device that supports the screen projection function will be identified and automatically added to the device list
  • the media data can be projected from the terminal device to the display device for playback.
  • the terminal device and the display device can be connected to the same Wifi network, and the method for the terminal to push media data to the display device is not limited.
  • the screen projection method can also be performed across networks. In this application, the specific implementation manner of screen projection is not limited.
  • the terminal device A sends a screen projection request to the display device, and the display device can ask the user whether to agree to the screen projection of the terminal device A through a UI inquiry pop-up window, etc., and can provide several option controls, such as "this consent”, “disagree” and “permanent consent”.
  • the user chooses "Permanent Agree", which is equivalent to permanently authorizing terminal device A.
  • the display device can maintain a list of trusted devices, and supports users to add, delete and modify device information with screen projection permission in the list of trusted devices.
  • Device information includes but not limited to device ID, MAC Address, etc., that is, the user can customize the authorization list on the display device side.
  • the display device receives the screen projection request sent by the terminal device B, it queries whether the terminal device B is included in the trusted device list according to the device information carried in the screen projection request. If terminal device B is included in the list of trusted devices, a screen projection connection between terminal device B and the display device is established; if terminal device B is not included in the list of trusted devices, the display device does not respond to the screen projection request, that is, no connection with The screen projection connection of terminal device B.
  • the authorization and connection mode of screen projection by the terminal device are not limited to the examples in this application.
  • each screen projection window corresponds to a terminal device that is successfully connected to the screen projection, and displays the screen projection data sent by the terminal device.
  • the display effect of multiple screen projection windows on the screen projection interface affects the user's perception, and there are many display problems: for example, the arrangement of multiple screen projection windows is not compact enough, resulting in a lot of blank space in the screen projection interface; The distribution of the screen projection windows is chaotic, and the overall layout is not regular; some screen projection windows are small in size, so that users cannot clearly see the details of the video in the window; If it is changed to a vertical screen display, the video display uniformity between the terminal device and the display device is poor, and so on.
  • the present application aims to plan and lay out multiple screen projection windows to improve the display effect of the screen projection interface.
  • an embodiment of the present application provides a multi-channel projection display method, the method is configured to be executed by the controller 250 in the display device, that is, the controller 250 is the execution subject of the method, and the method includes the following steps :
  • Step S01 according to the currently counted number of M screen projection channels and the media configuration information respectively received from the M terminal devices, match the layout among the M screen projection windows suitable for the screen;
  • M is used to indicate the number of screen projection channels, that is, the number of terminal devices that have established screen projection connections with the display device.
  • the number of screen projection channels will change dynamically with the connection and disconnection of screen projection connections.
  • the numbers are counted in order to accurately plan the typesetting and layout of the screencasting interface.
  • M is 1, single-channel projection is performed; when M is greater than or equal to 2, multiple-channel projection is performed; M can be zero, indicating that the display device has not received the projection data sent by the connected device, and the projection will not be displayed
  • the interface is also equivalent to an empty layout on the screen projection interface.
  • the media configuration information includes the aspect ratio of the video projected by the terminal device, that is, the ratio of the horizontal width to the vertical height when displaying the video.
  • the aspect ratio can be used to identify the display orientation.
  • the aspect ratio is greater than When or is equal to 1, that is, the width is greater than or equal to the height, the display mode is displayed in landscape mode; when the aspect ratio is less than 1, that is, the width is smaller than the height, the display mode is displayed in portrait mode.
  • the creation of the projection window can be constrained according to the aspect ratio and screen orientation indicated in the media configuration information, so that the projection window and the corresponding terminal device can maintain the same aspect ratio and display orientation, which can avoid media correlation after projection. Misconfiguration problem.
  • the media configuration information may also include a screen aspect ratio of the terminal device.
  • the aspect ratio is greater than or equal to 1, that is, the width is greater than or equal to the height, the display mode is in landscape mode; when the aspect ratio is less than 1, that is, the width is smaller than the height, the display mode is in portrait mode.
  • the media configuration information may further include a media category.
  • the display device After the display device reads the media configuration information, it may know the media category to be screen-casted, so as to display screen-casting windows with different effects according to the media category.
  • the projection window can display visualized projection data, and the projection window should match the aspect ratio and screen orientation of the video/image; for text-based media, the projection window can Visually display readable text information, but the aspect ratio and screen orientation of the projection window are not limited; for audio-only media, the projection window cannot display visual images, and voice playback controls can be set in the projection window (such as small Speaker), when playing the audio stream of the projection screen, the voice playback control can be presented as a dynamic special effect of "playing".
  • the aspect ratio and screen orientation of the projection window corresponding to the pure audio stream are not limited.
  • the sequence numbers may be allocated according to the time sequence of the screen projection connection, for example, the earlier the screen projection is connected, the smaller the sequence number.
  • the screen projection screen is pre-typeset at the moment of screen projection connection, and after receiving the screen projection data, the screen is displayed according to the pre-typesetting method.
  • the media configuration information of the display device may be to record video at a fixed resolution, such as 1920*1080, 1280*720, and so on.
  • the media configuration information of the display device may record video according to the ratio of the screen, and the ratio of the current screen is not the ratio of 16:9/9:16 of the large-screen terminal, but other ratios such as 17:9, 21:16, etc.
  • M screen projection windows are constructed according to M sets of size information, and the screen projection windows and their corresponding terminal devices maintain the same aspect ratio and display orientation. Since each set of size information defines a specific width and height of a screen projection window, a screen projection window matching the size information can be constructed. After the sizes of the M screen projection windows are all determined, it is also necessary to plan a specific display position of each screen projection window on the screen interface.
  • the position distribution information is calculated according to the media configuration information received from the terminal device, the position distribution information includes the coordinates (xi, yi) of a preset reference point Pointi on the M screen projection windows, and the preset It is assumed that the coordinates (xi, yi) of the reference point Pointi are used to locate the position of the screen projection window on the screen. Since the screen projection window is generally a rectangle, some key points in the rectangle can be selected by the preset reference point. Optionally, the preset reference point is any one of the four vertices of the rectangle or the center point of the rectangle, etc. The selection principle is to Subject to calculation and positioning. Finally, according to the calculated location distribution information, the created M screen projection windows are positioned and arranged, so that the sizes and positions of the M screen projection windows are determined and the layout is formed.
  • Figure 20(a) exemplarily shows the screen projection interface in the scenario where the number of screen projection channels is 1 and the video stream is a single vertical screen.
  • the screen resolution of the display device is 1920*1080 (that is, the default aspect ratio of the display device screen is 16:9), and the media configuration file sent by terminal device A indicates that the aspect ratio is 9:16 (vertical screen) as an example, then
  • Figure 20(b) exemplarily shows the screen projection interface in the scenario where the number of screen projection channels is 1 and the video stream is a single horizontal screen.
  • the screen resolution of the display device is 1920*1080 (that is, the default aspect ratio of the display device screen is 16:9), and the media configuration file sent by terminal device A indicates that the aspect ratio is 16:9 (horizontal screen), for example, then
  • the projection window 1 to be created is also a horizontal screen.
  • the upper left corner of the screen is used as the default reference point Point1
  • the upper left corner of the screen projection window 2 is used as the preset reference point Point2.
  • the layout after matching and generating the layout, during the process of displaying multiple screen projection windows according to the layout on the screen projection interface and loading and displaying the screen projection data of each channel, there may be new screen projection channels accessed at any time, or there may be At least one terminal device is disconnected from the screen projection connection, that is, the number of screen projection channels may change during this period, and the layout needs to be updated according to the changed screen projection channels to realize adaptive adjustment of the layout following the screen projection channels.
  • the adjustment involves the number of screen projection windows in the layout, the size of each screen projection window, and the relative position distribution of each screen projection window.
  • the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 16:9 (horizontal screen), the controller 250 detects that the number of projection screens is changed from 1 to 2, then the layout can be updated to the version shown in Figure 21(a) or Figure 21(b); for example, in Figure 21(a) or Figure 21(b)
  • terminal device B suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1, and can update the layout to the version shown in Figure 20(b).
  • Fig. 21(c) and Fig. 21(d) are two kinds of screen-casting interfaces in the scene where the number of screen-casting channels is 2 and one horizontal screen and one vertical screen according to some embodiments, and Fig. 21(c) shows the horizontal screen The screen projection window is displayed on the left, and the screen projection window for the vertical screen is displayed on the right; Figure 21(d) shows the screen projection window for the vertical screen on the left, and the screen projection window for the landscape screen is displayed on the right.
  • the display device has connected to the two-way screen projection initiated by terminal device A and terminal device B.
  • the screen resolution of the display device is 1920*1080, and the media configuration file sent by terminal device A indicates that the aspect ratio is 16:9 ( Horizontal screen), the media configuration file sent by terminal device B indicates that the aspect ratio is 9:16 (vertical screen) as an example, then the created projection window 1 (corresponding to terminal device A) is a horizontal Corresponding to terminal device B) is a vertical screen.
  • terminal device B suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1 , the layout can be updated to the version of the single horizontal screen window (screen projection window 1) full-screen exclusive screen shown in FIG. 20(b).
  • terminal device A suddenly disconnects the screen projection connection with the display device, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1 , the layout can be updated to a version similar to that shown in Figure 20(a) in which the single vertical screen window (screen projection window 2) monopolizes the screen.
  • the upper left corner of the screen projection window 1 is used as the preset reference point Point1
  • the upper left corner of the screen projection window 2 is used as the preset reference point Point2. Considering that the two vertical screen windows are evenly spaced along the horizontal direction, the viewing effect is better.
  • the display device establishes a screen projection connection with terminal device B, and the media configuration information sent by terminal device B indicates that the aspect ratio is 9: 16 (vertical screen), the controller 250 detects that the number of projection screens has been changed from 1 to 2, and the layout can be updated to the version with two channels and two vertical screens as shown in FIG. 21(e).
  • terminal device A or terminal device B disconnects the screen projection connection, and the controller 250 detects that the number of screen projection channels has changed from 2 to 1, then it can The layout is updated to a version similar to that shown in Figure 20(a) where a single vertical window occupies the screen.
  • the controller detects that When the number of screen projection channels becomes 0, the screen projection interface will be exited, and the UI display interface will return to that before screen projection.
  • the adaptability of the layout matching mechanism can refer to the previous example.
  • the layout matching can be It has the characteristics of adaptability, flexibility and diversity, etc., and this application will not list and repeat them one by one.
  • the layout forms of each screen projection window are different, resulting in a variety of layouts. 21(a) and 21(b)
  • the aforementioned two-way-horizontal-vertical screen scenario includes at least the two versions of FIG. 21(c) and 21(d).
  • the maximum area sum Smax is calculated, and then the version corresponding to Smax is used as the selected target layout, and the target layout is displayed on the screen projection interface, so that each projection window can occupy the screen more fully and to the maximum extent, and each projection screen
  • the layout between the windows is compact, and the blank area is reduced as much as possible to improve the display effect of the screen projection interface. Since the screen projection window is rectangular, the area of a single screen projection window is equal to the product of its width and height.
  • the screen resolution of the display device is 1920*1080
  • the media configuration file sent by terminal device A indicates that the aspect ratio is 17:9 (horizontal screen)
  • terminal device B sends
  • the media configuration file indicated that the ratio of width to height is 16:9 (horizontal screen), that is, the media of the two screens have different ratios of width to height
  • two screen projection windows are constructed.
  • Screen projection window 1 corresponds to terminal device A.
  • Screen projection window 2 corresponds to terminal device B.
  • version 1 of the layout is arranged side by side in a horizontal direction.
  • windowWidth1*windowHeight1 960 *508
  • Figure 22(c) exemplarily shows a screen-casting interface in a dual-vertical-screen scene with two screen-casting channels.
  • the screen-casting windows of two vertical screens are arranged side by side in the horizontal direction , and the screen recording resolutions of the two terminal devices are different.
  • the screen recording resolution of terminal device A is 1920*1080
  • the resolution of terminal device B’s screen recording is 2040*1080.
  • the media sent by terminal device A and terminal device B The configuration file indicates that the aspect ratio is 9:16 (vertical screen) and 9:17 (vertical screen), then the created screen projection window 1 (corresponding to terminal device A) and screen projection window 2 (corresponding to terminal device B) Both are vertical screens, but the horizontal display sizes of the two projection windows are different.
  • Figure 22(d) exemplarily shows another screen-casting interface in a dual-vertical-screen scene with the number of screen-casting channels being 2.
  • the screen-casting windows of two vertical screens are also arranged side by side in the horizontal direction Arrangement, the screen recording resolution of terminal device A is 2040*1080, and the screen recording resolution of terminal device B is 1920*1080.
  • the result is shown in Figure 22(d). Layout.
  • any version can be randomly selected as the target layout.
  • the screen projection windows in the layout may or may not be sorted according to the time when the screen projection is accessed.
  • the media configuration information sent by terminal device A indicates that its aspect ratio is 16:9 (horizontal screen)
  • the media configuration information sent by terminal device C indicates that its aspect ratio is 9:16 (vertical screen)
  • the media configuration information sent by terminal device B indicates that its aspect ratio is 9:16 (vertical screen), which corresponds to three-way A scene with one horizontal screen and two vertical screens.
  • three versions of the layout can be matched.
  • Figure 23(a) illustrates version 1
  • Figure 23(b) illustrates version 2
  • Figure 23(c) illustrates version 3.
  • any version can be output as the target layout.
  • Step S201 the terminal device side: when receiving the user's screen projection operation on the media, send a screen projection request to the display device.
  • the screen projection request includes at least device information of the terminal device;
  • Step S202 the display device side: upon receiving the screen projection request, establish a screen projection connection with the terminal device, and feed back authorized information to the terminal device;
  • Step S203 on the display device side: add one to the cumulative number of projection channels;
  • Step S204 the terminal device side: when receiving the authorized information, send the projection data and media configuration information to the display device;
  • the display device may inquire whether the terminal device has screen projection permission. If the terminal device has the permission to cast the screen, the display device will feed back the authorized information to the terminal device; if the terminal device does not have the permission to cast the screen, the display device will feed back unauthorized information to the terminal device, so that the unauthorized information received by the terminal device After the information, the projection data and media configuration information will no longer be sent.
  • the way for the display device to inquire about the screen projection permission of the terminal device is not limited to asking the user whether to authorize the terminal device through a UI pop-up window, or querying the list of trusted devices maintained by the device.
  • Step S205 the display device side: receiving the screen projection data and media configuration information sent by the terminal device, and querying the currently counted number M of screen projection channels.
  • M is greater than or equal to 1;
  • Step S209 the display device side: calculate the position distribution information according to the screen resolution and the size information of the M groups.
  • the position distribution information includes coordinates of preset reference points on the M projection screen windows, and the coordinates of the preset reference points are used to locate the position of the screen projection window on the screen;
  • Step S210 the display device side: according to the location distribution information, locate and arrange the M screen projection windows to obtain the layout.
  • each version can be screened based on the principle of maximum area, and the target layout with the largest sum of the areas of the projection windows can be output;
  • Step S211 display device side: display the layout in the screen projection interface, and display the screen projection data sent by the corresponding terminal device in each screen projection window in the layout.
  • the display device can accurately project the screen projection data to the designated screen projection window for display;
  • Step S212 the display device side: monitor the number of screen projection channels in real time during the process of displaying the screen projection interface;
  • Step S214 display device side: exit the screen projection interface when it is detected that the number of screen projection channels has decreased to zero.
  • this application realizes dynamic adaptive layout matching on the projection interface, so that the typesetting and layout of M projection windows are more suitable for the screen , the multi-channel projection windows are displayed in the largest range on the screen, and the distribution of the projection windows is more compact, reducing the blank space of the projection interface.
  • the aesthetics of the screen projection interface can also be considered, such as the uniformity and symmetry of the distribution of the screen projection windows, the relative balance of the size of the windows, etc., and the size and position distribution of each screen projection window can be set.
  • the obtained layout significantly improves the display effect of the screen projection interface, thereby enhancing the user's viewing experience of the screen projection.
  • the display device after the display device is started, it can directly enter the display interface of the signal source selected last time, or the signal source selection interface, wherein the signal source can be a preset video-on-demand program, or an HDMI interface, a live TV interface
  • the display can display the content obtained from different signal sources.
  • mobile terminals such as mobile phones and tablet computers have more and more functions, and more and more users use mobile terminals to watch videos and browse files.
  • mobile terminals such as mobile phones and tablet computers
  • display devices such as televisions and desktop monitors
  • the screen projection application in the mobile terminal when performing screen projection, the screen projection application in the mobile terminal is first started, and after the screen projection application in the mobile terminal is started, it automatically searches for a display device that can perform screen projection in the local area network currently connected to the mobile terminal , and then the display interface of the mobile terminal displays the device names of each display device that can cast a screen in a list. After the user selects and clicks the device name of the display device that needs to be projected in the display interface of the mobile terminal according to the actual screen projection needs, the mobile terminal can establish a screen projection connection with the display device selected by the user.
  • the mobile terminal After the mobile terminal is connected to the display device for screen projection, the mobile terminal can send the data displayed on the mobile terminal to the display device, so as to watch the video in the mobile terminal on the display device.
  • the screen projection method in the related art cannot simultaneously perform screen projection and playback on one display device for content sent through multiple screen projection protocols.
  • an embodiment of the present application provides a display device, including: a display and a controller, wherein the display is configured to display a user interface.
  • a playback application and at least one screen projection protocol application can be run in the controller, and the at least one screen projection protocol application can be connected to the playback application, and each screen projection protocol application can independently perform data communication with the playback application.
  • Each screen projection protocol application adopts a different screen projection protocol.
  • the existing screen projection protocols include: DLNA projection protocol, Miracast projection protocol, Airplay projection protocol and other projection protocols.
  • the DLNA screen projection protocol can support third-party media on the mobile terminal The push and projection of the media data of resources;
  • the Miracast projection protocol can support the mirror projection of the real-time screen of the mobile terminal screen;
  • the Airplay projection protocol can not only support the push projection of the media data of the third-party media resources on the mobile terminal, but also can It supports the mirror projection of the real-time screen of the screen of the mobile terminal;
  • other screen projection protocols can be the protocols developed by each display device manufacturer.
  • the controller in this embodiment is configured To perform the following steps:
  • Step 702 Create a first playback window on the user interface through the playback application, and the first playback window is used to play the first content to be projected.
  • the first mobile terminal may be the only mobile terminal that performs screen projection connection with the display device, or may be any one of multiple mobile terminals that perform screen projection connection with the display device.
  • the first screen projection request may be a screen projection request sent by the first mobile terminal to the display device through any screen projection protocol.
  • the first screen-casting protocol application is a screen-casting protocol application that adopts the same screen-casting protocol as the first screen-casting request, and the first play window may be the only play window created by the play application, or multiple play windows created by the play application. Play any of the windows.
  • the controller receives the screen projection request.
  • the screen protocol application creates a playback window for playback, but sends the first content to be projected to the playback application through the first screen projection protocol application, and the playback application creates a playback window (ie, the first playback window) for playback.
  • the corresponding screen projection protocol application responds to the screen projection request, and the playback application will continue to run at this time, so the first playback window will not be affected, and will Continue to play the first content to be projected.
  • the controller is further configured to perform the following steps:
  • the second mobile terminal and the first mobile terminal may be the same mobile terminal, or two different mobile terminals, and the second screen projection protocol is applied to use the same screen projection as the second screen projection request.
  • the screencasting protocol application of the protocol, the first screencasting request and the second screencasting request are two different screencasting requests sent by the user based on two different content to be screencasted, and the two screencasting requests can use the same projection
  • the screen protocol can be used to send out, or different screen projection protocols can be used to send out.
  • the first screencasting protocol application and the second screencasting protocol application are the same screencasting protocol application, if the two screencasting requests use different screencasting protocols sent, then the first screen projection protocol application and the second screen projection protocol application are two different screen projection protocol applications.
  • the second playback window and the first playback window are independent playback windows that can exist on the display at the same time, and the two playback windows may overlap partially or not at all.
  • the playback application can adaptively set the size and position of the first playback window and the second playback window, so that the two playback windows can be completely displayed on the display. Or, the playback application can adjust the size and position of the first playback window and the second playback window according to the user's preferences.
  • the content to be projected on the mobile terminal is generally divided into two types: one is the media data of the third-party media resources on the mobile terminal, and the other is the real-time screen of the mobile terminal.
  • the controller before sending the first screen-to-be-cast content transmitted by the first mobile terminal through the screen-casting connection to the playback application, the controller is further configured to:
  • the first screen-to-be-cast content through the first screen-casting protocol application; wherein, if the first screen-to-be-cast content belongs to media data of a third-party media resource, then the first screen-to-be-cast content is The link address of the target web page, if the first screen-to-be-projected content belongs to the real-time screen, then the first screen-to-be-projected content is the screen video stream of the first mobile terminal.
  • the first screen projection protocol application When receiving the first screen projection request, the first screen projection protocol application responds, and negotiates a protocol command with the screen projection application that sent the first screen projection request. After the negotiation is completed, the first screen projection protocol application acquires the first mobile terminal The first content to be screened sent.
  • the link address of the target webpage may be sent to the player via the broadcast method of Android or other inter-process communication methods. application.
  • the playing application After receiving the link address of the target web page, the playing application creates a first play window, connects to the target link address, and then plays the video content obtained from the target link address in the first play window.
  • the screen projection application in the first mobile terminal sends the first screen projection request to the display device through the DLNA screen projection protocol
  • the DLNA screen projection protocol application in the display device controller responds, and the screen projection application performs an agreement command Negotiation. After the negotiation is completed, the projection connection is established.
  • the content to be projected sent by the projection application through the projection connection is the URL address of a source in a third-party video software
  • the DLNA projection protocol application will receive the The URL address of the URL is notified to the playback application through the broadcast of android or other inter-process communication methods.
  • the playback application creates a playback window (or playback control) to play the film source corresponding to the URL address.
  • the first screen-to-be-cast content received by the first screen-casting protocol application is the screen video stream of the first mobile terminal
  • the first screen-to-be-cast content can be sent to the playback application by sharing memory.
  • the controller is configured as:
  • a first shared memory is created, and the first shared memory has a unique index name.
  • the screen video stream of the first mobile terminal is stored in the first shared memory in real time, and a first read instruction is sent to the playback application, and the first read
  • the instruction fetch includes the index name of the first shared memory.
  • the playback application can read the screen video stream from the first shared memory in real time, and play the video stream from the first shared memory in the first playback window. Screen video stream read in real time.
  • the Miracast screen-casting protocol application in the display device controller responds, and the command of the protocol with the screen-casting application Negotiation, after the negotiation is completed, the Miracast screen projection protocol application creates a shared memory, that is, the first shared memory, and names the first shared memory, and then passes the name of the first shared memory through android broadcast or other communication methods between processes , notify the playback application, and at the same time, the Miracast screen projection protocol application receives the screen video stream transmitted in real time from the first mobile terminal through other data transmission protocols, writes it into the first shared memory, and the playback application creates a playback window, from The real-time screen video stream is read from the first shared memory for playback.
  • the inter-process communication mode may be a communication mode such as a named pipe or a socket socket.
  • the screen projection application in the first mobile terminal stops the screen projection operation, it notifies the first screen projection protocol application in the display device controller through the screen projection protocol, and the first screen projection protocol application notifies the playback application to stop playing.
  • the first screen-casting protocol application can delete the created first shared memory.
  • the screen projection method provided by this application will be described below with reference to specific examples. Assume that there are four mobile terminals that have sent screen projection requests to the display device. Suppose that the first mobile terminal pushes a source of video software A to the display device through the DLNA projection protocol, and the second mobile terminal pushes a source of video software A to the display device through the DLNA projection protocol. The display device pushes a video source of video software B, the third mobile terminal pushes its own screen image to the display device through the miracast screen projection protocol, and the fourth mobile terminal pushes its own screen image to the display device through the airplay screen projection protocol.
  • Each protocol application in the display device controller: DLNA protocol application, miracast protocol application, and airplay protocol application respectively respond to the four screen projection requests. As shown in Figure 30, the interaction process between the four mobile terminals and the display device is as follows:
  • the first mobile terminal pushes a video source of video software A to the display device through the DLNA projection protocol, and the DLNA projection protocol application receives the projection request and interacts with the first mobile terminal using the standard DLNA protocol.
  • the URL address of a video source of video software A is received, and the DLNA projection protocol application starts the playback application, and transmits the URL address to the playback application through process communication, and the playback application creates a playback window player 1 (on the monitor
  • the display form on the user interface is a playback window), and the URL address is connected to play the video.
  • the second mobile terminal pushes a source of video software B to the display device through the DLNA projection protocol, and the DLNA projection protocol application receives the projection request and interacts with the second mobile terminal using the standard DLNA protocol.
  • the URL address of a video source of video software B is received, the DLNA projection protocol application starts the playback application, and transmits the URL address to the playback application through process communication, and the playback application creates a playback window player 2 (on the monitor The form on the user interface is the second playback window), and connect to the URL address to play the video.
  • the third mobile terminal pushes its own screen image to the display device through the miracast screen projection protocol.
  • the miracast screen projection protocol application receives the screen projection request, conducts miracast connection command negotiation, and transmits the screen image data after the negotiation is completed , the miracast screencasting protocol application creates a shared memory named miracast_sharememry1, and at the same time notifies the playback application to read the video stream from the shared memory miracast_sharememry1 in real time, and at the same time the playback application creates a playback window player3 at a suitable position on the display interface (on the display user interface The form of expression is the third play window) to play the video stream.
  • the fourth mobile terminal pushes its own screen image to the display device through the airplay screen projection protocol, and the airplay screen projection protocol application receives the screen projection request, conducts airplay connection command negotiation, and transmits the screen image data after the negotiation is completed , the airplay projection protocol application creates a shared memory named airplay_sharememry1, and at the same time notifies the playback application to read the video stream from the shared memory airplay_sharememry1 in real time, and at the same time, the playback application creates a playback window player4 at a suitable position on the display interface (on the display user interface The form of expression is the fourth playback window) to play the video stream.
  • a playback application is independently developed in the controller.
  • the controller can use multiple screen projection
  • the protocol application independently interacts with multiple mobile terminals, establishes a projection connection, and then forwards the content to be projected to the playback application.
  • the playback application receives a content to be projected, it will create a playback window for playback.
  • multiple independent playback windows will be created accordingly to play. In this way, the content to be projected through multiple screen projection protocols can be played on the same display device at the same time.
  • Step 3102 the first screen projection protocol application responds to the first screen projection request, and establishes a screen projection connection with the first mobile terminal, and the first screen projection protocol application uses Protocol application of the same projection protocol;
  • Step 3103 the first mobile terminal sends the first screen-to-be-cast content to the first screen-casting protocol application through the screen-casting connection;
  • Step 3104 the first screen-casting protocol application sends the first content to be screen-casted to the playback application;
  • Step 3105 the playing application creates a first playing window on the user interface displayed on the display, and the first playing window is used to play the first content to be mirrored.
  • the controller receives the screen projection request.
  • the screen protocol application creates a playback window for playback, but sends the first content to be projected to the playback application through the first screen projection protocol application, and the playback application creates a playback window (ie, the first playback window) for playback.
  • the corresponding screen projection protocol application responds to the screen projection request, and the playback application will continue to run at this time, so the first playback window will not be affected, and will Continue to play the first content to be projected. In this way, it is possible to simultaneously play on the display device the content to be projected sent through multiple screen projection protocols.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

La présente demande concerne un dispositif d'affichage (200), un terminal de communication (300) et un procédé d'affichage dynamique d'image sur un écran projeté. Le procédé consiste : lorsqu'un utilisateur déclenche un changement de nombre de terminaux de communication (300) connectés à un dispositif d'affichage (200), à mesurer le nombre de connexions des terminaux de communication (300) pour planifier un mode de disposition d'une image sur un écran projeté ; et ensuite, à envoyer aux terminaux de communication (300) un message de notification comprenant un paramètre d'affichage d'image sur un écran projeté, de telle sorte que les terminaux de communication (300) peuvent générer, selon le paramètre d'affichage, des données de projection d'écran conformes au mode de disposition, et à afficher les données de projection d'écran dans le dispositif d'affichage (200).
PCT/CN2022/102439 2021-08-02 2022-06-29 Dispositif d'affichage, terminal de communication et procédé d'affichage dynamique d'image sur un écran projeté WO2023011058A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202280053896.2A CN117837150A (zh) 2021-08-02 2022-06-29 显示设备、通信终端及投屏画面动态显示方法

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
CN202110881335.9A CN113613072B (zh) 2021-08-02 2021-08-02 一种多路投屏显示方法及显示设备
CN202110881335.9 2021-08-02
CN202110881496.8 2021-08-02
CN202110881496.8A CN114286152A (zh) 2021-08-02 2021-08-02 显示设备、通信终端及投屏画面动态显示方法
CN202111265530.5A CN113992963A (zh) 2021-10-28 2021-10-28 一种显示设备及投屏方法
CN202111265530.5 2021-10-28

Publications (1)

Publication Number Publication Date
WO2023011058A1 true WO2023011058A1 (fr) 2023-02-09

Family

ID=85155158

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/102439 WO2023011058A1 (fr) 2021-08-02 2022-06-29 Dispositif d'affichage, terminal de communication et procédé d'affichage dynamique d'image sur un écran projeté

Country Status (2)

Country Link
CN (1) CN117837150A (fr)
WO (1) WO2023011058A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116347145A (zh) * 2023-03-27 2023-06-27 广州南翼信息科技有限公司 多设备屏幕共享方法、投屏装置、显示装置及共享系统
CN117156190A (zh) * 2023-04-21 2023-12-01 荣耀终端有限公司 投屏管理方法和装置

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150201193A1 (en) * 2012-01-10 2015-07-16 Google Inc. Encoding and decoding techniques for remote screen sharing of media content using video source and display parameters
WO2017042983A1 (fr) * 2015-09-07 2017-03-16 株式会社オプティム Dispositif de partage d'écran, procédé de partage d'écran, système de partage d'écran et programme s'y rapportant
CN111601120A (zh) * 2020-04-21 2020-08-28 江苏大学 一种无线传屏显示系统与显示方法
CN112235626A (zh) * 2020-10-15 2021-01-15 Oppo广东移动通信有限公司 视频渲染方法、装置、电子设备及存储介质
CN112394892A (zh) * 2019-08-15 2021-02-23 北京字节跳动网络技术有限公司 投屏方法、投屏设备、移动终端、存储介质
CN112423102A (zh) * 2020-11-18 2021-02-26 努比亚技术有限公司 一种小窗投屏控制方法、设备及计算机可读存储介质
CN112988102A (zh) * 2021-05-11 2021-06-18 荣耀终端有限公司 投屏方法和装置
CN113613072A (zh) * 2021-08-02 2021-11-05 海信视像科技股份有限公司 一种多路投屏显示方法及显示设备
CN114286152A (zh) * 2021-08-02 2022-04-05 海信视像科技股份有限公司 显示设备、通信终端及投屏画面动态显示方法

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150201193A1 (en) * 2012-01-10 2015-07-16 Google Inc. Encoding and decoding techniques for remote screen sharing of media content using video source and display parameters
WO2017042983A1 (fr) * 2015-09-07 2017-03-16 株式会社オプティム Dispositif de partage d'écran, procédé de partage d'écran, système de partage d'écran et programme s'y rapportant
CN112394892A (zh) * 2019-08-15 2021-02-23 北京字节跳动网络技术有限公司 投屏方法、投屏设备、移动终端、存储介质
CN111601120A (zh) * 2020-04-21 2020-08-28 江苏大学 一种无线传屏显示系统与显示方法
CN112235626A (zh) * 2020-10-15 2021-01-15 Oppo广东移动通信有限公司 视频渲染方法、装置、电子设备及存储介质
CN112423102A (zh) * 2020-11-18 2021-02-26 努比亚技术有限公司 一种小窗投屏控制方法、设备及计算机可读存储介质
CN112988102A (zh) * 2021-05-11 2021-06-18 荣耀终端有限公司 投屏方法和装置
CN113613072A (zh) * 2021-08-02 2021-11-05 海信视像科技股份有限公司 一种多路投屏显示方法及显示设备
CN114286152A (zh) * 2021-08-02 2022-04-05 海信视像科技股份有限公司 显示设备、通信终端及投屏画面动态显示方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116347145A (zh) * 2023-03-27 2023-06-27 广州南翼信息科技有限公司 多设备屏幕共享方法、投屏装置、显示装置及共享系统
CN116347145B (zh) * 2023-03-27 2023-10-31 广州南翼信息科技有限公司 多设备屏幕共享方法、投屏装置、显示装置及共享系统
CN117156190A (zh) * 2023-04-21 2023-12-01 荣耀终端有限公司 投屏管理方法和装置

Also Published As

Publication number Publication date
CN117837150A (zh) 2024-04-05

Similar Documents

Publication Publication Date Title
US10896019B2 (en) Screen sharing system, screen sharing method, and storage medium
US20240053944A1 (en) Display apparatus and method for controlling screen projections from multiple devices to same screen
WO2017148294A1 (fr) Procédé, dispositif et terminal mobile de commande d'appareil basé sur terminal mobile
CN113613072B (zh) 一种多路投屏显示方法及显示设备
US20180077461A1 (en) Electronic device, interractive mehotd therefor, user terminal and server
CN114286165B (zh) 一种显示设备、移动终端、投屏数据传输方法及系统
WO2023011058A1 (fr) Dispositif d'affichage, terminal de communication et procédé d'affichage dynamique d'image sur un écran projeté
WO2020248714A1 (fr) Procédé et dispositif de transmission de données
US20160205427A1 (en) User terminal apparatus, system, and control method thereof
CN114286152A (zh) 显示设备、通信终端及投屏画面动态显示方法
CN118435566A (zh) 显示设备、智能家居系统及用于显示设备的多屏控制方法
KR20200116378A (ko) 복수의 방송 채널의 영상을 표시하는 장치 및 방법
WO2024045770A1 (fr) Procédé et appareil de commande de contenu, support de stockage, et dispositif électronique
US11662971B2 (en) Display apparatus and cast method
WO2023024630A1 (fr) Dispositif d'affichage, dispositif terminal et procédé d'affichage de contenu
CN116567333A (zh) 一种显示设备及多窗口的画质显示方法
US10863233B2 (en) Wireliss docking system for audio-video
CN116980554A (zh) 一种显示设备及视频会议界面显示方法
CN114286320A (zh) 一种显示设备、移动终端及蓝牙连接方法
CN114827700B (zh) 投屏内容的显示方法、装置、终端及存储介质
WO2024222032A1 (fr) Procédé d'affichage et dispositif d'affichage
WO2024021036A1 (fr) Procédé de commande de modèle, appareil, dispositif, système et support de stockage informatique associés
CN113992963A (zh) 一种显示设备及投屏方法
CN115857854A (zh) 显示设备及投屏画面切换方法
CN117809592A (zh) 显示设备和图层刷新率设置方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22851773

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202280053896.2

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22851773

Country of ref document: EP

Kind code of ref document: A1