WO2023024630A1

WO2023024630A1 - Display device, terminal device, and content display method

Info

Publication number: WO2023024630A1
Application number: PCT/CN2022/096383
Authority: WO
Inventors: 宋子全; 庞秀娟; 马晓燕; 刘美玉; 刘承龙; 肖成创
Original assignee: 海信视像科技股份有限公司
Priority date: 2021-08-27
Filing date: 2022-05-31
Publication date: 2023-03-02
Also published as: CN117859331A

Abstract

Disclosed are a display device and a terminal. A terminal device may establish a screen projection connection with the display device. Each time the display device receives a push, the display device must detect a resource type of a screen projection and a current display state, and formulate an update strategy adapted to a current scene for a display. The update strategy indicates in which target playback window to load and display newly received data to be played back, and other content such as the number of windows and layout adjustment may also be involved. According to the present application, a connection instruction of a first terminal is received, the number of second terminals is obtained, an interactive window is generated according to the number of second terminals, and according to an instruction generated by a control in the interactive window, the display is controlled to display screen projection content of the first terminal.

Description

Display device, terminal device and content display method

This application requires the application number 202110997833.X submitted on August 27, 2021, the application number 202110998023.6 submitted on August 27, 2021, the application number 202110998018.5 submitted on August 27, 2021, and the application number submitted on August 27, 2021. The priority of the Chinese patent application with application number 202111133007.7 filed on September 27, 2021, the entire content of which is incorporated in this application by reference.

technical field

The present invention relates to the technical field of content display, in particular to a display device, a terminal device and a content display method.

Background technique

A display device is a display product that can realize two-way human-computer interaction and integrate multiple functions such as audio-visual, entertainment, and data. In order to meet the diverse needs of users, the display device is equipped with various applications such as screen projection, audio-visual, entertainment, etc., and interacts and exchanges information with users through the user interface.

In some application scenarios, users can initiate screen projection to display devices through terminals such as mobile phones and PCs, so as to push media resources such as images, audio, and video in small-screen terminals to display devices for large-screen playback. Currently, the display device only supports media resources that are pushed and played in one-way projection. For example, the display device is currently displaying media resource A. At this time, if a mobile phone pushes media resource B to the display device, the display device must stop playing media resource A. Only then can media resource B be loaded and played. Although this method can avoid display conflicts between media resources, it cannot meet the needs of users to watch multiple media resources simultaneously, and the application scenarios are limited.

Contents of the invention

According to some embodiments of the present application, a display device is provided, including:

a display for displaying images;

a communicator for communicating with one or more terminal devices;

a controller coupled to the display and communicator and configured to:

receiving a screen projection request from a first terminal device among the one or more terminal devices,

In response to the screen projection request, an update policy of the display is determined according to the current display state of the display, so as to play resources corresponding to the screen projection request.

According to some embodiments of the present application, a method for displaying content is provided, including:

Description of drawings

FIG. 1 is a usage scenario of a display device according to some embodiments;

FIG. 2 is a block diagram of a hardware configuration of a control device 100 according to some embodiments;

FIG. 3 is a block diagram of a hardware configuration of a display device 200 according to some embodiments;

FIG. 4 is a software configuration diagram in a display device 200 according to some embodiments;

Fig. 5 is the UI schematic diagram that comprises multi-channel video window in playing interface;

Fig. 6 is the UI diagram that comprises audio window in playing interface;

Fig. 7 is the UI diagram that comprises image window in playing interface;

Fig. 8 is a schematic diagram of UI update after a new video push is received by the display device when displaying the playback interface of Fig. 6;

Fig. 9 is a schematic diagram of displaying the display device currently displaying the playback interface of Fig. 6 and displaying the first prompt pop-up window when a new video push is received;

FIG. 10 is a schematic diagram of UI updating when the user selects the first replacement control in the first prompt pop-up window;

Fig. 11 is a schematic diagram of a second prompt pop-up window displayed when the display device currently displays the playback interface of Fig. 5 when a new video push is received;

12 is a schematic diagram of UI update when the user selects the second replacement control in the second prompt pop-up window;

Fig. 13(a) is a schematic diagram of displaying a third prompt pop-up window when a new audio push is newly received when the display device is currently displaying two channels of video and one channel of image;

Fig. 13(b) is a schematic diagram of UI update when the user selects the second newly added control in the third prompt pop-up window shown in Fig. 13(a);

Fig. 13(c) is a schematic diagram of UI update when the user selects the third replacement control in the third prompt pop-up window shown in Fig. 13(a);

Fig. 14(a) is a schematic diagram of displaying a third prompt pop-up window when a new audio push is received when the display device currently displays three channels of video and one channel of image;

Fig. 14(b) is a schematic diagram of UI update when the user selects the third replacement control in the third prompt pop-up window shown in Fig. 14(a);

Fig. 15(a) is a schematic diagram of a scene when a new image is received when the display device displays one image and one image;

Fig. 15(b) is a UI schematic diagram after updating the playing interface shown in Fig. 15(a);

Figure 16(a) is a schematic diagram of displaying a third prompt pop-up window when a new image push is received when the display device is currently displaying one video and one audio;

Fig. 16(b) is a schematic diagram of UI update when the user selects the second newly added control in the third prompt pop-up window shown in Fig. 16(a);

Fig. 16(c) is a schematic diagram of UI update when the user selects the third replacement control in the third prompt pop-up window shown in Fig. 16(a);

Fig. 17(a) is a schematic diagram of displaying a third prompt pop-up window when a new image push is received when the display device is currently displaying four channels of video;

Fig. 17(b) is a schematic diagram of UI update when the user selects the third replacement control in the third prompt pop-up window shown in Fig. 17(a);

18 is a schematic diagram of the playback interface after the user switches the window focus;

Fig. 19 is a flow chart of a resource synchronous playback method;

FIG. 20 is a flow chart of another resource synchronous playback method;

Fig. 21 is a flowchart of a display method for multi-channel projection according to some embodiments;

Figure 22 is a schematic diagram of a first user interface according to some embodiments;

Figure 23 is a schematic diagram of a second user interface according to some embodiments;

Fig. 24 is a schematic diagram of yet another first user interface according to some embodiments;

Fig. 25 is a schematic diagram of yet another second user interface according to some embodiments;

Fig. 26 is a schematic diagram of yet another first user interface according to some embodiments;

Fig. 27 is a schematic diagram of yet another second user interface according to some embodiments;

Fig. 28 is a schematic diagram of yet another first user interface according to some embodiments;

Fig. 29 is a schematic diagram of yet another first user interface according to some embodiments;

Fig. 30 is a schematic diagram of yet another second user interface according to some embodiments;

Fig. 31 is a schematic diagram of yet another first user interface according to some embodiments;

Fig. 32 is a schematic diagram of yet another first user interface according to some embodiments;

Fig. 33 is a schematic diagram of yet another first user interface according to some embodiments;

Figure 34 is a schematic diagram of another second user interface according to some embodiments;

Figure 35 is a schematic diagram of another first user interface according to some embodiments;

Figure 36 is a schematic diagram of another second user interface according to some embodiments;

Fig. 37 is a schematic diagram of the display effect of mirroring and projecting the screen of the mobile device to the display device 200 according to some embodiments;

Fig. 38 is a schematic diagram of another display effect of mirroring a mobile device to a display device 200 according to some embodiments;

Fig. 39 is a schematic flowchart of a method for controlling audio output of mirroring screen projection according to some embodiments;

Fig. 40 is a schematic diagram of the display effect of mirroring and projecting screens from multiple mobile devices to the display device 200 according to some embodiments;

Fig. 41 is a timing diagram of a method for controlling audio output of screen mirroring according to some embodiments.

Detailed ways

The following will describe the exemplary embodiments of the present application with reference to the accompanying drawings in the exemplary embodiments of the present application. Apparently, the described exemplary embodiments are only some of the embodiments of the present application, but not all of the embodiments.

Fig. 1 is a schematic diagram of a usage scenario of a display device according to an embodiment. As shown in FIG. 1 , the display device 200 also performs data communication with the server 400 , and the user can operate the display device 200 through the smart device 300 or the control device 100 .

In some implementations, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes at least one of infrared protocol communication, Bluetooth protocol communication, and other short-distance communication methods, and the display device is controlled wirelessly or wiredly. Device 200. The user can control the display device 200 by inputting user instructions through at least one of buttons on the remote control, voice input, and control panel input.

In some embodiments, the smart device 300 may include any one of a mobile terminal, a tablet computer, a computer, a notebook computer, an AR/VR device, and the like.

In some implementations, the smart device 300 can also be used to control the display device 200 . For example, the display device 200 is controlled using an application program running on the smart device.

In some implementation manners, the smart device 300 and the display device may also be used for data communication.

In some implementations, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the module for acquiring voice commands configured inside the display device 200 can directly receive the user's voice command control , the user's voice command control can also be received through the voice control device provided outside the display device 200 .

The display device 200 also performs data communication with the server 400 . The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200 .

FIG. 2 is a block diagram of a configuration of the control device 100 according to the exemplary embodiment. As shown in FIG. 2 , the control device 100 includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply. The control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, and play an intermediary role between the user and the display device 200 .

FIG. 3 is a block diagram showing a hardware configuration of a display device 200 according to an exemplary embodiment.

In some embodiments, the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface. at least one.

The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, the first interface to the nth interface for input/output.

The display 260 includes a display screen component for presenting a picture, and a drive component for driving image display, for receiving an image signal output from the controller, for displaying video content, image content and a menu control interface component, and a user control UI interface wait.

The display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

The communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220 .

In some embodiments, the user interface may be used to receive a control signal from the control device 100 (such as an infrared remote controller, etc.).

The external device interface 240 may include, but is not limited to, any of the following: high-definition multimedia interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. or multiple interfaces. It may also be a composite input/output interface formed by the above-mentioned multiple interfaces.

In some embodiments, the tuner-demodulator 210 receives broadcast TV signals through wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or cable broadcast TV signals.

In some implementations, the controller 250 and the tuner-demodulator 210 may be located in different split devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box wait.

The user input interface 280 is an interface that can be used to receive control input (such as: a physical button on the display device body, or others).

Referring to Fig. 4, in some embodiments, the system is divided into four layers, from top to bottom are the application program (Applications) layer (abbreviated as "application layer"), application framework (Application Framework) layer (abbreviated as "framework layer") "), Android runtime (Android runtime) and system library layer (referred to as "system runtime layer"), and the kernel layer.

The above embodiments introduce the hardware/software architecture and function realization of the display device. In some application scenarios, the terminal device can push media data to the display device. For example, the user can project the media resources of interest to the display device for large-screen playback through some application programs with screen projection functions; or the terminal device and the display The device establishes a screen projection connection through certain protocols, such as DLNA (Digital Living Network Alliance, Digital Living Network Alliance) protocol, etc.; or, the terminal device and the display device perform mirror projection to mirror the media data currently displayed by the terminal device Synchronously push to the display device side.

When the user plays media data on the terminal device and selects screen projection push, he can select other devices for playing the media data from the device list, and the display device that supports the screen projection function will be recognized and automatically added to the device list In , after the user selects the display device from the device list, the media data can be projected from the terminal device to the display device for playback. When using the projection method to push media data, the terminal device and the display device generally need to be connected to the same Wifi network, and the method for the terminal to push media data to the display device is not limited. Of course, in other technologies, the screen projection method can also be performed across networks. In this application, there is no limitation on the specific manner of screen projection.

The terminal device can push the media data to the display device only after the terminal device is successfully connected to the display device. For every media resource available on the Web, such as HTML documents, images, videos, audio and programs, etc., a general URI (Uniform Resource Identifier, Uniform Resource Identifier) performs resource location, that is, URI is equivalent to providing an address for locating resources. Optionally, the terminal device may push the URI of the media data to be projected to the display device, and the display device may locate the resource after receiving the URI, and then download and play the corresponding media data. Wherein, the terminal device may be a mobile terminal such as a mobile phone, a tablet computer, a smart wearable device (such as a smart bracelet, etc.), a notebook computer, or a desktop computer, a TV, and other devices with screen projection and display functions.

In other exemplary implementations, the terminal device may also push the URL (Uniform Resource Locator, Uniform Resource Locator) of the media data, that is, the network address of the media resource, to the display device, so that the display device accesses the URL and downloads the media data; Alternatively, the terminal device can also directly synchronize images, video streams, audio streams, etc. to the display device in real time.

In some implementations, if the display device only supports playing a single-channel media resource, optionally, when the user initiates screen projection, the terminal device can generate a stop instruction, and the screen projection request sent to the display device carries the stop instruction And the projection data such as URI/URL of the resource, after the display device receives the projection request and recognizes the stop command in the projection request, it will first stop playing the current broadcast resource to release the occupied playback resource, and then load the And play the projection data. Optionally, or the terminal device does not generate a stop instruction, but directly pushes the screen projection data to the display device; when the display device receives the screen projection data, the controller of the display device generates a stop instruction, and sends the stop instruction to the The player of the broadcasting resource, so as to control the player of the broadcasting resource to close and stop playing the broadcasting resource. For a display device that only supports single-channel resource playback, if the display device has on-broadcast resources when it receives the projection data pushed by the terminal, the user will have to interrupt the media resource being watched before jumping to the playback projection screen resources, which cannot meet the needs of users to watch multiple media resources synchronously.

In some implementations, users can customize whether the display device has the ability to play multiple resources synchronously. Optionally, for example, a playback capability setting page is added to the setting menu of the display device. In this page, the user can choose to enable Or turn off the ability to play multiple resources synchronously. If the user disables the multi-channel resource synchronous playback capability, that is, the display device only supports single-channel resource playback, the display device needs to close the player of the currently broadcast resource and release the occupied playback resources before starting to broadcast the projection data; If the user enables the synchronous playback capability of multiple resources, a new player can be created to load and play the projected data without stopping the playback of the currently broadcasting resources. If the user has not participated in the definition and adjustment of the playback capability, the multi-channel resource synchronous playback capability can be set to be enabled by default. After enabling the multi-channel resource synchronous playback function, the display device can receive one or more media resources pushed by a single terminal device, and can also receive one or more media resources pushed by multiple terminal devices at the same time, so as to adapt to more Application scenarios.

In some implementations, before sending a screen projection request, the terminal device needs to search for nearby display devices that can establish a screen projection connection. When a certain display device A is found, the terminal device can send a query command to the display device A, and the display device After receiving and responding to the query instruction, A queries and feeds back its own resource playback capability to the terminal device, so that the terminal device knows in advance whether the display device A supports simultaneous playback of multiple resources.

In some implementations, when the terminal device sends a screen projection request to display device A, if it knows that display device A only supports playing a single-channel media resource, optionally, the terminal device generates a stop instruction and makes the screen projection request simultaneously Carry the data such as the stop command and the URI/URL of the screen projection resource. When the display device A receives the screen projection request, it parses the screen projection request and separates the stop command and the screen projection data. If the display device A currently has the broadcast resource, Then respond to the stop command first, close the player of the current broadcast resource, and re-create a new player, and then use the new player to load and play the projection data pushed by the terminal; if it is displayed that device A currently has no broadcast resource, then Create a player directly to load and play screencast data.

In some implementations, when the terminal device sends a screen casting request to display device A, if it knows that display device A only supports playing a single-channel media resource, optionally, the terminal device may first generate a stop instruction, and send the stop instruction to Display device A; when display device A receives a stop instruction, it detects whether it currently has broadcast resources. If display device A currently has broadcast resources, it responds to the stop instruction, closes the player of the current broadcast resource, and sends a message to the terminal The device sends a stopped message; when the terminal device receives the stopped message, it sends the projection data to the display device A, and when the display device A receives the projection data, it creates a player to load and play the projection data. When display device A receives the stop instruction, if it detects that it has no resources currently on the air, it sends a screen projection data acquisition request to the terminal device, and terminal device A responds to the screen projection data acquisition request and sends the screen projection data to display device A , display device A uses the created player to load and play the projection data.

When the terminal device sends a screen projection request to display device A, if it knows that display device A supports synchronous playback of multiple resources, it does not generate a stop command, but only sends screen projection data to display device A, and when display device A receives the projection data , it is allowed to create a new player to load and play the projection data, regardless of whether there are currently broadcast resources.

After the user initiates a screen casting request to display device A, the terminal device can only send screen projection data to display device A without considering whether display device A supports synchronous playback of multiple resources, and display device A itself decides whether to stop broadcasting. resource. When display device A receives the screen projection data pushed by the terminal, it first inquires its own resource synchronous playback capability. For broadcasting resources, the controller can generate a stop command, and send the stop command to the player 1 corresponding to the broadcasting resource, so as to close the player 1, and then create a new player 2 to load and play the projection data; or, display If device A finds that it does not have the ability to play multi-channel resources synchronously (that is, only supports single-channel resource playback), and detects that there is no currently broadcast resource, it does not need to generate a stop command, and directly creates a new player to load and play the projected screen resource .

In some implementations, when the display device A receives the projection data pushed by the terminal, if it finds that it has the ability to play multiple resources synchronously, it can choose to create another player to load it regardless of whether it currently has resources that are being broadcast. And play the projection data.

In some implementations, before the terminal device pushes the screen projection data to the display device, it can also actively access the display device to identify the current resource playback capability of the display device. Optionally, the display device can be in the relevant screen projection protocol/ Describe its resource playback capability in the service. As an example, for example, set the target field in the rendererdevicedesc.xml file. The target field is, for example, <modelDescription>multiplay</modelDescription>. If the terminal device accesses the rendererdevicedesc.xml file and reads If the target field is found, it is known that the display device has the ability to play multiple resources synchronously, and the terminal device does not generate a stop command, but directly pushes the projection data; if the terminal device does not read the target field, it is equivalent to reading that the target field is empty , then it is known that the display device only has the ability to play single-channel media resources, then the terminal device can generate a stop command, and use the stop command to trigger the display device to stop playing the current broadcast resource.

In some implementations, the flag bit can also be set in the target field, and the terminal device can know the playback capability of the display device by accessing the target field and reading the value of the flag bit, for example, the value of the flag bit is set to When it is 1, it indicates that the display device has the ability to play multiple resources synchronously; if the value of the flag bit is set to 0, it indicates that the display device does not have the ability to play multi-channel resources synchronously, but only supports the playback of a single media resource.

In some implementations, the media resources pushed by the terminal device can be divided into video types, audio types, and image types according to resource types, and the display device can use video players, audio players, and image players, etc., so as to realize Play by category, and support different types of players to run at the same time.

In some implementations, multiple video players are created in order to realize multi-channel projection, and each video player can control video playback, including but not limited to video volume, resolution, speed, playback progress (such as pause , seek, fast forward, fast rewind) and other aspects of adjustment and control. Correspondingly, as shown in the UI example of Figure 5, on the display of the display device (hereinafter also referred to as the playback interface), create and display a plurality of windows for displaying videos or images, hereinafter referred to as playback windows, and each playback window displays A video frame output by a video player. The play window in the embodiment of the present application can play dynamic video images, and can also display or present static images.

In some implementations, since the display device generally only supports the output and playback of a single channel of audio, after all, outputting multiple channels of audio at the same time will cause sound mixing, making it impossible for the user to hear the sound of each audio resource, so no matter the display device is pushed How many channels of audio resources, the display device only runs one audio player, for example, the audio player is currently playing audio resource 1, if the display device receives the push of audio resource 2, the audio player needs to stop playing audio resource 1, but Replace it with starting audio resource 2 to avoid playback conflicts between audio resources. The audio player can control audio playback, including but not limited to audio volume, double speed, sound effects, playback progress (such as pause, seek, forward, back), song switching, playback mode (single loop, random playback, sequential playback) and other aspects of regulation and control. Correspondingly, as shown in the UI example in Figure 6, on the playback interface of the display device, only one playback window corresponding to the audio type is created, and the playback window can display audio playback information, and the audio playback information includes but is not limited to song title, singer, Lyrics, song duration, playback progress bar, pause/play controls, background images, etc. Audio-related content. When the user triggers an instruction to push the next song, the information of the next song will be directly displayed on the display device, and the next song will be played.

It should be noted that the audio player and power amplifier mentioned here are different concepts. The audio player is an audio player program, which is equivalent to an application; while the power amplifier is a hardware level that provides a physical channel for outputting audio. The power amplifier may include a built-in power amplifier And the external power amplifier, the built-in power amplifier is the speaker of the display device, and the external power amplifier is the external power amplifier device connected to the display device through Bluetooth, HDMI, etc., such as audio, headphones, etc.

In some embodiments, the terminal device can store local images, or download and manage themed images through APPs such as wallpaper applications and photo album applications. Optionally, the local or image-related applications of the terminal device can save images in the form of albums. Including at least one image, after the terminal device pushes the album to the display device, the display device creates an image player, and the image player is used to display the images in the album, and to control the display of the album according to the specified display mode, the display mode Including but not limited to switching images forward, switching images backward, displaying images sequentially, displaying images in a loop, etc. Since the display device supports dynamic image switching, it is only necessary to create an image player to realize image display switching. Correspondingly, as shown in the UI example in Figure 7, only one playback window corresponding to the image type is created on the playback interface of the display device, and the playback window is associated with the image player, and only a single image is displayed in the playback window at a time. A switching control is set in the window, referring to Fig. 7, which may specifically include a forward switching control and a backward switching control, and the user selects the forward switching control, then switches to display the previous image of the current image; if the user selects the backward switching control Switch the control to switch to display the next image of the current image. When switching, you can perform image positioning according to the order of the images in the album.

In some implementations, the user can manually operate the switching control in the image window to switch images in a specified order (forward or backward), and if the user does not operate, the static display of the current image will be maintained; or, it can also be set to Album sorting automatically rotates images. For example, if an album includes image 1, image 2, and image 3, it will be displayed in the order of image 1 → image 2 → image 3 → image 1 → image 2 → ...; or, it can also be displayed in accordance with image 1 →Image 2→Image 3 After traversing and displaying the images in the album for a round, the current image stays at the last image 3, and the user can switch images by operating the switch control.

In some embodiments, for example, a landscape image A is currently displayed in the image window of the display device. At this time, the display device receives the family photo album B pushed by a certain terminal device, so there is no need to create a new image player and image window, and control the original The image window stops displaying the landscape image A, and instead loads and displays the image in the newly received family album B. Optionally, the landscape image A and the family album B can be combined to obtain a screen-casting album. The screen-casting albums are sorted such that the landscape image A is in the front, and the family album B is in sequence. In this way, the screen-casting album can be played in a loop or In the form of manual switching by the user, the user can also browse to the previously replaced landscape image A.

In some implementations, the display device can maintain a screen-casting album, and sort and store the screen-casting images according to the time order in which the images are pushed, and can automatically display the images in the screen-casting album in a loop after each newly pushed image is displayed. historical images to provide users with a better image browsing experience.

In some embodiments, the image may be a static image, or may also be a dynamic image. For example, a segment of a certain duration can be intercepted from a video and made into a dynamic image; for another example, multiple consecutive frames of images can be dynamically spliced to obtain a dynamic image in a format such as gif.

In some implementations, as shown in Figures 5 to 7, for any type of playback window, an exit control can be set in the playback window to support the user to exit the playback of a certain channel of media resources at any time. For example, referring to Figure 5, when the user selects the exit control in the playback window 4 corresponding to video 4, the video player corresponding to video 4 will be released, the playback of video 4 will be stopped, and the playback window 4 will be destroyed at the same time, and the playback interface The total number of windows in the middle is changed from 4 to 3, and the window layout can be readjusted. The original window layout in Figure 5 presents a "four-square grid" array of 4 playback windows. The adjusted window layout is, for example, playback window 1 and playback window 2 are parallel on top, and the playback window 3 is centered on the bottom alone.

In some implementations, the number of windows in the playback interface will change accordingly every time a channel is added or exited. Therefore, a reasonable The position distribution of each window is set to provide an optimal window layout adapted to the screen and the current playing scene, which is not specifically limited in this embodiment.

In one example, referring to Figure 6, when the user selects the exit control in the playback window 2 corresponding to the audio, the audio player will be released, the currently pushed audio will stop playing, and the playback window 2 will be destroyed at the same time, and the playback The total number of windows in the interface changes from 2 to 1, and there is only one playback window 1 that plays video 1. At this time, the playback window 1 can be switched to full-screen mode. When the audio data pushed by the terminal is received again, the audio player and the corresponding playback window need to be recreated.

In one example, referring to FIG. 7 , if the user selects the exit control in the playback window 3 for displaying images, the image player will be released, and the currently pushed image/album will be stopped from being displayed, and the playback window 3 will be displayed simultaneously. It will also be destroyed, causing the total number of windows in the playback interface to change from 3 to 2. It is also necessary to re-adjust the window layout, for example, adjust the playback window 1 and playback window 2 to be vertical/left-right. When the image data newly pushed by the terminal is subsequently received, the image player and the corresponding playback window need to be recreated.

In some embodiments, for example, if the display device is playing a live sports event on CCTV5, and suddenly receives a movie 2 pushed by terminal device C, two play windows are created, play window 1 displays the live broadcast of CCTV5, and play window 2 displays The video stream of movie 2. Therefore, when creating a window, you can make the playback window 2 have an exit control, and the playback window 1 has no exit control. That is, the resource that the user initially watched cannot exit before accepting the projection. After playing, the display device automatically returns to the playback channel before the screen projection, or the display device automatically resumes playing the media resource that the user was watching before the screen projection.

In some implementations, since the video and audio pushed by the terminal have a preset total duration requirement, for example, the preset total duration of a certain song is 4 minutes, the preset total duration of a certain movie is 90 minutes, etc., so the display device can monitor The playback process of each screen projection resource, when it is detected that the playback process of a certain screen projection resource has reached the end, that is, the broadcast duration is equal to the preset total duration, it means that the user has not voluntarily exited the playback of the projection screen resource during this period, and the projection If the screen resource is finished playing naturally, the video/audio player corresponding to the screen projection resource will be released, and the corresponding playback window will be destroyed at the same time, so as to completely exit the screen projection.

In some implementations, the multi-channel media data received by the display device is assigned to be displayed in different play windows of the same play interface, for example, the display device is currently displaying video 1 and audio 1, that is, the current display state is obtained as having two Play window 1 corresponds to the video type, and play window 2 corresponds to the audio type. At this time, the display device receives the projection push of video 2. For example, a new play window 3 can be added to display video 2, or the play window can be 1 or playing window 2 is replaced by displaying video 2; for another example, when the display device receives the push of audio 2, since playing window 2 is playing audio 1, and the sound power amplifier generally only supports playing single-channel audio data, playing window 2 can be stopped Play Audio 1, and is replaced by Play Audio 2.

It can be seen that both the resource type of the projection screen and the current display state of the display device will affect the final presentation of the playback interface, wherein the current display state includes the number of playback windows currently displayed on the display interface, and the resource type corresponding to each playback window, For example, the current display state of FIG. 6 is {window 1-video; window 2-audio}.

In some exemplary implementations, each time the display device receives a push, it needs to detect the resource type and the current display state of the projected screen, so as to formulate an update strategy for the playback interface that adapts to the current playback scene, and the update strategy indicates In which target playback window to load and display the newly received data to be played, other content such as the number of windows and layout adjustments can also be involved, so as to avoid playback conflicts between media resources and support users to simultaneously browse through multiple windows. Push media resources to improve user experience.

In some implementations, after the display device enables the multi-channel resource synchronous playback function, the display device can receive multiple media resources pushed by a single terminal, or one or more media resources pushed by multiple terminals respectively, that is, theoretically display There is no upper limit to the number of projection channels of the device, but considering that if the playback interface is to accommodate more playback windows, the size of each window must be compressed, and the video, audio playback information, and images displayed in the window will be compressed synchronously. If it is too much, the user cannot see the content and details of the media resource clearly, and the viewing experience is poor. In this regard, this application provides a preset upper limit value, which is used to restrict the maximum number of playback windows allowed to be displayed in the playback interface. The preset upper limit value is not as large as possible, but as large as possible. Considering the possibility of visual viewing, this application does not limit the value of the preset upper limit. For example, if the preset upper limit is 4, the display device has the ability to support simultaneous playback of 4 channels of media resources. On the basis of the preset upper limit constraints, the following will provide corresponding update strategies for different playback scenarios.

In some implementations, if the data to be played pushed by the terminal device to the display device is a video type, since the video can be played by creating an independently running video player and window, instead of sharing the same playback data such as audio or image types player and window, so it is detected whether the current number of playback windows on the playback interface is less than the preset upper limit.

In some implementations, an update strategy is configured as follows: when the display device receives a new video push, if the current number of playback windows is less than the preset upper limit, a new one can be created and added in the playback interface for The target playback window is used to display the newly received data to be played. The playback status of other playback windows remains unchanged, and then re-match the window layout of the playback interface according to the updated total number of playback windows.

As an example, take the display device currently displaying the playback interface shown in FIG. 6 and the preset upper limit value is equal to 4 as an example, the current playback interface includes a playback window 1 displaying video 1, and playing audio 1 (synchronously displaying audio 1 Audio playback information) playback window 2. At this time, the display device receives the video 2 newly pushed by the terminal device D. Since the current number of windows does not reach the upper limit, as shown in Figure 8, a playback window 3 can be directly created and added (i.e. target playback window), and load and display video 2 in playback window 3, at this time the playback status of playback window 1 and playback window 2 remains unchanged, that is, playback window 1 still keeps playing video 1, and playback window 2 still keeps playing audio 1 . Since the total number of playback windows is updated from 2 to 3, the window layout needs to be readjusted. In the example in Figure 8, the window layout changes so that playback window 1 and playback window 2 are side by side in the upper part of the interface along the horizontal direction, and playback window 3 is centered alone in the lower part of the interface. The overall presentation is similar to the inverted "pin" layout. It should be noted that when the total number of playback windows changes, the window layout adjustment form is not limited.

Optionally, layout templates corresponding to different numbers of playback windows can be preset. Taking the preset upper limit equal to 4 as an example, when the number of playback windows is 1, the layout template can be set as a single-window exclusive screen (i.e. full-screen display); When the number of playback windows is 2, the layout template is set to display double windows side by side or side by side; when the number of playback windows is 3, the layout template is set to a positive "pin" or inverted "pin" layout, for example, and the three windows are arranged in parallel. The three windows are arranged in the same row, or Play Window 1 is located alone on the left of the interface, Play Window 2 and Play Window 3 are located side by side on the right of the interface, or Play Window 1 and Play Window 2 are located side by side on the left of the interface, and Play Window 3 is located alone The right part of the interface, etc.; when the number of playback windows is 4, the layout template, for example, is set in the form of a four-square grid. When the number of windows changes, the local template corresponding to the updated total number of playback windows is called, and each playback window is sorted and displayed according to the layout template, and the associated media data is displayed in each playback window.

Optionally, the aforementioned sorting display is, for example, window sorting according to the time when each channel of media data is pushed/received by the display device; or, priority sorting is performed according to resource type, for example, the display device receives 4 channels of media data pushed by screen projection , where two channels are of video type, one channel is of image type, and the other channel is of audio type, then the sequence will be sorted according to the priority of video > audio > image, that is, playback window 1 and playback window 2 display two channels of video respectively, and playback window 3 displays audio, Among them, playback window 4 displays images, playback window 1 and playback window 2 belong to the same priority, and can be sorted internally at the same level according to the time when media data is pushed/received. There is only one window of each type for audio and image at most. It should be noted that the window sorting rules are not limited to the examples in this embodiment of the application.

In the foregoing embodiments, when the number of windows in the playback interface does not reach the upper limit, the newly added windows are used to load and display newly received video data. In other exemplary implementations, an update strategy is configured as follows: when the display device receives a new video push, if the number of current playback windows is less than the preset upper limit, a first pop-up window pops up, and according to The user updates the playing interface accordingly through the operation of the first prompt pop-up window.

In some implementations, referring to the example in FIG. 9 , the first prompt pop-up window includes a new control and a first replacement control. If the user selects the new control, then create and add a playback window in the playback interface as the target playback window (ie, playback window 3), and load and display the newly received data to be played (video 2) in the target playback window, The playback state of other playback windows (i.e. playback window 1 and playback window 2) remains unchanged, and, according to the total number of playback windows after updating, re-match the window layout of the playback interface, that is, the update under the new mechanism of the aforementioned embodiment the same way.

In some implementations, the first replacement control is generated according to the current display state. Referring to the examples in FIG. 6 and FIG. 9 , the current display state includes two playback windows, one is a video window and the other is an audio window, then The generated first replacement control includes two sub-controls, namely "Replace Video Window" and "Replace Audio Window". Users can choose which channel of media data to keep and which channel to replace according to their viewing tendency. For example, the If the user has finished listening to Audio 1 and does not want to listen to it again, he can select "Replace Audio Window", then the selected and replaced playback window 2 will become the target playback window, the power amplifier device will stop playing Audio 1, and the playback window 2 will stop displaying audio For playback information, playback window 2 is replaced with display video 2, and the playback status of playback window 1 remains unchanged. The UI update after the replacement is shown in Figure 10, and the update shows that two videos are playing. When using this replacement mechanism to update the playback interface, the number of windows in the playback interface does not change, only the media data displayed in one of the target playback windows is replaced, so there is no need to adjust the window layout. On-broadcast resources are replaced with new push resources, providing more feasibility for window display in the playback interface, as well as adaptability to playback scenarios/user interests.

In some implementations, an update strategy is configured as follows: when the display device receives a new video push, if the number of current playback windows is equal to the preset upper limit, in this scenario, no new windows can be added on the playback interface. Then a second prompt pop-up window pops up, and the playback interface is updated accordingly according to the user's operation through the second prompt pop-up window.

In some implementations, referring to the example in FIG. 5 , the current display state of the display device includes 4 playback windows, respectively displaying video 1, video 2, video 3 and video 4, a total of four video channels, with a preset upper limit equal to 4 as an example, in this playback scenario, the display device receives the video 5 pushed by the terminal device E. Since the total number of playback windows has reached the upper limit, it can display the second prompt pop-up window as shown in Figure 11. The second prompt pop-up window includes cancel control and a second replacement control. If the user selects the cancel control, the screen projection is automatically canceled, that is, the playback of the video 5 pushed by the terminal device E is cancelled, and the playback interface does not update.

In some implementations, the second replacement control is generated according to the current display state. Referring to the examples in FIG. 5 and FIG. 11 , the current display state includes four video windows, and the corresponding generated The second replacement control includes four sub-controls, which are "replace the first video window", "replace the second video window", "replace the third video window" and "replace the fourth video window". When the total number of windows reaches the upper limit, and the user does not want to cancel the playback of the currently newly pushed media data, he can choose one of the four video windows as the target playback window according to his interests and intentions. For example, if the user does not want to continue watching the third video, he can choose If the "replace the third video window" sub-control is set, as shown in Figure 12, the playback window 3 stops playing the video 3, but is replaced by the playback status of the playback video 5, the playback window 1, the playback window 2 and the playback window 4 constant. This embodiment can ensure that the total number of playback windows always meets the upper limit requirement, and through the replacement mechanism, the newly received projection screen resource can replace a certain channel of old media resources, avoiding the situation that the new resource is canceled before it starts playing.

In the aforementioned embodiment, when the total number of playback windows reaches the upper limit, the second prompt pop-up window is used to inquire whether to cancel screen projection or perform resource replacement. In other exemplary implementations, an update strategy is also provided: when the display device receives a new video push, if the current number of playback windows is equal to the preset upper limit value, then according to the preset rules, from the current playback window Filter out the target playback window from all windows, and make the target playback window stop playing the original media data, and replace it with the newly received data to be played, and the playback status of other playback windows remains unchanged. In this embodiment, when the number of windows reaches the upper limit, the second prompt pop-up window is not displayed to ask the user, but the target playback window for resource replacement is automatically screened out according to preset rules.

In some implementations, for example, the preset rule is to replace the previous projection resource. Taking the playback scene in FIG. 5 as an example, for the currently pushed video 5, the previous push is video 4. The corresponding playback window 4 is selected as the target playback window, and the playback window 4 stops playing the original video 4, and is replaced by loading and starting video 5; for another example, the preset rule is to replace the current oldest media resource in the playback interface, That is, the resource that is pushed the earliest among the multi-channel resources is to replace the playback window 1 with the playback video 5; as another example, the preset rules can also be set from other angles, such as replacing the resource with the lowest playback volume/popularity, and replacing the video duration The shortest resources, replace resources that appear to freeze or even black screen, etc. Specifically, the preset rules may be set and adjusted according to actual application scenarios, which are not specifically limited in this application.

As mentioned above, for media data of audio type and image type, multiple channels of audio data share the same audio player and audio window, and multiple channels of image data share the same image player and image window. For the convenience of generalization and unified description, This application names the corresponding resource type when multiple channels of the same type of media data share the same player and window as the target type, and the target type is audio type or image type. When the display device receives the media resource of the target type newly pushed by the terminal device, the update strategy is slightly different from the video type.

In some embodiments, if it is detected that the data to be played newly pushed by the terminal device belongs to the target type, it is further detected whether there is a target play window corresponding to the same target type in the playback interface. If there is a target playback window corresponding to the same target type in the playback interface, then stop playing the original media data currently played by the target playback window, and be replaced by the data to be played, and the playback status of other playback windows remains unchanged. Taking Figure 6 as an example, in the playback interface, playback window 1 plays video 1, and playback window 2 plays audio 1. At this time, the display device receives audio 2 pushed by the terminal device. Audio 2 belongs to the audio type in the target type, and in the playback interface If there is a playback window 2 corresponding to the audio type, then the playback window 2 is the target playback window, so that the playback window 2 stops playing audio 1, and is replaced by playing audio 2 and synchronously displaying the playback information of audio 2, and the playback window 1 keeps playing The status is unchanged.

In some implementations, if there is no target playback window corresponding to the same target type in the playback interface, it is further detected whether the number of playback windows currently included in the playback interface is less than a preset upper limit. If the number of playback windows does not reach the upper limit, it is supported to update the playback interface with a new mechanism, that is, create and add a playback window in the playback interface as the target playback window, display the newly received media data in the target playback window, and other playback windows The playback status of the player remains unchanged, and, according to the updated total number of playback windows, re-match the window layout of the playback interface. Taking Figure 7 as an example, in the current playback interface of the display device, the playback window 1 displays video 1, the playback window 2 displays video 2, and the playback window 3 displays images. At this time, the display device receives the audio 1 pushed by the terminal device F, and in the playback interface There is currently no audio window, create and add a playback window 4, and display the playback information of the pushed audio 1 in the playback window 4, and control the power amplifier to start playing audio 1 synchronously.

In some implementations, if there is no target playback window corresponding to the same target type in the playback interface, and the number of playback windows does not reach the upper limit, a third prompt pop-up window is displayed. Referring to the example in Fig. 13(a), in this example, the display device currently includes two channels of video and one channel of images, the playback window 1 plays video 1, the playback window 2 plays video 2, and the playback window 3 plays images. At this time, the display device receives A channel of audio 1 is pushed, and the third prompt pop-up window displayed at this time includes the second new control and the third replacement control. If the user selects the second newly added control, as shown in the example in FIG. 13(b), create and add a playback window 4 for playing audio 1, and adjust the window layout to a four-square grid.

In one example, the third replacement control is generated according to the current display state, including three sub-controls of "replace the first video window", "replace the second video window" and "replace the image window". 3. Replace control Select a target playback window, replace the target playback window with Play Audio 1, and the playback status and window layout of other windows remain unchanged. For example, the user selects the "replace the first video window" sub-control, as shown in Figure 13(c), and the playback window 1 is selected as the target playback window, then the playback window 1 stops playing video 1 and is replaced by playing audio 1.

In some implementations, if there is no target playback window corresponding to the same target type in the playback interface, and the number of playback windows reaches the upper limit (take the preset upper limit equal to 4 as an example), a third prompt pop-up window is displayed. Referring to the example in Figure 14(a), in this example, the display device currently includes three channels of video and one channel of images, the playback window 1 plays video 1, the playback window 2 plays video 2, the playback window 3 plays images, and the playback window 4 plays video 3, At this time, the display device receives a push of audio 1, and the third pop-up window displayed at this time includes a second cancel control and a third replacement control. If the user selects the second cancel control, the loading and playing of audio 1 is canceled, and the playback interface remains the status quo. The third replacement control is generated according to the current display state, including four sub-controls of "replace the first video window", "replace the second video window", "replace the third video window" and "replace the image window". The user selects a target playback window through the third replacement control, and replaces the target playback window with Play Audio 1, and the playback status and window layout of other windows remain unchanged. For example, the user selects the "replace image window" sub-control, as shown in Figure 14(b), and the playback window 3 is selected as the target playback window, then the playback window 1 stops displaying images and is replaced by playing audio 1.

Of course, when the number of windows reaches the upper limit, the third prompt pop-up window may not be displayed for inquiry, but the target playback window may be selected from the current playback interface according to the aforementioned preset rules. For specific implementation methods, please refer to the implementation of the aforementioned video type section example, which will not be repeated here. When the display device receives an image push, the update policy of the playback interface is the same as when it receives an audio push.

Referring to the example in Figure 15(a), the display device currently displays one video and one image, the playback window 1 plays video 1, and the playback window 2 displays image 1. At this time, the display device receives image 2 pushed by the terminal device K, and it will play Window 2 is used as the target playback window, as shown in Figure 15(b), stop displaying image 1 in playback window 2, and load image 2 to replace display in playback window 2, and the playback status and window layout of playback window 1 remain unchanged.

Referring to the example in Figure 16(a), the display device is currently broadcasting one video and one audio, the playback window 1 plays audio 1, and the playback window 2 plays video 1. At this time, the display device receives image 1 pushed by terminal device A, and displays The third prompt pops up. If the user selects the second added control in the third pop-up window, as shown in Figure 16(b), a playback window 3 for displaying image 1 is created and added, and the window layout is adjusted. The third replacement control includes two sub-controls of "Replace Video Window" and "Replace Audio Window". If the user selects the "Replace Audio Window" sub-control, the playback of Audio 1 will stop, and the playback window 1 will stop displaying the playback information of Audio 1. And be replaced by display image 1, as shown in Figure 16(c), at this time the playback interface is updated to display image 1 in playback window 1, and display video 1 in playback window 2, and the window layout can remain unchanged.

Referring to the example in Figure 17(a), the display device is currently broadcasting four channels of video, the playback window 1 displays video 1, the playback window 2 displays video 2, the playback window 3 displays video 3, and the playback window 4 displays video 4, at this time the display device After receiving the image 1 pushed by the terminal device B, a third prompt pop-up window is displayed. If the user selects the second cancel control in the third prompt pop-up window, the display device cancels the display of image 1, and the playback interface maintains the status quo. The third replacement control includes four sub-controls "replace the first video window", "replace the second video window", "replace the third video window" and "replace the fourth video window". Replace the "Second Video Window" sub-control, as shown in Figure 17(b), the playback window 2 stops displaying video 2, and is replaced by loading and displaying image 1, playback of playback window 1, playback window 3, and playback window 4 The state as well as the window layout remain unchanged.

In some embodiments, if the display device obtains the data stream through the URI/URL pushed by the terminal, assuming that the display device currently displays the upper limit of four videos, the playback window 1 displays video 1, the playback window 2 displays video 2, and the playback window 3 displays video 3. The playback window 4 displays the video 4. At this time, the display device receives the URI of the video 5 pushed by a certain terminal. Referring to FIG. 11, the second prompt pop-up window is displayed. If the user selects the cancel control in the second prompt pop-up window, Then the display device does not need to use the URI to obtain the video data of video 5; if the user selects the "replace the fourth video window" sub-control in the second prompt pop-up window, the display device stops obtaining and receiving the video stream of video 4, and passes The newly received URI locates the resource of video 5, and loads and displays the obtained video stream of video 5 in the playback window 4, thereby completing the resource replacement of the playback window 4.

In some exemplary implementations, the user can switch the window focus by operating the remote control. Optionally, the display device can highlight the focus window. For example, in FIG. 18, a rectangular focus frame is set on the focus window, and For example, the focus window can be scaled appropriately so that the user can intuitively distinguish where the focus is. As shown in FIG. 18 , the playback window 3 has focus after the user operates.

Since the user generally selects a target power amplifier to output audio from currently connected and available power amplifier devices, the display device generally only supports single-channel audio playback. Video resources can also be subdivided into pure video resources and audio and video resources. Pure video resources have only video images but no audio content. Audio and video resources have audio content synchronized with video images. For example, movie images have corresponding vocal lines. When resources are played in multi-window mode, for example, currently there are 3 video windows and 1 audio window, if at least one of the 3 video windows plays audio and video resources, it means that at least two playback windows can play sound. It involves the question of which channel the target power amplifier outputs.

In some implementations, the output of the target power amplifier can be determined by referring to the real-time window focus, and when the operation of switching the window focus is received, the window where the current focus is located (for ease of distinction and description, this application is named the first playback window) , and determine whether the media data currently played by the first playback window (for ease of distinction and description, this application is named the first media data) contains audio content. For audio and video resources, pure audio resources, images with background music and other types of media data, all contain audio content; for pure video resources, images without background music, plain text and other types of media data, audio content is not included .

In some implementation manners, if the first media data being played in the current focus window includes audio content, the target power amplifier is controlled to output the audio content. If the first media data being played in the current focus window does not contain audio content, you can turn on the mute mode, or control the target power amplifier to jump to play the audio data associated with the second playback window, which is currently displaying audio playback information The playback window, that is, the only audio window shared by multiple audio resources, that is, when the focus window cannot output sound, it will remain silent or output the audio data currently played by the audio player.

Taking Figure 13(b) as an example, in the updated playback interface, play window 1 plays video 1, play window 2 plays video 2, play window 3 plays images, and play window 4 plays audio 1, assuming that video 1 is a pure video resource , Video 2 is an audio and video resource. If the current focus is on playback window 1, since video 1 has no audio content, playback window 1 will continue to play video 1, and the target power amplifier has no audio output (equivalent to a mute state), or, in order to avoid poor viewing experience caused by silent output, you can also Control the target power amplifier to output audio 1; if the current focus is on playback window 2, because video 2 is an audio and video resource, it has audio content, so control the target power amplifier to output the audio content contained in video 2; if the current focus is on playback window 3, because pure If the image has no audio content, the playback window 3 can continue to display the image, the target power amplifier is muted, or switched to output audio 1; if the current focus is on the playback window 4, since the playback window 4 is playing audio 1, the target power amplifier is controlled to output audio directly 1.

In some implementations, the target power amplifier only outputs the audio content of a window resource. For example, when the target power amplifier is outputting the audio content of video 2, it will stop outputting audio 1. If the user switches the focus to the playback window 4, the target power amplifier will switch To output audio 1, stop outputting the audio content of video 2, so as to avoid playback conflicts between audios.

In other exemplary implementations, audio output is not limited to the aforementioned focus control mode. For example, if there is currently an audio window in the playback interface, the audio data associated with the audio window will be output first; Output the audio content in the audio and video resources. When there are multiple audio and video resources, for example, the earliest audio and video resources can be output first, or the latest audio and video resources can be output first. The audio output control mode in the scenario of synchronous playback of multiple resources is not limited to that described in the embodiments of the present application.

The aforementioned embodiments and UI presentations mainly focus on resources of video, audio and image types. In other playback scenarios, resource types may also be text, graphic, and so on. Optionally, for text-type resources, for example, a text window can be created for display, or in the form of floating subtitles, which can be scrolled and displayed on the upper layer of the playback interface; for graphic-type resources, for example, a graphic window can be created for display. For the text type and graphic type, if the playback is performed in window mode, the update strategy can refer to the aforementioned image type, which will not be repeated here.

It should be noted that, for resource types not mentioned in this application, a matching playback form can be adopted according to the characteristics of different types of resources. In addition, according to different playback scenarios, preset upper limit value settings, user interaction operations, and various types of resources pushed by the terminal, an adaptive update strategy is formulated, and the update strategy can be appropriately modified and extended to achieve more Feasibility in application scenarios. The UIs involved in the drawings and embodiments of the present application are only exemplary, and the actual design and application shall prevail.

Referring to the descriptions of the foregoing embodiments, in some implementations, referring to FIG. 19 , the present application provides a method for synchronously playing resources. The method is executed by the controller 250 in the display device, and includes the following procedural steps:

Step S01, when receiving data to be broadcast pushed by a terminal device, determine an update strategy for the playback interface according to the resource type of the data to be broadcast and the current display state of the playback interface. Wherein, the current display state includes the number of playback windows currently included in the playback interface, and the resource type corresponding to each playback window.

Step S02, load and play the data to be played in the target play window indicated by the update policy.

Referring to the descriptions of the foregoing embodiments, in some implementations, referring to FIG. 20 , the present application also provides another more detailed resource synchronous playback method, including the following procedural steps:

Step S101, when receiving the data to be broadcast pushed by the terminal device, detect the resource type of the data to be broadcast. If the to-be-broadcast data is of video type, branch line logic 1 is executed; if the to-be-broadcast data is of target type such as audio or image, branch line logic 2 is executed.

Branch line logic one, including steps S102 to S106:

Step S102, judging whether the current number of playback windows is less than a preset upper limit. If the judgment result of step S102 is yes, then execute step S103 and step S104, otherwise, execute step S105 and step S106. That is, when the number of windows does not reach the upper limit, the window addition mechanism can be implemented; if the number of windows has reached the upper limit, the window replacement mechanism can be implemented, so as to ensure that the number of windows does not exceed the preset upper limit.

Step S103, adding a new playing window in the playing interface as a target playing window, loading and playing the data to be played in the target playing window, and keeping the playing states of other playing windows unchanged.

Step S104, re-matching the window layout of the playback interface according to the updated total number of playback windows.

Step S105, acquiring the target playback window selected by the user for resource replacement.

Step S106, stop playing the original media data before the target playback window is selected, make the target playback window display the data to be played, and keep the playback status and window layout of other playback windows unchanged.

Branch line logic two, including step S107 to step S113:

Step S107, detecting whether there is an object playing window corresponding to the same object type in the playing interface. If the judgment result of step S107 is yes, then step S108 is executed, otherwise, step S109 is executed.

Step S108, stop playing the original media data currently played by the target play window, make the target play window display the data to be played, and keep the play status and window layout of other play windows unchanged.

Step S109, judging whether the current number of playback windows is less than a preset upper limit. If the judgment result of step S109 is yes, then execute step S110 and step S111 , otherwise, execute step S112 and step S113 . That is, when the number of windows does not reach the upper limit, the window addition mechanism can be implemented; if the number of windows has reached the upper limit, the window replacement mechanism can be implemented, so as to ensure that the number of windows does not exceed the preset upper limit.

Step S110, adding a new playback window in the playback interface as a target playback window, loading and playing the data to be played in the target playback window, and keeping the playback status of other playback windows unchanged.

Step S111, adjusting the window layout of the playback interface according to the updated total number of playback windows.

Step S112, acquiring the target playback window selected by the user for resource replacement.

Step S113, stop playing the original media data before the target playback window is selected, make the target playback window display the data to be played, and keep the playback status and window layout of other playback windows unchanged.

In the method illustrated in FIG. 20 and FIG. 19 , for adding or replacing a window, it is possible to query by displaying a prompt pop-up window and support the user to select a target window for resource replacement, or to filter the replacement target according to preset rules.

The user can project the content on the terminal to the display device, and watch the projected content through the display device, which can provide the user with a better viewing experience. However, after the display device has been connected to one terminal, another terminal also wants to connect to the display device. In the related art, this is achieved by cutting off the connected terminal and displaying on the display interface the content of the latest terminal connected to the display device. . In this way, the connected terminal is suddenly disconnected, and the display content on the display device changes suddenly, which brings bad experience to the user.

For ease of description, in this embodiment of the application, the user interface of the display device mentioned is the first user interface, the user interface of the first terminal is the second user interface, the first terminal is a terminal newly connected to the display device, and the second The terminal is a terminal already connected to the display device.

In order to solve the above-mentioned technical problems, an embodiment of the present application provides a display method for multi-channel projection, which provides the user with selectable controls on the second user interface, so that the user can select the display mode corresponding to the control according to the needs , displaying the screen projection content of the first terminal on the first user interface according to the display manner. Specifically, the display method may include adding a screen display, or replacing a certain path of the second terminal screen display, and the like.

As shown in FIG. 21, FIG. 21 is a flow chart of a multi-channel projection display method according to some embodiments, including:

In the embodiment of the present application, the terminal can be a mobile terminal such as a mobile phone, a tablet computer, a smart wearable device (such as a smart bracelet, etc.), a notebook computer, or a device with screen projection and display functions such as a desktop computer and a smart TV.

The terminal receives user input for connecting to the display device, and sends a connection instruction to the display device.

In this embodiment of the present application, the screen projection between the first terminal and the display device may refer to displaying on the display device a picture played in real time on the first terminal. The screen projection can also refer to when the first terminal plays a certain audio and video, displaying the content of the audio and video on the display device, the first terminal sends the media asset address of the played audio and video to the display device, and the display device Play the audio and video according to the media asset address. In some embodiments, the screen projection includes recording the screen played in real time on the first terminal, the first terminal sends the recorded data to the display device, and the display device plays the data.

In an example, when the first terminal is playing audio and video, the screen projection control is called out through a user operation, and the user presses the screen projection control through the touch screen. The first terminal searches for a screen-projecting device, displays the screen-projecting device in a list on the second user interface, and the user presses the screen-projecting device control, and if the screen-projecting device control corresponds to the display device, then The mobile terminal triggers the connection action and sends a connection instruction to the display device.

In one example, the user can press the mirroring screen projection control on the user interface of the mobile phone terminal, and the display device displays a device that can mirror the screen projection with the mobile phone terminal, and the user presses the device control. Correspondingly, at this time, the mobile phone triggers a connection action and sends a connection command to the display device.

In some embodiments, the user can control the display of the screen projection control on the second user interface through related operations. In one example, when media asset data is played in full screen on the terminal, the user can touch the display screen, and functional controls will be displayed on the interface at this time, and the functional controls can include pause/play control, screen projection control, and/or, close / Turn on the barrage control, etc. The user can touch the projection control set on the user interface, and the page jumps to display a list of connection terminals to be selected. The list includes device controls that can correspond to the display device projection device. Select a device control and send a connection command to The device that the device control corresponds to. In some embodiments, the device may be a display device. In another example, the user slides down/up the screen to call out the function menu of the terminal, and the function menu displays a screen mirroring control, that is, a screen mirroring control, etc., selects the screen mirroring control, and sends a connection command to the display equipment.

When the display device receives the connection instruction sent by the first terminal, it acquires the number of second terminals currently connected to the display device. In some embodiments, the number of the second terminals may be multiple, and screen projection contents of multiple terminals may be simultaneously displayed on the display of the display device. In some embodiments, the number of the second terminals may be stored in the display device in the form of a list, the list records the information of the connected second terminals, and records the numbers of all the second terminals connected to the second terminals, when the When the display device is connected to a terminal, the number of the second terminal counted will be increased by one. When the display device is disconnected from the second terminal, the screen projection content corresponding to the second terminal on the display device will no longer be displayed, and the list will be updated at the same time. Reduce the number of second terminals by one. When the process of the second terminal exits abnormally or is killed (kill), the display device disconnects the connection of the corresponding second terminal, and the screen projection content corresponding to the second terminal on the display device is no longer displayed, and the list will be updated while updating the list. The number of two terminals is reduced by one. In this embodiment, obtaining the number of second terminals may be implemented by calling the list and searching for the number of second terminals recorded in the list.

The display device sends indication information to the terminal, where the indication information is used to indicate the number of the second terminal to the first terminal, and trigger the first terminal to generate an interaction window according to the number of the second terminal, where the interaction window includes controls. In some embodiments, assuming that the number of second terminals is n, the indication information may be defined as MIRROR_PEERNUM:n.

The terminal receives the indication information sent by the display device, determines the control according to the number of the second terminal, and displays the control on the second user interface in the form of an interactive window.

In the embodiment of the present application, the number of the second terminal can determine the controls displayed in the interactive window of the first terminal, and the user can select different controls to realize different display modes of the screen projection content of the first terminal on the display device , which can include adding a screen display, or replacing a terminal screen display of a certain channel, etc.

When the number of the second terminal is the first preset number, the first terminal determines that the control includes a new window control. Exemplarily, when the first preset number is 0, the second terminal number is 0. At this time, there is no second terminal connected to the display device. On the interface, the controls at this time include newly added window controls.

As shown in Figure 22, the screen projection content of the second terminal is not displayed on the first user interface, the display device sends instruction information to the first terminal, and the first terminal receives the instruction information sent by the display device, the instruction information includes the second The number of terminals. At this time, the control determined by the first terminal according to the number of the second terminal is a new window control. In some embodiments, selecting the new window control can make a new playback window on the first user interface for displaying Screencasting content of the first terminal. As shown in FIG. 23 , in FIG. 23 , the controls on the second user interface include a new window control 701, and the user selects the new window control 701 to realize the real-time display of the first terminal to the display device. As shown in FIG. 24, the user can see all content displayed on the interface with the first terminal. In another example, as shown in FIG. 25, the controls on the second user interface include a new window control 901. After the user selects the new window control 901, the link address of the media asset data currently played by the first terminal is sent to A display device. The display device finds the media asset data according to the link address, and plays the media asset data, as shown in FIG. 26 , which shows the same media asset data played by the first terminal.

In some embodiments, when the display device plays a local data resource or a data resource directly obtained from a server, the data resource is not the screen projection content sent by the second terminal, and when the number of the second terminals is the first preset number When , it is determined that the controls include new window controls and replacement controls. When the new window control is selected, a new window is added on the first user interface, and the playback window corresponding to the resource originally played by the device will be displayed, and displayed on the first user interface at the same time as the newly added playback window. There are two playback windows on the screen, and the original playback resources and the screen projection content of the first terminal are displayed on the first user interface at the same time. When the replacement control is selected, the first user interface displays the originally played resource and displays the projected screen content of the first terminal, which can be implemented in the following manner. The playing window is canceled, and the screen projection content of the first terminal is displayed at the canceled playback window, or the playback window corresponding to the original playback resource is retained, and the screen projection content of the first terminal is displayed in the playback window.

When the number of the second terminal is greater than the first preset number and less than the second preset number, the first terminal determines that the control includes a new window control and a replacement control corresponding to the second terminal. In the embodiment of the present application, in order to avoid excessive operating pressure of the display device or to allow users to see clear pictures and improve user experience, the display device may limit the number of terminals connected to it, and the second preset number is The upper limit of the number of playing windows on the user interface, for example, the second preset number may be four.

In the embodiment of the present application, when the display device plays local data resources or data resources obtained directly from the server, the replacement control can also replace the originally played resources on the first user interface. In some embodiments, when ensuring that the playback windows on the first user interface are not larger than the second preset number, the original playback resources can be displayed on the newly added window while displaying the screen projection content of the first terminal. on the first user interface.

In some embodiments, when there is at least one second terminal connected to the display device, the screen projection content of the second terminal is displayed on the first user interface, and at this time, the user can be provided with the screen projection content of the first terminal on the display device A variety of display methods, one can be to add a new window on the display interface, and display the screen projection content of the first terminal in the newly added window; another method can display the screen projection content of the first terminal in an alternative way , specifically, the user can, according to his or her own wishes, replace the screen projection content that does not want to continue playing in the user interface. In some embodiments, when identifying replaceable screen projection content, it can be identified by connecting the terminal to the display device in order . The replacement control can be marked as replacing the first way, replacing the second way, and replacing the third way when naming. When the first channel is selected, the screen projection content of the first terminal connected to the display device will be replaced with the screen projection content of the first terminal. In some embodiments, when the replacement control is selected, the display device no longer displays the screen projection content of the terminal corresponding to the replacement control, and displays the screen projection content of the first terminal at the position where the screen projection content of the original connected terminal is displayed.

The display device sends instruction information to the first terminal, and the first terminal receives the instruction information. If the display device has a second terminal connected, as shown in Figure 26, the screen projection content of the second terminal is displayed on the first user interface At this time, the first terminal determines the controls according to the number of the second terminals. As shown in FIG. 27 , a new window control 1101 and a replacement first-way control 1102 are displayed on the second user interface. When the user selects the new control 1101, a new playback window is added on the first user interface. The newly added playback window is used to play the projected content, as shown in Figure 28. After the new playback window is added, the current first user interface There are two playback windows on the user interface, and the two playback windows are distributed on the user interface. The size of the two playback windows can be set to the same size, and the playback content of the original playback window does not change.

In some embodiments, the size of the playback window can be manually adjusted by the user operating the control device, so that when the user is more interested in the screen projection content in a certain playback window, the playback window can be enlarged, and the corresponding playback window The playback content can occupy a larger display interface, which is convenient for users to watch.

In some embodiments, if the user chooses to replace the first control 1102, the screen projection content of the original connected terminal will no longer be played on the first user interface, but the screen projection content of the first terminal will be played, for example, as shown in Figure 29 shown. The connection between the original terminal and the display device is disconnected.

The display device sends instruction information to the first terminal, and the first terminal receives the instruction information. If the display device has two second terminals connected, as shown in Figure 28, the screen projection contents of the two second terminals are displayed on the first terminal. On the user interface, the controls determined according to the number of the second terminals at this time, as shown in FIG. 30 , display a new window control 1401 , a replacement first control 1402 and a replacement second control 1403 on the second user interface. If the window control 1401 is selected, a playback window is added in the first user interface, as shown in FIG. 31 , there are currently three playback windows in FIG. 31 . Select and replace the first control 1402, and then the first user interface is shown in FIG. 32 . Select and replace the second-way control 1403, and then the first user interface is shown in FIG. 33 . In the embodiment of the present application, the user can disconnect the originally connected second terminal from the display device by selecting a different replacement control, and display the projected screen content of the first terminal at the position where the original projected screen content is displayed.

The display device sends instruction information to the first terminal, and the first terminal receives the instruction information. If the display device has three second terminals connected, as shown in Figure 31, the content of the screen projection of the three second terminals is displayed on the first terminal. On the user interface, the controls determined according to the number of the second terminals at this time, as shown in FIG. Replace the third control 1804. After the above controls are selected, the first user interface can refer to the interface changes when two second terminals are connected, and will not be described in detail here.

The display device sends instruction information to the first terminal, and the first terminal receives the instruction information, and when the number of the second terminals is a second preset number, it is determined that the control includes the control of no longer projecting the screen, and the connected terminal The corresponding replacement control. Exemplarily, the second preset number is four. If there are 4 second terminals connected to the display device, as shown in Figure 35, the projection content of the four second terminals are all displayed on the first user interface. At this time, the controls determined according to the number of the second terminals are as follows: As shown in FIG. 36 , the second user interface displays a no-screen projection control 2001 , a replacement first control 2002 , a second replacement control 2003 , a third replacement control 2004 and a fourth replacement control 2005 . After the replacement control is selected, the first user interface can refer to the interface changes when two second terminals are connected, and will not be described in detail again. When the user selects the no-screen-casting control 2001, the first user interface does not change, and the first terminal and the display device do not perform the screen-casting operation.

In this embodiment of the present application, when the display device is connected to multiple terminals, there is no limit to the arrangement of the projection content of multiple terminals on the first display interface. For example, all the projection content of multiple terminals can be displayed horizontally. All the projected content can also be displayed vertically, part of the projected content can be displayed horizontally, and part of the projected content can be displayed vertically.

In some embodiments, the arrangement direction of the screen-casting content on the display interface of the first display device can be determined according to the display direction of the screen-casting content. For example, when the screen-casting content is a movie, the screen-casting content can be The content is displayed horizontally on the first display interface, which is convenient for the user to watch the movie on the display interface of the display device.

The first terminal receives a user input for selecting a control on the interaction window, and sends an instruction generated according to the control to the display device. In the embodiment of the present application, when a new window control is selected, a new window command is generated correspondingly, when a replacement control is selected, a replacement command is correspondingly generated, and when a no screen projection control is selected, a no screen projection command is correspondingly generated.

The first receiving terminal sends an instruction generated according to selecting the control in the interaction window, and controls the display to display the projected screen content according to the instruction. In this embodiment of the application, the first terminal only needs to ensure that the screen projection content is sent to the display device before the display device displays the projection content, and this application does not limit the timing of sending the screen projection content.

After receiving the corresponding instruction from the first terminal based on the control, the display device adapts and displays the first user interface according to the user's selection. If the user selects a new window control, the first user interface and the second terminal screen Create a new playback window in the layout; if the user chooses to replace a certain control, the projection content corresponding to the replacement control will no longer be displayed, and will be displayed at the position where the original projection content connected to the second terminal is displayed. Screencasting content of the first terminal. If the user selects the no-screen-casting control, the screen-casting content of the first terminal will not be displayed on the first user interface.

In some embodiments, the selected control can be implemented by the user pressing the position corresponding to the control on the touch screen of the first terminal. In other embodiments, the selected control can be realized after the user presses the position corresponding to the control on the touch screen of the first terminal, and then press the second The corresponding position of the control is determined on the touch screen of a terminal. As shown in FIG. 23, the selection of the new window control can be realized after the user presses the position corresponding to the new window control on the touch screen, and then presses the position corresponding to the control.

In some embodiments, the instruction includes an instruction to add a new window, and the step of controlling the display to display the projected screen content according to the instruction includes: adding a new playback window on the current user interface, and displaying the projected screen in the newly added playback window content. The playback content in other playback windows on the first user interface remains unchanged. In order to ensure the rationality of screen area allocation, the positions of other playback windows may change, so that when a new playback window is added, all playback windows are within the user's The layout on the interface is more reasonable.

In some embodiments, the first user interface includes at least one play window, and the screen projection content of the terminal connected to the display device is played in the play window;

The instruction includes a replacement instruction, the replacement instruction includes replacement information, the replacement information is used to indicate the playback window information corresponding to the replaced second terminal, the playback window information can be the order of the playback window in all playback windows, and the replacement The information is determined by the replacement control selected by the user. Exemplarily, when the user selects to replace the first control, the replacement information is the playback window that is arranged first among all the playback windows. The step of controlling the display to display the projected screen content according to the instruction includes: disconnecting the second terminal corresponding to the replacement information, creating a new play window at the play window where the original projected screen content is played, and converting the first The screencast content of the terminal is displayed in the newly created playback window.

In some embodiments, the instruction includes an instruction not to cast the screen, and the step of controlling the display to display the screen projection content according to the instruction includes: rejecting the connection with the first terminal, and maintaining the display state of the user interface unchanged.

In some embodiments, the method further includes: judging whether the condition of not sending the indication information is met; if the condition is met, not sending the indication information; if the condition is not met, sending the indication information.

In some embodiments, the condition for not sending the indication information includes that the upper limit value of the number of playing windows on the first user interface is a specific value. Exemplarily, the specific value is 0. In an example, when the specific value is 0, the display device only receives one terminal connection, at this time, the default is to replace the second terminal of the original connection, that is, the content of the second terminal of the original connection is no longer displayed, The screen projection content of the first terminal is displayed at the position where the projection screen content originally connected to the second terminal is displayed, and no instruction information needs to be sent at this time. In addition, if the display device does not currently have a second terminal connected to it, a new window is directly created on the first user interface, and the screen projection content is displayed in the newly created window. In this solution, no instruction information is also sent.

In the embodiment of the present application, displaying the screen projection content of the first terminal at the position where the screen projection content originally connected to the second terminal is displayed can be realized in various ways. In some embodiments, the connection of the second terminal currently corresponding to the display device is disconnected, and a playback window is created in its position for playing the screen-casting content of the first terminal. In some embodiments, the screen-casting content of the first terminal may be displayed by using an original play window connected to the screen-casting content of the second terminal.

In some embodiments, the terminal sends an instruction generated by the control selected by the user to the display device, and the command may be defined as follows: "MIRROR_INDEX: m", where m represents replacement information corresponding to the control selected by the user. m is -1 and means no longer screen projection control, m is 0 means add window control, m is 1 means replace the first control, m is 2 means replace the second control, m is 3 means replace the third control , m is 4 means to replace the fourth control.

The above content is the solution described from the perspective of the first terminal as the main body, and the solution is briefly described from the perspective of the display device. The display device receives the connection instruction of the first terminal, obtains the number of the second terminals, generates and sends indication information, the indication information is used to indicate the number of the second terminals to the first terminal, and triggers the first terminal to generate an interactive window according to the number of the second terminals receiving an instruction corresponding to the control generated according to the number of the second terminal selected by the user in the interaction window, and executing and controlling the display to display the screen projection content of the first terminal according to the instruction.

In the above embodiments, a multi-channel screen projection display method, display device and terminal, in some embodiments of the present application, the user can select the display mode of the screen projection content on the display device according to needs, so as to improve the user experience. The method includes: receiving a connection instruction from the first terminal, acquiring the number of the second terminal, wherein the second terminal is a terminal already connected to the display device; sending indication information, and the indication information is used to indicate the second terminal to the first terminal. the number of terminals, and trigger the first terminal to generate an interaction window according to the number of the second terminal, the interaction window includes controls, and the controls on the interaction window are determined according to the number of the second terminals; receiving the information sent by the first terminal according to selecting the The instruction generated by the control in the interactive window is used to control the display to display the screen projection content of the first terminal according to the instruction.

Based on the above display device 200, the user can project the content displayed on the mobile device to the display device 200 through mirroring, and display the screen content of the mobile device through the display device. Wherein, the mobile device may be the terminal device or terminal described in the above embodiments, for example, it may be the first terminal device or the second terminal device. Screen mirroring is to cast the screen content of the mobile device to the display device 200 in real time. Figure 37 and Figure 38 are schematic diagrams of the display effect of mirroring the screen of the mobile device to the display device 200 according to some embodiments, as shown in Figure 37 It is a schematic diagram of the display effect of the operation interface of the mobile device. FIG. 38 is a schematic diagram of the display effect of the mobile device viewing media resources, such as videos in the mobile phone album.

At present, the operating system installed on the mobile device is usually the Android system. The mobile device records the screen in real time through the screen recording application, and transmits the recorded video stream to the display device to realize mirroring and projection. However, the screen recording application is an application developed by a third party, that is, the screen recording application does not belong to the system level application, and can only allow the recording of the current display content of the operating system screen, but does not allow the operating system to record the currently playing audio data and system audio. Therefore, when the user mirrors the screen of the mobile device to the display device, the sound of the mobile device cannot be output through the display device. The corresponding audio is played through the display device 200, resulting in poor experience. In order to improve user experience, some embodiments of the present application provide a display device and an audio output control method for screen mirroring.

The audio output control process of mirroring screen projection provided by some embodiments of the present application is described below with reference to the accompanying drawings.

Fig. 39 is a schematic flowchart of a method for controlling audio output of screen mirroring according to some embodiments. The present application provides a display device in some embodiments, the display device 200 includes a display 260, a speaker, and a controller 250, the controller 250 is connected to the display 260 and the speaker in communication, and the controller 250 is configured It is the process of implementing the audio output control of mirroring screen projection in FIG. 39 . With reference to Figure 39, the audio output control process of the mirroring projection screen is as follows:

S701: In response to a screen projection request sent by a mobile device, control the display to display a playback interface, and control to start a Bluetooth protocol service of the display device, where the playback interface is used to display a screen projection interface of the mobile device.

In some embodiments, when the user needs to mirror and project the screen content displayed on the mobile device to the display device 200, the mobile device may send a screen projection request to the display device 200. The display device 200 receives and responds to the screen projection request sent by the mobile device, and establishes a connection with the mobile device based on a network connection protocol. After the connection is established, the mobile device can record the screen in real time, record and compress the currently displayed content on the screen into a video stream and transmit it to the display device 200, where the video stream is the screen projection interface of the mobile device. The display device 200 controls the display 260 to display the playback interface, and displays the screen projection interface of the mobile device on the playback interface.

In some embodiments, both the mobile device and the display device 200 can download the same screen-casting application, the mobile device and the display device 200 establish a connection relationship based on the screen-casting application, and the mobile device sends a video stream to the display device 200 through the screen-casting application. Here, in the screen casting application of the mobile device, you can set the "Start Screen Casting" control. When the mobile device receives the user's click on the "Start Screen Casting" control, the recording screen function will be triggered. While recording the screen in real time, the The recorded screen is compressed and transmitted to the display device 200 .

In some embodiments, when the display device 200 establishes a connection with the mobile device through the network protocol, the display device 200 needs to detect whether the Bluetooth protocol service of the machine is enabled at the same time. If it detects that the Bluetooth protocol service has not been started, it needs to immediately control the Bluetooth The protocol service is started, so that the display device 200 responds to the audio transmission request of the mobile device based on the Bluetooth protocol service.

In some embodiments, the display device controls the Bluetooth protocol service to connect with the mobile device through the Bluetooth A2DP (Advanced Audio Distribution Profile, Bluetooth audio transmission model agreement) protocol.

In some embodiments, the display device 200 may respond to screen projection requests from multiple mobile devices, that is, may receive screen projection connections from multiple mobile devices. If the mobile device includes a first mobile device and a second mobile device, when receiving a screen projection request from the second mobile device, the display device 200 controls the display and simultaneously displays the screen projection interface sent by the first mobile device And the screen casting interface sent by the first mobile device, wherein the time when the first mobile device sends the screen casting request to the display device is earlier than the time when the second mobile device sends the screen casting request to the display device. Fig. 40 is a schematic diagram of a display effect of mirroring and projecting screens of multiple mobile devices to the display device 200 according to some embodiments. As shown in FIG. 40 , taking three mobile devices as an example, the first mobile device, the second mobile device and the third mobile device all project screens with the display device 200 , and the display device 200 displays three mobile devices on the same screen on the display 260 . The screen projection interface of the device.

In some embodiments, when there are multiple mobile devices that send screen casting requests, for example, both the first mobile device and the second mobile device send screen casting requests to the display device 200, and the display device 200 receives the first mobile device and the second mobile device. 2. The screen projection request sent by the mobile device creates the same number of playback controls as the mobile device in the playback interface, wherein multiple playback controls belong to the same application. For example, multiple mobile devices and the display device 200 are installed with the same screen-casting application, and the display device 200 receives screen-casting requests from multiple mobile devices based on one application. When the first mobile device establishes a connection with the display device 200, the display device 200 creates a first playback control on the playback interface, and controls the display 260 to display the first playback control in full screen. When the display device 200 establishes a connection with the second mobile device again, the display device 200 creates a second play control on the playback interface; Create a third playback control. Wherein, the playback control is used to display the screen projection interface of the mobile device. The display effect of the three playback controls may be as shown in FIG. 40 . The display device 200 controls the playback controls to respectively acquire and display corresponding screen projection interfaces of the mobile device.

In some embodiments, when there are multiple mobile devices that send screen casting requests, for example, both the first mobile device and the second mobile device send screen casting requests to the display device 200, and the display device 200 receives the first mobile device and the second mobile device. 2. The screen projection request sent by the mobile device controls the user interface displayed on the display to display the same number of display windows as the mobile device, wherein the plurality of display windows respectively display the playback interface, and the plurality of display windows respectively belong to for different applications. For example, multiple mobile devices are installed with different screen projection applications. Taking three mobile devices as an example, they are respectively installed with NFC (Near Field Communication, approach communication), Airplay (wireless screen projection technology), DLNA (DIGITAL LIVING NETWORK ALLIANCE, digital life network alliance) screen projection applications of three projection protocols, and the display device 200 also establishes network connections with each mobile device based on the projection applications supporting these three projection protocols. The display device 200 respectively receives screen projection requests from corresponding mobile devices based on different applications. When the first mobile device establishes a connection with the display device 200, the display device 200 starts the corresponding application, and controls the display 260 to display the first display window in full screen, and the screen projection interface of the first mobile device is displayed in the first display window . When the display device 200 establishes a connection with the second mobile device again, the display device 200 starts a corresponding application, and controls the first display window to be placed on one side of the display 260, and displays the second mobile device on the other side of the display 260. A display window, where a screen projection interface of the second mobile device is displayed in the second display window. The display device 200 controls the display window to acquire and display corresponding screen projection interfaces of the mobile device.

S702: Receive an audio transmission request sent by the mobile device, and use the Bluetooth protocol service to establish a Bluetooth transmission channel with the mobile device.

In some embodiments, after the display device 200 establishes a connection with the mobile device based on the network connection protocol, the mobile device may send an audio transmission request to the display device 200, and notify the display device 200 that the transfer device will perform sound transmission through the audio transmission request. The audio transmission request content may be as follows: MP_BLUEA2DP_CONNECT_REQUEST (based on Bluetooth A2DP connection request).

In some embodiments, in the screen-casting application of the mobile device, a "transmit sound" control can be set, and when the transfer device receives a user's click on the "transmit sound" control, the mobile device is triggered to send an audio transmission request to the display device 200 , the display device 200 responds to the audio transmission request, and establishes a connection with the mobile device using the Bluetooth protocol service.

In some embodiments, if the mobile device includes a first mobile device and a second mobile device, when receiving an audio transmission request from the second mobile device, the display device 200 controls to stop receiving the audio transmission sent by the first mobile device. audio data, and receive the audio data sent by the second mobile device, so that the speaker stops the output of the audio data of the first mobile device, and controls the speaker to output the audio of the second mobile device data, wherein the time when the first mobile device sends the audio transmission request to the display device is earlier than the time when the second mobile device sends the audio transmission request to the display device. That is to say, when there are multiple mobile devices that send the screen projection request, the display device 200 responds to the last mobile device that sends the audio transmission request, and establishes a Bluetooth transmission channel with the last mobile device that sends the audio transmission request.

In some embodiments, the mobile device includes a first mobile device and a second mobile device, the first mobile device sending the audio transmission request to the display device earlier than the second mobile device sending the audio transmission request to the display device time of request. The display device 200 receives the audio transmission request sent by the second mobile device, and detects the current Bluetooth connection status of the display device 200 . If the display device 200 has established a Bluetooth transmission channel with the first mobile device, disconnect the Bluetooth transmission channel with the first mobile device. That is to say, after the display device 200 has established a Bluetooth connection with the first mobile device, the display device 200 needs to send an instruction to disconnect the Bluetooth connection to the first mobile device, and first disconnect the direct connection with the first mobile device, for example, The display device 200 sends MP_BLUEA2DP_DISCONNECT (to disconnect the Bluetooth A2DP connection) to the first mobile device.

In some embodiments, the first mobile device updates the Bluetooth connection status of the mobile device after receiving the instruction sent by the display device 200 representing disconnection of the Bluetooth connection.

In some embodiments, after the display device 200 disconnects the Bluetooth connection with the first mobile device, the second mobile device that subsequently sends an audio transmission request feeds back an instruction indicating that the Bluetooth connection is allowed, for example, the display device 200 sends an instruction to the second mobile device Send MP_BLUEA2DP_CONNECT_REQUEST_ACK (based on Bluetooth A2DP connection request confirmation). The second mobile device resends an audio transmission request to the display device 200 after receiving the instruction for running the Bluetooth connection sent by the display device 200 .

In some embodiments, the display device 200 receives the audio transmission request resent by the second mobile device according to the command indicating that the Bluetooth connection is allowed, and establishes a Bluetooth transmission channel with the second mobile device.

In some embodiments, when detecting the current Bluetooth connection state of the display device 200, if the display device 200 has not established a Bluetooth transmission channel with the first mobile device, the display device 200 responds to the first mobile device. The audio transmission request sent by the second mobile device directly establishes a Bluetooth transmission channel with the second mobile device.

In some embodiments, after receiving the audio transmission request sent by the mobile device, the display device 200 uses a default PIN code to pair with the mobile device in the background, so that when the first connection is made, the display device 200 and the mobile device None of the mobile devices mentioned above will pop up the Bluetooth pairing dialog box. That is, when the mobile device and the display device 200 are connected via Bluetooth, they are paired in the background through a default PIN code, and a Bluetooth transmission channel is established after the pairing is completed. In this way, if the mobile device and the display device 200 are connected with Bluetooth for the first time, neither the mobile device nor the display device 200 will pop up a Bluetooth pairing dialog box, and the user does not need to operate the Bluetooth pairing process, so as to avoid disturbing the user to watch the screen projection interface and improve user experience. experience.

S703: Receive audio data sent by the mobile device through the Bluetooth transmission channel, and control the speaker to output the audio data.

In some embodiments, after the display device 200 establishes a Bluetooth transmission channel with the mobile device, the mobile device 200 may send audio data to the display device 200 through the Bluetooth transmission channel, so that the display device 200 outputs corresponding audio data through the speaker.

In some embodiments, if the display device 200 establishes a connection with the mobile device through the network connection protocol, the screen content can be projected first, and then the audio data can be transmitted after the display device 200 establishes a Bluetooth connection channel with the mobile device. For example, if the user displays the screen projection interface of the mobile device and then displays the device 200, he can first search for the content that needs to be screened, such as, if he finds a video recorded by himself in the album, when the user clicks on the video to play, Then trigger the mobile device to establish a Bluetooth connection channel with the display device 200 , and then send the corresponding audio data of the video to the display device 200 for playback. Of course, the user can also trigger the mobile device to establish a Bluetooth connection channel with the display device 200 before playing the video.

The audio output control process of mirroring screen projection in some embodiments of the present application will be further described below with reference to the accompanying drawings.

Fig. 41 is a timing diagram of a method for controlling audio output of screen mirroring according to some embodiments. As shown in FIG. 41 , the display device 200 is connected to the first mobile device and the second mobile device for screen projection, and the screen contents of the first mobile device and the second mobile device are displayed on the display 260 of the display device 200 . The first mobile device first sends an audio transmission request to the display device 200, and after receiving the audio transmission request sent by the first mobile device, the display device 200 feeds back to the first mobile device an instruction indicating that Bluetooth connection is allowed. After receiving the instruction, the first mobile device and the display device are paired through the default PIN code. After the pairing is successful, the first mobile device and the display device 200 establish a Bluetooth connection channel, and the first mobile device sends audio data to the display device 200, and the display After receiving the audio data, the device 200 controls the speaker to play. At this time, the second mobile device sends an audio transmission request to the display device 200, and the display device 200 detects the current Bluetooth connection status of the machine, and after detecting that it has been connected to the first mobile device with Bluetooth, it first sends an indication message to the first mobile device. The instruction of opening the Bluetooth connection disconnects the Bluetooth connection with the first mobile device, and stops playing the audio data of the first mobile device. The first mobile device updates its own Bluetooth connection status after receiving the instruction sent by the display device 200 representing disconnection of the Bluetooth connection. After the display device 200 disconnects the Bluetooth connection with the first mobile device, it sends an instruction to the second mobile device indicating that the Bluetooth connection is allowed. After the second mobile device receives the instruction, both the second mobile device and the display device pass the default PIN code. After pairing, after the pairing is successful, the second mobile device establishes a Bluetooth connection channel with the display device 200, the second mobile device sends audio data to the display device 200, and the display device 200 controls the speaker to play after receiving the audio data.

In this application, when the display device and the mobile device are mirroring the screen, the mobile device divides the screen projection interface and the corresponding audio data into two channels and transmits them to the display device. The audio data of the mobile device is displayed through the Bluetooth transmission channel. Device output greatly improves user experience.

Some embodiments of the present application also provide an audio output control method for screen mirroring, the method includes: the display device 200 responds to the screen projection request sent by the mobile device, controls the display 260 to display a playback interface, and controls to start the display device Bluetooth protocol service, wherein the playback interface is used to display the screen projection interface of the mobile device. The display device 200 receives the audio transmission request sent by the mobile device, and uses the Bluetooth protocol service to establish a Bluetooth transmission channel with the mobile device. The display device 200 receives the audio data sent by the mobile device through the Bluetooth transmission channel, and controls the speaker to output the audio data.

In some embodiments, when there are multiple mobile devices that send screen casting requests, the display device 200 receives multiple screen casting requests sent by the mobile devices, and creates an image with the same data as the mobile device in the playback interface. A playback control, wherein multiple playback controls belong to the same application. The display device 200 controls the playback controls to respectively acquire and display corresponding screen projection interfaces of the mobile device.

In some embodiments, when there are multiple mobile devices that send the screen projection request, the display device 200 responds to the last mobile device that sends the audio transmission request, and establishes a Bluetooth transmission channel with the last mobile device that sends the audio transmission request.

The above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed above. Many modifications and variations are possible in light of the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various modified embodiments suitable for specific use considerations.

Claims

A display device comprising:

a display for displaying images;

a communicator for communicating with one or more terminal devices;

a controller coupled to the display and communicator and configured to:

receiving a screen projection request from a first terminal device among the one or more terminal devices,

In response to the screen projection request, an update policy of the display is determined according to the current display state of the display, so as to play resources corresponding to the screen projection request.
The display device according to claim 1, wherein the current display state of the display includes displaying one or more windows for displaying images, and the controller is further configured to:

In response to the number of the one or more windows being less than the preset upper limit, and the resource type of the resource corresponding to the screen projection request is video, creating a target window as a window for playing the resource corresponding to the screen projection request, and maintaining the state of the one or more windows; and

According to the updated total number of windows, re-layout the display of the one or more windows and the target window in the display area of the display.
The display device according to claim 1, wherein the current display state of the display includes displaying one or more windows for displaying images, and the controller is further configured to:

In response to the number of the one or more windows being less than the preset upper limit, and the resource type of the resource corresponding to the screen projection request is video, control the display to pop up a first prompt pop-up window, wherein the first prompt pop-up window including the first added control; and

In response to the selection operation of the first newly added control, create a target window as a window for playing the resource corresponding to the screen projection request, and keep the state of the one or more windows unchanged, and, according to the updated The total number of windows, rearranging the display of the one or more windows and the target window in the display area of the display.
The display device according to claim 1, wherein the current display state of the display includes displaying one or more windows for displaying images, and the controller is further configured to:

In response to the number of the one or more windows being less than the preset upper limit, and the resource type of the resource corresponding to the screen projection request is video, control the display to pop up a first prompt pop-up window, wherein the first prompt pop-up window includes the first A replacement control, in response to a selection operation on the first replacement control, use the window selected by the first replacement control as a target window for playing the resource corresponding to the screen projection request, and stop playing the target The original media data before the window is selected, so that the selected target window displays the resources corresponding to the screen projection request, and the playback status of other windows remains unchanged.
The display device according to claim 1, wherein the current display state of the display includes displaying one or more windows for displaying images, and the controller is further configured to:

In response to the number of the one or more windows being equal to the preset upper limit, and the resource type of the resource corresponding to the screen casting request is video, control the display to pop up a second prompt pop-up window, wherein the second prompt pop-up window includes a cancel control; and

In response to the selection operation of the cancel control, the playing of the resource corresponding to the screen projection request is cancelled, and the display is not updated.
The display device according to claim 1, wherein the current display state of the display includes displaying one or more windows for displaying images, and the controller is further configured to:

In response to the number of the one or more windows being equal to the preset upper limit, and the resource type of the resource corresponding to the screen projection request is video, then control the display to pop up a second prompt pop-up window, wherein the second prompt pop-up window A second replacement control is included, and in response to a selection operation on the second replacement control, the window selected through the second replacement control is used as the target window for playing the resource corresponding to the screen projection request, and the playback of the resource corresponding to the screen projection request is stopped. The original media data before the target window is selected, so that the selected target window displays the resources corresponding to the screen projection request, and the playback status of other windows remains unchanged.
The display device according to claim 1, wherein the current display state of the display includes displaying one or more windows for displaying images, the controller is further configured to:

In response to the number of the one or more windows being equal to the preset upper limit, and the resource type of the resource corresponding to the screen casting request is video, according to the preset rules, the current one or more windows from the display Screen out the target window as the resource corresponding to the screen projection request for playing; and

Stop playing the original media data before the target window is selected, make the selected target window display the resource corresponding to the screen projection request, and keep the playback status of other windows unchanged.
The display device according to claim 1, wherein the current display state of the display includes displaying one or more windows for displaying images, and the controller is further configured to:

In response to the resource type of the resource corresponding to the screen casting request being a target type, detecting whether there is a target window corresponding to the same target type in the one or more windows, wherein the target type is audio or image;

In response to the target window corresponding to the same target type in the one or more windows, stop playing the original media data currently played by the target window, make the target window display the resource corresponding to the screen projection request, and other windows The playback state of the .
The display device according to claim 8, wherein the current display state of the display includes displaying one or more windows for displaying images, and the controller is further configured to:

In response to no object window corresponding to the same object type in the one or more windows, detecting whether the number of the one or more windows is less than a preset upper limit;

In response to the number of the one or more windows being less than a preset upper limit, creating a target window as a window for playing resources corresponding to the screen projection request, and keeping the state of the one or more windows unchanged, and, according to After updating the total number of windows, rearrange the display of the one or more windows and the target window in the display area of the display.
The display device according to claim 8, wherein the current display state of the display includes displaying one or more windows for displaying images, and the controller is further configured to:

In response to no object window corresponding to the same object type among the one or more windows, detecting whether the number of the one or more windows is less than a preset upper limit;

In response to the number of the one or more windows being equal to the preset upper limit, control the display to pop up a third prompt pop-up window, the third prompt pop-up window includes a third replacement control; in response to the third replacement control The selected operation is to use the window selected by the third replacement control as the target window for playing the resource corresponding to the screen projection request, stop playing the original media data before the target window is selected, and make the selected The target window plays the resource corresponding to the screen projection request, and the playback status of other windows remains unchanged.
The display device according to claim 1, wherein the current display state of the display includes displaying one or more windows for displaying images, and the display device further includes a built-in power amplifier or an interface for connecting to an external power amplifier, so The above controller is further configured to execute:

In response to the resource type of the resource corresponding to the screen projection request being audio, display audio playback information in a target window that is a window for playing the resource corresponding to the screen projection request, and control the built-in power amplifier or external power amplifier to play audio data synchronously ;
The display device according to claim 1, wherein the current display state of the display includes displaying one or more windows for displaying images, and the display device further includes a built-in power amplifier or an interface for connecting to an external power amplifier, so The above controller is further configured to execute:

In response to the operation of switching window focus, obtain the first window where the current focus of the one or more windows is located, and determine whether the first media data currently played by the first window contains audio content;

In response to the first media data including audio content, control a built-in power amplifier or an external power amplifier to play the audio content.
The display device according to claim 1, wherein the current display state of the display includes displaying one or more windows for displaying images, and the display device further includes a built-in power amplifier or an interface for connecting to an external power amplifier, so The above controller is further configured to execute:

In response to the operation of switching window focus, obtain the first window where the current focus of the one or more windows is located, and determine whether the first media data currently played by the first window contains audio content;

In response to the fact that the first media data does not contain audio content, turn on the silent mode, or control the built-in power amplifier or external power amplifier to jump to play the audio data associated with the second window, wherein the second window is the one or A window that currently displays audio playback information among multiple windows.
The display device according to claim 1, wherein the current display state of the display includes displaying one or more windows for displaying images, and the display device further includes a built-in power amplifier or an interface for connecting to an external power amplifier, so The above controller is further configured to execute:

In response to the operation of switching window focus, obtain the first window where the current focus of the one or more windows is located, and determine whether the first media data currently played by the first window contains audio content;

In response to the fact that the first media data does not contain audio content, control the built-in power amplifier or the external power amplifier to jump to play the audio data associated with the second window, wherein the second window is currently displayed in the one or more windows Window for audio playback information.
The display device according to claim 1, wherein the controller is further configured to:

Before receiving the resources pushed by the terminal device, set the target field correspondingly according to the user-defined multi-channel resource synchronous playback capability;

Wherein, the target field is used to enable the terminal device to query whether the display device has the ability to play multiple resources synchronously by reading the target field before pushing the resource;

The target field is also used to enable the terminal device to push the resources corresponding to the screen projection request to the display device when it is found that the display device has the capability of synchronously playing multiple resources;

The target field is also used to enable the terminal device to send a stop command to the display device when it is found that the display device does not have the ability to play multiple resources synchronously, the stop command is used to instruct the display device to stop playing the current media data, and After receiving the completion information fed back by the display device in response to the stop instruction, push the resources corresponding to the screen projection request to the display device.
The display device according to claim 1, wherein the screen casting request includes a first resource push request and a second resource push request, wherein the media types of the first resource and the second resource are different, or, the The media types of the first resource and the second resource are video types, wherein the controller is configured to determine an update policy of the display in the following manner:

In response to the first resource push request, displaying a first window on the display to play the first resource corresponding to the first resource push request;

In response to the second resource push request, add a second window on the display to play the second resource corresponding to the second resource push request in the second window, and keep playing the first resource in the first window. a resource.
The display device according to claim 16, wherein the display device further comprises an audio output interface, and

The controller is configured as:

In response to the second resource push request, output the sound data corresponding to the second resource from the audio output interface.
The display device according to claim 1, wherein the screen projection request includes a first resource push request and a second resource push request, wherein the media types of the first resource and the second resource are both audio types or an image type, the controller is configured to determine an update strategy for the display as follows:

In response to the first resource push request, displaying a first window on the display to play the first resource corresponding to the first resource push request;

In response to the second resource push request, instead of creating a second window, switch from playing the first resource to playing the second resource in the first window.
The display device according to claim 16, wherein the first resource has a first playback duration, the second resource has a second playback duration, and the controller is further configured to execute:

When it is detected that the playback duration of any one of the first resource and the second resource reaches the preset playback duration, the corresponding window is destroyed, and the remaining windows are kept for normal playback.
The display device according to claim 19, wherein the one window corresponds to one media player, and the controller is further configured to perform:

When it is detected that the playback duration of any one of the first resource and the second resource reaches the preset playback duration, the media player corresponding to the resource is released, and the corresponding window is destroyed at the same time, and the remaining windows are reserved for normal playback.
The display device according to claim 1, wherein the screen projection request includes a connection instruction, wherein the controller is configured to determine an update policy of the display in the following manner:

Acquiring the number of second terminal devices in response to a connection instruction of the first terminal device, where the second terminal device is a terminal device that has been connected to the display device;

sending indication information to the first terminal device, where the indication information is used to indicate the quantity of the second terminal equipment, and trigger the first terminal equipment to generate an interaction window according to the quantity of the second terminal equipment, the interaction window includes controls, and the interaction window The control is determined according to the quantity of the second terminal device;

receiving an instruction generated and sent by the first terminal device according to selecting the control in the interaction window, and controlling the display to play the resource corresponding to the screen projection request according to the instruction.
The display device according to claim 21, wherein the instruction includes a new window instruction, and the controller is configured to perform the following steps to control the display to play the resource corresponding to the screen projection request according to the instruction: A new window is added on the first user interface, and the resource corresponding to the screen projection request is played in the newly added window.
The display device according to claim 21, wherein the first user interface includes at least one window, and the resources of the second terminal device are played in the window;

The instruction includes a replacement instruction, the replacement instruction includes replacement information, and the replacement information is used to indicate the window information of the second terminal device corresponding to the resource to be replaced; the controller is configured to perform the following steps to control the display according to The instruction plays the resource corresponding to the screen projection request: disconnecting the second terminal device corresponding to the replacement information, canceling the window of the second terminal device, creating a new window at the window of the second terminal device, and The resources of the first terminal device are displayed in a newly created window.
The display device according to claim 21, wherein the instruction includes an instruction not to project the screen, and the controller is configured to perform the following steps to control the display to display the video corresponding to the screen projection request according to the instruction: Resource: refuse to connect with the first terminal device, and keep the display state of the user interface unchanged.
The display device according to claim 21, wherein the controller is further configured to:

Judging whether the conditions for not sending instruction information are met;

In response to meeting the condition, not sending indication information;

In response to not meeting the condition, send indication information.
The display device according to claim 25, wherein the condition for not sending the indication information includes that an upper limit value of the number of windows on the first user interface is a specific value.
A content display method, comprising:

receiving a screen projection request from a first terminal device among the one or more terminal devices,

In response to the screen projection request, an update policy of the display is determined according to the current display state of the display, so as to play resources corresponding to the screen projection request.
According to the method according to claim 27, the screen casting request includes a first resource push request and a second resource push request, wherein the media types of the first resource and the second resource are different, or, the second resource The media types of the first resource and the second resource are both video types, wherein determining the update policy of the display includes:

In response to the first resource push request, displaying a first window on the display to play the first resource corresponding to the first resource push request;

In response to the second resource push request, add a second window on the display to play the second resource corresponding to the second resource push request in the second window, and keep playing the first resource in the first window. a resource.