WO2023142655A1

WO2023142655A1 - Screen projection method, display device, and intelligent device

Info

Publication number: WO2023142655A1
Application number: PCT/CN2022/135421
Authority: WO
Inventors: 高红; 焉丽飞; 张伟丽; 吕甜甜; 肖成禹; 于渚
Original assignee: 海信视像科技股份有限公司
Priority date: 2022-01-27
Filing date: 2022-11-30
Publication date: 2023-08-03
Also published as: CN118556405A

Abstract

The present application discloses a screen projection method, a display device, and an intelligent device. For a display device end: a screen projection instruction input by a user is received, and connectable devices are scanned for, a connectable device being a screen projection sending terminal using a screen projection function; a screen projection request is sent to a target device among the connectable devices, the screen projection request being used for instructing the target device to establish a screen projection connection with the display device; media data sent by the target device is received in response to the screen projection request, and the media data is played. Multi-screen interaction and screen sharing are implemented while a screen projection operation is made to be more efficient and convenient, scenario adaptability and intelligence of a screen projection function are improved, and the screen projection use experience of a user is improved.

Description

Screen projection method, display device and smart device

Cross References to Related Applications

This disclosure claims the priority of the Chinese patent application filed on January 27, 2022, with application number 202210100384.9; and filed on February 08, 2022, with application number 202210118550.8, the entire contents of which are incorporated in this disclosure by reference.

technical field

The present application relates to the technical field of display devices, and in particular to a screen projection method, a display device and a smart device.

Background technique

In some application scenarios, users can initiate screen projection to display devices through smart terminals such as mobile phones and tablets, so as to push media resources such as video, audio, and images in small-screen terminals to large-screen display devices for playback, realizing multiple screen interaction and screen sharing. The user needs to initiate screen projection on the smart terminal. The smart terminal scans for other devices that can be projected. The user then selects a target device. The smart terminal establishes a screen projection connection with the target device. After the connection is successful, the target device receives and displays the information sent by the smart terminal. media data.

However, in some business scenarios, this screen projection method may have problems such as inconvenient operation and insufficient intelligence of the screen projection response. For example, the user performs the screencasting operation on the screencasting sender, and then watches the screencasting content on the screencasting receiver. If the distance between the screencasting sender and the screencasting receiver is far, the process will consume the user’s time and limit the screencasting. Functional usage scenarios, and low intelligence.

Contents of the invention

The present disclosure provides a display device, including: a display for displaying an image and/or a user interface; a communicator for communicating with a smart device; a controller connected to the display and the controller and configured to: receive The screen projection command input by the user scans the connectable device, and the connectable device is a screen projection sending terminal that enables the screen projection function; sends a screen projection request to the target device in the connectable device, and the screen projection request is used to indicate the target device and the display device Establish a screen projection connection; receive the media data sent by the target device in response to the screen projection request, and control the display to play the media data.

The present disclosure also provides a smart device, including: a display, for displaying an image and/or a user interface; a communicator, for communicating with the display device; and a controller, configured to: receive a screen projection request sent by the display device ;The screen projection request is sent by the display device in response to the screen projection instruction input by the user, after determining that the target device is a smart device from the scanned connectable devices, and the connectable device is a screen projection sending terminal with the screen projection function enabled; Screen request, establish a screen projection connection with the display device; after the screen projection connection is successful, send media data to the display device.

The present disclosure also provides a screen projection method for a display device, including: the display device receives a screen projection instruction input by the user, scans for a connectable device, and the connectable device is a screen projection sending terminal that enables the screen projection function; The target device among the connectable devices sends a screen projection request, and the screen projection request is used to instruct the target device to establish a screen projection connection with the display device; the display device receives the media data sent by the target device in response to the screen projection request, and plays the media data.

The present disclosure also provides a screen projection method for a smart device, including: the smart device receives a screen projection request sent by a display device; If the target device is determined to be a smart device, the connectable device is a screen projection sending terminal with the screen projection function enabled; the smart device responds to the screen projection request and establishes a screen projection connection with the display device; after the screen projection connection is successful, the smart device sends a screen projection to The display device sends media data.

Description of drawings

FIG. 1 shows a usage scenario of a display device;

FIG. 2 shows a hardware configuration block diagram of a control device 100;

FIG. 3 shows a hardware configuration block diagram of a display device 200;

FIG. 4 shows a software configuration diagram in a display device 200;

Figure 5 shows a speech recognition network architecture diagram;

Fig. 6 shows an example diagram of screen projection interaction scenarios;

FIG. 7 shows a scene diagram 1 in which a user inputs voice on a display device;

Figure 8 shows the UI display effect of the display device when there is no connectable device in the scene;

FIG. 9 shows the UI display effect 1 of the display device when there are multiple connectable devices in the scene;

Fig. 10 shows the UI display effect 2 of the display device when there are multiple connectable devices in the scene;

Fig. 11 shows a schematic diagram of a target device displaying a screen casting request;

Fig. 12 shows the UI display effect diagram of the display device after the target device rejects the screen projection request;

Fig. 13 shows a schematic diagram of synchronous playback between the target device and the display device after the screen projection connection is successful;

FIG. 14 shows the UI display effect of the display device when the screen projection is exited when the display device is connected to the target device for the first time;

FIG. 15 shows the second scene where the user inputs voice on the display device;

Fig. 16 shows the UI display effect diagram of the display device when the device specified by the user in the screen casting instruction cannot be scanned by the display device;

FIG. 17 shows a flow chart of the first screen projection method executed by the display device;

FIG. 18 shows a flow chart of a second screen projection method executed by the display device;

Fig. 19 shows a flow chart of a screen projection method executed by a smart device;

Fig. 20 shows a screen projection interaction logic diagram;

Fig. 21 shows a schematic diagram of a user interface of a display device;

Fig. 22 shows a flow chart of a screen projection method;

Fig. 23 shows a schematic diagram of a user interface of another display device;

Fig. 24 shows a schematic diagram of a user interface of another display device;

Fig. 25 shows a schematic diagram of a user interface of another display device;

Fig. 26 shows a schematic diagram of a user interface of another display device;

Fig. 27 shows a schematic diagram of switching the use mode of a Wifi module;

Fig. 28 shows a schematic diagram of a user interface of another display device;

Fig. 29 shows another schematic diagram of switching the use mode of the Wifi module.

Detailed ways

In order to make the purpose and implementation of the present disclosure clearer, the following will clearly and completely describe the exemplary implementations of the present disclosure in conjunction with the accompanying drawings in the exemplary embodiments of the present disclosure. Obviously, the described exemplary embodiments are only the present invention. Some, but not all, embodiments are disclosed.

It should be noted that the brief description of terms in the present disclosure is only for the convenience of understanding the implementations described below, and is not intended to limit the implementations of the present disclosure. These terms are to be understood according to their ordinary and usual meaning unless otherwise stated.

The terms "first", "second", and "third" in the description and claims of the present disclosure and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean limiting specific sequential or sequential unless otherwise noted. It is to be understood that the terms so used are interchangeable under appropriate circumstances.

Fig. 1 is a schematic diagram of a usage scenario of a display device according to an embodiment. As shown in FIG. 1 , the display device 200 also performs data communication with the server 400 , and the smart device 300 or the control device 100 can be used by the user to operate the display device 200 .

In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes at least one of infrared protocol communication, Bluetooth protocol communication, and other short-distance communication methods, and the display device is controlled wirelessly or wiredly. Device 200. At least one mode of inputting user commands such as buttons on the remote control, voice input, and control panel input can be selected by the user to control the display device 200 .

In some embodiments, the smart device 300 may include any one of a mobile terminal, a tablet computer, a computer, a notebook computer, an AR/VR device, and the like.

In some embodiments, the smart device 300 can also be used to control the display device 200 . For example, the display device 200 is controlled using an application program running on the smart device.

In some embodiments, the smart device 300 and the display device may also be used for data communication.

In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the module for obtaining voice commands configured inside the display device 200 can directly receive the user's voice command control , the user's voice command control can also be received through the voice control device provided outside the display device 200 .

In some embodiments, the display device 200 also performs data communication with the server 400 . The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200 . The server 400 may be one cluster, or multiple clusters, and may include one or more types of servers.

In some embodiments, the software steps executed by one step execution body can be migrated to another step execution body for data communication with it for execution as needed. Exemplarily, the software steps executed by the server can be migrated to be executed by the display device in data communication with it as required, and vice versa.

FIG. 2 shows a block diagram of the configuration of the control device 100 according to the exemplary embodiment. As shown in FIG. 2 , the control device 100 includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply. The control device 100 can receive the user's input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, and play an intermediary role between the user and the display device 200 .

In some embodiments, the communication interface 130 is used for communicating with the outside, and includes at least one of a WIFI chip, a Bluetooth module, NFC or an alternative module.

In some embodiments, the user input/output interface 140 includes at least one of a microphone, a touch pad, a sensor, a button or an alternative module.

FIG. 3 shows a block diagram of a hardware configuration of a display device 200 according to an exemplary embodiment.

In some embodiments, the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface. at least one. The user interface is used to receive input signals, and the controller 250 is respectively connected to the display 260 and the user interface.

In some embodiments the controller includes a CPU, a video processor, an audio processor, a graphics processor, a RAM, a ROM, a first interface to an nth interface for input/output.

In some embodiments, the display 260 includes a display screen component for presenting images, and a drive component for driving image display, for receiving image signals output from the controller, and displaying video content, image content, and menu manipulation interface. Components and user manipulation of the UI interface, etc.

In some embodiments, the display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

In some embodiments, the tuner-demodulator 210 receives broadcast TV signals through wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or cable broadcast TV signals.

In some embodiments, the communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220 .

In some embodiments, the detector 230 is used to collect signals of the external environment or interaction with the outside. For example, the detector 230 includes a light receiver, which is a sensor for collecting ambient light intensity; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environmental scenes, user attributes or user interaction gestures, or , the detector 230 includes a sound collector, such as a microphone, for receiving external sound.

In some embodiments, the external device interface 240 may include, but is not limited to, the following: high-definition multimedia interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), Any one or more interfaces such as RGB ports. It may also be a composite input/output interface formed by the above-mentioned multiple interfaces.

In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different split devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box wait.

In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operations of the display device 200 . For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 may perform an operation related to the object selected by the user command.

In some embodiments, the object may be any of a selectable object, such as a hyperlink, icon, or other operable control. The operations related to the selected object include: displaying operations connected to hyperlink pages, documents, images, etc., or executing operations corresponding to the icon.

In some embodiments, the controller includes a central processing unit (Central Processing Unit, CPU), a video processor, an audio processor, a graphics processing unit (Graphics Processing Unit, GPU), a RAM (Random Access Memory, RAM), a ROM (Read -Only Memory, ROM), at least one of the first interface to the nth interface for input/output, a communication bus (Bus) and the like.

CPU processor. It is used to execute the operating system and application program instructions stored in the memory, and to execute various application programs, data and content according to various interactive instructions received from the outside, so as to finally display and play various audio and video content. A CPU processor may include multiple processors. For example, including a main processor and one or more sub-processors.

In some embodiments, the graphics processor is configured to generate various graphic objects, such as at least one of icons, operation menus, and user input instruction display graphics. The graphics processor includes an arithmetic unit, which performs calculations by receiving various interactive instructions input by users, and displays various objects according to display attributes; it also includes a renderer, which renders various objects obtained based on the arithmetic unit, and the above-mentioned rendered objects are used to be displayed on the display.

In some embodiments, the video processor is used to receive an external video signal and perform video decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, image synthesis, etc. according to the standard codec protocol of the input signal. At least one of the processing can obtain a signal that can be directly displayed or played on the display device 200 .

In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, and the like. Wherein, the demultiplexing module is used for demultiplexing the input audio and video data streams. The video decoding module is used to process the demultiplexed video signal, including decoding and scaling. An image synthesis module, such as an image synthesizer, is used to superimpose and mix the graphic generator with the zoomed video image according to the GUI signal input by the user or generated by itself, so as to generate an image signal available for display. The frame rate conversion module is used for converting the input video frame rate. The display formatting module is used to convert the received frame rate to the video output signal, and change the signal to conform to the display format signal, such as outputting RGB data signal.

In some embodiments, the audio processor is used to receive an external audio signal, perform decompression and decoding according to the standard codec protocol of the input signal, and perform at least one of processing such as noise reduction, digital-to-analog conversion, and amplification processing , to obtain a sound signal that can be played on a loudspeaker.

In some embodiments, the graphical user interface (GUI) displayed on the display 260 provides icons or controls for the user to input user commands, and the user input interface receives user input commands through the graphical user interface (GUI). Alternatively, the way of inputting user commands may be to input a specific sound or gesture, and then the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

In some embodiments, the user interface 280 is an interface that can be used to receive control input (such as: physical buttons on the display device body, or others).

In some embodiments, the system of the display device may include a kernel (Kernel), a command parser (shell), a file system and application programs. Together, the kernel, shell, and file system make up the basic operating system structure, and they allow users to manage files, run programs, and use the system. After power-on, the kernel starts, activates the kernel space, abstracts hardware, initializes hardware parameters, etc., runs and maintains virtual memory, scheduler, signal and inter-process communication (IPC). After the kernel starts, the Shell and user applications are loaded. After the application is started, it is compiled into machine code to form a process.

As shown in FIG. 4 , the system of the display device is divided into three layers, which are application layer, middleware layer and hardware layer from top to bottom.

The application layer mainly includes common applications on TV, and application framework (Application Framework), among which, commonly used applications are mainly applications developed based on the browser browser, such as: HTML5APPs; and native applications (Native APPs);

Application Framework (Application Framework) is a complete program model that has all the basic functions required by standard application software, such as: file access, data exchange, etc. frame).

Native APPs can support online or offline, message push or local resource access.

The middleware layer includes middleware such as various TV protocols, multimedia protocols, and system components. Middleware can use the basic services (functions) provided by the system software to connect various parts of the application system or different applications on the network, so as to achieve the purpose of resource sharing and function sharing.

The hardware layer mainly includes the HAL interface, hardware and drivers. The HAL interface is a unified interface for all TV chips to connect, and the specific logic is implemented by each chip. Drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

In order to clearly illustrate the embodiments of the present disclosure, a speech recognition network architecture provided by the embodiments of the present disclosure will be described below with reference to FIG. 5 .

Referring to FIG. 5 , the smart device is used to receive input information and output a processing result of the information. The speech recognition service device is an electronic device deployed with a speech recognition service, the semantic service device is an electronic device deployed with a semantic service, and the business service device is an electronic device deployed with a business service. The electronic device here may include a server, a computer, etc., and the speech recognition service, semantic service (also called a semantic engine) and business service here are web services that can be deployed on the electronic device, wherein the speech recognition service is used for audio Recognized as text, the semantic service is used for semantic analysis of the text, and the business service is used to provide specific services such as the weather query service of Moji Weather, the music query service of QQ Music, etc. In an embodiment, there may be multiple entity service devices deployed with different business services in the architecture shown in FIG. 5 , or one or more functional services may be integrated in one or more entity service devices.

In some embodiments, the following is an example description of the process of processing the information input to the smart device based on the architecture shown in FIG. process:

[Speech Recognition]

After receiving the query sentence input by voice, the smart device can upload the audio of the query sentence to the voice recognition service device, so that the voice recognition service device can recognize the audio as text through the voice recognition service and return it to the smart device. In one embodiment, before uploading the audio of the query sentence to the speech recognition service device, the smart device may perform denoising processing on the audio of the query sentence, where the denoising processing may include steps such as removing echo and environmental noise.

[semantic understanding]

The smart device uploads the text of the query sentence recognized by the speech recognition service to the semantic service device, so that the semantic service device can perform semantic analysis on the text through the semantic service to obtain the business field and intention of the text.

[semantic response]

According to the semantic analysis result of the text of the query statement, the semantic service device sends a query instruction to the corresponding business service device to obtain the query result given by the business service. The smart device can obtain and output the query result from the semantic service device. As an embodiment, the semantic service device can also send the semantic analysis result of the query sentence to the smart device, so that the smart device can output the feedback sentence in the semantic analysis result.

It should be noted that the architecture shown in FIG. 5 is only an example, and does not limit the protection scope of the present disclosure. In the embodiments of the present disclosure, other architectures may also be used to implement similar functions. For example, all or part of the three processes may be completed by a smart terminal, which will not be repeated here.

The above embodiments introduce the hardware/software architecture and function realization of the display device. For a device terminal that is the same as or similar to the display device, it can be used as the object to be projected, that is, the receiving end of the screen projection; for other smart devices such as smartphones, tablets, laptops, etc., if they support the screen projection function, they can be used as the screen projection receiver. screen sender. Through the screen projection interaction between the display device and the smart device, the media resources or screen playback content in the smart device can be projected to the display device, so as to realize the sharing and playback of media on different terminals and different screens, in teaching, games, conferences, etc. widely used in the scene.

In the existing implementation, the wireless network (wifi) switch and the screen projection switch of the smart device 1 can be turned on, so that the LAN is connected to the Internet and the screen projection function is turned on. When device 1 initiates screen projection, smart device 1 will search for other smart devices in the local area network that can receive screen projection data to form the first device list. After the user selects a display device in the first device list, the smart device and display device After the screen projection connection is established and the screen projection connection is successful, the smart device sends the screencast media data to the display device, and the display device plays the media data.

In some embodiments, the smart device and the display device can establish a screen projection connection through certain protocols, such as DLNA (Digital Living Network Alliance, Digital Living Network Alliance) protocol, Miracast protocol, etc.; Mirror projection, such as Airplay, etc., so that the media data image currently being displayed by the smart device is pushed to the display device side synchronously.

In some embodiments, after the smart device is successfully connected to the display device, for each available media resource on the Web, such as HTML documents, images, videos, audio and programs, a general URI (Uniform Resource Identifier , Uniform Resource Identifier) for resource location, that is, the URI is equivalent to providing an address for locating resources. Optionally, the smart device can push the URI of the media data that needs to be projected to the display device. After receiving the URI, the display device can locate the resource, and then download and play the corresponding media data.

In some embodiments, the smart device may also push the URL (Uniform Resource Locator, Uniform Resource Locator) of the media data, that is, the network address of the media resource, to the display device, so that the display device accesses the URL and downloads the media data.

In some embodiments, the smart device can also directly encode media data such as images, videos, and audio, and synchronize them to the display device in real time in the form of a media stream; after receiving the media stream, the display device decodes the media stream and play. After the screen projection connection is successful, the transmission mode of the media data from the smart device to the display device is not limited to the examples of the embodiments of the present disclosure.

Considering the poor adaptability of the screen projection function discussed in the background technology, insufficient intelligence of the screen projection response, and inconvenient operation, in some embodiments, referring to (a) in FIG. 6 , the display device receives the projection screen input by the user Screen command, the display device side actively scans for connectable devices in response to screen projection commands. Connectable devices refer to screen projection sending terminals that have enabled the screen projection function (that is, the screen projection switch The device identifiers of the devices are combined into a second device list. The device identifiers can be device names, device models, etc., which are convenient for users to identify devices. The second device list can be used to select a target device, and the display device will automatically send a screen projection to the target device. Request, the target device responds to the screen projection request, and establishes a screen projection connection with the display device. The screen projection connection is equivalent to building a data transmission channel, so that the screen projection data can be transmitted from the target device to the display device.

In some embodiments, referring to (b) in FIG. 6 , the smart device initiates the screen projection after responding to the user's operation, and the smart device scans the devices that can be projected to obtain the first device list, based on the selection in the first device list After displaying the device, the smart device establishes a screen projection connection with the display device, and then sends the media data to be projected to the display device.

It can be seen that compared with (b) in Figure 6, (a) in Figure 6, the execution end that triggers the screen projection operation, device scanning, and sending the screen projection request is completely opposite, and the actual scene can be adapted to any kind of screen projection mode, for example, in scenes such as teaching and conferences, the screen projection mode 1 shown in (a) in Figure 6 can be used to ensure efficient and convenient screen projection operations and avoid interfering with business processes in the scene. For example, the teacher only needs to enter a screen projection command on the display device, and the display device can actively scan and initiate a screen projection request to the laptop. The teacher only needs to stand at position B and refer to the projected courseware to explain. Moving between location A and location B does not require other screen projection assistance, thus reducing the time spent on screen projection operations and will not interfere with the normal teaching process, allowing students to focus more on the knowledge shared by projection screens; another example, user A wants Project the exciting video currently playing on your mobile phone to the TV in the living room, so as to share the video with user B who is watching TV in the living room, but user A is not in the living room, then the projection mode of the example (b) in Figure 6 The second can be implemented by user A. The disclosure realizes the scene adaptability of the screen projection mode, so that the user can select a suitable screen projection mode according to the scene, personal habits and other factors.

In some embodiments, for screen projection mode 2, if the smart device A does not support certain screen projection protocols, after the smart device A sends a screen projection request to the display device, it will prompt that it cannot connect or the connection fails; Mode 1, if the smart device A does not support the screen projection protocol, the screen mirroring function, or the switch status of the screen projection switch of the smart device is off, it will be displayed that the device is in the scanning stage, and the device of the smart device A cannot be scanned and read at all. Information, that is, smart device A is not a connectable device, so by displaying the scan results on the device side, it can be reflected to a certain extent that there is a smart device with an abnormal screen projection connection problem in the scene. For example, the display device is the screen projection device determined by the user. The display device can be used to play the video of smart device B, but the display device does not scan the smart device B. If it detects that the screen projection switch of smart device B is turned off, the smart device can be turned on B's screen projection switch, after the display device re-scans and scans to smart device B, then a screen projection connection with smart device B can be established.

In some embodiments, the input of the screen projection instruction can trigger and start the processing logic of the screen projection mode 1, wherein the form of the screen projection instruction is not limited, for example, the screen projection instruction can be voice input, if the display device has a voice collector, For example, a microphone, etc., can realize the sound collection function, collect the user's voice command, for example, detect the semantically similar voice information spoken by the user such as "cast screen", "I want to cast screen", "cast screen from mobile phone", and display The semantic service at the bottom of the device receives the collected voice information, and sends the voice information to a semantic server such as Semantic Cloud. The semantic server analyzes the voice information, such as keyword extraction and semantic recognition, and then feeds back the analysis results to the semantic service. , the semantic service then recognizes the result of the analysis, for example, if it recognizes that the user's voice intention is screen projection, the display device will confirm that it has received the screen projection instruction. The screen projection is invoked on the display device through voice, without manual operation by the user, which improves the efficiency and convenience of the screen projection, and realizes the rapid response and function trigger of the projection screen. For the adaptability of speech processing logic and architecture, refer to the example in FIG. 5 and related descriptions, and details will not be repeated here.

In some embodiments, the sound collector of the display device can be in the always-on state, so that the display device can collect user voice information at all times, but this method may involve privacy security risks, or lead to invalid, irrelevant sound collection and analysis, It also results in increased power consumption of the device. Optionally, the display device can turn on the sound collector when receiving a preset operation instruction, the preset operation instruction is not limited, for example, after the voice key on the remote controller responds to the user's operation, the display device receives the key value sent by the remote controller Finally, if it is recognized that the user has triggered the voice key, the voice collector will be turned on to collect the voice information input by the user. The operation of long pressing the voice key can keep the voice collector in the continuous collection state. After the voice key is released, the voice collector will Stop collecting, and the display device can collect a piece of voice information during the period from when the voice key is pressed to when it is released. Optionally, after the voice key is released, if the voice key is not triggered again within a preset time period, the sound collector can be automatically turned off.

In some embodiments, the screen projection instruction can also be input in other ways, such as clicking the screen projection button on the remote control, clicking the screen projection control set in the page under a certain access path of the display device; or through preset gesture control, the display The device collects user gesture images, and through gesture recognition, the user input is a screen projection command; or, with the help of the communication connection between the mobile terminal and the display device, the screen projection command is issued through the mobile terminal, and so on.

In some embodiments, taking the voice input projection command as an example, referring to FIG. 7, the display device receives the "screen projection" spoken by the user, the display device collects the voice information, and sends it to the voice server for analysis. According to the semantic server feedback As a result of the analysis of the user's voice intention, it is determined that the screen projection command has been received. Since the screen projection command does not specify the device information of the screen projection sender, it is necessary to set the target device according to the preset rules, and then send the The target device sends a screen mirroring request.

In some embodiments, the display device actively scans the connectable devices in response to the screen projection command to obtain a device list. For ease of distinction, the device list scanned by the display device is named the second device list. The second device list includes The device identifiers of the scanned S connectable devices, the device identifiers are not limited to the device name, device model, etc., according to the S value status of the second device list, the preset rules for setting the target device can be configured.

In some embodiments, the preset rule 1 can be configured as follows: if the value of S is zero, it means that the second device list is an empty set, indicating that the display device has not scanned for available devices, and there is no connectable device in the current scene. Refer to FIG. 8 The UI example of , can display the first prompt pop-up window 80 on the current interface layer. Optionally, the first prompt pop-up window 80 includes a first prompt message 81, a first exit control 82 and a first rescan control 83. The first prompt message 81 is used to prompt that there is currently no connectable device, such as the first prompt in FIG. An example of a prompt message 81 is "Currently no connectable device has been scanned, you can try to turn on the screen projection switch of the target device and re-scan". If the first exit control 82 is selected, then exit the screen projection and end the screen projection mode 1. Processing logic: if the user expects the screen projection sender to be the smart device 1, then switch the screen projection switch of the smart device 1 to the on state, and then trigger the first rescanning control 83, and the display device will rescan for connectable devices, and can Scan to the smart device 1, the smart device 1 is the target device selected by the user, the display device sends a screen projection request to the smart device 1 to build a screen projection connection between the display device and the smart device 1, so as to receive the media data pushed by the smart device 1 .

In some embodiments, preset rule 1 can also be configured as follows: if the value of S is 1, it means that the second device list is a single-element set, indicating that the display device only scans one available device, that is, there is only one available device in the current scene. If the device is connected, the user does not need to select it. By default, the only device that can be connected is set as the target device, and the display device sends a screen projection request to the target device.

In some embodiments, preset rule 1 can also be configured as follows: if the value of S is greater than 1, it means that the second device list is a multi-element set, indicating that the display device has scanned multiple available devices, that is, there are multiple devices in the current scene If the device can be connected, the user needs to cooperate with the selected target to make the device on the current interface layer. Referring to the examples in FIG. 9 and FIG. 10 , the device list 90 can be displayed. Optionally, referring to FIG. 9 , the device list 90 includes device identifiers 91 and confirmation controls 92 of S scanned connectable devices, each device identifier has a corresponding selection control 911, and the device list 90 includes For example, if the target device selected by the user is the smart device 2, the selection control 911 of the smart device 2 is switched to the selected state, and then the confirmation control 92 is selected, that is, it is confirmed that the smart device 2 is set as the target device, and the display device Send a screencasting request to smart device 2.

Optionally, referring to FIG. 10, the device list 90 may only include the device identifiers 91 of S connectable devices, without setting the confirmation control 92 and each device identifier has a corresponding selection control 911. In this example, the display device can be set Focus, the focus is linked with the user's action of switching devices. Assuming that the user's expected screencasting sender is the smart device 2, it can be realized by moving the focus so that the focus falls on the smart device 2, and then trigger the OK button on the remote control. That is, it is confirmed that the smart device 2 is selected as the target device, and the display device sends a screen projection request to the smart device 2 . The presentation effect of the focus is not limited, for example, setting the focus to be highlighted, or adding a focus frame, or enlarging the device logo with the focus to a certain extent, etc., so that the focus is eye-catching and easy to identify.

In some embodiments, if the empty selection operation in the device list 90 of FIG. 9 is detected, it is determined that the confirmation control 92 is selected, or it is detected that the cancel control 93 is selected, or it is detected that the return operation is performed in the interface of FIG. 10 , By default, the target device selected by the user is canceled. Optionally, exit the screen projection and end the processing logic of screen projection mode 1.

In some embodiments, the target device receives the screen casting request. Optionally, it can accept the screen casting request by default and establish a screen casting connection with the display device, or, referring to the example in FIG. 11 , on the current interface of the target device Display a screen projection request pop-up window 111, which includes a screen projection request information 112, an acceptance control 113 and a rejection control 114 in the screen projection request pop-up window 111, and the screen projection request information 112 should include the device identification of the display device as the screen projection request/receiving end, For example, the example in FIG. 11 is "The living room TV initiates a screen projection request, do you accept it?" If the user wants to accept the screen projection request, the acceptance control 113 is selected, and the screen projection request pop-up window 111 is closed, and the target device is established with the living room TV. The screen projection connection, and after the connection is successful, the screen projection data is sent to the living room TV; if the user does not want to accept the screen projection request, trigger the rejection control 114 to be selected, then close the screen projection request pop-up window 111, and send it to the living room TV Screencasting rejection indication information; referring to the example in Figure 12, after the display device receives the screencasting rejection indication information, it displays a prompt message on the current interface, for example, prompting "The target device has rejected your screencasting request, please try again later or Switch to other devices”, the prompt information can be viewed by the user, and the user can choose to initiate a screen projection request to the target device again, or switch to other connectable screen projection senders, or stop initiating screen projection requests to any connectable devices.

In some embodiments, if the target device accepts the screen projection request of the display device, a screen projection connection with the display device is established. After the screen projection connection is successful, referring to the example in FIG. 13 , the target device synchronously pushes the media data played on the current screen On the display device side, the display device receives the media data, loads and plays the media data in the screen projection interface, and realizes screen mirroring and resource sharing at both ends of the screen projection interaction.

In some embodiments, referring to FIG. 13 , after the screen projection connection is successful, the first screen projection end control 131 is displayed on the playback interface of the target device, the second screen projection end control 132 is displayed on the screen projection interface of the display device, and the first screen projection end control 132 is displayed. If any one of the screen control 131 and the second end screen projection control 132 is selected, it is necessary to exit the screen projection interaction process.

In some embodiments, if the target device detects that the first end screen projection control 131 is selected, the target device disconnects the screen projection connection with the display device, cancels the display of the first end screen projection control 131, and sends a message to the display device The first exit screen projection instruction information; the display device exits the screen projection interface after receiving the first exit screen projection instruction information.

In some embodiments, if the display device detects that the second end screen projection control 132 is selected, the display device disconnects the screen projection connection with the target device, exits the screen projection interface, and sends the second screen projection exit to the target device Instruction information: the target device receives the second exit screen projection instruction information, and cancels the display of the first end screen projection control 131 .

In some embodiments, the display device can maintain a screen projection connection list, the screen projection connection list includes several historical connection records, each historical connection record records the connection information of a smart device that has established a screen projection connection with the display device, and the connection information It is not limited to include device identification, screen casting times, permission type and connection time, etc. Among them, for example, when the display device and the smart device 2 are successfully connected each time the screen projection is successfully connected, the number of screen projections of the smart device 2 is cumulatively increased by one; the permission type is used to represent the continuous screen projection permission of the smart device, assuming that the smart device 2 has the projection Screen continuous permission, the display device is allowed to automatically initiate a screen projection request to the smart device 2 when it scans the smart device 2; the connection time is the time when the smart device and the display device successfully establish a screen projection connection. When the display device and a new smart device D create a screen projection connection for the first time, a connection record D between the smart device D and the display device is generated, the connection record D is inserted into the screen projection connection list, and the number of screen projections in the connection record D is updated to 1. The device ID and connection time in the connection information are generally fixed, and the number of screen projections will accumulate. If the permission type is permanent permission, it will remain unchanged. If the permission type is no permission or temporary permission, the permission type may change with the user's authorization operation. change.

In some embodiments, if the display device detects that the second end screen projection control 132 is selected, or the display device receives the first exit screen projection instruction information sent by the target device, it queries the connection list of the target device from the screen projection connection list. Display the number of screen projection times of the display device. If the number of screen projection times is 1, it means that the target device is connected to the display device for the first time. Referring to the example in Figure 14, the display device displays a third prompt pop-up window 141 before exiting the screen projection interface. The pop-up window 141 includes authorization query information 142, a single authorization control 143 (that is, "Yes, only next time continuous" in Figure 14), and a permanent authorization control 144 (that is, "Yes, permanent continuous" in Figure 14) and deny authorization control 145 (ie "No" in FIG. 14). Among them, the authorization inquiry information 142 is used to inquire whether to grant the continuous permission of screen projection to the target device. For example, in FIG. , that is, refuse to grant the continuous screen projection permission for smart device 2, then close the third prompt pop-up window, the current screen projection connection between the display device and the target device is disconnected, and the next time the display device scans to the smart device 2, it will not automatically Initiate a screen projection request to the smart device 2 and connect with it, and the next time the screen projection connection is disconnected, a third prompt pop-up window 141 still needs to be displayed to inquire about the permission type of the smart device 2 .

In some embodiments, if it is detected that the permanent authorization control 144 is selected, that is, the smart device 2 is agreed to grant the permanent continuous screen projection permission, the third prompt pop-up window 141 is closed, and the current screen projection connection between the display device and the target device is disconnected. Open, and update the connection information of the smart device 2 in the screen projection connection list, that is, add the second permission identifier in the connection information of the smart device 2, and the second permission identifier is used to indicate that the smart device 2 has a permanent continuous screen projection permission, when any subsequent screen projection connection between the smart device 2 and the display device is disconnected, there is no need to display the third prompt pop-up window 141, that is, the continuous screen projection permission of the smart device 2 has permanent validity.

In some embodiments, if it is detected that the single authorization control 143 is selected, that is, it is agreed to grant the smart device 2 a single continuous screen projection permission, then the third prompt pop-up window 141 is closed to display the current screen projection between the device and the target device The connection is disconnected, and the first permission identifier is added to the connection information of the smart device 2. The first permission identifier is used to indicate that the smart device 2 has a single screen projection continuous permission. Assume that the current projection between the display device and the smart device 2 If the number of times is i, when the display device scans the smart device 2 next time, it will automatically complete the screen projection connection with the smart device 2, and the screen projection times corresponding to the smart device 2 will become i+1. At this time, the single time of the smart device 2 If the continuous screen projection permission fails, the display device will revoke the first permission identifier of the smart device 2, so that when the i+1th screen projection connection between the display device and the smart device 2 is disconnected, the third prompt pop-up window 141 can be displayed again , to determine the subsequent permission status of the smart device 2 for continuous screen projection. The permission types of different smart devices can be set according to factors such as the frequency of screen projection to smart devices, usage habits, and scene requirements.

In some embodiments, if the display device detects that the second end screen projection control 132 is selected, or the display device receives the first exit screen projection instruction information sent by the target device, it queries the connection list of the target device from the screen projection connection list. Display the screen casting times of the device. If the screen casting times are greater than 1, it means that the target device and the display device are not connected for the first time. Then identify the permission type corresponding to the target device. If the target device has the second permission identifier, that is, the permission type is permanent screen projection continuous permission, then when the display device quits screen projection, the third prompt pop-up window 141 will not be displayed, and the current screen projection connection with the target device will be directly disconnected; If the target device has the first permission ID, that is, the permission type is the single screen projection continuous permission, or, if the target device does not have the first permission ID and the second permission ID, that is, the target device permission type is no continuous screen projection permission, the display device needs to display the third prompt pop-up window 141 again when exiting the screen projection, and update the permission type of the target device according to the user's authorization operation instruction. For example, the target device under the i-th (i greater than 1) screen projection connection has no continuous screen projection permission. When the i-th screen projection connection is disconnected, if it is detected that the single authorization control 143 is selected, it is the target device. If it is detected that the selected one is the permanent authorization control 144, then the second authorization identification is added for the target device; if it is detected that the selected one is the denial authorization control 145, then the permission type of the target device remains unchanged. Change.

In some embodiments, the preset rules for setting the target device can be defined and configured based on the screen projection connection list. It can be understood that the number of screen projections is 0 before the target device and the display device are connected for the first screen projection. After the first screen projection connection is successful, the number of screen projections becomes 1. Before the second screen projection connection is successful, the number of screen projections remains unchanged. Therefore, the default adaptation rule is: the display device responds to the screen projection command, Scan the current connectable devices to get the device list, then compare the device list with the screen projection connection list, query the M first devices that currently have the first permission ID or the second permission ID from the device list, the first device uses For a smart device that has the permission to cast screens continuously, if the M value is equal to 1, no matter whether the S value of the current device list is greater than 1 or not, the only first device queried will be set as the target device without being selected by the user .

In some embodiments, the preset rules for adaptation are further as follows: if M is greater than 1, it means that all M first devices have the permission to continue screen projection. Choose one of the devices as the target device. Optionally, the first device with the highest priority can be set as the target device according to a preset priority. The preset priority is, for example, sorted according to the number of screen projections. The more screen projections, the higher the priority, or, The preset priority is sorted according to the connection time, the closer the connection time is to the current time, the higher the priority. The preset priority is not limited to the example of this embodiment. The way of configuring the preset rules is not limited to the examples of the present disclosure.

In some embodiments, if the display device only supports single-channel projection, only one target device can be set; if the display device supports multi-channel projection, it is allowed to set one or more targets while avoiding playback conflicts Devices, such as display devices that support N-way projection, if N is greater than 1, N projection windows can be created and displayed in the screen-casting interface, and the display device can receive media data from N target devices The media data is loaded and displayed in the N projection windows respectively, and the projection windows in the N projection windows have independent controls for ending the projection, and the projection windows in the N projection windows can be independently controlled.

In some embodiments, referring to the example in FIG. 15 , when the screen casting instruction is voice input, the voice content may be in the form of "screen projection + target device identifier 151", that is, the device corresponding to the target device identifier specified in the screen casting instruction is At the projection sending end, the sound collector collects the voice information, and then sends the voice information to the semantic server. The semantic server feeds back the analysis result, and the display device recognizes the analysis result. For example, if the voice information is "screen projection from smart device 3", the user The voice intent is screen projection, and the target device is smart device 3.

In some embodiments, when it is recognized that the parsing result contains the target device identifier, the display device scans to obtain a device list, and then further detects whether the target device identifier is included in the device list. If the target device ID is included in the device list, it indicates that the smart device A specified by the user in the screen projection command has been scanned by the display device, and the smart device A is a connectable device, then the smart device A is used as the target device, and the projection is sent to the target device. screen request and establish a screen mirroring connection.

In some embodiments, if the target device identifier is not included in the device list, it means that the display device cannot scan the smart device B specified by the user in the screen projection instruction, and the smart device B is currently an unconnectable device. Referring to the example in FIG. 16 , in A second prompt pop-up window 160 is displayed on the current interface of the display device, and the second prompt pop-up window 160 includes a second prompt message 161, a connection control 162 (i.e. "Yes" in FIG. 16 ), a second rescan control 163 (i.e. "No, rescan" among Fig. 16) and the second exit control 164 (i.e. "No, exit" among Fig. 16), the second prompt information 161 is used for prompting that no specified device has been scanned, and asks whether to switch to the current For example, the prompt in Figure 16 reads "Your specified smart device B has not been scanned, please check smart device B, is it switched to the current connectable device?", after the second prompt message 161 is for the user to view, if detected When the connection control 162 is selected, that is, it is determined to switch to a connectable device, the second prompt pop-up window 160 is closed, and the target device is set according to the aforementioned preset rules, which will not be repeated here. Then send a screen projection request to the target device, and establish a screen projection connection between the display device and the target device.

In some embodiments, since the smart device B cannot be scanned, the second prompt information 161 can be used for the user to check the abnormal situation of the smart device B, for example, check whether the smart device B has turned on the screen projection switch, whether it supports the screen projection function, etc., if If it is determined that smart device B does not support screen projection at all, then it can be determined that the second exit control 164 is selected, that is, it refuses to switch to a connectable device, the display device closes the second prompt pop-up window 160, and exits the processing flow of screen projection mode 1; if determined The screen projection switch of smart device B is turned off, and the screen projection switch has been turned on, then the second rescanning control 163 can be selected, that is, the switch to the connectable device is refused, the display device closes the second prompt pop-up window 160, and rescans After scanning, the smart device B can be searched and scanned, and the smart device B is specially specified in the screen projection command, so after re-scanning, set the smart device B as the target device.

Based on the description of the foregoing embodiments, in some embodiments, FIG. 17 illustrates the first screen projection method executed by the display device, the method is executed by the controller 250 of the display device, and includes the following program steps:

Step S171, receiving a screen projection instruction input by the user, and scanning for connectable devices. The connectable device is a screen projection sending terminal with the screen projection function enabled.

Step S172, sending a screen projection request to the target device among the connectable devices. The screen projection request is used to instruct the target device to establish a screen projection connection with the display device.

Step S173, receiving the media data sent by the target device in response to the screen projection request, and playing the media data.

Based on the description of the foregoing embodiments, in some embodiments, FIG. 18 illustrates the second screen projection method executed by the display device, the method is executed by the controller 250 of the display device, and includes the following procedural steps:

Step S181, collect voice information input by the user, and send the voice information to the semantic server.

Step S182, receiving the analysis result fed back by the semantic server, and identifying the user's voice intention as screen projection.

Step S183, scanning for connectable devices to obtain a device list. The device list includes device IDs that display the connectable devices scanned by the device.

Step S184, judging whether the parsing result contains the target device identifier. If the parsing result does not contain the target device identifier, execute step S185; if the parsing result contains the target device identifier, execute step S186.

Step S185, setting a target device according to preset rules, and sending a screen projection request to the target device. After step S185 is executed, go to step S1812.

Step S186, detecting whether the target device identifier is included in the device list. The device list includes device IDs of S connectable devices scanned by the display device. If the target device ID is included in the device list, step S187 is performed; if the target device ID is not included in the device list, step S188 is performed.

Step S187, sending a screen projection request to the smart device indicated by the target device identifier. After step S187 is executed, go to step S1812.

Step S188, prompting that the smart device specified by the target device identifier has not been scanned.

Step S189, inquire whether to switch to the current connectable device. If it is agreed to switch to the current connectable device, execute step S1810; if reject to switch to the current connectable device, execute step S1811.

Step S1810, setting a target device according to preset rules, and sending a screen projection request to the target device. After step S1810 is executed, go to step S1812.

Step S1811, re-scanning for connectable devices or exiting screen projection.

Step S1812, receiving the media data sent by the target device in response to the screen projection request, and controlling the display to play the media data.

Optionally, in step S185 and step S1810, the preset rule is configured as:

Get the current S value of the device list; the S value is used to represent the total number of connectable devices scanned by the display device.

If the S value of the device list is detected to be zero, it will prompt that there is currently no device that can be connected, and it is allowed to exit the screen mirroring or re-scan for connectable devices;

If it is detected that the S value is equal to 1, set the only connectable device as the target device;

If it is detected that the S value is greater than 1, display the device list, and receive the selected operation on the target device in the device list.

Optionally, in step S185 and step S1810, the preset rule can also be configured as:

Query the M first devices in the device list that currently have continuous permission to cast screens;

If the value of M is equal to 1, the only first device found in the query is set as the target device;

If the value of M is greater than 1, according to the preset priority, the first device with the highest priority is set as the target device.

Based on the description of the foregoing embodiments, in some embodiments, FIG. 19 illustrates a screen projection method executed by a smart device, the method is executed by a controller in the smart device, and includes the following procedural steps:

Step S191, receiving a screen projection request sent by the display device. The screen projection request is sent by the display device in response to the screen projection instruction input by the user, after determining that the target device is a smart device from the scanned connectable devices, and the connectable device is a screen projection sending terminal with the screen projection function enabled.

Step S192, in response to the screen projection request, establish a screen projection connection with the display device.

In step S193, after the screen projection connection is successful, the media data is sent to the display device.

In screen projection mode 1, after the smart device as the screen projection sender turns on the screen projection switch, it does not need to actively scan the screen projection device, but passively waits for the scan of the display device, and then receives and responds to the screen projection request sent by the display device After that, build a screen projection connection with the display device, and then share the specified projection data or screen content to the display device; and in screen projection mode 2, after the smart device turns on the screen device, the display device does not actively scan for connectable devices, but waits for the scan and designation from the sender, then receives and responds to the screen projection request sent by the smart device, establishes a screen projection connection with the smart device, and then receives the projection from the smart device. incoming media data. Specifically, the projection mode can be adapted according to the actual scene, thereby expanding the scene adaptability of the projection function. The voice projection improves the intelligence of the projection, and provides users with a more efficient and convenient projection use and interactive experience. In addition, the software and hardware configuration of the smart device is not limited to the examples disclosed in the present disclosure, and the smart device needs to support a screen projection function and have a media playback capability.

In the above embodiment, the screen projection interaction involves the display device and the smart device. The display device is the screen projection receiver, and the target device/smart device is the screen projection sender. Voice, UI interaction and other forms of input, the display device responds to the screen projection command, scans the currently connected devices, and if the target device that the user expects to send media resources can be scanned, the screen projection request can be sent to the target device. The device can establish a screen projection connection with the display device in response to the screen projection request. After the screen projection connection is successful, the target device can synchronize its current media data to the display device, so as to realize the projection of the media data of the target device to the display device for playback. . Taking the teaching scene as an example, the teacher only needs to input a screen projection command on the display device, and the display device can actively scan and initiate a screen projection request to the laptop. The teacher only needs to stand at position B and refer to the projected courseware to explain , there is no need to move between position A and position B, and there is no need for other screen projection assistance, so the time spent on screen projection operation is reduced, and the normal teaching process will not be disturbed, so that students can pay more attention to the knowledge shared by projection screen. It can be seen that, while realizing multi-screen interaction and screen sharing, the present disclosure makes the screen projection operation more efficient and convenient, thereby improving the scene adaptability and intelligence of the screen projection function, and improving the user's screen projection experience.

In some embodiments, FIG. 20 illustrates a screen projection interaction logic between a display device and a smart device, involving two interactive terminals: device A (screen projection sender/target device) and device B (screen projection receiver/display device) , and cooperate with user interaction. Figure 20 at least includes the following steps:

S2001. Determine that the screen projection switch of device A is on.

S2002. Device B receives a screen projection instruction from a user.

Device B detects the screen-casting command, and the screen-casting command is not limited to voice commands, button commands, and the like.

S2003. Device B starts the configured screen projection service/protocol in response to the screen projection instruction.

Device B starts the configured screen projection service/protocol to trigger screen projection mode 1.

This embodiment takes Miracast as an example. Miracast projection is not limited to device A and device B being in the same local area network, and does not require the use of external connection lines and other materials, and does not require device A or device B to install specific executable files. Both device A and device B need to support screen mirroring, so that screen sharing between different terminals can be realized.

S2004. Device B activates the wifi switch, and initializes wifip2pManager.

S2005. Device B invokes discoverPeers in the P2P protocol to actively scan for connectable P2P devices, and obtains a device list.

S2006. Device B sends a Peers_changed broadcast through wifip2pManager.

S2007. The device B listens to the Peers_changed broadcast, and sets the target device as the device A according to the device list and the screen projection instruction, and referring to the preset rules.

Here, the solution in FIG. 18 may be referred to, and details are not repeated here.

S2008. Device B encapsulates wifip2pconfig of device A.

S2009. Device B sends a screen projection request to device A.

S2010, the device B performs a P2P pairing connection with the device A.

S2011. Device B acquires information such as the IP address and port number of device A.

S2012. Device B initiates a TCP (Transmission Control Protocol, Transmission Control Protocol) connection to device A.

Device A and device B successfully complete the TCP connection.

S2013, device A and device B perform RTSP (Real Time Streaming Protocol, real-time streaming protocol) negotiation under the Miracast service.

S2014. Device A establishes a UDP (User Datagram Protocol, User Datagram Protocol) connection with device B.

UDP sends encapsulated IP packets without establishing a connection.

S2015. Device A transmits the Miracast data stream to device B through UDP.

S2016. Device B displays Miracast projection data.

The screen projection connection in the present disclosure covers P2P connection, TCP connection and UDP connection, and these connections need to be disconnected when the screen projection connection is disconnected. While realizing multi-screen interaction and screen sharing, the disclosure makes the screen projection operation more efficient and convenient, thereby improving the scene adaptability and intelligence of the screen projection function, and improving the user's screen projection experience.

The Miracast screen projection mentioned in the above embodiments is an essential function of electronic devices such as display devices and mobile phones. The Miracast projection function utilizes the peer-to-peer (Peer-to-peer) network transmission function, based on the Wifi display standard protocol (standard protocol, WFD protocol for short). The screen image and sound data of terminals that support the wireless screen projection function, such as mobile phones or tablets, are transmitted to other external devices for display, so as to achieve the purpose of screen sharing. Exemplarily, the screen image and sound data on the terminal may be transmitted to the display device, so that the display device can display it. For the support of the screen projection function, the Wifi module on the display device needs to support P2P and STA mode (STAtion). When the display device needs to use the screen projection function, the Wifi module needs to work in the P2P mode. When the display device is connected to the Internet normally, the Wifi The module needs to work in STA mode.

The Wifi module can use the time-slice time-division multiplexing technology, and work in both P2P mode and STA mode at the same time, allowing users to cast screens anytime and anywhere while using the display device functions normally. Time-division multiplexing Wifi module means that the Wifi module switches between P2P mode and STA mode in turn. The Wifi module can be converted from STA mode to P2P mode, then converted from P2P mode to STA mode, and then continue to switch from STA mode to P2P pattern, and so on.

However, when the display device is in the Miracast screen projection function, mosaics often appear in the picture displayed on the display device, as shown in FIG. 21 , which shows a schematic diagram of a user interface of a display device according to some embodiments. The user interface appears to display mosaics, which affects the user's viewing.

Therefore, when using the Miracast screen projection function, a new solution is proposed for the mosaic situation on the screen.

Specifically, by setting the Wifi module on the display, so that the Wifi module is in the P2P mode for a longer time than in the STA mode, and controls the Wifi module to receive the media data sent by the target device in response to the screen projection request. This can ensure that a large amount of projection data will not be lost, and mosaic will not appear.

After analysis, it is found that this situation is caused by long-term packet loss when the terminal sends screen projection data to the display device. Since the Wifi module is time-division multiplexed, when the Wifi module is in STA mode, it cannot receive the projection data, so the Wifi module misses the projection data, that is, packet loss occurs. If a large amount of projection data is not received, it will Affects the picture on the display device, resulting in mosaic.

In order to solve the above technical problems, the embodiment of the present disclosure provides a screen projection method, in which the Wifi module is set to be in the P2P mode for a longer time than the STA mode time, so as to ensure that a large amount of screen projection data will not be lost and mosaic will not appear . As shown in FIG. 22, FIG. 22 shows a flow chart of a screen projection method according to some embodiments. The method includes S2210-S2230. The method is described in detail below:

In the foregoing embodiments, an example is described in which the display device sends a screen projection request to the target device. In an actual application process, the target device may also send a screen projection request to the display device. It can be set according to the user's needs and the functions of the display device and the target device. Next, take the target device sending a screen casting request to the display device as an example to describe the screen casting method.

S2210. Receive a screen projection request sent by the target device, and complete a screen projection connection with the target device.

In the embodiment of the present disclosure, the target device may respond to the user's operation, so that the target device sends a screen projection request to the display device. In some embodiments, the target device may be a terminal that supports a wireless screen projection function, such as a mobile phone or a tablet. Exemplarily, as shown in FIG. 23, FIG. 23 shows a schematic diagram of a user interface of another display device according to some embodiments. In FIG. 23, a list of screen-casting devices searched by the target device is displayed, and the device list All the devices in can realize the screen projection function with the display device, and the device list includes three device names named "terminal A", "terminal B" and "terminal C". When the terminal is a touch-screen mobile phone, the touch-screen mobile phone can respond to the position corresponding to the user's hand pressing the device control to achieve the purpose of selecting the device. When the position corresponding to the terminal C control on the touch-screen mobile phone is touched, and terminal C is When displaying the device, send a screencasting request to the display device. In the embodiments of the present disclosure, there is no limitation on how to call up the screen projection device list on the target device, and any method that can realize the display device to display the screen projection device list is acceptable.

In some embodiments, the devices in the screen-casting device list are arranged according to the number of screen-casting times, and the devices with more screen-casting times are arranged in front of the device list. In some embodiments, other arrangements can also be used to arrange the devices in the screen projection device list. For example, the arrangement order of the devices is preset in the display device. The screen-casting devices are pre-set in order to sort the devices that can currently realize the screen-casting connection with the display device.

In some embodiments, when the display device receives the screen projection request, a notification message is displayed on the display device, and the notification message may be used to ask the user whether to agree to screen projection between the display device and the terminal. The display device judges whether the user agrees to cast the screen between the display device and the target device according to the user's operation. If it is detected that the screen projection between the display device and the target device is confirmed, the display device sends an instruction to agree to the screen projection to the target device, and the target device and the display device perform projection. screen connection. After the screen projection connection is executed, the target device can send screen projection data to the display device.

As shown in FIG. 24, FIG. 24 shows a schematic diagram of another user interface of a display device according to some embodiments. In FIG. 24, the content of the query message is "whether you agree to the target device to perform screen projection". Along with the query message, an agree control and a disagree control are displayed. The control device allows the user to move the focus to the consent control and press the confirmation key on the control device. At this time, the screen projection consent command is generated, and the screen projection consent command is sent to the target device, and the target device and the display device perform screen projection connection. In addition, the control device can also allow the user to move the focus to the disagreement control, and in response to the user's pressing operation on the confirmation key on the control device, at this time, a disagreement screen projection command is generated. At this time, the display device does not agree with the target device. Execute the screencast connection.

It should be noted that, in the embodiment of the present disclosure, when the focus is moved to the control, the border of the control can be thickened. In addition, other forms can also be used to indicate that the control is selected. Exemplarily, when the focus moves to the control, the Controls can change shape, from square to circular, etc., and when the control is selected, the control will be enlarged according to the preset ratio. For example, the display area of the video control on the user interface is the same. When a certain control is selected , the display area of the control is 1.2 times larger than that of the original control. Since the present disclosure does not limit the focus on the control and the shape of the control, other forms that can facilitate the user to distinguish that the video control is selected can be accepted.

In some embodiments, other methods can be used to realize the generation of the screen projection instruction and the generation of the screen projection instruction of disapproval. For example, a preset button is set on the control device. Instruction; when the second preset button is pressed, a screen projection instruction that does not agree is generated. The inquiry message can also be played in the form of voice, and the display device detects whether the user agrees to the voice of screen projection according to the content of the inquiry message. Exemplarily, the display device plays "whether you agree to the target device's screen projection", detects the user's "agree" answer within a certain period of time, and generates an instruction to agree to the screen projection at this time. It is also possible to detect "disagree" answered by the user within a certain period of time, and at this time, a disagree screen projection instruction is generated. In addition, if the user's reply is not detected within a certain period of time, in order to ensure that the display device does not play the screen projection data of the terminal at will, a disapproval screen projection instruction can be generated after the user has heard the inquiry message played by the display device. In some embodiments, while the query message is displayed on the user interface of the display device, the query message is played by voice, so that the user can be reminded to check the query message and avoid missing the projected screen content.

In some embodiments, before receiving the screen projection request sent by the target device, the method further includes:

Receive a power-on command, and determine whether the state of the screen projection switch is on. In some embodiments, the power-on button on the press control device can be pressed by the user to generate a power-on command. In some other embodiments, the power button provided on the display device can also be pressed by the user to generate a power-on instruction.

In the embodiment of the present disclosure, when the display device is turned on, the use mode of the Wifi module is determined according to the state of the screen projection switch. In the embodiment of the present disclosure, a screen projection switch is provided on the user interface of the display device, and the screen projection switch controls the use of the screen projection function of the display device. The setting of the screen projection switch is convenient for the user to set whether to use the screen projection function. Exemplarily, as shown in FIG. 25 , a schematic diagram of a user interface of another display device according to some embodiments is shown in FIG. 25 , and a screen projection switch is displayed on the display interface in FIG. 25 . When the user wants to use the screen projection function, the state of the screen projection switch is set to the on state, and the screen projection can be performed at any time. For users who do not use the screen mirroring function frequently, the status of the screen mirroring switch can be set to off.

Exemplarily, referring to Fig. 25 again, in Fig. 25, the state of the screen projection switch is off, the control device can be operated by the user, and the focus can be moved to the screen projection switch control, and the confirmation key or right button on the control device can be used by the user Press to switch the status of the screen projection switch to on. As shown in FIG. 26 , a schematic diagram of a user interface of another display device according to some embodiments is shown in FIG. 26 . In FIG. 26 , the screen projection switch is turned on. When the status of the screen projection switch is on, press the confirmation button on the control device, or press the left button to switch the status of the screen projection switch to off. Refer to the screen projection switch in Figure 25 again. The state of the screen projection switch is Disabled.

In some embodiments, if the state of the screen projection switch is on, set the usage mode of the Wifi module to the third mode, wherein the time in STA mode in the third mode is longer than the time in P2P mode. For users who often use the screen mirroring function, the status of the screen mirroring switch can be set to on. When the display device is turned on, the usage mode of the Wifi module is the third mode.

In the embodiment of the present disclosure, if the status of the screen projection switch is on, it means that the user wants to use the screen projection function. Before receiving the screen projection request from the terminal, set the Wifi module to be in STA mode for a longer time than in P2P mode , which can meet the needs of users for normal use of the Internet and ensure that users can use the Internet function normally. In addition, the Wifi module spends less time in P2P mode, and it will not have much impact, because there is no real screen projection at this time, and no screen projection data is transmitted to the display device, so the display device can still be used for less time in P2P mode. In the discovered state, specifically, when the terminal searches for a device that can cast a screen, the display device can be discovered, and the device discovered by the display device can be displayed in the screen projection device list. For example, refer to FIG. 23 again. The discovered devices can be displayed in the list of screen-casting devices shown in FIG. 7 . When the display device is in the discovered state, the display device can be searched for screen projection at any time. After the screen projection connection is completed, you can wait for the terminal to send the screen projection data.

The usage mode of the display device is the third mode. For example, as shown in FIG. 27, FIG. 27 shows a schematic diagram of switching the usage mode of a Wifi module according to some embodiments, switching from STA mode to P2P mode, Then switch from P2P mode to STA mode for a time period, if a time period is 500 unit time, in STA mode is 310 unit time, P2P mode and STA mode switching time is 70 unit time, in STA mode is 500-310- 140=50 unit time. In actual work, the Wifi module is constantly switching between P2P mode and STA mode, and the time in P2P mode is longer than that in STA mode. In Figure 27, first stay in STA mode for 310 units of time, then switch from STA mode to P2P mode for 70 units of time, continue to stay in P2P mode for 50 units of time, then switch from P2P mode to STA mode for 70 units of time, and then repeatedly enter STA mode, according to this The regularity continues to switch the mode the Wifi module is in.

In some embodiments, if the state of the screen projection switch is off, then set the use mode of the Wifi module to the fourth mode, wherein the fourth mode is the STA mode. In the embodiment of the present disclosure, when the status of the screen projection switch is off, it means that the user does not want to use the screen projection function for the time being. After the display device is turned on, the use mode of the Wifi module is directly set to STA mode, so that the Wifi module will not be time-shared. Multiplexing can make the Wifi module completely in STA mode and improve the performance of the display device connected to the Internet.

To give an example, user A is an elderly person, and generally does not use the screen projection function, so the state of the screen projection switch is set to off. However, I often watch online TV dramas on the Internet, so setting the Wifi module to STA mode can save resources and improve the normal network speed and quality in single mode STA mode.

When the usage mode of the Wifi module is the fourth mode, an instruction to start the P2P mode is received, and it is judged whether it is currently controlled by the first terminal. In the embodiment of the present disclosure, when the Wifi module is in the STA mode, the display device cannot perform the screen projection operation. If you want to use the screen projection function later, the display device can be used by the user to continue to operate, and continue to enable the display device to realize the screen projection function.

In the embodiment of the present disclosure, when there is a need to use the screen projection function after the state of the screen projection switch is set to off, at this time, in addition to setting the state of the screen projection switch to the on state, it can also be used separately on the display device. An application that is set only to start the P2P mode, and use this application to realize the screen projection function.

The instruction to start the P2P mode may be generated by the user moving the focus to the application control for starting the P2P mode through the control device and pressing the confirmation key on the control device. Exemplarily, as shown in FIG. 28 , FIG. 12 shows a schematic diagram of a user interface of another display device according to some embodiments, and application controls for starting the P2P mode are displayed on the user interface in FIG. 28 . In the embodiments of the present disclosure, the specific location where the display device displays the application control for starting the P2P mode, and how to control the display of the application control on the user interface are not limited.

In some embodiments, the application control for starting the P2P mode can be set on the home page interface of the display device. In other embodiments, the application control for starting the P2P mode can also be set on the secondary interface. For example, in The first-level interface, that is, the home page interface, displays setting controls. The control device allows the user to move the focus to the setting control and press the confirmation key on the control device. The user interface jumps to the second-level corresponding to the setting control. interface, the application control for starting the P2P mode is displayed on the secondary interface, and of course other controls related to setting display device properties can also be displayed on the secondary interface.

In the embodiment of the present disclosure, after receiving the instruction to start the P2P mode, it is judged whether it is currently controlled by the first terminal. In the embodiment of the present disclosure, the display device determines the usage mode of the Wifi module according to its own actual situation.

In the embodiment of the present disclosure, the display device may be controlled by the control device, and the display device may also be controlled by an application configured in the first terminal for controlling the display device, and the application is similar to a virtual control device. Judging whether it is currently controlled by the first terminal may be understood as whether the display device is connected to an application set in the first terminal for controlling the display device. In some embodiments, the application for controlling the display device is in the same local area network as the display device, and when the Wifi module of the display device is in STA mode, it can control the display device.

If it is controlled by the first terminal, set the use mode of the Wifi module to the first mode, wherein the time of the P2P mode in the first mode is longer than the time of the STA mode. In the embodiment of the present disclosure, after the instruction to start the P2P mode is received, the operation to be performed is determined according to the current situation of the display device, that is, whether the display device is controlled by the first terminal.

In some embodiments, if the display device is controlled by the first terminal, that is, connected to an application in the first terminal that controls the display device, the usage mode of the Wifi module is set to the first mode. In the embodiment of the present disclosure, when an instruction to start the P2P mode is received, it means that the user wants to use the screen projection function, and when the display device is controlled by the first terminal, the Wifi module is time-division multiplexed to set the usage mode of the Wifi module for the first mode.

In the embodiment of the present disclosure, the connection between the application for controlling the display device in the first terminal and the display device requires that the Wifi module is in the STA mode. In addition, in order to ensure that the Wifi module will not miss a large amount of projection data, set the Wifi module to be in P2P mode for a longer time than in STA mode. Setting the use mode of the Wifi module to the first mode in this way can not only ensure that the application for controlling the display device set on the first terminal can still control the display device, but also prevent the Wifi module from missing a large amount of projection data.

If the display device is not controlled by the first terminal, the use mode of the Wifi module is set to the second mode. In some embodiments, if the display device is not controlled by the first terminal, that is, it is not connected to the application used to control the display device in the first terminal, then the use mode of the Wifi module is set to the second mode, and the second mode is only P2P model. This is because the application used to control the display device in the first terminal is not connected to the display device, and does not require the Wifi module to be in STA mode, so directly set the Wifi module of the display device as a P2P module, which can completely avoid missing the Wifi module Screen data.

The above are the steps performed before the screen projection connection with the target device is completed. The following describes the steps performed by the method after the screen projection connection with the target device is completed.

S2220. Set the Wifi module, so that the time of the Wifi module in the P2P mode is longer than the time in the STA mode. In the embodiment of the present disclosure, after the display device and the target device complete the screen projection connection, the target device will send the screen projection data to the display device so that the display device can display the screen projection data. In order to ensure that the Wifi module in the display device does not A large amount of projection data will be missed, and the time for the Wifi module to be in P2P mode is longer than that in STA mode.

In some embodiments, the method for setting the Wifi module includes: judging whether it is currently controlled by the first terminal. The display device now needs to perform miracast projection, so it is necessary to set a better state so that there will be no mosaic on the screen of the display device. First, it is determined whether the current display device is controlled by the first terminal. In some embodiments, whether the current display device is controlled by the first terminal may be determined by determining whether the display device is connected to an application for controlling the display device. If the application used to control the display device is connected, the Wifi module needs to be time-division multiplexed, increase the time-sharing parameters of the P2P mode, reduce the time-sharing parameters of the STA, and set the time of the Wifi module in P2P mode to be longer than the time in STA mode. If there is no application connected to control the display device, just set the overall mode of the display device to P2P mode, which can reduce the packet loss rate.

In some embodiments, judging whether it is currently controlled by the first terminal may be understood as whether the display device is connected to an application set in the first terminal for controlling the display device. The application for controlling the display device has been introduced above, and will not be repeated here.

Specifically, if it is controlled by the first terminal, the usage mode of the Wifi module is set to the first mode, wherein the time of the P2P mode in the first mode is longer than the time of the STA mode. In some embodiments, if the display device is controlled by the first terminal, that is, connected to the application that controls the display device in the first terminal, then the usage mode of the Wifi module is set to the first mode, and in the first mode, it is still time-division multiplexing The WIFI module, this is because the application of the control display device in the first terminal requires the Wifi module to be in STA mode, but in order to ensure that the Wifi module will not miss a large amount of projection data, the Wifi module is set to be in P2P mode The time is longer than the time in STA mode, so that it can not only ensure that the application for controlling the display device set on the first terminal can still control the display device, but also prevent the Wifi module from missing a large amount of projection data.

Exemplarily, one time period is 500 unit time, that is, switching from STA mode to P2P mode, and then switching from P2P mode to STA mode. In P2P mode, it takes 300 unit time, the switching time between P2P mode and STA mode is 70 unit time, in STA mode, it is 500-300-140=60 unit time, as shown in Figure 29, which shows that according to some embodiments Another schematic diagram of switching the use mode of the Wifi module, first in STA mode for 60 unit time, then STA mode is converted to P2P mode for 70 unit time, continue to be in P2P mode for 300 unit time, and then P2P mode is converted to STA mode for 70 unit time , and then repeatedly enter the STA mode, and continue to switch the mode of the Wifi module according to this rule.

To give an example, user B likes to cast the content on the terminal to the display device to share with his family, so the screen mirroring switch is set to on by default. In this way, you can cast your screen at any time every time you use it. If the control device at home is not found, an application for controlling the display device set on the terminal can be used to control the display device. The display device performs a screen projection operation, and the whole display device finds that the display device has been connected to the application for controlling the display device. At this time, the Wifi module is time-division multiplexed, so that the display device can be controlled by the application for controlling the display device. Screencasting is also possible. When the Wifi module is time-division multiplexed, it is necessary to increase the time in P2P mode and reduce the time in STA mode. Because the application used to control the display device is a small amount of data for the control of the display device, the time spent in the STA mode can be reduced. However, the amount of projection data transmitted by the projection screen is relatively large, and the time in the P2P mode needs to be increased. The control device at home was found in the back, and the display device was disconnected from the application used to control the display device. At this time, the display device is in the screen projection state. In order to ensure the reliability of receiving packets of the display device, the picture quality is guaranteed.

If it is not controlled by the first terminal, then set the usage mode of the Wifi module to the second mode, wherein the second mode is the P2P mode. In some embodiments, if the display device is not controlled by the first terminal, that is, it is not connected to the application controlling the display device in the first terminal, then the usage mode of the Wifi module is set to the second mode, and the second mode is only the P2P mode . This is because if there is no application controlling the display device in the first terminal connected to the display device, the Wifi module does not need to be in STA mode, so directly set the Wifi module of the display device as a P2P module, which can avoid the Wifi module missing connection screen data.

In the embodiment of the present disclosure, the display device searches for the display device at any time in the normal use of the Internet function scene to cast the screen. After the display device successfully casts the screen, the Wifi The use mode of the module is set to only P2P mode or to increase the time of P2P mode to ensure data quality. After the screen projection is over, return to the coexistence mode of P2P mode and STA mode again.

S2230. Control the Wifi module to receive the screen projection data sent by the target device; play the screen projection data. In the embodiment of the present disclosure, after the setting of the Wifi module is completed, the set Wifi module is controlled to receive the screen projection data sent by the target device. Use the display device to play the projection data.

In some embodiments, the screen projection end instruction is received, the screen projection connection is disconnected from the target device, and the status of the screen projection switch is determined. In the embodiment of the present disclosure, the user can perform the operation of disconnecting the screen projection on the display device, and the operation of disconnecting the screen projection can also be performed on the target device. After disconnecting the screen projection connection with the target device, judge the state of the screen projection switch. If the state of the screen projection switch is on, set the use mode of the Wifi module to the third mode, where the STA mode time in the third mode is longer than P2P mode time. In the embodiment of the present disclosure, when the status of the screen projection switch is on, it means that the user wants to use the display device to realize the screen projection function, but at this time the screen projection operation is not actually performed, and the function is mainly realized by using the display device to connect to the Internet , so set the time of the Wifi module in STA mode to be longer than the time of P2P mode. This will not only enable the display device to connect to the Internet normally and realize the Internet access function, but also enable the display device to be searched by the terminal and realize screen projection at any time.

If the state of the screen projection switch is off, set the usage mode of the Wifi module to the fourth mode, wherein the fourth mode is the STA mode. In the embodiment of the present disclosure, if the status of the screen projection switch is off, it means that the user does not want the display device to implement the screen projection function, so the use mode of the Wifi module is directly set to the STA mode.

In the embodiment of the present disclosure, after completing the screen projection connection with the target device, use the Wifi module whose P2P mode time is longer than the STA mode time to receive the screen projection data, so as to ensure that the data will not be missed, or that the screen projection data will be missed in a small amount In the case of avoiding mosaic.

An embodiment of the present disclosure further provides a display device, including: a display, a user interface, and a controller. Among them, the display is used to display images and/or user interfaces; the user interface is used to receive input signals; the controllers connected to the display and the user interface are used to perform: receive the screen projection request sent by the target device, and communicate with the target device Complete the screen projection connection, set the Wifi module so that the Wifi module is in P2P mode longer than the time in STA mode; control the Wifi module to receive the projection data sent by the target device; play the projection data.

In some embodiments, the controller is configured to set the Wifi module according to the following steps: determine whether it is currently controlled by the first terminal; if it is controlled by the first terminal, set the use mode of the Wifi module to the first mode, wherein the first mode The time in the P2P mode is longer than the time in the STA mode; if it is not controlled by the first terminal, set the use mode of the Wifi module to the second mode, wherein the second mode is the P2P mode.

In some embodiments, before receiving the screen projection request sent by the target device, the controller is further configured to: receive a power-on command, and determine whether the state of the screen projection switch is on; if the state of the screen projection switch is on, then Set the use mode of the Wifi module to the third mode, where the STA mode time in the third mode is longer than the P2P mode time.

In some embodiments, the controller is also used to execute: if the state of the screen projection switch is off, then set the usage mode of the Wifi module to the fourth mode, wherein the fourth mode is the STA mode; when the usage mode of the Wifi module is In the fourth mode, an instruction to start the P2P mode is received, and it is judged whether it is currently controlled by the first terminal; if it is controlled by the first terminal, the use mode of the Wifi module is set as the first mode; if it is not controlled by the first terminal, then Set the use mode of the Wifi module to the second mode.

In some embodiments, the controller is also used to perform: receiving the screen projection end instruction, disconnecting the screen projection connection with the target device, and judging the state of the screen projection switch; if the state of the screen projection switch is on, then setting the use of the Wifi module The mode is the third mode; if the state of the screen projection switch is off, set the usage mode of the Wifi module to the fourth mode.

For the content executed by the controller in the display device disclosed above, reference may be made to the above.

In the above embodiment, a screen projection method and a display device, in which the Wifi module is set to be in the P2P mode for a time longer than the STA mode time, so as to ensure that a large amount of screen projection data will not be lost and mosaic will not occur. The method includes: receiving a screen projection request sent by a target device, completing a screen projection connection with the target device, setting the Wifi module so that the time of the Wifi module in the P2P mode is longer than the time in the STA mode; controlling the Wifi module to receive the screen projection data sent by the target device ; Play the projection data.

The UI and its interactive operation methods provided in the present disclosure are only exemplary, and do not constitute a limitation to the present application, and the actual UI design and application shall prevail. In addition, the user's response/interaction operation on the UI interface in the present disclosure is not limited to the execution through the controls in the interface, and voice commands can also be input. Taking Figure 14 as an example, when the display device displays the UI interface in Figure 14, the user can say After the voice information of "permanent continuous connection" is displayed, and the result of device recognition and analysis is displayed, it is learned that the user's voice intention is to grant the smart device 2 the permission of permanent screen projection and continuous connection.

According to the screen projection command form, the device scan result of the screen projection receiver, the selection rules of the screen projection sender, the type of screen projection protocol/function, the screen projection connection method adapted to different projection protocols, the screen projection data transmission method, the projection In terms of the system and operating status of both ends of the screen interaction, the screen projection requirements of the business scene, etc., the screen projection mode of the present disclosure is flexibly and adaptively modified and expanded. The screen projection mode and its interaction logic are not limited to the examples of the present disclosure. The underlying processing logic of screen projection interaction between device A and device B can be adapted to different systems and different screen projection protocols/functions, and is not limited to the example shown in FIG. 20 .

In some embodiments, the present disclosure also provides a computer-readable non-volatile storage medium, which can store a program. When the computer storage medium is located in the display device, the program execution may include the program steps of the above-mentioned screen projection method at the display device end; when the computer storage medium is located in the smart device, the program execution may include the above-mentioned screen projection method at the smart device end program steps. Wherein, the computer storage medium may be a magnetic disk, an optical disk, a read-only memory (English: Read-Only Memory, ROM for short), or a random access memory (English: Random Access Memory, RAM for short).

For convenience of explanation, the above description has been made in conjunction with specific implementation manners. However, the above exemplary discussion is not intended to be exhaustive or to limit the implementations to the precise forms disclosed above. Many modifications and variations are possible in light of the above teachings. The selection and description of the above embodiments are for better explaining the content of the present disclosure, so that those skilled in the art can use the embodiments better.

Claims

A display device comprising:

displays for displaying images and/or user interfaces;

The communicator is used to communicate with the smart device;

a controller, coupled to said display and said communicator, configured to perform:

After receiving the screen projection instruction input by the user, scan the connectable device, and the connectable device is a screen projection sending terminal with the screen projection function enabled;

Sending a screen projection request to a target device in the connectable devices, where the screen projection request is used to instruct the target device to establish a screen projection connection with the display device;

receiving the media data sent by the target device in response to the screen projection request, and controlling the display to play the media data.
According to the display device according to claim 1, the display device further comprises a sound collector, and the communicator is also used for communication connection with the semantic server, then the controller is further configured to execute:

Controlling the sound collector to collect voice information;

sending the voice information to the semantic server, and the semantic server is used to parse the voice information;

receiving and identifying the analysis result fed back by the semantic server;

Identifying that the user's voice intention indicated by the analysis result is screen projection, that is, receiving the screen projection instruction.
According to the display device according to claim 2, the controller is further configured to send a screen projection request to the target device among the connectable devices in the following manner:

Recognizing that the parsing result contains the target device identifier, and detecting whether the target device identifier is included in the device list; the device list includes the device identifiers of connectable devices scanned by the display device;

Detecting that the target device identifier is included in the device list, sending a screen projection request to the smart device indicated by the target device identifier.
According to the display device according to claim 3, the controller is further configured to send a screen projection request to the target device among the connectable devices in the following manner:

It is detected that the target device identifier is not included in the device list, and a second prompt pop-up window is displayed, and the second prompt pop-up window is used to prompt that the smart device specified by the target device identifier has not been scanned, and ask whether to switch to Current connectable devices;

After receiving an instruction to refuse to switch to a connectable device, close the second prompt pop-up window, re-scan for connectable devices or exit the screen mirroring;

After receiving an instruction to agree to switch to a connectable device, close the second prompt pop-up window, set the target device according to preset rules, and send a screen projection request to the target device.
According to the display device according to claim 3, the controller is further configured to send a screen projection request to the target device among the connectable devices in the following manner:

Recognizing that the parsing result does not contain the target device identifier, setting the target device according to preset rules, and sending a screen projection request to the target device.
According to the display device according to claim 4 or 5, the preset rule for setting the target device is configured as:

Obtain the current S value of the device list; the S value is used to represent the total number of connectable devices scanned by the display device;

If it is detected that the S value of the device list is zero, a first prompt pop-up window is displayed, and the first prompt pop-up window is used to prompt that there is currently no connectable device, and it is allowed to exit the screen projection or re-scan for connectable devices;

If it is detected that the S value is equal to 1, the only connectable device is set as the target device;

If it is detected that the S value is greater than 1, display the device list, and receive a selection operation on the target device in the device list.
According to the display device according to claim 4 or 5, the controller is further configured to perform:

After receiving the exit projection screen instruction, query the number of screen projection times between the display device and the target device; the number of screen projection times is cumulatively increased by one each time the screen projection connection is successful;

If it is inquired that the number of times of screen projection is equal to 1, the control display displays a third prompt pop-up window, and the third prompt pop-up window is used to ask whether the target device is granted continuous permission for screen projection; the continuous permission for screen projection It is used to instruct the display device to automatically send a screen projection request to the first device when the first device is scanned;

Receiving an instruction of refusing to grant continuous screen projection permission to the target device, closing the third prompt pop-up window, and disconnecting the screen projection connection with the target device;

After receiving an instruction agreeing to grant a single screen projection continuous permission to the target device, close the third prompt pop-up window, disconnect the screen projection connection with the target device, and add the first Authorization ID;

After receiving an instruction agreeing to grant the target device a permanent continuous screen projection permission, close the third prompt pop-up window, disconnect the screen projection connection with the target device, and add a second permission for the target device logo.
The display device according to claim 7, the controller is further configured to perform:

After receiving the screen-casting exit command, if the number of screen-casting times is greater than 1, identify the permission type corresponding to the target device;

If the permission type is permanent continuous screen projection permission, the third prompt pop-up window is not displayed, and the screen projection connection with the target device is disconnected;

If the permission type is no screen projection continuous permission or single screen projection continuous permission, the third prompt pop-up window is displayed again, and the permission type of the target device is updated according to the user's authorization operation instruction.
According to the display device according to claim 7, the preset rule for setting the target device is configured as:

Querying M first devices that currently have the first authorization identifier or the second authorization identifier in the device list;

If the M value is equal to 1, setting the unique first device found as the target device;

If the value of M is greater than 1, according to the preset priority, the first device with the highest priority is set as the target device.
The display device according to claim 1, the controller is further configured to:

Set the Wifi module so that the time of the Wifi module in P2P mode is greater than the time in STA mode;

Controlling the Wifi module to receive the media data sent by the target device after responding to the screen projection request.
According to the display device according to claim 10, the controller is configured to set the Wifi module according to the following steps:

judging whether it is currently controlled by the first terminal;

If controlled by the first terminal, the use mode of the Wifi module is set to the first mode, wherein the time of the P2P mode in the first mode is greater than the time of the STA mode;

If it is not controlled by the first terminal, then set the usage mode of the Wifi module to the second mode, wherein the second mode is the P2P mode.
According to the display device according to claim 10, the controller, before receiving the screen projection request sent by the target device, is further configured to:

Receive the power-on command, and judge whether the state of the screen projection switch is on;

If the state of the screen projection switch is on, then set the use mode of the Wifi module to the third mode, wherein the STA mode time in the third mode is greater than the P2P mode time.
The display device according to claim 12, the controller is further configured to:

If the state of the screen projection switch is off, then the use mode of the Wifi module is set to the fourth mode, wherein the fourth mode is the STA mode;

When the use mode of the Wifi module is the fourth mode, an instruction to start the P2P mode is received, and it is judged whether it is currently controlled by the first terminal; if it is controlled by the first terminal, the use mode of the Wifi module is set as the first mode; if not Controlled by the first terminal, the use mode of the Wifi module is set to the second mode.
The display device according to claim 12, the controller is further configured to:

receiving the screen projection end instruction, disconnecting the screen projection connection with the target device, and judging the state of the screen projection switch;

If the state of the screen projection switch is on, then the use mode of the Wifi module is set to be the third mode; if the state of the screen projection switch is off, then the use mode of the Wifi module is set to the fourth mode.
A smart device comprising:

displays for displaying images and/or user interfaces;

The communicator is used for communicating with the display device;

a controller, coupled to said display and said communicator, configured to perform:

Receive the screen projection request sent by the display device; the screen projection request is sent by the display device after determining that the target device is the smart device from the scanned connectable devices in response to the screen projection instruction input by the user, and the connectable device It is the sending terminal for screen projection with the screen projection function enabled;

Responding to the screen projection request, establishing a screen projection connection with the display device;

After the screen projection connection is successful, the media data is sent to the display device.
A screen projection method, the method comprising:

The display device receives the screen projection instruction input by the user, scans the connectable device, and the connectable device is a screen projection sending terminal with the screen projection function enabled;

The display device sends a screen projection request to a target device in the connectable devices, where the screen projection request is used to instruct the target device to establish a screen projection connection with the display device;

The display device receives the media data sent by the target device in response to the screen projection request, and plays the media data.
According to the screen projection method according to claim 16, the display device receiving the media data sent by the target device after responding to the screen projection request comprises:

The display device sets the Wifi module so that the Wifi module is in P2P mode for a longer time than in STA mode;

The display device controls the Wifi module to receive the media data sent by the target device in response to the screen projection request.
The method according to claim 17, the method for setting the Wifi module comprises:

judging whether it is currently controlled by the first terminal;

If controlled by the first terminal, the use mode of the Wifi module is set to the first mode, wherein the time of the P2P mode in the first mode is greater than the time of the STA mode;

If it is not controlled by the first terminal, then set the usage mode of the Wifi module to the second mode, wherein the second mode is the P2P mode.
According to the method according to claim 17, before receiving the screen projection request sent by the first terminal, the method further comprises:

Receive the power-on command, and judge whether the state of the screen projection switch is on;

If the state of the screen projection switch is on, then set the use mode of the Wifi module to the third mode, wherein the STA mode time in the third mode is greater than the P2P mode time.
The method of claim 19, further comprising:

If the state of the screen projection switch is off, then the use mode of the Wifi module is set to the fourth mode, wherein the fourth mode is the STA mode;

When the use mode of the Wifi module is the fourth mode, an instruction to start the P2P mode is received, and it is judged whether it is currently controlled by the first terminal; if it is controlled by the first terminal, the use mode of the Wifi module is set as the first mode; if not Controlled by the first terminal, the use mode of the Wifi module is set to the second mode.
The method of claim 19, further comprising:

receiving the screen projection end instruction, disconnecting the screen projection connection from the first terminal, and judging the state of the screen projection switch;

If the state of the screen projection switch is on, then the use mode of the Wifi module is set to be the third mode; if the state of the screen projection switch is off, then the use mode of the Wifi module is set to the fourth mode.
A screen projection method, the method comprising:

The smart device receives the screen projection request sent by the display device; the screen projection request is sent by the display device after determining that the target device is the smart device from the scanned connectable devices in response to the screen projection instruction input by the user. The connected device is a screen projection sending terminal with the screen projection function enabled;

The smart device establishes a screen projection connection with the display device in response to the screen projection request;

After the screen projection connection is successful, the smart device sends media data to the display device.