WO2021213379A1 - 投屏显示方法、系统、终端设备和存储介质 - Google Patents

投屏显示方法、系统、终端设备和存储介质 Download PDF

Info

Publication number
WO2021213379A1
WO2021213379A1 PCT/CN2021/088371 CN2021088371W WO2021213379A1 WO 2021213379 A1 WO2021213379 A1 WO 2021213379A1 CN 2021088371 W CN2021088371 W CN 2021088371W WO 2021213379 A1 WO2021213379 A1 WO 2021213379A1
Authority
WO
WIPO (PCT)
Prior art keywords
window
display
source device
application
data
Prior art date
Application number
PCT/CN2021/088371
Other languages
English (en)
French (fr)
Inventor
陈光军
王波
朱培
方定庚
文平
吴同刚
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to US17/919,963 priority Critical patent/US20230176806A1/en
Priority to EP21792145.1A priority patent/EP4123437A4/en
Publication of WO2021213379A1 publication Critical patent/WO2021213379A1/zh

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/147Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41Structure of client; Structure of client peripherals
    • H04N21/4104Peripherals receiving signals from specially adapted client devices
    • H04N21/4122Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41Structure of client; Structure of client peripherals
    • H04N21/4104Peripherals receiving signals from specially adapted client devices
    • H04N21/4126The peripheral being portable, e.g. PDAs or mobile phones
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H04N21/4312Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/4363Adapting the video stream to a specific local network, e.g. a Bluetooth® network
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/4363Adapting the video stream to a specific local network, e.g. a Bluetooth® network
    • H04N21/43637Adapting the video stream to a specific local network, e.g. a Bluetooth® network involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/443OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
    • H04N21/4438Window management, e.g. event handling following interaction with the user interface
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47End-user applications
    • H04N21/472End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47End-user applications
    • H04N21/485End-user interface for client configuration
    • H04N21/4858End-user interface for client configuration for modifying screen layout parameters, e.g. fonts, size of the windows

Definitions

  • This application belongs to the technical field of terminal control, and in particular relates to a projection screen display method, system, terminal device and storage medium.
  • Projection display refers to the method of projecting the screen content of the source device onto the screen of the target device for display.
  • the user can project the application window of the mobile phone screen onto a large-screen TV for display to obtain a clearer and more delicate picture display effect.
  • screen projection software such as AirPlay and ApowerMirror are usually used to realize the screen display.
  • these screen projection software are based on the current user-visible window content of the source device to perform the screen projection, and cannot display the activated but invisible window content on the source device. Project the screen to the target device for display.
  • the embodiments of the present application provide a screen projection display method, system, terminal device, and storage medium, which can project the invisible window content on the source device to the target device for display.
  • an embodiment of the present application provides a screen projection display method, which is applied to a target device to be screened, and the method includes:
  • the window data After receiving the projection request information sent by the source device, obtain the window data of the application that the source device has started, the window data including the correspondence of at least two windows opened by the same application that the source device has started Display content;
  • the target device that is projected on the screen obtains the window data of the application that the source device has started.
  • These window data include not only the content of the window currently visible to the source device, but also the source device that is started but not visible.
  • the target device displays the window data in its own display interface, so that the invisible window contents on the source device can be projected to the target device for display.
  • the obtaining window data of the application that has been started by the source device may include:
  • the window component of each window is in an active state and the layer of each window component is in a visible state, and each window component is in a visible state.
  • the window data of the windows all include the corresponding window identifier;
  • the displaying the window data of the started application program may include:
  • the respective windows are constructed and displayed respectively.
  • the window components of the running windows of the source device's application are set to the active state in advance, and the layers of these window components are set to the visible state; then, the source device sets these running windows ( The window data including the currently visible windows of the source device and the invisible windows running in the background are sent to the target device; the target device can distinguish the window data of each different window according to the window identifier contained in each window data, and then can construct separately Obtain different windows; in addition, because the window components of these windows are activated and the layers are visible, the content of these windows can be displayed on the display interface of the target device, that is, the content of the invisible windows on the source device can be projected on the screen To display on the target device.
  • each of the windows includes a first window and a second window
  • the first window is a window not currently displayed by the source device
  • the second window is a window currently displayed by the source device
  • the respective windows may include:
  • the first window is added to the first area for display, and the second window is added to the second area for display.
  • the windows to be displayed on the target device include window A and window B.
  • Window A is the window that the source device is running but is not currently displayed
  • window B is the window currently displayed by the source device, which can be displayed in the target device’s display interface.
  • the first window to the first area for display, and adding the second window to the second area for display may further include:
  • the fourth window is added to the second area and displayed above the second window.
  • the first window to the first area for display, and adding the second window to the second area for display may further include:
  • the sixth window is added to the second area for display, and the second window is moved to the first area and covered To the top of the first window.
  • the window data of each of the windows further includes corresponding application information, window component information, and window attribute information, and the separately constructing and displaying the respective windows may include:
  • a corresponding window is constructed and displayed in the display interface.
  • window component information, and window attribute information contained in the window data of each window After obtaining the application information, window component information, and window attribute information contained in the window data of each window, the number, size, display position, and display mode of the windows to be displayed can be determined according to these information, and then constructed in the display interface And display the corresponding window. How to divide the area in the display interface, how many areas to divide, the size of the display window in each area, the display mode of the window, etc. can be set according to these information.
  • the corresponding window in the display interface can also include:
  • the first instruction, the target application information, and the target window component information are sent to the source device, so as to instruct the source device to determine the target application information and the target window component information respectively.
  • Operating target application and target window, and operating the target window of the target application based on the first instruction.
  • the target device can send the detected operation instruction, and the target application information and target window component information contained in the window data of the corresponding window to the source device, so that the source device can determine the target application and target window corresponding to the operation instruction, thereby
  • the target window in the target application performs corresponding operations.
  • the input event on the target device will be sent to the source device through the cross-device channel, and the source device will simulate the same event and send it to the corresponding window of the application for execution.
  • the second instruction is sent to the source device to instruct the source device to keep the current display interface unchanged, and based on the The second instruction operates the corresponding window running in the background of the source device.
  • the content displayed by the target device does not change, but when the user operates on the target device, the content displayed on the source device does not change, and the two devices are independent of each other.
  • the corresponding operation instruction will be sent to the source device, and the source device will operate the window X running in the background based on the operation instruction without affecting the content currently displayed on the source device , And then the source device transmits the refreshed window content of the background window X to the target device, and finally the content of the window X displayed by the target device will be refreshed along with it.
  • the embodiment of the present application provides another screen projection display method, which is applied to the source device of the screen projection, and the method includes:
  • the window data of the launched application is sent to the target device to instruct the target device to display the information of the launched application Window data, where the window data includes display content corresponding to at least two windows opened by the same application that has been started by the source device.
  • the source device sends the projection request information to the target device, and after receiving the confirmation information returned by the target device based on the projection request information, sends the window data of the launched application to the target device to indicate all
  • the target device displays the window data of the started application program.
  • the target device that is projected on the screen obtains the window data of the application that the source device has started.
  • These window data include not only the window content currently visible on the source device, but also the invisible window content of the source device.
  • the target device is in These window data are displayed in its own display interface, so that the invisible window content on the source device can be projected to the target device for display.
  • the sending window data of the started application to the target device to instruct the target device to display the window data of the started application may include:
  • the window components of each of the windows are in an activated state and the layers of each of the window components are in a visible state, and the window data of each of the windows includes a corresponding window identifier.
  • the target device can distinguish the window data of each different window according to the window identifier contained in each window data, and then can construct each different window separately; in addition, because the window components of these windows are activated and the layer is visible, it can be set in the target The contents of these windows are displayed on the display interface of the device.
  • the target device before sending the window data of each opened window in the started application to the target device, it may further include:
  • the window component of the new window is set to the active state, and the layer of the window component of the new window is set to the visible state.
  • the source device detects that an application has opened a new window, it sets the window component of the new window to the active state, and sets the layer of the window component to the visible state. Regardless of whether the new window is currently displayed or running in the background, the properties of the window components of the new window are specially set for subsequent projection to the target device for display.
  • an embodiment of the present application provides another projection display method, the method includes:
  • the source device sends a screencast request message to the target device
  • the target device After receiving the screen projection request information, the target device returns confirmation information to the source device based on the screen projection request information;
  • the source device After receiving the confirmation information, the source device sends window data of the started application to the target device, where the window data includes at least two windows opened by the same application that has been started by the source device Corresponding display content;
  • the target device obtains and displays the window data of the started application program.
  • the source device sends the screen projection request information to the target device.
  • the target device After receiving the screen projection request information, the target device returns confirmation information to the source device based on the screen projection request information.
  • the source device sends the window data of the started application to the target device, and finally the target device obtains and displays the window data of the application that has been started by the source device.
  • These window data include not only the content of the window currently visible on the source device, but also the content of the invisible window that the source device has started.
  • the target device displays these window data in its own display interface, so as to realize the invisible window content on the source device. Project the screen to the target device for display.
  • an embodiment of the present application provides a projection display system, including:
  • the source device is configured to send screen request information to the target device, and after receiving the confirmation information returned by the target device based on the screen request information, send window data of the started application to the target device,
  • the window data includes display content corresponding to at least two windows opened by the same application that has been started by the source device;
  • the target device is configured to, after receiving the screen projection request information sent by the source device, return confirmation information based on the screen projection request information to the source device to obtain and display the window data of the started application.
  • an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and running on the processor.
  • the processor executes the computer program, Implement the steps of the projection display method as proposed in the first aspect of the embodiment of the present application, or implement the steps of the projection display method as proposed in the second aspect of the embodiment of the present application.
  • an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements what is proposed in the first aspect of the embodiments of the present application.
  • the embodiments of the present application provide a computer program product that, when the computer program product runs on a terminal device, causes the terminal device to execute the steps of the projection display method proposed in the first aspect of the embodiments of the present application, or execute Such as the steps of the projection screen display method proposed in the second aspect of the embodiment of the present application.
  • the embodiment of the present application has the beneficial effect that the invisible window content on the source device can be projected to the target device for display, which expands the comprehensiveness and flexibility of the projected display and is simple to operate. Strong practicality.
  • FIG. 1 is a schematic diagram of the hardware structure of a mobile phone to which the projection display method provided by an embodiment of the present application is applicable;
  • FIG. 2 is a schematic diagram of a projection screen display system provided by an embodiment of the present application.
  • FIG. 3 is a software architecture diagram of a projection display system provided by an embodiment of the present application.
  • FIG. 4 is a flowchart of a screen projection display method applied to a target device being screened according to an embodiment of the present application
  • FIG. 5 is a flowchart of another screen projection display method applied to a target device to be screened provided by an embodiment of the present application
  • FIG. 6 is a flowchart of a projection screen display method applied to a projection screen source device according to an embodiment of the present application
  • FIG. 7 is an interaction flowchart of a projection screen display method provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of the application of the projection screen display method provided by an embodiment of the present application in an actual scenario
  • FIG. 9 is a schematic diagram of the screen display of the tablet computer after the mobile phone returns to the desktop in the scene shown in FIG. 8;
  • FIG. 10 is a schematic diagram of the properties of the window component of window A and the window component of window B in the scene shown in FIG. 8;
  • FIG. 11 is a schematic diagram of layer division of window data of window A and window B in the scene shown in FIG. 8;
  • FIG. 12 is a schematic diagram of the application of the projection display method provided by an embodiment of the present application in a same source projection scenario;
  • FIG. 12 is a schematic diagram of the application of the projection display method provided by the embodiment of the present application in a heterogeneous projection scenario
  • FIG. 13 is a schematic diagram of a terminal device provided by an embodiment of the present application.
  • the projection display method provided by the embodiments of this application can be applied to mobile phones, tablet computers, wearable devices, in-vehicle devices, augmented reality (AR)/virtual reality (VR) devices, notebook computers, and super mobile personal On terminal devices or servers such as ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc., the embodiments of this application do not impose any restrictions on the specific types of terminal devices and servers.
  • UMPCs ultra-mobile personal computers
  • PDAs personal digital assistants
  • TVs large-screen TVs
  • the terminal device may be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a wireless local loop (Wireless Local Loop, WLL) station, Personal Digital Assistant (PDA) devices, handheld devices with wireless communication functions, computing devices or other processing devices connected to wireless modems, in-vehicle devices, car networking terminals, computers, laptop computers, handheld communication devices , Handheld computing equipment, satellite wireless equipment, wireless modem cards, TV set top boxes (STB), customer premise equipment (customer premise equipment, CPE), and/or other equipment used to communicate on wireless devices, and download
  • a first-generation communication device for example, a mobile terminal in a 5G network or a mobile terminal in a public land mobile network (PLMN) network that will evolve in the future.
  • PLMN public land mobile network
  • the wearable device can also be a general term for applying wearable technology to intelligently design daily wear and develop wearable devices, such as glasses, gloves, Watches, clothing and shoes, etc.
  • a wearable device is a portable device that is directly worn on the body or integrated into the user's clothes or accessories.
  • Wearable devices are not only a kind of hardware device, but also realize powerful functions through software support, data interaction, and cloud interaction.
  • wearable smart devices include full-featured, large-sized, complete or partial functions that can be implemented without relying on smart phones, such as smart watches or smart glasses, and only focus on a certain type of application function, and need to be used in conjunction with other devices such as smart phones. , Such as all kinds of smart bracelets and smart jewelry for physical sign monitoring.
  • Fig. 1 shows a block diagram of a part of the structure of a mobile phone provided in an embodiment of the present application.
  • the mobile phone includes: a radio frequency (RF) circuit 110, a memory 120, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a wireless fidelity (WiFi) module 170, and a processor 180 , And power supply 190 and other components.
  • RF radio frequency
  • the structure of the mobile phone shown in FIG. 1 does not constitute a limitation on the mobile phone, and may include more or fewer components than those shown in the figure, or a combination of some components, or different component arrangements.
  • the RF circuit 110 can be used for receiving and sending signals during the process of sending and receiving information or talking. In particular, after receiving the downlink information of the base station, it is processed by the processor 180; in addition, the designed uplink data is sent to the base station.
  • the RF circuit includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like.
  • the RF circuit 110 can also communicate with the network and other devices through wireless communication.
  • the above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (Code Division) Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE)), Email, Short Messaging Service (SMS), etc.
  • GSM Global System of Mobile Communication
  • GPRS General Packet Radio Service
  • CDMA Code Division Multiple Access
  • WCDMA Wideband Code Division Multiple Access
  • LTE Long Term Evolution
  • Email Short Messaging Service
  • the memory 120 may be used to store software programs and modules.
  • the processor 180 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 120.
  • the memory 120 may mainly include a storage program area and a storage data area.
  • the storage program area may store an operating device, an application program required by at least one function (such as a sound playback function, an image playback function, etc.), etc.; Data created by the use of mobile phones (such as audio data, phone book, etc.), etc.
  • the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.
  • the input unit 130 may be used to receive inputted digital or character information, and generate key signal input related to user settings and function control of the mobile phone.
  • the input unit 130 may include a touch panel 131 and other input devices 132.
  • the touch panel 131 also known as a touch screen, can collect user touch operations on or near it (for example, the user uses any suitable objects or accessories such as fingers, stylus, etc.) on the touch panel 131 or near the touch panel 131. Operation), and drive the corresponding connection device according to the preset program.
  • the touch panel 131 may include two parts: a touch detection device and a touch controller.
  • the touch detection device detects the user's touch position, and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it To the processor 180, and can receive and execute the commands sent by the processor 180.
  • the touch panel 131 can be implemented in multiple types such as resistive, capacitive, infrared, and surface acoustic wave.
  • the input unit 130 may also include other input devices 132.
  • the other input device 132 may include, but is not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), trackball, mouse, and joystick.
  • the display unit 140 may be used to display information input by the user or information provided to the user and various menus of the mobile phone.
  • the display unit 140 may include a display panel 141.
  • the display panel 141 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.
  • the touch panel 131 can cover the display panel 141. When the touch panel 131 detects a touch operation on or near it, it transmits it to the processor 180 to determine the type of the touch event, and then the processor 180 responds to the touch event. The type provides corresponding visual output on the display panel 141.
  • the touch panel 131 and the display panel 141 are used as two independent components to realize the input and input functions of the mobile phone, but in some embodiments, the touch panel 131 and the display panel 141 can be integrated. Realize the input and output functions of the mobile phone.
  • the mobile phone may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors.
  • the light sensor may include an ambient light sensor and a proximity sensor.
  • the ambient light sensor can adjust the brightness of the display panel 141 according to the brightness of the ambient light.
  • the proximity sensor can close the display panel 141 and/or when the mobile phone is moved to the ear. Or backlight.
  • the accelerometer sensor can detect the magnitude of acceleration in various directions (usually three-axis), and can detect the magnitude and direction of gravity when it is stationary.
  • the audio circuit 160, the speaker 161, and the microphone 162 can provide an audio interface between the user and the mobile phone.
  • the audio circuit 160 can transmit the electrical signal converted from the received audio data to the speaker 161, which is converted into a sound signal for output by the speaker 161; on the other hand, the microphone 162 converts the collected sound signal into an electrical signal, and the audio circuit 160 After being received, it is converted into audio data, and then processed by the audio data output processor 180, and then sent to, for example, another mobile phone via the RF circuit 110, or the audio data is output to the memory 120 for further processing.
  • WiFi is a short-distance wireless transmission technology.
  • the mobile phone can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 170. It provides users with wireless broadband Internet access.
  • FIG. 1 shows the WiFi module 170, it is understandable that it is not a necessary component of the mobile phone, and can be omitted as needed without changing the essence of the application.
  • the processor 180 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. Various functions and processing data of the mobile phone can be used to monitor the mobile phone as a whole.
  • the processor 180 may include one or more processing units; preferably, the processor 180 may integrate an application processor and a modem processor, where the application processor mainly processes operating devices, user interfaces, application programs, etc. , The modem processor mainly deals with wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 180.
  • the mobile phone also includes a power source 190 (such as a battery) for supplying power to various components.
  • a power source 190 such as a battery
  • the power source can be logically connected to the processor 180 through a power management device, so that functions such as charging, discharging, and power consumption management can be managed by the power management device.
  • the mobile phone may also include a camera, including at least one conventional RGB camera, and at least one infrared camera or hyperspectral camera.
  • a camera including at least one conventional RGB camera, and at least one infrared camera or hyperspectral camera.
  • the position of the camera on the mobile phone may be front-mounted or rear-mounted, which is not limited in the embodiment of the present application.
  • the mobile phone may include a single camera, a dual camera, or a triple camera, etc., which is not limited in the embodiment of the present application.
  • a mobile phone may include three cameras, of which one is the main camera, one is a wide-angle camera, and one is a telephoto camera.
  • the multiple cameras may be all front-mounted, or all rear-mounted, or partly front-mounted and another part rear-mounted, which is not limited in the embodiment of the present application.
  • the mobile phone may also include a Bluetooth module, etc., which will not be repeated here.
  • FIG. 2 is a schematic diagram of a projection display system according to an embodiment of the present application.
  • the projection display system includes a source device A and a target device B.
  • Both the source device A and the target device B can be various types of terminal devices, such as mobile phones, tablets, PCs, large-screen TVs and other devices.
  • the source device A and the target device B can be connected in various wired or wireless ways, such as communication data lines, Bluetooth, Wifi, etc., which are not limited in this application.
  • you need to connect the source device A and the target device B such as using a communication data line connection, Wifi interconnection, or starting a Bluetooth connection.
  • a dialog box will pop up on the display interface of source device A to prompt the user to connect. After the user clicks the connection is successful, you can Using the projection display method proposed in this application, each application window running on the source device A is projected onto the screen of the target device B for display.
  • the source device A is used to send the screen projection request information to the target device B, and after receiving the confirmation information returned by the target device B based on the screen projection request information, send the window data of the launched application to the target device B.
  • the target device B; the target device B is configured to, after receiving the projection request information sent by the source device A, return confirmation information based on the projection request information to the source device A, and obtain and display the Window data of the launched application.
  • the target device B may be specifically used to: obtain window data of each window that has been opened in the application program that the source device has started, and the window components of each window are in an active state and each The layers of the window components are in a visible state, and the window data of each window includes a corresponding window identifier; according to the window data of each window and the window identifier contained in the window data, each window is constructed and displayed respectively .
  • each of the windows includes a first window and a second window
  • the first window is a window not currently displayed by the source device
  • the second window is a window currently displayed by the source device
  • the target Device B can be specifically used for:
  • the first window is added to the first area for display, and the second window is added to the second area for display.
  • target device B may also be used for:
  • the fourth window is added to the second area and displayed above the second window.
  • target device B may also be used for:
  • the sixth window is added to the second area for display, and the second window is moved to the first area and covered To the top of the first window.
  • the window data of each of the windows also includes corresponding application information, window component information, and window attribute information, and the target device B can be specifically used for:
  • a corresponding window is constructed and displayed in the display interface.
  • target device B can also be used for:
  • the first instruction, the target application information, and the target window component information are sent to the source device, so as to instruct the source device to determine the target application information and the target window component information respectively.
  • Operating target application and target window, and operating the target window of the target application based on the first instruction.
  • target device B can also be used for:
  • the second instruction is sent to the source device to instruct the source device to keep the current display interface unchanged, and based on the The second instruction operates the corresponding window running in the background of the source device.
  • source device A may be specifically used for:
  • each window Send the window data of each window opened in the started application to the target device to instruct the target device to construct and display the window data of each window and the window identifier contained in the window data.
  • Each window wherein the window component of each window is in an active state and the layer of each window component is in a visible state, and the window data of each window includes a corresponding window identifier.
  • source device A may also be used for:
  • the window component of the new window is set to the active state, and the layer of the window component of the new window is set to the visible state.
  • Fig. 3 is a software architecture diagram of a projection display system provided by an embodiment of the present application.
  • the left half of Figure 3 is a diagram of the software architecture of the source device.
  • the source device is a device that actively casts a screen, such as a mobile phone.
  • the software architecture of the source device can be divided into two parts: the application layer and the framework layer.
  • the application layer includes system application modules and third-party application modules, which are used to manage various applications running on the device.
  • the frame layer includes a window management module, a cross-device management module, a data management module, and a drive module.
  • the window management module is used to manage the display modes of all windows of the application, including the coordinate size of the window display, the window display level, and the window display attributes, etc.
  • the cross-device management module is responsible for the proximity discovery, authentication, connection between the source device and the target device, application status awareness collaboration on the source device side, file drag and drop, message notification and other functions;
  • the data management module is responsible for the source device and the target device Transmission of projected audio, video, layer and other data streams between the screens, and is responsible for reverse event (that is, the event triggered by the target device to control the source device) control and other functions;
  • the driver module belongs to the underlying driver and is responsible for discovery and authentication , Configure the connection work, and provide driving events (such as device discovery, device information, device disappearance, etc.) to the cross-device management module, and execute commands issued by the cross-device management module, such as connect, disconnect and other commands.
  • the right half of Figure 3 is the software architecture diagram of the target device.
  • the target device is the device that is projected on the screen, such as a flat panel or a large-screen TV.
  • the software architecture of the target device can also be divided into two parts: the application layer and the framework layer.
  • the application layer includes the projection display APP, which is responsible for displaying the projection interface on the target device, and is also responsible for the management of the UI display interface.
  • the framework layer includes a window management module, a cross-device management module, a data management module, and a driver module.
  • the role of each functional module can refer to the role of the corresponding functional module in the framework layer of the source device.
  • FIG. 4 shows a flowchart of a projection display method applied to a target device to be projected according to an embodiment of the present application, including:
  • the user can click to send a screencast request message in the source device.
  • the target device After the target device receives the screencast request message, it can return a screencast confirmation message to the source device, and then obtain the Window data of an application that has been started by the source device, where the window data includes display content corresponding to at least two windows opened by the same application that has been started by the source device.
  • These window data may be layer data of more than one window opened by the same application that has been started by the source device, or layer data of multiple windows opened by different applications that have been started by the source device. Obviously, this part of the window data includes not only the content of the window currently visible by the source device, but also the content of the window that is currently activated but not visible by the source device.
  • the user can also click to send the screen request message on the target device. After the source device receives the screen request message, it will send the window of the application that the source device has started. Data to the target device.
  • the target device After receiving the window data of the application program that the source device has started, the target device can display this part of the window data on its own display screen. Since this part of the window data contains the contents of the windows that have been started but not visible on the source device, it is possible to project the contents of the invisible windows on the source device to the target device for display. Specifically, information such as the number, size, and application program of the windows to be displayed can be determined according to the obtained window data, and then the corresponding number and position of the window display area can be divided in the device display interface, and the display area is displayed in each area. The contents of the various windows that the source device has started.
  • the target device that is projected on the screen obtains the window data of the application that the source device has started.
  • These window data include not only the content of the window currently visible to the source device, but also the source device that is started but not visible.
  • the target device displays the window data in its own display interface, so that the invisible window contents on the source device can be projected to the target device for display.
  • Fig. 5 shows a flowchart of another screen projection display method applied to a target device being screened provided by an embodiment of the present application, including:
  • the target device After receiving the screen projection request information sent by the source device, the target device obtains the window data of each window that has been opened in the application program that the source device has started. For example, if the source device has started application A and opened windows A 1 and A 2 in application A, the target device will obtain the window data of A 1 and the window data of A 2 respectively; if the source device has started the application A and B, windows A 1 and A 2 are opened in application A, and windows B 1 , B 2 and B 3 are opened in application B, then the target device obtains the window data of A 1 and the window of A 2 respectively data, the window data B 1, B 2 of the window data and the window data of B 3.
  • the source device In order to display the content of each window in the target device, the source device needs to perform special processing on each window of the application in advance, so that the window components of each window are activated and the layers of each window component are Visible status.
  • the basic unit for realizing the differentiation and display of the windows in the application is the Activity component, that is, the window component.
  • the status of the Activity component is resumed (active state), and the layer visiable attribute is true (that is, the layer is visible); for windows that are not currently visible on the source device, The status of the Activity component is paused or stopped (stopped or terminated), and the layer's visiable attribute is false (that is, the layer is in an invisible state).
  • the status of the Activity component of each invisible window in the source device is set to resumed, and the layer visiable property is set to true.
  • the target device receives window data of the same window (the window currently displayed by the source device), so there is no need to distinguish which window of which application the received window data belongs to.
  • the target device needs to distinguish which window the acquired window data belongs to, so the source device needs to add information such as a window identifier to the window data.
  • the window data of each window includes a corresponding window identifier, so that when the target device subsequently displays this part of the window data, it can construct and display the content of each window based on the window data of different windows.
  • the target device After the target device obtains the window data of each window opened in the application that the source device has started, it constructs and displays each window according to the window data of each window and the window identifier contained in the window data. . For example, if the target device obtains the window data of window A and window B that the source device has started, two areas can be divided into the device display interface, and window A can be constructed based on the window data of window A. One area is displayed, and window B is constructed based on the window data of window B and displayed in another area.
  • the window data of each of the windows further includes corresponding application information, window component information, and window attribute information, and the separately constructing and displaying the respective windows may include:
  • the application information may include key information such as the identifier of the application, userid, and so on. These information mark a specific application and are used to distinguish data between applications.
  • Window component information such as Activity information. There may be multiple activities in an application, so the window data needs to carry various attribute information of the Activity, such as the token handle of the activity, the component name, and whether it is the main interface and other information.
  • Window attribute information such as Window information.
  • An Activity may contain multiple Window layers.
  • the Window layer is the basic unit of layer display. That is, an Activity may include multiple Window layers.
  • the window layer defines the layer size, Key information such as spacing. In actual operation, this part of information can be added to the Buffer data of the window data layer. For example, header information can be added to the Buffer data to mark the information of each layer.
  • the number, size, display position, and display mode of the windows to be displayed can be determined according to these information, and then constructed in the display interface And display the corresponding window. How to divide the area in the display interface, how many areas to divide, the size of the display window in each area, the display mode of the window, etc. can be set according to these information.
  • the application mode can be obtained in the application information, such as the navigation mode, and which window is the home page at the same time, so that the target device can display the home page window on the left side of the screen, and other windows stacked on the right side of the screen.
  • the corresponding window in the display interface can also include:
  • the target device constructs and displays the corresponding windows in the display interface
  • the user can operate these windows in the display interface of the target device, which is equivalent to performing reverse control of the corresponding windows in the source device.
  • the original process is to directly transmit the event coordinates and type information back to the source device.
  • the event that is transmitted back to the original device needs to increase the handle of the window component.
  • the information indicates which window the coordinate of the event falls on, so as to determine which window of the source device is to be operated.
  • the target device can send the detected operation instruction, and the target application information and target window component information contained in the window data of the corresponding window to the source device, so that the source device can determine the target application and target window corresponding to the operation instruction, thereby
  • the target window in the target application performs corresponding operations.
  • the input event on the target device will be sent to the source device through the cross-device channel, and the source device will simulate the same event and send it to the corresponding window of the application for execution. After the event is executed, if the content of the corresponding window of the source device is refreshed, the content of the corresponding window in the display interface of the target device will also be refreshed accordingly.
  • each of the windows includes a first window and a second window
  • the first window is a window not currently displayed by the source device
  • the second window is a window currently displayed by the source device
  • the respective windows may include:
  • the first window is added to the first area for display, and the second window is added to the second area for display.
  • the windows to be displayed on the target device include window A and window B.
  • Window A is the window that the source device is running but is not currently displayed
  • window B is the window currently displayed by the source device, which can be displayed in the target device’s display interface.
  • the first window to the first area for display, and adding the second window to the second area for display may further include:
  • the third window is added to the second area for display, and the second window is closed; if it is detected from the second window If the fourth window is opened in the window, the fourth window is added to the second area and displayed above the second window.
  • the first window to the first area for display, and adding the second window to the second area for display may further include:
  • the fifth window opened from the first window is detected, the fifth window is added to the second area for display, and the second window is closed; if it is detected from the second window Add the sixth window to the second area for display, and move the second window to the first area and cover it to be displayed above the first window .
  • the new window will continue to be added to the right area, that is, it will be overlaid on the top of the E window for display, and the system will close the E window after completion. If you continue to open a new window from the F window, the new window will be overlaid on top of the F window, and the F window will be moved to the left area, overlaid on the top of the B window, and so on.
  • the above display mode is the dual-window display mode of the target device, which depends on the application window currently running on the source device. If there is only one application window currently running, the only application window can be added to the left area for display.
  • the target device can also be set to single-window or multi-window display mode, that is, it can display only one application window currently running on the source device, or it can display multiple or all application windows currently running on the source device at the same time. It only needs to be displayed on the source device. It is enough to divide the corresponding number of window display areas in the interface.
  • the target device can switch from the dual-window display mode to the single-window display mode, and display the desktop content of the source device in a single window, which is a kind of same-source screencasting method.
  • the content displayed by the target device does not change, but when the user operates on the target device, the content displayed on the source device does not change, and the two devices are independent of each other.
  • the navigation mode and shopping mode described above are also supported.
  • the user operates on the source device as long as the operation object is not a window displayed on the target device, the display content of the target device does not change.
  • the corresponding operation instruction will be sent to the source device, and the source device will operate the window X running in the background based on the operation instruction, without composing the content currently displayed on the source device After that, the source device then transmits the refreshed window content of the background window X to the target device, and finally the content of window X displayed by the target device will be refreshed along with it.
  • the source device can run two displays (display0 and display1), where display0 displays the original content of the source device, and this content is visible to the user on the source device; the other display1 displays the application window projected to the target device Content. This content is not visible to the user on the source device, that is, it runs in the background of the source device. Events on the target device are transmitted to the application window on the source device display1 through the cross-device transmission channel.
  • the embodiment of this application will pre-set the window components of the running windows of the application of the source device to the active state before performing the projection display, and set the layers of these window components to the visible state; then, the source device sets these The window data of the running windows (which can include the windows currently visible on the source device and the invisible windows running in the background) are sent to the target device; the target device can distinguish the window data of each different window according to the window identifier contained in each window data , And then can build different windows separately; in addition, because the window components of these windows are activated and the layers are visible, the content of these windows can be displayed on the display interface of the target device, which means that the source device is invisible The contents of the window are projected to the target device for display.
  • Fig. 6 shows a flow chart of a screen projection display method applied to a screen projection source device provided by an embodiment of the present application, including:
  • the execution subject of the embodiments of the present application is the source device. After the source device and the target device are connected, the user can click on the source device to send a screencast request message to the target device.
  • the user can operate on the target device, agree to perform the projection display, and click Back to confirm the message to the source device.
  • the source device sends the window data of the started application to the target device to instruct the target device to display the window data of the application that has been started by the source device.
  • These window data may be layer data of more than one window opened by the same application that has been started by the source device, or layer data of multiple windows opened by different applications that have been started by the source device.
  • this part of the window data includes not only the content of the window currently visible by the source device, but also the content of the window that is currently activated but not visible by the source device.
  • the target device After the target device receives the window data of the application that the source device has started, it can display this part of the window data on its own display screen. Since this part of the window data contains the contents of the windows that have been started but not visible on the source device, it is possible to project the contents of the invisible windows on the source device to the target device for display. Specifically, the target device can determine the number, size, and application programs of the windows to be displayed according to the obtained window data, and then divide the corresponding number and position of the window display area in the device display interface, and each area is respectively Display the contents of the different windows that the source device has started.
  • the sending window data of the started application to the target device to instruct the target device to display the window data of the started application may include:
  • the window components of each of the windows are in an activated state and the layers of each of the window components are in a visible state
  • the window data of each of the windows includes a corresponding window identifier.
  • the target device can distinguish the window data of each different window according to the window identifier contained in each window data, and then can construct each different window separately; in addition, because the window components of these windows are activated and the layer is visible, it can be set in the target The contents of these windows are displayed on the display interface of the device.
  • the target device before sending the window data of each opened window in the started application to the target device, it may further include:
  • the window component of the new window is set to the active state, and the layer of the window component of the new window is set to the visible state.
  • the source device detects that an application has opened a new window, it sets the window component of the new window to the active state, and sets the layer of the window component to the visible state. Regardless of whether the new window is currently displayed or running in the background, the properties of the window components of the new window are specially set so that the screen can be subsequently projected to the target device for display.
  • the window management module in the software architecture diagram shown in Figure 3 can be used to modify the window component properties of each window.
  • the window management module has absolute management rights to all application windows, can set the properties of the window components, and manage the window components. Life cycle state.
  • the source device sends the screen projection request information to the target device, and after receiving the confirmation information returned by the target device based on the screen projection request information, sends the window data of the launched application to all
  • the target device is used to instruct the target device to display the window data of the started application program.
  • the target device that is projected on the screen obtains the window data of the application that the source device has started.
  • These window data include not only the window content currently visible on the source device, but also the invisible window content of the source device.
  • the target device is in These window data are displayed in its own display interface, so that the invisible window content on the source device can be projected to the target device for display.
  • FIG. 7 shows an interaction flow chart of a projection screen display method provided by an embodiment of the present application, including:
  • the source device sends screen projection request information to the target device;
  • the user can click on the source device to send a screencast request message to the target device.
  • the target device After receiving the screen projection request information, the target device returns confirmation information to the source device based on the screen projection request information.
  • the user can click on the target device to agree to the screen projection display operation, and return confirmation information to the source device based on the screen projection request information.
  • the source device After receiving the confirmation information, the source device sends window data of the started application to the target device.
  • the source device After receiving the confirmation information, the source device sends the window data of the started application to the target device.
  • These window data may be layer data of more than one window opened by the same application that has been started by the source device, or layer data of multiple windows opened by different applications that have been started by the source device.
  • the target device obtains and displays window data of the started application.
  • the target device After the target device receives the window data of the application that the source device has started, it can display this part of the window data on its own display screen. Since this part of the window data contains the contents of the windows that have been started but not visible on the source device, it is possible to project the contents of the invisible windows on the source device to the target device for display.
  • the source device first sends the screen projection request information to the target device.
  • the target device After receiving the screen projection request information, the target device returns confirmation information to the source device based on the screen projection request information.
  • the source device sends the window data of the started application to the target device, and finally the target device obtains and displays the window data of the application that has been started by the source device.
  • These window data include not only the content of the window currently visible on the source device, but also the content of the invisible window that the source device has started.
  • the target device displays these window data in its own display interface, so as to realize the invisible window content on the source device. Project the screen to the target device for display.
  • FIG. 8 is a schematic diagram of the application of the projection screen display method proposed in this application in an actual scenario.
  • the source device of the screen projection is a mobile phone
  • the target device of the projection screen is a tablet computer
  • the mobile phone and the tablet computer can be connected via Bluetooth, data cable or wifi.
  • operations such as device discovery and device connection can be performed via Bluetooth, and the screened window data can be transmitted via wifi, which has a lower data transmission delay.
  • the application window currently displayed on the mobile phone is Window B, and a certain application window running in the background is Window A.
  • the user can click a multi-screen collaboration button in the notification bar of the tablet, and then the tablet will pop up a projection screen to display the APP waiting interface.
  • the mobile phone A prompt box will pop up, asking the user to confirm the screencast connection; when the user clicks on the phone to connect and the connection is successful, the phone will send the window data of the started window A and B to the tablet; the tablet is receiving the window After the data, the A window and the B window are displayed on the display interface at the same time.
  • the size of the A window and the B window displayed on the tablet computer may be the same as the screen size of the mobile phone.
  • the number and position of the display windows can be set arbitrarily. For example, if the mobile phone has started running 10 application windows, 10 areas can be arbitrarily divided on the screen of the tablet computer and the 10 application windows can be displayed separately.
  • the tablet computer displays window A on the far left, and window B on the right of window A.
  • the user can set the screen display mode in the tablet computer, such as navigation mode or shopping mode, and the display mode of the corresponding new window opening, please refer to the description of the navigation mode or shopping mode mentioned above.
  • the user can perform reverse operation control on window A and window B on the tablet computer.
  • it When an operation is clicked on the tablet, it will simulate an identical operation event to the phone, trigger the phone to perform the corresponding operation, perform the refresh of the window content, and then the window content displayed on the tablet will follow the refresh.
  • window component Activity A of window A In the background of the mobile phone system, special processing is required for the properties of the window component Activity A of window A. As shown in Figure 10, the status of Activity A is set to resumed (active), and the layer visiable is set to true (layer visible ). In addition, since window B is the window currently displayed by the mobile phone, the status of its window component Activity B is also resumed, and the layer visiable is also true.
  • the tablet computer After the tablet computer receives the window data of window A and window B transmitted by the mobile phone, it needs to distinguish which application and which window the obtained window data belongs to. Specifically, header information can be added to the Buffer data of the window layer. To mark the properties of the layer. Assuming that both window A and window B are windows started by application M, the layer division diagram of window data is shown in Figure 11.
  • the attribute information of application M can be added to the layer buffer data of application M, such as application name and userid And other information, used to mark a specific application, in order to realize the data distinction between different applications.
  • Window component attribute information of window A to the layer buffer data of Activity A
  • window component attribute information of window B to the layer buffer data of Activity B, such as Activity's token handle, component name, and whether it is the main Interface and other information.
  • An Activity may contain multiple windows. Window is the basic unit of layer display. Key information such as layer size and spacing can be added to the layer buffer data of Window. After the tablet computer obtains window data with relevant layer attribute information, it can determine the display mode of each window according to the attribute information.
  • Figure 12 is a schematic diagram of the interface for switching from the same source screen to the different source screen for the mobile phone screen-casting tablet computer.
  • (A) in FIG. 12 is a scene where the same source screen is projected, and what the tablet computer displays is the screen content of the mobile phone, for example, the same window B is displayed.
  • the tablet computer When the user clicks a button in the interface of the tablet computer, it can trigger the way to enter the different source screen projection. At this time, even if the mobile phone is switched back to the desktop display, the tablet computer still displays the application window W 1 and application window W 2 running in the background that the mobile phone has started.
  • the mobile phone can run two displays (display0 and display1), of which display0 displays the original content of the mobile phone; the other virtual display1 displays the application window (ie application window W 1 and application window W 2 ) that is projected to the tablet.
  • Content This content is invisible to the user on the mobile phone. It runs in the background of the mobile phone and is visible to the user after being projected to the tablet.
  • the event triggered on the tablet is transmitted to the application window on the mobile phone display1 through the cross-device transmission channel.
  • the content displayed on the tablet computer does not change.
  • the content displayed on the mobile phone does not change and is independent of each other.
  • the event generated by the user operating the application window W 1 and the application window W 2 in the tablet interface is transmitted to W 1 and W 2 displayed in the display1 of the mobile phone background through the cross-device transmission channel, and the event processing is completed, and the mobile phone background is refreshed after W 1 and W 2 interface content, to cast screen tablet interface content of W 1 and W 2 with the refresh.
  • the application windows displayed by the target device run on the source device regardless of whether the screen is cast from the same source or from a different source.
  • the same source screen directly transfers the display content of the source device (display0) to the target device for display, and the heterogeneous screen transfer transfers the display content of the source device running on display1 in the background to the target device. show.
  • This application realizes the simultaneous display of multiple windows activated by the source device on the target device through the accurate management of window data transmission across devices, which expands the comprehensiveness and flexibility of the projection screen, and is simple to operate and has strong practicability.
  • the embodiments of the present application also provide a computer-readable storage medium that stores a computer program that, when executed by a processor, implements the steps of each projection display method as proposed in the present application.
  • the embodiments of the present application also provide a computer program product, which when the computer program product runs on a terminal device, causes the terminal device to execute the steps of each projection display method proposed in this application.
  • FIG. 13 is a schematic structural diagram of a terminal device provided by an embodiment of this application.
  • the terminal device 8 of this embodiment includes: at least one processor 80 (only one is shown in FIG. 13), a processor, a memory 81, and a processor that is stored in the memory 81 and can be processed in the at least one processor.
  • the terminal device 8 may be a computing device such as a mobile phone, a desktop computer, a notebook, a palmtop computer, and a cloud server.
  • the terminal device may include, but is not limited to, a processor 80 and a memory 81.
  • FIG. 13 is only an example of the terminal device 8 and does not constitute a limitation on the terminal device 8. It may include more or less components than shown in the figure, or a combination of certain components, or different components. , For example, can also include input and output devices, network access devices, and so on.
  • the so-called processor 80 may be a central processing unit (Central Processing Unit, CPU), and the processor 80 may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSPs), and application specific integrated circuits (Application Specific Integrated Circuits). , ASIC), ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the memory 81 may be an internal storage unit of the terminal device 8 in some embodiments, such as a hard disk or memory of the terminal device 8. In other embodiments, the memory 81 may also be an external storage device of the terminal device 8, such as a plug-in hard disk equipped on the terminal device 8, a smart media card (SMC), and a secure digital (Secure Digital, SD) card, Flash Card, etc. Further, the memory 81 may also include both an internal storage unit of the terminal device 8 and an external storage device.
  • the memory 81 is used to store an operating device, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 81 can also be used to temporarily store data that has been output or will be output.
  • the disclosed device and method may be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the modules or units is only a logical function division.
  • there may be other division methods for example, multiple units or components may be It can be combined or integrated into another device, or some features can be omitted or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the computer program can be stored in a computer-readable storage medium. When executed by the processor, the steps of the foregoing method embodiments can be implemented.
  • the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate forms.
  • the computer-readable medium may include at least any entity or device capable of carrying computer program code to a terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), and a random access memory (RAM, Random Access Memory), electric carrier signal, telecommunications signal and software distribution medium.
  • ROM read-only memory
  • RAM random access memory
  • electric carrier signal telecommunications signal and software distribution medium.
  • U disk mobile hard disk, floppy disk or CD-ROM, etc.
  • computer-readable media cannot be electrical carrier signals and telecommunication signals.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • User Interface Of Digital Computer (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

本申请适用于终端控制技术领域,提供了一种投屏显示方法、系统、终端设备和存储介质。该投屏显示方法包括:目标设备在接收到源设备发送的投屏请求信息后,获取所述源设备已启动的应用程序的窗口数据,所述窗口数据包括所述源设备已启动的同一个应用程序打开的至少两个窗口对应的显示内容;所述目标设备显示所述已启动的应用程序的窗口数据。被投屏的目标设备获取到的是源设备已启动的应用程序的窗口数据,这些窗口数据既包括源设备当前可见的窗口内容,也包括源设备已启动但不可见的窗口内容,目标设备在自身的显示界面中显示这些窗口数据,从而实现将源设备上不可见的窗口内容投屏到目标设备上显示。

Description

投屏显示方法、系统、终端设备和存储介质
本申请要求于2020年4月20日提交国家知识产权局、申请号为202010312360.0、申请名称为“投屏显示方法、系统、终端设备和存储介质”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
技术领域
本申请属于终端控制技术领域,尤其涉及一种投屏显示方法、系统、终端设备和存储介质。
背景技术
投屏显示是指将源设备的屏幕内容投影到目标设备的屏幕上进行显示的方法。例如,用户可以将手机屏幕的应用窗口投影到大屏电视上显示,以获得更清晰、更细腻的画面显示效果。
目前,通常采用AirPlay、ApowerMirror等投屏软件来实现投屏显示,然而这些投屏软件都是基于源设备当前用户可见的窗口内容进行投屏,无法将源设备上已启动但不可见的窗口内容投屏到目标设备上显示。
发明内容
有鉴于此,本申请实施例提供了一种投屏显示方法、系统、终端设备和存储介质,可以将源设备上不可见的窗口内容投屏到目标设备上显示。
第一方面,本申请实施例提供了一种投屏显示方法,应用于被投屏的目标设备,所述方法包括:
在接收到源设备发送的投屏请求信息后,获取所述源设备已启动的应用程序的窗口数据,所述窗口数据包括所述源设备已启动的同一个应用程序打开的至少两个窗口对应的显示内容;
显示所述已启动的应用程序的窗口数据。
在本申请实施例中,被投屏的目标设备获取到的是源设备已启动的应用程序的窗口数据,这些窗口数据既包括源设备当前可见的窗口内容,也包括源设备已启动但不可见的窗口内容,目标设备在自身的显示界面中显示这些窗口数据,从而实现将源设备上不可见的窗口内容投屏到目标设备上显示。
进一步的,所述获取所述源设备已启动的应用程序的窗口数据可以包括:
获取所述源设备已启动的应用程序中已打开的各个窗口的窗口数据,每个所述窗口的窗口组件均为激活状态且每个所述窗口组件的图层均为可见状态,每个所述窗口的窗口数据均包含对应的窗口标识;
所述显示所述已启动的应用程序的窗口数据可以包括:
根据所述各个窗口的窗口数据以及窗口数据中包含的窗口标识,分别构建并显示所述各个窗口。
在进行投屏显示前,预先将源设备的应用程序的已运行窗口的窗口组件设置为激活状态,并且将这些窗口组件的图层设置为可见状态;然后,源设备将这些已运行的 窗口(可以包括源设备当前可见的窗口,以及后台运行的不可见窗口)的窗口数据发送至目标设备;目标设备根据各个窗口数据中包含的窗口标识,能够区分各个不同窗口的窗口数据,进而能够分别构建得到各个不同的窗口;另外,由于这些窗口的窗口组件已激活且图层可见,故可以在目标设备的显示界面中显示这些窗口的内容,也即实现将源设备上不可见的窗口内容投屏到目标设备上显示。
进一步的,所述各个窗口包括第一窗口和第二窗口,所述第一窗口为所述源设备当前未显示的窗口,所述第二窗口为所述源设备当前显示的窗口,所述显示所述各个窗口可以包括:
在显示界面中划分出第一区域和第二区域;
将所述第一窗口添加到所述第一区域中显示,并将所述第二窗口添加到所述第二区域中显示。
比如,目标设备上要显示的窗口包括A窗口和B窗口,其中A窗口是源设备已运行但当前未显示的窗口,B窗口是源设备当前显示的窗口,则可以在目标设备的显示界面中划分一左一右两个区域,两个区域的尺寸可以相同也可以不同,每个区域的尺寸可以和源设备的屏幕尺寸相同。然后,将A窗口添加到左边的区域中显示,并将B窗口添加到右边的区域中显示。
更进一步的,在将所述第一窗口添加到所述第一区域中显示,并将所述第二窗口添加到所述第二区域中显示之后,还可以包括:
若检测到从所述第一窗口中打开的第三窗口,则将所述第三窗口添加到所述第二区域中显示,并关闭所述第二窗口;
若检测到从所述第二窗口中打开的第四窗口,则将所述第四窗口添加到所述第二区域中且覆盖到所述第二窗口的上方显示。
这属于导航模式的投屏显示方式,同样以上述例子进行说明,若检测到从A窗口中打开的新窗口C(用户在目标设备的显示界面中对A窗口进行操作打开C窗口),则将C窗口添加到该右边的区域中显示,即覆盖到B窗口上方显示,C窗口显示完毕后系统会关闭B窗口。若检测到从B窗口中打开的新窗口D,则将D窗口添加到该右边的区域中,覆盖到B窗口的上方层叠显示。
更进一步的,在将所述第一窗口添加到所述第一区域中显示,并将所述第二窗口添加到所述第二区域中显示之后,还可以包括:
若检测到从所述第一窗口中打开的第五窗口,则将所述第五窗口添加到所述第二区域中显示,并关闭所述第二窗口;
若检测到从所述第二窗口中打开的第六窗口,则将所述第六窗口添加到所述第二区域中显示,并将所述第二窗口移动到所述第一区域中且覆盖到所述第一窗口的上方显示。
这属于购物模式的投屏显示方法,同样以上述例子进行说明,若检测到从A窗口中打开的新窗口E,则将E窗口添加到该右边的区域中显示,即覆盖到B窗口上方显示,E窗口显示完毕后系统会关闭B窗口。若检测到从B窗口中打开的新窗口F,则将F窗口添加到该右边的区域中(即原来B窗口的位置)显示,同时将B窗口移动到左边的区域中,覆盖到A窗口的上方层叠显示。
进一步的,每个所述窗口的窗口数据还包含对应的应用程序信息、窗口组件信息和窗口属性信息,所述分别构建并显示所述各个窗口可以包括:
根据每个所述窗口的窗口数据包含的应用程序信息、窗口组件信息和窗口属性信息,确定待显示窗口的数量、尺寸、显示位置以及显示方式;
按照所述待显示窗口的数量、尺寸、显示位置以及显示方式,在显示界面中构建并显示相应的窗口。
在获得每个所述窗口的窗口数据包含的应用程序信息、窗口组件信息和窗口属性信息后,可以根据这些信息确定待显示窗口的数量、尺寸、显示位置以及显示方式,然后在显示界面中构建并显示相应的窗口。在显示界面中如何分区域,分多少个区域,每个区域显示窗口的大小,窗口的显示方式等均可以根据这些信息设定。
更进一步的,在显示界面中构建并显示相应的窗口之后,还可以包括:
若检测到对构建并显示的任意一个窗口进行操作的第一指令,则获取所述任意一个窗口的窗口数据包含的目标应用程序信息和目标窗口组件信息;
将所述第一指令、所述目标应用程序信息和所述目标窗口组件信息发送至所述源设备,以指示所述源设备根据所述目标应用程序信息和所述目标窗口组件信息分别确定待操作的目标应用程序和目标窗口,并基于所述第一指令对所述目标应用程序的所述目标窗口进行操作。
目标设备可以将检测到的操作指令,以及对应窗口的窗口数据包含的目标应用程序信息和目标窗口组件信息发送给源设备,以便源设备确定该操作指令对应的目标应用程序和目标窗口,从而对该目标应用程序中的目标窗口进行相应的操作。总的来说,目标设备上的输入事件,会通过跨设备通道发送到源设备,源设备再模拟一个同样的事件,发送给应用程序的相应窗口执行。
进一步的,在分别构建并显示所述各个窗口之后,还可以包括:
当检测到所述源设备当前的显示界面发生变化时,保持显示所述各个窗口的内容;
当检测到对构建并显示的所述各个窗口进行操作的第二指令后,将所述第二指令发送至所述源设备,以指示所述源设备保持当前的显示界面不变,并基于所述第二指令对所述源设备后台运行的对应窗口进行操作。
这属于异源投屏的方式,用户在源设备上操作时,目标设备显示的内容不变,而用户在目标设备上操作时,源设备显示的内容不变,两个设备相互独立。用户在目标设备上对某个窗口X进行操作时,相应的操作指令会发送至源设备,源设备基于该操作指令对后台运行的窗口X进行操作,不会对源设备当前显示的内容构成影响,然后源设备再将后台窗口X刷新的窗口内容传输至目标设备,最终目标设备显示的窗口X的内容会随着刷新。
第二方面,本申请实施例提供了另一种投屏显示方法,应用于投屏的源设备,所述方法包括:
发送投屏请求信息至目标设备;
在接收到所述目标设备基于所述投屏请求信息返回的确认信息后,发送已启动的应用程序的窗口数据至所述目标设备,以指示所述目标设备显示所述已启动的应用程序的窗口数据,所述窗口数据包括所述源设备已启动的同一个应用程序打开的至少两 个窗口对应的显示内容。
源设备发送投屏请求信息至目标设备,且在接收到所述目标设备基于所述投屏请求信息返回的确认信息后,发送已启动的应用程序的窗口数据至所述目标设备,以指示所述目标设备显示所述已启动的应用程序的窗口数据。被投屏的目标设备获取到的是源设备已启动的应用程序的窗口数据,这些窗口数据既包括源设备当前可见的窗口内容,也包括源设备已启动但不可见的窗口内容,目标设备在自身的显示界面中显示这些窗口数据,从而实现将源设备上不可见的窗口内容投屏到目标设备上显示。
进一步的,所述发送已启动的应用程序的窗口数据至所述目标设备,以指示所述目标设备显示所述已启动的应用程序的窗口数据可以包括:
发送已启动的应用程序中已打开的各个窗口的窗口数据至所述目标设备,以指示所述目标设备根据所述各个窗口的窗口数据以及窗口数据中包含的窗口标识,分别构建并显示所述各个窗口;
其中,每个所述窗口的窗口组件均为激活状态且每个所述窗口组件的图层均为可见状态,每个所述窗口的窗口数据均包含对应的窗口标识。
目标设备根据各个窗口数据中包含的窗口标识,能够区分各个不同窗口的窗口数据,进而能够分别构建得到各个不同的窗口;另外,由于这些窗口的窗口组件已激活且图层可见,故可以在目标设备的显示界面中显示这些窗口的内容。
进一步的,在发送已启动的应用程序中已打开的各个窗口的窗口数据至所述目标设备之前,还可以包括:
若检测到任意一个已启动的应用程序打开一个新窗口,则将所述新窗口的窗口组件设置为激活状态,并将所述新窗口的窗口组件的图层设置为可见状态。
源设备只要检测到某个应用程序打开了一个新窗口,则将该新窗口的窗口组件设置为激活状态,并将该窗口组件的图层设置为可见状态。无论该新窗口是当前显示还是后台运行,该新窗口的窗口组件的属性均特殊设置,以便后续投屏到目标设备中显示。
第三方面,本申请实施例提供了另一种投屏显示方法,所述方法包括:
源设备发送投屏请求信息至目标设备;
所述目标设备在接收到所述投屏请求信息后,基于所述投屏请求信息返回确认信息至所述源设备;
所述源设备在接收到所述确认信息后,发送已启动的应用程序的窗口数据至所述目标设备,所述窗口数据包括所述源设备已启动的同一个应用程序打开的至少两个窗口对应的显示内容;
所述目标设备获取并显示所述已启动的应用程序的窗口数据。
首先由源设备发送投屏请求信息至目标设备,该目标设备在接收到该投屏请求信息后,基于该投屏请求信息返回确认信息至该源设备。该源设备在接收到该确认信息后,发送已启动的应用程序的窗口数据至该目标设备,最后该目标设备获取并显示源设备已启动的应用程序的窗口数据。这些窗口数据既包括源设备当前可见的窗口内容,也包括源设备已启动但不可见的窗口内容,目标设备在自身的显示界面中显示这些窗口数据,从而实现将源设备上不可见的窗口内容投屏到目标设备上显示。
第四方面,本申请实施例提供了一种投屏显示系统,包括:
源设备,用于发送投屏请求信息至目标设备,并在接收到所述目标设备基于所述投屏请求信息返回的确认信息后,发送已启动的应用程序的窗口数据至所述目标设备,所述窗口数据包括所述源设备已启动的同一个应用程序打开的至少两个窗口对应的显示内容;
目标设备,用于在接收到所述源设备发送的投屏请求信息后,基于所述投屏请求信息返回确认信息至所述源设备,获取并显示所述已启动的应用程序的窗口数据。
第五方面,本申请实施例提供了一种终端设备,包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序,所述处理器执行所述计算机程序时实现如本申请实施例第一方面提出的投屏显示方法的步骤,或者实现如本申请实施例第二方面提出的投屏显示方法的步骤。
第六方面,本申请实施例提供了一种计算机可读存储介质,所述计算机可读存储介质存储有计算机程序,所述计算机程序被处理器执行时实现如本申请实施例第一方面提出的投屏显示方法的步骤,或者实现如本申请实施例第二方面提出的投屏显示方法的步骤。
第七方面,本申请实施例提供了一种计算机程序产品,当计算机程序产品在终端设备上运行时,使得终端设备执行如本申请实施例第一方面提出的投屏显示方法的步骤,或者执行如本申请实施例第二方面提出的投屏显示方法的步骤。
上述第二方面至第七方面所能实现的技术效果,可以参照第一方面所述的技术效果,在此不再赘述。
本申请实施例与现有技术相比存在的有益效果是:可以将源设备上不可见的窗口内容投屏到目标设备上显示,扩展了投屏显示的全面性与灵活性,且操作简单,实用性强。
附图说明
图1是本申请实施例提供的投屏显示方法所适用于的手机的硬件结构示意图;
图2是本申请实施例提供的一种投屏显示系统的示意图;
图3是本申请实施例提供的一种投屏显示系统的软件架构图;
图4是本申请实施例提供的一种应用于被投屏的目标设备的投屏显示方法的流程图;
图5是本申请实施例提供的另一种应用于被投屏的目标设备的投屏显示方法的流程图;
图6是本申请实施例提供的一种应用于投屏的源设备的投屏显示方法的流程图;
图7是本申请实施例提供的一种投屏显示方法的交互流程图;
图8是本申请实施例提供的投屏显示方法在实际场景下应用的一种示意图;
图9是图8所示的场景中,当手机返回桌面后,平板电脑屏幕显示的示意图;
图10是图8所示的场景中,窗口A的窗口组件和窗口B的窗口组件的属性示意图;
图11是图8所示的场景中,窗口A、窗口B的窗口数据的图层划分示意图;
图12中的(A)是本申请实施例提供的投屏显示方法在同源投屏场景下应用的一 种示意图;
图12中的(B)是本申请实施例提供的投屏显示方法在异源投屏场景下应用的一种示意图;
图13是本申请实施例提供的一种终端设备的示意图。
具体实施方式
以下描述中,为了说明而不是为了限定,提出了诸如特定装置结构、技术之类的具体细节,以便透彻理解本申请实施例。然而,本领域的技术人员应当清楚,在没有这些具体细节的其它实施例中也可以实现本申请。在其它情况中,省略对众所周知的装置、电路以及方法的详细说明,以免不必要的细节妨碍本申请的描述。
以下实施例中所使用的术语只是为了描述特定实施例的目的,而并非旨在作为对本申请的限制。如在本申请的说明书和所附权利要求书中所使用的那样,单数表达形式“一个”、“一种”、“所述”、“上述”、“该”和“这一”旨在也包括例如“一个或多个”这种表达形式,除非其上下文中明确地有相反指示。还应当理解,在本申请实施例中,“一个或多个”是指一个、两个或两个以上;“和/或”,描述关联对象的关联关系,表示可以存在三种关系;例如,A和/或B,可以表示:单独存在A,同时存在A和B,单独存在B的情况,其中A、B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。
本申请实施例提供的投屏显示方法可以应用于手机、平板电脑、可穿戴设备、车载设备、增强现实(augmented reality,AR)/虚拟现实(virtual reality,VR)设备、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer,UMPC)、上网本、个人数字助理(personal digital assistant,PDA)、大屏电视等终端设备或者服务器上,本申请实施例对终端设备和服务器的具体类型不作任何限制。
例如,所述终端设备可以是WLAN中的站点(STAION,ST),可以是蜂窝电话、无绳电话、会话启动协议(Session InitiationProtocol,SIP)电话、无线本地环路(Wireless Local Loop,WLL)站、个人数字处理(Personal Digital Assistant,PDA)设备、具有无线通信功能的手持设备、计算设备或连接到无线调制解调器的其它处理设备、车载设备、车联网终端、电脑、膝上型计算机、手持式通信设备、手持式计算设备、卫星无线设备、无线调制解调器卡、电视机顶盒(set top box,STB)、用户驻地设备(customer premise equipment,CPE)和/或用于在无线装置上进行通信的其它设备以及下一代通信装置,例如,5G网络中的移动终端或者未来演进的公共陆地移动网络(Public Land Mobile Network,PLMN)网络中的移动终端等。
作为示例而非限定,当所述终端设备为可穿戴设备时,该可穿戴设备还可以是应用穿戴式技术对日常穿戴进行智能化设计、开发出可以穿戴的设备的总称,如眼镜、手套、手表、服饰及鞋等。可穿戴设备即直接穿在身上,或是整合到用户的衣服或配件的一种便携式设备。可穿戴设备不仅仅是一种硬件设备,更是通过软件支持以及数据交互、云端交互来实现强大的功能。广义穿戴式智能设备包括功能全、尺寸大、可不依赖智能手机实现完整或者部分的功能,如智能手表或智能眼镜等,以及只专注于某一类应用功能,需要和其它设备如智能手机配合使用,如各类进行体征监测的智能手环、智能首饰等。
以所述终端设备为手机为例。图1示出的是与本申请实施例提供的手机的部分结构的框图。参考图1,手机包括:射频(Radio Frequency,RF)电路110、存储器120、输入单元130、显示单元140、传感器150、音频电路160、无线保真(wireless fidelity,WiFi)模块170、处理器180、以及电源190等部件。本领域技术人员可以理解,图1中示出的手机结构并不构成对手机的限定,可以包括比图示更多或更少的部件,或者组合某些部件,或者不同的部件布置。
下面结合图1对手机的各个构成部件进行具体的介绍:
RF电路110可用于收发信息或通话过程中,信号的接收和发送,特别地,将基站的下行信息接收后,给处理器180处理;另外,将设计上行的数据发送给基站。通常,RF电路包括但不限于天线、至少一个放大器、收发信机、耦合器、低噪声放大器(Low Noise Amplifier,LNA)、双工器等。此外,RF电路110还可以通过无线通信与网络和其他设备通信。上述无线通信可以使用任一通信标准或协议,包括但不限于全球移动通讯装置(Global System of Mobile communication,GSM)、通用分组无线服务(General Packet Radio Service,GPRS)、码分多址(Code Division Multiple Access,CDMA)、宽带码分多址(Wideband Code Division Multiple Access,WCDMA)、长期演进(Long Term Evolution,LTE))、电子邮件、短消息服务(Short Messaging Service,SMS)等。
存储器120可用于存储软件程序以及模块,处理器180通过运行存储在存储器120的软件程序以及模块,从而执行手机的各种功能应用以及数据处理。存储器120可主要包括存储程序区和存储数据区,其中,存储程序区可存储操作装置、至少一个功能所需的应用程序(比如声音播放功能、图像播放功能等)等;存储数据区可存储根据手机的使用所创建的数据(比如音频数据、电话本等)等。此外,存储器120可以包括高速随机存取存储器,还可以包括非易失性存储器,例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。
输入单元130可用于接收输入的数字或字符信息,以及产生与手机的用户设置以及功能控制有关的键信号输入。具体地,输入单元130可包括触控面板131以及其他输入设备132。触控面板131,也称为触摸屏,可收集用户在其上或附近的触摸操作(比如用户使用手指、触笔等任何适合的物体或附件在触控面板131上或在触控面板131附近的操作),并根据预先设定的程式驱动相应的连接装置。可选的,触控面板131可包括触摸检测装置和触摸控制器两个部分。其中,触摸检测装置检测用户的触摸方位,并检测触摸操作带来的信号,将信号传送给触摸控制器;触摸控制器从触摸检测装置上接收触摸信息,并将它转换成触点坐标,再送给处理器180,并能接收处理器180发来的命令并加以执行。此外,可以采用电阻式、电容式、红外线以及表面声波等多种类型实现触控面板131。除了触控面板131,输入单元130还可以包括其他输入设备132。具体地,其他输入设备132可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆等中的一种或多种。
显示单元140可用于显示由用户输入的信息或提供给用户的信息以及手机的各种菜单。显示单元140可包括显示面板141,可选的,可以采用液晶显示器(Liquid Crystal Display,LCD)、有机发光二极管(Organic Light-Emitting Diode,OLED)等形式来配 置显示面板141。进一步的,触控面板131可覆盖显示面板141,当触控面板131检测到在其上或附近的触摸操作后,传送给处理器180以确定触摸事件的类型,随后处理器180根据触摸事件的类型在显示面板141上提供相应的视觉输出。虽然在图1中,触控面板131与显示面板141是作为两个独立的部件来实现手机的输入和输入功能,但是在某些实施例中,可以将触控面板131与显示面板141集成而实现手机的输入和输出功能。
手机还可包括至少一种传感器150,比如光传感器、运动传感器以及其他传感器。具体地,光传感器可包括环境光传感器及接近传感器,其中,环境光传感器可根据环境光线的明暗来调节显示面板141的亮度,接近传感器可在手机移动到耳边时,关闭显示面板141和/或背光。作为运动传感器的一种,加速计传感器可检测各个方向上(一般为三轴)加速度的大小,静止时可检测出重力的大小及方向,可用于识别手机姿态的应用(比如横竖屏切换、相关游戏、磁力计姿态校准)、振动识别相关功能(比如计步器、敲击)等;至于手机还可配置的陀螺仪、气压计、湿度计、温度计、红外线传感器等其他传感器,在此不再赘述。
音频电路160、扬声器161,传声器162可提供用户与手机之间的音频接口。音频电路160可将接收到的音频数据转换后的电信号,传输到扬声器161,由扬声器161转换为声音信号输出;另一方面,传声器162将收集的声音信号转换为电信号,由音频电路160接收后转换为音频数据,再将音频数据输出处理器180处理后,经RF电路110以发送给比如另一手机,或者将音频数据输出至存储器120以便进一步处理。
WiFi属于短距离无线传输技术,手机通过WiFi模块170可以帮助用户收发电子邮件、浏览网页和访问流式媒体等,它为用户提供了无线的宽带互联网访问。虽然图1示出了WiFi模块170,但是可以理解的是,其并不属于手机的必须构成,完全可以根据需要在不改变本申请的本质的范围内而省略。
处理器180是手机的控制中心,利用各种接口和线路连接整个手机的各个部分,通过运行或执行存储在存储器120内的软件程序和/或模块,以及调用存储在存储器120内的数据,执行手机的各种功能和处理数据,从而对手机进行整体监控。可选的,处理器180可包括一个或多个处理单元;优选的,处理器180可集成应用处理器和调制解调处理器,其中,应用处理器主要处理操作装置、用户界面和应用程序等,调制解调处理器主要处理无线通信。可以理解的是,上述调制解调处理器也可以不集成到处理器180中。
手机还包括给各个部件供电的电源190(比如电池),优选的,电源可以通过电源管理装置与处理器180逻辑相连,从而通过电源管理装置实现管理充电、放电、以及功耗管理等功能。
尽管未示出,手机还可以包括摄像头,其中至少一个常规的RGB摄像头,以及至少一个红外摄像头或者高光谱摄像头。可选地,摄像头在手机的上的位置可以为前置的,也可以为后置的,本申请实施例对此不作限定。
可选地,手机可以包括单摄像头、双摄像头或三摄像头等,本申请实施例对此不作限定。
例如,手机可以包括三摄像头,其中,一个为主摄像头、一个为广角摄像头、一 个为长焦摄像头。
可选地,当手机包括多个摄像头时,这多个摄像头可以全部前置,或者全部后置,或者一部分前置、另一部分后置,本申请实施例对此不作限定。
另外,尽管未示出,手机还可以包括蓝牙模块等,在此不再赘述。
图2是本申请实施例提出的一种投屏显示系统的示意图,该投屏显示系统包括源设备A和目标设备B。源设备A和目标设备B均可以是各种类型的终端设备,比如手机、平板、PC、大屏电视等设备,也即本申请适用于手机投平板、手机投大屏电视、手机投手机、手机投PC、平板投平板、PC投平板、PC投大屏电视等所有设备类型组合的屏幕互投。
源设备A和目标设备B可以通过各类有线或者无线的方式连接,比如通讯数据线、蓝牙、Wifi等方式,本申请对此不做限定。在进行投屏显示之前,需要将源设备A和目标设备B连接,比如采用通讯数据线连接,Wifi互连或者启动蓝牙连接等方式。在将源设备A和目标设备B连接后,可以在源设备A上触发启动投屏,也可以在目标设备B上触发启动投屏,比如在目标设备B上,可以通过下拉通知栏,点击某个多屏协同显示的按钮,之后弹出相应的投屏显示APP的待连接的显示界面,与此同时源设备A的显示界面上会弹出对话框提示用户进行连接,用户点击连接成功后,即可采用本申请提出的投屏显示方法,将源设备A上运行的各个应用窗口投屏到目标设备B的屏幕中显示。
具体的,源设备A用于发送投屏请求信息至目标设备B,并在接收到所述目标设备B基于所述投屏请求信息返回的确认信息后,发送已启动的应用程序的窗口数据至所述目标设备B;目标设备B用于在接收到所述源设备A发送的投屏请求信息后,基于所述投屏请求信息返回确认信息至所述源设备A,获取并显示所述已启动的应用程序的窗口数据。
进一步的,所述目标设备B具体可以用于:获取所述源设备已启动的应用程序中已打开的各个窗口的窗口数据,每个所述窗口的窗口组件均为激活状态且每个所述窗口组件的图层均为可见状态,每个所述窗口的窗口数据均包含对应的窗口标识;根据所述各个窗口的窗口数据以及窗口数据中包含的窗口标识,分别构建并显示所述各个窗口。
进一步的,所述各个窗口包括第一窗口和第二窗口,所述第一窗口为所述源设备当前未显示的窗口,所述第二窗口为所述源设备当前显示的窗口,所述目标设备B具体可以用于:
在显示界面中划分出第一区域和第二区域;
将所述第一窗口添加到所述第一区域中显示,并将所述第二窗口添加到所述第二区域中显示。
进一步的,所述目标设备B还可以用于:
若检测到从所述第一窗口中打开的第三窗口,则将所述第三窗口添加到所述第二区域中显示,并关闭所述第二窗口;
若检测到从所述第二窗口中打开的第四窗口,则将所述第四窗口添加到所述第二区域中且覆盖到所述第二窗口的上方显示。
进一步的,所述目标设备B还可以用于:
若检测到从所述第一窗口中打开的第五窗口,则将所述第五窗口添加到所述第二区域中显示,并关闭所述第二窗口;
若检测到从所述第二窗口中打开的第六窗口,则将所述第六窗口添加到所述第二区域中显示,并将所述第二窗口移动到所述第一区域中且覆盖到所述第一窗口的上方显示。
进一步的,每个所述窗口的窗口数据还包含对应的应用程序信息、窗口组件信息和窗口属性信息,所述目标设备B具体可以用于:
根据每个所述窗口的窗口数据包含的应用程序信息、窗口组件信息和窗口属性信息,确定待显示窗口的数量、尺寸、显示位置以及显示方式;
按照所述待显示窗口的数量、尺寸、显示位置以及显示方式,在显示界面中构建并显示相应的窗口。
更进一步的,所述目标设备B还可以用于:
若检测到对构建并显示的任意一个窗口进行操作的第一指令,则获取所述任意一个窗口的窗口数据包含的目标应用程序信息和目标窗口组件信息;
将所述第一指令、所述目标应用程序信息和所述目标窗口组件信息发送至所述源设备,以指示所述源设备根据所述目标应用程序信息和所述目标窗口组件信息分别确定待操作的目标应用程序和目标窗口,并基于所述第一指令对所述目标应用程序的所述目标窗口进行操作。
更进一步的,所述目标设备B还可以用于:
当检测到所述源设备当前的显示界面发生变化时,保持显示所述各个窗口的内容;
当检测到对构建并显示的所述各个窗口进行操作的第二指令后,将所述第二指令发送至所述源设备,以指示所述源设备保持当前的显示界面不变,并基于所述第二指令对所述源设备后台运行的对应窗口进行操作。
进一步的,所述源设备A具体可以用于:
发送已启动的应用程序中已打开的各个窗口的窗口数据至所述目标设备,以指示所述目标设备根据所述各个窗口的窗口数据以及窗口数据中包含的窗口标识,分别构建并显示所述各个窗口;其中,每个所述窗口的窗口组件均为激活状态且每个所述窗口组件的图层均为可见状态,每个所述窗口的窗口数据均包含对应的窗口标识。
进一步的,所述源设备A还可以用于:
若检测到任意一个已启动的应用程序打开一个新窗口,则将所述新窗口的窗口组件设置为激活状态,并将所述新窗口的窗口组件的图层设置为可见状态。
源设备A和目标设备B之间进行投屏显示的具体工作原理与实施步骤,可以参见下文所述的各个投屏显示方法的实施例。
图3是本申请实施例提供的一种投屏显示系统的软件架构图。
图3的左半部分,是源设备的软件架构图,源设备是主动投屏的设备,比如可以是一个手机。源设备的软件架构可以分为应用层和框架层两部分,其中应用层包含系统应用模块和三方应用模块,用于管理设备上运行的各类应用程序。框架层包括窗口管理模块、跨设备管理模块、数据管理模块和驱动模块,其中窗口管理模块用于管理 应用程序的所有窗口的显示方式,包括窗口显示的坐标大小、窗口显示层级和窗口显示属性等;跨设备管理模块负责源设备和目标设备之间的靠近发现、鉴权、连接、源设备侧应用状态感知协同、文件拖入拽出、消息通知等功能;数据管理模块负责源设备和目标设备之间传输投屏的音频、视频以及图层等数据码流,并且负责反向事件(即通过目标设备触发的对源设备进行控制的事件)控制等功能;驱动模块属于底层驱动,负责发现认证、配置连接的工作,并向跨设备管理模块提供驱动事件(比如发现设备、设备信息、设备消失等事件),以及执行跨设备管理模块下发的命令,比如连接、断开等命令。
图3的右半部分,是目标设备的软件架构图,目标设备是被投屏的设备,比如可以是一个平板或者大屏电视。目标设备的软件架构同样可分为应用层和框架层两部分,其中应用层包含投屏显示APP,负责在目标设备上显示投屏的界面,同时负责UI显示界面相关的管理工作。框架层包括窗口管理模块、跨设备管理模块、数据管理模块和驱动模块,各个功能模块的作用可以参照上述源设备的框架层中对应的功能模块的作用。
现有的投屏显示方法都是基于源设备当前用户可见的显示窗口内容进行投屏,具有一定的局限性,本申请为打破该局限性而提出一种新的投屏显示方法。
图4示出了本申请实施例提供的一种应用于被投屏的目标设备的投屏显示方法的流程图,包括:
401、在接收到源设备发送的投屏请求信息后,获取所述源设备已启动的应用程序的窗口数据;
当源设备和目标设备连接后,用户可以在源设备中点击发送一个投屏请求信息,目标设备在接收到该投屏请求信息后,可以返回一个投屏确认信息至该源设备,然后获取该源设备已启动的应用程序的窗口数据,所述窗口数据包括所述源设备已启动的同一个应用程序打开的至少两个窗口对应的显示内容。这些窗口数据,可以是源设备已启动的同一个应用程序打开的一个以上窗口的图层数据,也可以是源设备已启动的不同应用程序打开的多个窗口的图层数据。显然,这部分窗口数据既包括源设备当前可见的窗口内容,也包括源设备当前已启动但不可见的窗口内容。
需要说明的是,当源设备和目标设备连接后,用户也可以于目标设备端点击发送投屏请求信息,源设备在接收到该投屏请求信息后,发送源设备已启动的应用程序的窗口数据至目标设备。
402、显示所述已启动的应用程序的窗口数据。
目标设备在接收到所述源设备已启动的应用程序的窗口数据之后,可以在自身的显示屏中显示这部分窗口数据。由于这部分窗口数据包含源设备上已启动但不可见的窗口内容,因此实现了将源设备上不可见的窗口内容投屏到目标设备上显示。具体的,可以根据获取到的窗口数据确定待显示的窗口的数量、大小、所属应用程序等信息,然后在设备显示界面中划分出相应数量和位置的窗口显示区域,在各个区域中分别显示该源设备已启动的各个不同窗口的内容。
在本申请实施例中,被投屏的目标设备获取到的是源设备已启动的应用程序的窗口数据,这些窗口数据既包括源设备当前可见的窗口内容,也包括源设备已启动但不 可见的窗口内容,目标设备在自身的显示界面中显示这些窗口数据,从而实现将源设备上不可见的窗口内容投屏到目标设备上显示。
图5示出了本申请实施例提供的另一种应用于被投屏的目标设备的投屏显示方法的流程图,包括:
501、在接收到源设备发送的投屏请求信息后,获取所述源设备已启动的应用程序中已打开的各个窗口的窗口数据;
目标设备在接收到源设备发送的投屏请求信息后,获取所述源设备已启动的应用程序中已打开的各个窗口的窗口数据。比如,若源设备已启动应用程序A,并在应用程序A中打开了窗口A 1和A 2,则目标设备分别获取A 1的窗口数据和A 2的窗口数据;若源设备已启动应用程序A和B,在应用程序A中打开了窗口A 1和A 2,在应用程序B中打开了窗口B 1、B 2和B 3,则目标设备分别获取A 1的窗口数据、A 2的窗口数据、B 1的窗口数据、B 2的窗口数据和B 3的窗口数据。
为了在目标设备中显示各个窗口的内容,需要源设备预先对应用程序的各个窗口做特殊处理,使得每个所述窗口的窗口组件均为激活状态且每个所述窗口组件的图层均为可见状态。比如,对于Android系统的应用程序,实现应用内窗口区分显示的基本单位是Activity组件,也即窗口组件。一般情况下,对于源设备当前显示的可见窗口,其Activity组件的状态为resumed(激活状态),且图层visiable属性为true(即图层为可见状态);对于源设备当前不可见的窗口,其Activity组件的状态为paused或者stoped(停止或终止状态),且图层visiable属性为false(即图层为不可见状态)。为了在目标设备中显示不可见的窗口,首先需要通过系统软件架构中的窗口服务功能,将源设备中各个不可见窗口的Activity组件的状态设置为resumed,且图层visiable属性设置为true。
在传统的投屏方案中,目标设备接收到的都是同一个窗口(源设备当前显示的窗口)的窗口数据,因此不需要区分接收到的窗口数据属于哪个应用程序的哪个窗口。而在本申请实施例中,目标设备需要区分获取到的窗口数据分别属于哪个窗口,故源设备需要往窗口数据中添加窗口标识等信息。每个所述窗口的窗口数据均包含对应的窗口标识,这样目标设备在后续显示这部分窗口数据时,就能基于不同窗口的窗口数据分别构建并显示各个窗口的内容。
502、根据所述各个窗口的窗口数据以及窗口数据中包含的窗口标识,分别构建并显示所述各个窗口。
目标设备在获取到所述源设备已启动的应用程序中已打开的各个窗口的窗口数据之后,根据所述各个窗口的窗口数据以及窗口数据中包含的窗口标识,分别构建并显示所述各个窗口。比如,目标设备获取的是源设备已启动的窗口A的窗口数据以及窗口B的窗口数据,则可以在设备显示界面中划分出两块区域,根据窗口A的窗口数据构建出窗口A,并在其中一块区域中显示,根据窗口B的窗口数据构建出窗口B,并在另一块区域中显示。
进一步的,每个所述窗口的窗口数据还包含对应的应用程序信息、窗口组件信息和窗口属性信息,所述分别构建并显示所述各个窗口可以包括:
(1)根据每个所述窗口的窗口数据包含的应用程序信息、窗口组件信息和窗口属 性信息,确定待显示窗口的数量、尺寸、显示位置以及显示方式;
(2)按照所述待显示窗口的数量、尺寸、显示位置以及显示方式,在显示界面中构建并显示相应的窗口。
在每个窗口的窗口数据中,还可以添加对应的应用程序信息、窗口组件信息和窗口属性信息。其中,应用程序信息可以包括应用程序的标识、userid等关键信息,这些信息标志某一个特定的应用程序,用于应用程序之间的数据区分。窗口组件信息,例如Activity信息,一个应用程序内可能存在多个Activity,所以窗口数据中需要带上Activity的各种属性信息,比如Activity的token句柄、component组件名、是否是主界面等信息。窗口属性信息,例如Window信息,一个Activity可能包含多个Window图层,Window图层是图层显示的基本单位,即一个Activity可能包括多个Window图层,Window图层中定义了图层大小、间距等关键信息。在实际操作中,这部分信息可以添加在窗口数据图层的Buffer数据中,比如可以在Buffer数据中增加头部信息,以标记各个图层的信息。
在获得每个所述窗口的窗口数据包含的应用程序信息、窗口组件信息和窗口属性信息后,可以根据这些信息确定待显示窗口的数量、尺寸、显示位置以及显示方式,然后在显示界面中构建并显示相应的窗口。在显示界面中如何分区域,分多少个区域,每个区域显示窗口的大小,窗口的显示方式等均可以根据这些信息设定。应用程序信息中可以获取应用的模式,比如是导航模式,同时可以获取哪个窗口是主页,这样目标设备可以把主页窗口显示在屏幕左边,其它窗口层叠显示在屏幕右边。
更进一步的,在显示界面中构建并显示相应的窗口之后,还可以包括:
(1)若检测到对构建并显示的任意一个窗口进行操作的第一指令,则获取所述任意一个窗口的窗口数据包含的目标应用程序信息和目标窗口组件信息;
(2)将所述第一指令、所述目标应用程序信息和所述目标窗口组件信息发送至所述源设备,以指示所述源设备根据所述目标应用程序信息和所述目标窗口组件信息分别确定待操作的目标应用程序和目标窗口,并基于所述第一指令对所述目标应用程序的所述目标窗口进行操作。
目标设备在显示界面中构建并显示相应的窗口之后,用户可以在目标设备在显示界面中对这些窗口进行操作,相当于执行对源设备中相应窗口的反向控制。用户在目标设备上点击投屏应用的窗口时,原始流程是直接将事件坐标、类型信息反向传输给源设备,而在本申请中,反向传输给原设备的事件需要增加窗口组件的句柄信息,表明该事件的坐标是落在哪个窗口上的,从而确定是对源设备的哪个窗口进行操作。目标设备可以将检测到的操作指令,以及对应窗口的窗口数据包含的目标应用程序信息和目标窗口组件信息发送给源设备,以便源设备确定该操作指令对应的目标应用程序和目标窗口,从而对该目标应用程序中的目标窗口进行相应的操作。总的来说,目标设备上的输入事件,会通过跨设备通道发送到源设备,源设备再模拟一个同样的事件,发送给应用程序的相应窗口执行。在事件执行后,源设备对应窗口的内容若刷新,则目标设备的显示界面中该对应窗口的内容也会跟随刷新。
进一步的,所述各个窗口包括第一窗口和第二窗口,所述第一窗口为所述源设备当前未显示的窗口,所述第二窗口为所述源设备当前显示的窗口,所述显示所述各个 窗口可以包括:
(1)在显示界面中划分出第一区域和第二区域;
(2)将所述第一窗口添加到所述第一区域中显示,并将所述第二窗口添加到所述第二区域中显示。
比如,目标设备上要显示的窗口包括A窗口和B窗口,其中A窗口是源设备已运行但当前未显示的窗口,B窗口是源设备当前显示的窗口,则可以在目标设备的显示界面中划分一左一右两个区域,两个区域的尺寸可以相同也可以不同,每个区域的尺寸可以和源设备的屏幕尺寸相同。然后,将A窗口添加到左边的区域中显示,并将B窗口添加到右边的区域中显示。
更进一步的,在将所述第一窗口添加到所述第一区域中显示,并将所述第二窗口添加到所述第二区域中显示之后,还可以包括:
若检测到从所述第一窗口中打开的第三窗口,则将所述第三窗口添加到所述第二区域中显示,并关闭所述第二窗口;若检测到从所述第二窗口中打开的第四窗口,则将所述第四窗口添加到所述第二区域中且覆盖到所述第二窗口的上方显示。
这属于导航模式的投屏显示方式,同样以上述例子进行说明,若检测到从A窗口中打开的新窗口C(用户在目标设备的显示界面中对A窗口进行操作打开C窗口),则将C窗口添加到该右边的区域中显示,即覆盖到B窗口上方显示,C窗口显示完毕后系统会关闭B窗口。若检测到从B窗口中打开的新窗口D,则将D窗口添加到该右边的区域中,覆盖到B窗口的上方层叠显示。假如继续从A窗口中再次打开1个新窗口,则该新窗口会继续添加到该右边的区域中,即覆盖到C窗口上方显示,完毕后系统会关闭C窗口。而假如从D窗口中继续打开一个新窗口,则该新窗口覆盖到D窗口的上方层叠显示,以此类推。
更进一步的,在将所述第一窗口添加到所述第一区域中显示,并将所述第二窗口添加到所述第二区域中显示之后,还可以包括:
若检测到从所述第一窗口中打开的第五窗口,则将所述第五窗口添加到所述第二区域中显示,并关闭所述第二窗口;若检测到从所述第二窗口中打开的第六窗口,则将所述第六窗口添加到所述第二区域中显示,并将所述第二窗口移动到所述第一区域中且覆盖到所述第一窗口的上方显示。
这属于购物模式的投屏显示方法,同样以上述例子进行说明,若检测到从A窗口中打开的新窗口E,则将E窗口添加到该右边的区域中显示,即覆盖到B窗口上方显示,E窗口显示完毕后系统会关闭B窗口。若检测到从B窗口中打开的新窗口F,则将F窗口添加到该右边的区域中(即原来B窗口的位置)显示,同时将B窗口移动到左边的区域中,覆盖到A窗口的上方层叠显示。假如继续从A窗口中再次打开1个新窗口,则该新窗口会继续添加到该右边的区域中,即覆盖到E窗口上方显示,完毕后系统会关闭E窗口。而假如从F窗口中继续打开一个新窗口,则该新窗口覆盖到F窗口的上方层叠显示,同时将F窗口移动到该左边的区域中,覆盖到B窗口的上方层叠显示,以此类推。
上述显示方式为目标设备的双窗口显示模式,依赖于源设备当前运行的应用窗口,假如当前运行的应用窗口只有1个,则可以将该唯一的应用窗口添加到该左边的区域 中显示。
目标设备还可以设置单窗口或者多窗口显示模式,即可以只显示1个源设备当前运行的应用窗口,也可以同时显示多个或全部源设备当前运行的应用窗口,只需要在源设备的显示界面中划分出相应数量的窗口显示区域即可。
另外,若检测到源设备点击回到桌面,则目标设备可以由双窗口显示模式切换至单窗口显示模式,以单窗口显示源设备的桌面内容,这属于一种同源投屏方式。
进一步的,在分别构建并显示所述各个窗口之后,还可以包括:
(1)当检测到所述源设备当前的显示界面发生变化时,保持显示所述各个窗口的内容;
(2)当检测到对构建并显示的所述各个窗口进行操作的第二指令后,将所述第二指令发送至所述源设备,以指示所述源设备保持当前的显示界面不变,并基于所述第二指令对所述源设备后台运行的对应窗口进行操作。
这属于异源投屏的方式,用户在源设备上操作时,目标设备显示的内容不变,而用户在目标设备上操作时,源设备显示的内容不变,两个设备相互独立。在进行异源投屏时,同样支持上文所述的导航模式和购物模式。用户在源设备上操作时,只要操作对象不是目标设备上显示的窗口,则目标设备的显示内容没有变化。而用户在目标设备上对某个窗口X进行操作时,相应的操作指令会发送至源设备,源设备基于该操作指令对后台运行的窗口X进行操作,不会对源设备当前显示的内容构成影响,然后源设备再将后台窗口X刷新的窗口内容传输至目标设备,最终目标设备显示的窗口X的内容会随着刷新。
在实际操作中,源设备可以运行两块display(display0和display1),其中display0显示源设备的原始内容,此内容在源设备上对用户可见;另一块display1显示投屏到目标设备的应用窗口的内容,此内容在源设备上对用户不可见,即运行在源设备的后台,目标设备上的事件通过跨设备传输通道传递给源设备display1上的应用窗口。
本申请实施例在进行投屏显示前,会预先将源设备的应用程序的已运行窗口的窗口组件设置为激活状态,并且将这些窗口组件的图层设置为可见状态;然后,源设备将这些已运行的窗口(可以包括源设备当前可见的窗口,以及后台运行的不可见窗口)的窗口数据发送至目标设备;目标设备根据各个窗口数据中包含的窗口标识,能够区分各个不同窗口的窗口数据,进而能够分别构建得到各个不同的窗口;另外,由于这些窗口的窗口组件已激活且图层可见,故可以在目标设备的显示界面中显示这些窗口的内容,也即实现将源设备上不可见的窗口内容投屏到目标设备上显示。
图6示出了本申请实施例提供的一种应用于投屏的源设备的投屏显示方法的流程图,包括:
601、发送投屏请求信息至目标设备;
本申请实施例的执行主体是源设备,当源设备和目标设备连接后,用户可以在源设备中点击发送一个投屏请求信息至目标设备。
602、在接收到所述目标设备基于所述投屏请求信息返回的确认信息后,发送已启动的应用程序的窗口数据至所述目标设备,以指示所述目标设备显示所述已启动的应用程序的窗口数据。
用户可以在目标设备上操作,同意执行投屏显示,点击返回确认信息至该源设备。该源设备在接收到该确认信息后,发送已启动的应用程序的窗口数据至该目标设备,以指示该目标设备显示源设备已启动的应用程序的窗口数据。这些窗口数据,可以是源设备已启动的同一个应用程序打开的一个以上窗口的图层数据,也可以是源设备已启动的不同应用程序打开的多个窗口的图层数据。显然,这部分窗口数据既包括源设备当前可见的窗口内容,也包括源设备当前已启动但不可见的窗口内容。目标设备在接收到源设备已启动的应用程序的窗口数据之后,可以在自身的显示屏中显示这部分窗口数据。由于这部分窗口数据包含源设备上已启动但不可见的窗口内容,因此实现了将源设备上不可见的窗口内容投屏到目标设备上显示。具体的,目标设备可以根据获取到的窗口数据确定待显示的窗口的数量、大小、所属应用程序等信息,然后在设备显示界面中划分出相应数量和位置的窗口显示区域,在各个区域中分别显示该源设备已启动的各个不同窗口的内容。
进一步的,所述发送已启动的应用程序的窗口数据至所述目标设备,以指示所述目标设备显示所述已启动的应用程序的窗口数据可以包括:
发送已启动的应用程序中已打开的各个窗口的窗口数据至所述目标设备,以指示所述目标设备根据所述各个窗口的窗口数据以及窗口数据中包含的窗口标识,分别构建并显示所述各个窗口。
其中,每个所述窗口的窗口组件均为激活状态且每个所述窗口组件的图层均为可见状态,每个所述窗口的窗口数据均包含对应的窗口标识。目标设备根据各个窗口数据中包含的窗口标识,能够区分各个不同窗口的窗口数据,进而能够分别构建得到各个不同的窗口;另外,由于这些窗口的窗口组件已激活且图层可见,故可以在目标设备的显示界面中显示这些窗口的内容。
进一步的,在发送已启动的应用程序中已打开的各个窗口的窗口数据至所述目标设备之前,还可以包括:
若检测到任意一个已启动的应用程序打开一个新窗口,则将所述新窗口的窗口组件设置为激活状态,并将所述新窗口的窗口组件的图层设置为可见状态。
源设备只要检测到某个应用程序打开了一个新窗口,则将该新窗口的窗口组件设置为激活状态,并将该窗口组件的图层设置为可见状态。无论该新窗口是当前显示还是后台运行,该新窗口的窗口组件的属性均特殊设置,以便后续可投屏到目标设备中显示。具体的,可以采用图3所示软件架构图中的窗口管理模块来修改各个窗口的窗口组件属性,该窗口管理模块拥有对所有应用窗口的绝对管理权,可以设置窗口组件的属性,管理窗口的生命周期状态。
在本申请实施例中,源设备发送投屏请求信息至目标设备,且在接收到所述目标设备基于所述投屏请求信息返回的确认信息后,发送已启动的应用程序的窗口数据至所述目标设备,以指示所述目标设备显示所述已启动的应用程序的窗口数据。被投屏的目标设备获取到的是源设备已启动的应用程序的窗口数据,这些窗口数据既包括源设备当前可见的窗口内容,也包括源设备已启动但不可见的窗口内容,目标设备在自身的显示界面中显示这些窗口数据,从而实现将源设备上不可见的窗口内容投屏到目标设备上显示。
图7示出了本申请实施例提供的一种投屏显示方法的交互流程图,包括:
701、源设备发送投屏请求信息至目标设备;
当源设备和目标设备连接后,用户可以在源设备中点击发送一个投屏请求信息至目标设备。
702、所述目标设备在接收到所述投屏请求信息后,基于所述投屏请求信息返回确认信息至所述源设备;
目标设备在接收到该投屏请求信息后,用户可以在目标设备上点击同意投屏显示操作,基于该投屏请求信息返回确认信息至该源设备。
703、所述源设备在接收到所述确认信息后,发送已启动的应用程序的窗口数据至所述目标设备;
源设备在接收到所述确认信息后,发送已启动的应用程序的窗口数据至该目标设备。这些窗口数据,可以是源设备已启动的同一个应用程序打开的一个以上窗口的图层数据,也可以是源设备已启动的不同应用程序打开的多个窗口的图层数据。
704、所述目标设备获取并显示所述已启动的应用程序的窗口数据。
目标设备在接收到源设备已启动的应用程序的窗口数据之后,可以在自身的显示屏中显示这部分窗口数据。由于这部分窗口数据包含源设备上已启动但不可见的窗口内容,因此实现了将源设备上不可见的窗口内容投屏到目标设备上显示。
在本申请实施例中,首先由源设备发送投屏请求信息至目标设备,该目标设备在接收到该投屏请求信息后,基于该投屏请求信息返回确认信息至该源设备。该源设备在接收到该确认信息后,发送已启动的应用程序的窗口数据至该目标设备,最后该目标设备获取并显示源设备已启动的应用程序的窗口数据。这些窗口数据既包括源设备当前可见的窗口内容,也包括源设备已启动但不可见的窗口内容,目标设备在自身的显示界面中显示这些窗口数据,从而实现将源设备上不可见的窗口内容投屏到目标设备上显示。
应理解,上述实施例中各步骤的序号的大小并不意味着执行顺序的先后,各过程的执行顺序应以其功能和内在逻辑确定,而不应对本申请实施例的实施过程构成任何限定。
为便于理解,下面以几个实际应用场景来说明本申请提出的投屏显示方法。
图8是本申请提出的投屏显示方法在实际场景下应用的一种示意图。
在图8中,投屏的源设备是一个手机,被投屏的目标设备是一个平板电脑。
该手机和该平板电脑可以通过蓝牙、数据线或者wifi等方式连接。优选的,可以通过蓝牙进行设备发现,设备连接等操作,通过wifi传输投屏的窗口数据,这样具有较低的数据传输延迟。
手机当前显示的应用窗口是B窗口,后台运行的某个应用窗口是A窗口。在投屏操作时,当手机和平板电脑连接后,用户可以在平板电脑的通知栏处点击某个多屏协同按钮,然后平板电脑会弹出某个投屏显示APP的待连接界面,此时手机会弹出提示框,要求用户确认投屏连接;当用户在手机上点击连接并连接成功后,手机会将已启动运行的A窗口和B窗口的窗口数据发送至平板电脑;平板电脑在接收到窗口数据后,在显示界面上同时显示A窗口和B窗口。具体的,在平板电脑上显示的A窗口和B 窗口的大小尺寸,可以和手机的屏幕大小尺寸一致。显示窗口的数量和位置可以任意设置,比如若手机已启动运行10个应用窗口,则可以在平板电脑的屏幕中任意划分出10个区域并分别显示这10个应用窗口。
在图8中,平板电脑的最左边显示窗口A,窗口A的右边显示窗口B。用户可以在平板电脑中设置投屏显示的模式,比如导航模式或者购物模式,相应的新窗口打开的显示方式,可参照前文所述的关于导航模式或者购物模式的说明。
另外,用户可以在平板电脑上对窗口A和窗口B进行反向的操作控制。在平板电脑上点击操作时,会模拟一个完全相同的操作事件到手机,触发手机执行相应的操作,执行窗口内容的刷新,然后平板电脑上显示的窗口内容会跟随刷新。
在同源投屏的场景下,若用户在手机上点击回到桌面,则平板电脑会切换回单窗口的显示模式,仅显示手机的桌面内容,如图9所示。
在手机的系统后台,需要对窗口A的窗口组件Activity A的属性进行特殊处理,如图10所示,将Activity A的状态设置为resumed(激活),且图层visiable设置为true(图层可见)。另外,由于窗口B为手机当前显示的窗口,故其窗口组件Activity B的状态也为resumed,且图层visiable也是true。
平板电脑在接收到手机传输的窗口A和窗口B的窗口数据之后,需要区分获取到的窗口数据是属于哪个应用程序,哪个窗口的,具体可以在窗口图层的Buffer数据中增加头部信息,以标记图层的属性。假设窗口A和窗口B都为应用M启动的窗口,窗口数据的图层划分示意图如图11所示,可以在应用M的图层Buffer数据中添加应用M的属性信息,比如应用的名称、userid等信息,用于标志某个特定应用,以实现不同应用之间的数据区分。可以在Activity A的图层Buffer数据中添加窗口A的窗口组件属性信息,在Activity B的图层Buffer数据中添加窗口B的窗口组件属性信息,比如Activity的token句柄、component组件名、是否是主界面等信息。一个Activity可能包含多个Window,Window是图层显示的基本单位,可以在Window的图层Buffer数据中添加图层大小、间距等关键信息。平板电脑在获取到带有相关图层属性信息的窗口数据之后,可以根据这些属性信息来确定各个窗口的显示方式。
图12是手机投屏平板电脑,由同源投屏切换至异源投屏的界面示意图。其中,图12中的(A)是同源投屏的场景,平板电脑显示的是手机的屏幕内容,比如同为显示B窗口。
当用户于平板电脑的界面中点击某个按钮,可以触发进入异源投屏的方式。此时即便手机切换回桌面显示,平板电脑依然显示手机已启动的在后台运行的应用窗口W 1和应用窗口W 2,如图12中的(B)所示,已进入异源投屏场景。
具体的,手机可以运行两块display(display0和display1),其中display0显示手机的原始内容;另一块虚拟的display1显示投屏到平板电脑的应用窗口(即应用窗口W 1和应用窗口W 2)的内容,此内容在手机上对用户不可见,运行在手机的后台,投屏到平板电脑后才对用户可见,平板电脑上触发的事件通过跨设备传输通道传递给手机display1上的应用窗口。在异源投屏场景下,用户在手机上操作时,平板电脑上显示的内容不变,用户在平板电脑上操作投屏窗口的内容时,手机上显示的内容不变,相互独立。
用户在平板电脑界面中对应用窗口W 1和应用窗口W 2进行操作产生的事件,通过跨设备传输通道传递给手机后台display1中显示的W 1和W 2,完成事件的处理,待手机后台刷新W 1和W 2的界面内容之后,投屏到平板电脑中的W 1和W 2的界面内容随着刷新。
在本申请中,无论同源投屏还是异源投屏,目标设备显示的应用窗口均运行在源设备上。而在窗口数据的传输方面,同源投屏直接将源设备的显示内容(display0)传输到目标设备上显示,异源投屏将源设备运行在后台的display1上的显示内容传输到目标设备上显示。本申请通过对跨设备传输窗口数据的精确管理,实现在目标设备上同时显示源设备已启动的多个窗口,扩展了投屏显示的全面性与灵活性,且操作简单,实用性强。
本申请实施例还提供了一种计算机可读存储介质,所述计算机可读存储介质存储有计算机程序,所述计算机程序被处理器执行时实现如本申请提出的各个投屏显示方法的步骤。
本申请实施例还提供了一种计算机程序产品,当计算机程序产品在终端设备上运行时,使得终端设备执行本申请提出的各个投屏显示方法的步骤。
图13为本申请一实施例提供的终端设备的结构示意图。如图13所示,该实施例的终端设备8包括:至少一个处理器80(图13中仅示出一个)处理器、存储器81以及存储在所述存储器81中并可在所述至少一个处理器80上运行的计算机程序82,所述处理器80执行所述计算机程序82时实现上述任意投屏显示方法实施例中的步骤。
所述终端设备8可以是手机、桌上型计算机、笔记本、掌上电脑及云端服务器等计算设备。该终端设备可包括,但不仅限于,处理器80、存储器81。本领域技术人员可以理解,图13仅仅是终端设备8的举例,并不构成对终端设备8的限定,可以包括比图示更多或更少的部件,或者组合某些部件,或者不同的部件,例如还可以包括输入输出设备、网络接入设备等。
所称处理器80可以是中央处理单元(Central Processing Unit,CPU),该处理器80还可以是其他通用处理器、数字信号处理器(Digital Signal Processor,DSP)、专用集成电路(Application Specific Integrated Circuit,ASIC)、现成可编程门阵列(Field-Programmable Gate Array,FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
所述存储器81在一些实施例中可以是所述终端设备8的内部存储单元,例如终端设备8的硬盘或内存。所述存储器81在另一些实施例中也可以是所述终端设备8的外部存储设备,例如所述终端设备8上配备的插接式硬盘,智能存储卡(Smart Media Card,SMC),安全数字(Secure Digital,SD)卡,闪存卡(Flash Card)等。进一步地,所述存储器81还可以既包括所述终端设备8的内部存储单元也包括外部存储设备。所述存储器81用于存储操作装置、应用程序、引导装载程序(BootLoader)、数据以及其他程序等,例如所述计算机程序的程序代码等。所述存储器81还可以用于暂时地存储已经输出或者将要输出的数据。
所属领域的技术人员可以清楚地了解到,为了描述的方便和简洁,仅以上述各功 能单元、模块的划分进行举例说明,实际应用中,可以根据需要而将上述功能分配由不同的功能单元、模块完成,即将所述装置的内部结构划分成不同的功能单元或模块,以完成以上描述的全部或者部分功能。实施例中的各功能单元、模块可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中,上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能单元的形式实现。另外,各功能单元、模块的具体名称也只是为了便于相互区分,并不用于限制本申请的保护范围。上述装置中单元、模块的具体工作过程,可以参考前述方法实施例中的对应过程,在此不再赘述。
在上述实施例中,对各个实施例的描述都各有侧重,某个实施例中没有详述或记载的部分,可以参见其它实施例的相关描述。
本领域普通技术人员可以意识到,结合本文中所公开的实施例描述的各示例的单元及算法步骤,能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行,取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能,但是这种实现不应认为超出本申请的范围。
在本申请所提供的实施例中,应该理解到,所揭露的装置和方法,可以通过其它的方式实现。例如,以上所描述的装置实施例仅仅是示意性的,例如,所述模块或单元的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,例如多个单元或组件可以结合或者可以集成到另一个装置,或一些特征可以忽略,或不执行。另一点,所显示或讨论的相互之间的耦合或直接耦合或通讯连接可以是通过一些接口,装置或单元的间接耦合或通讯连接,可以是电性,机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的,作为单元显示的部件可以是或者也可以不是物理单元,即可以位于一个地方,或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外,在本申请各个实施例中的各功能单元可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时,可以存储在一个计算机可读取存储介质中。基于这样的理解,本申请实现上述实施例方法中的全部或部分流程,可以通过计算机程序来指令相关的硬件来完成,所述的计算机程序可存储于一计算机可读存储介质中,该计算机程序在被处理器执行时,可实现上述各个方法实施例的步骤。其中,所述计算机程序包括计算机程序代码,所述计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。所述计算机可读介质至少可以包括:能够将计算机程序代码携带到终端设备的任何实体或装置、记录介质、计算机存储器、只读存储器(ROM,Read-Only Memory)、随机存取存储器(RAM,Random Access Memory)、电载波信号、电信信号以及软件分发介质。例如U盘、移动硬盘、磁碟或者光盘等。在某些司法管辖区,根据立法和专利实践,计算机可读介质不可以是电载波信号和电信信号。
以上所述实施例仅用以说明本申请的技术方案,而非对其限制;尽管参照前述实施例对本申请进行了详细的说明,本领域的普通技术人员应当理解:其依然可以对前述各实施例所记载的技术方案进行修改,或者对其中部分技术特征进行等同替换;而这些修改或者替换,并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围,均应包含在本申请的保护范围之内。

Claims (15)

  1. 一种投屏显示方法,应用于被投屏的目标设备,其特征在于,所述方法包括:
    在接收到源设备发送的投屏请求信息后,获取所述源设备已启动的应用程序的窗口数据,所述窗口数据包括所述源设备已启动的同一个应用程序打开的至少两个窗口对应的显示内容;
    显示所述已启动的应用程序的窗口数据。
  2. 如权利要求1所述的投屏显示方法,其特征在于,所述获取所述源设备已启动的应用程序的窗口数据包括:
    获取所述源设备已启动的应用程序中已打开的各个窗口的窗口数据,每个所述窗口的窗口组件均为激活状态且每个所述窗口组件的图层均为可见状态,每个所述窗口的窗口数据均包含对应的窗口标识;
    所述显示所述已启动的所述应用程序的窗口数据包括:
    根据所述各个窗口的窗口数据以及窗口数据中包含的窗口标识,分别构建并显示所述各个窗口。
  3. 如权利要求2所述的投屏显示方法,其特征在于,所述各个窗口包括第一窗口和第二窗口,所述第一窗口为所述源设备当前未显示的窗口,所述第二窗口为所述源设备当前显示的窗口,所述显示所述各个窗口包括:
    在显示界面中划分出第一区域和第二区域;
    将所述第一窗口添加到所述第一区域中显示,并将所述第二窗口添加到所述第二区域中显示。
  4. 如权利要求3所述的投屏显示方法,其特征在于,在将所述第一窗口添加到所述第一区域中显示,并将所述第二窗口添加到所述第二区域中显示之后,还包括:
    若检测到从所述第一窗口中打开的第三窗口,则将所述第三窗口添加到所述第二区域中显示,并关闭所述第二窗口;
    若检测到从所述第二窗口中打开的第四窗口,则将所述第四窗口添加到所述第二区域中且覆盖到所述第二窗口的上方显示。
  5. 如权利要求3所述的投屏显示方法,其特征在于,在将所述第一窗口添加到所述第一区域中显示,并将所述第二窗口添加到所述第二区域中显示之后,还包括:
    若检测到从所述第一窗口中打开的第五窗口,则将所述第五窗口添加到所述第二区域中显示,并关闭所述第二窗口;
    若检测到从所述第二窗口中打开的第六窗口,则将所述第六窗口添加到所述第二区域中显示,并将所述第二窗口移动到所述第一区域中且覆盖到所述第一窗口的上方显示。
  6. 如权利要求2所述的投屏显示方法,其特征在于,每个所述窗口的窗口数据还包含对应的应用程序信息、窗口组件信息和窗口属性信息,所述分别构建并显示所述各个窗口包括:
    根据每个所述窗口的窗口数据包含的应用程序信息、窗口组件信息和窗口属性信息,确定待显示窗口的数量、尺寸、显示位置以及显示方式;
    按照所述待显示窗口的数量、尺寸、显示位置以及显示方式,在显示界面中构建并显示相应的窗口。
  7. 如权利要求6所述的投屏显示方法,其特征在于,在显示界面中构建并显示相应的窗口之后,还包括:
    若检测到对构建并显示的任意一个窗口进行操作的第一指令,则获取所述任意一个窗口的窗口数据包含的目标应用程序信息和目标窗口组件信息;
    将所述第一指令、所述目标应用程序信息和所述目标窗口组件信息发送至所述源设备,以指示所述源设备根据所述目标应用程序信息和所述目标窗口组件信息分别确定待操作的目标应用程序和目标窗口,并基于所述第一指令对所述目标应用程序的所述目标窗口进行操作。
  8. 如权利要求2至7中任一项所述的投屏显示方法,其特征在于,在分别构建并显示所述各个窗口之后,还包括:
    当检测到所述源设备当前的显示界面发生变化时,保持显示所述各个窗口的内容;
    当检测到对构建并显示的所述各个窗口进行操作的第二指令后,将所述第二指令发送至所述源设备,以指示所述源设备保持当前的显示界面不变,并基于所述第二指令对所述源设备后台运行的对应窗口进行操作。
  9. 一种投屏显示方法,应用于投屏的源设备,其特征在于,所述方法包括:
    发送投屏请求信息至目标设备;
    在接收到所述目标设备基于所述投屏请求信息返回的确认信息后,发送已启动的应用程序的窗口数据至所述目标设备,以指示所述目标设备显示所述已启动的应用程序的窗口数据,所述窗口数据包括所述源设备已启动的同一个应用程序打开的至少两个窗口对应的显示内容。
  10. 如权利要求9所述的投屏显示方法,其特征在于,所述发送已启动的应用程序的窗口数据至所述目标设备,以指示所述目标设备显示所述已启动的应用程序的窗口数据包括:
    发送已启动的应用程序中已打开的各个窗口的窗口数据至所述目标设备,以指示所述目标设备根据所述各个窗口的窗口数据以及窗口数据中包含的窗口标识,分别构建并显示所述各个窗口;
    其中,每个所述窗口的窗口组件均为激活状态且每个所述窗口组件的图层均为可见状态,每个所述窗口的窗口数据均包含对应的窗口标识。
  11. 如权利要求10所述的投屏显示方法,其特征在于,在发送已启动的应用程序中已打开的各个窗口的窗口数据至所述目标设备之前,还包括:
    若检测到任意一个已启动的应用程序打开一个新窗口,则将所述新窗口的窗口组件设置为激活状态,并将所述新窗口的窗口组件的图层设置为可见状态。
  12. 一种投屏显示方法,其特征在于,包括:
    源设备发送投屏请求信息至目标设备;
    所述目标设备在接收到所述投屏请求信息后,基于所述投屏请求信息返回确认信息至所述源设备;
    所述源设备在接收到所述确认信息后,发送已启动的应用程序的窗口数据至所述 目标设备,所述窗口数据包括所述源设备已启动的同一个应用程序打开的至少两个窗口对应的显示内容;
    所述目标设备获取并显示所述已启动的应用程序的窗口数据。
  13. 一种投屏显示系统,其特征在于,包括:
    源设备,用于发送投屏请求信息至目标设备,并在接收到所述目标设备基于所述投屏请求信息返回的确认信息后,发送已启动的应用程序的窗口数据至所述目标设备,所述窗口数据包括所述源设备已启动的同一个应用程序打开的至少两个窗口对应的显示内容;
    目标设备,用于在接收到所述源设备发送的投屏请求信息后,基于所述投屏请求信息返回确认信息至所述源设备,获取并显示所述已启动的应用程序的窗口数据。
  14. 一种终端设备,包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序,其特征在于,所述处理器执行所述计算机程序时实现如权利要求1至8任一项所述的投屏显示方法的步骤,或者实现如权利要求9至11任一项所述的投屏显示方法的步骤。
  15. 一种计算机可读存储介质,所述计算机可读存储介质存储有计算机程序,其特征在于,所述计算机程序被处理器执行时实现如权利要求1至8任一项所述的投屏显示方法的步骤,或者实现如权利要求9至11任一项所述的投屏显示方法的步骤。
PCT/CN2021/088371 2020-04-20 2021-04-20 投屏显示方法、系统、终端设备和存储介质 WO2021213379A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/919,963 US20230176806A1 (en) 2020-04-20 2021-04-20 Screen Projection Display Method and System, Terminal Device, and Storage Medium
EP21792145.1A EP4123437A4 (en) 2020-04-20 2021-04-20 SCREEN PROJECTION DISPLAY METHOD AND SYSTEM, TERMINAL APPARATUS AND STORAGE MEDIUM

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010312360.0A CN113542825B (zh) 2020-04-20 2020-04-20 投屏显示方法、系统、终端设备和存储介质
CN202010312360.0 2020-04-20

Publications (1)

Publication Number Publication Date
WO2021213379A1 true WO2021213379A1 (zh) 2021-10-28

Family

ID=78123630

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/088371 WO2021213379A1 (zh) 2020-04-20 2021-04-20 投屏显示方法、系统、终端设备和存储介质

Country Status (4)

Country Link
US (1) US20230176806A1 (zh)
EP (1) EP4123437A4 (zh)
CN (2) CN113542825B (zh)
WO (1) WO2021213379A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114501120A (zh) * 2022-01-11 2022-05-13 烽火通信科技股份有限公司 多终端无线投屏切换方法与电子设备

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114911442A (zh) * 2022-04-27 2022-08-16 Oppo广东移动通信有限公司 投屏方法、装置、终端、设备、计算机介质及程序产品
CN117093290A (zh) * 2022-05-13 2023-11-21 华为技术有限公司 窗口尺寸调整方法、相关装置及通信系统
CN117950612A (zh) * 2022-10-31 2024-04-30 荣耀终端有限公司 一种投屏方法和电子设备
CN117724669A (zh) * 2023-06-30 2024-03-19 荣耀终端有限公司 一种投屏显示方法及电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103379221A (zh) * 2012-04-23 2013-10-30 Lg电子株式会社 移动终端及其控制方法
US20160216850A1 (en) * 2015-01-26 2016-07-28 Hong Kong Applied Science and Technology Research Institute Company Limited Method for managing multiple windows on a screen for multiple users, and device and system using the same
CN106095237A (zh) * 2016-06-08 2016-11-09 联想(北京)有限公司 信息处理方法及电子设备
CN110515576A (zh) * 2019-07-08 2019-11-29 华为技术有限公司 显示控制方法及装置

Family Cites Families (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8966379B2 (en) * 2010-10-01 2015-02-24 Z124 Dynamic cross-environment application configuration/orientation in an active user environment
US10152190B2 (en) * 2003-12-15 2018-12-11 Open Invention Network, Llc Systems and methods for improved application sharing in a multimedia collaboration session
US20070174429A1 (en) * 2006-01-24 2007-07-26 Citrix Systems, Inc. Methods and servers for establishing a connection between a client system and a virtual machine hosting a requested computing environment
US8407605B2 (en) * 2009-04-03 2013-03-26 Social Communications Company Application sharing
EP2283421B1 (en) * 2008-05-20 2019-08-14 Citrix Systems, Inc. Methods and systems for using external display devices with a mobile computing device
JP5713551B2 (ja) * 2009-10-01 2015-05-07 キヤノン株式会社 情報処理システム、情報処理装置、及びそれらの制御方法
US20160124698A1 (en) * 2011-08-24 2016-05-05 Z124 Unified desktop triad control user interface for an application launcher
US20220004350A1 (en) * 2011-09-27 2022-01-06 Z124 Unified desktop triad control user interface for an application launcher
US9703468B2 (en) * 2011-09-27 2017-07-11 Z124 Unified desktop independent focus in an application manager
US9008177B2 (en) * 2011-12-12 2015-04-14 Qualcomm Incorporated Selective mirroring of media output
KR101522399B1 (ko) * 2011-12-23 2015-05-22 주식회사 케이티 휴대 단말에서 외부 디스플레이 기기로의 화면 표출 방법 및 그 휴대 단말
JP5942498B2 (ja) * 2012-03-14 2016-06-29 日本電気株式会社 画面送信装置、画面共有装置、画面送信方法、及び、画面送信プログラム
CN102841804B (zh) * 2012-06-27 2014-12-10 北京三星通信技术研究有限公司 同屏多前台任务处理方法及装置
US9201563B2 (en) * 2012-10-10 2015-12-01 Google Inc. Mobile device friendly window management for remote desktop
KR102064952B1 (ko) * 2013-07-12 2020-01-10 삼성전자주식회사 수신 데이터를 이용하여 어플리케이션을 운영하는 전자 장치
KR102144553B1 (ko) * 2013-08-30 2020-08-13 삼성전자주식회사 다중 디스플레이 방법, 저장 매체 및 전자 장치
EP3054378B1 (en) * 2013-10-04 2022-11-02 Sony Group Corporation Information processing device, information processing method, and program
KR20170096408A (ko) * 2016-02-16 2017-08-24 삼성전자주식회사 어플리케이션을 표시하는 방법 및 이를 지원하는 전자 장치
KR20180095399A (ko) * 2017-02-17 2018-08-27 삼성전자주식회사 화면을 공유하기 위한 전자 장치 및 방법
US10687021B2 (en) * 2017-11-27 2020-06-16 Blue Jeans Network, Inc. User interface with a hierarchical presentation of selection options for selecting a sharing mode of a video conference
CN109889885A (zh) * 2019-02-27 2019-06-14 努比亚技术有限公司 一种投屏控制方法、终端及计算机可读存储介质
CN110417992B (zh) * 2019-06-20 2021-02-12 华为技术有限公司 一种输入方法、电子设备和投屏系统
KR20210061199A (ko) * 2019-11-19 2021-05-27 삼성전자주식회사 디스플레이 기기, 모바일 기기, 디스플레이 기기의 화면 미러링 방법 및 모바일 기기의 화면 미러링 방법
KR20210074877A (ko) * 2019-12-12 2021-06-22 삼성전자주식회사 디스플레이 장치 및 그 동작방법
US11150861B1 (en) * 2020-06-25 2021-10-19 Coscreen, Inc. Apparatus and method for simultaneous multi-user screen and window sharing, capturing and coordination
US11481178B2 (en) * 2021-01-29 2022-10-25 Avaya Management L.P. Secure multiple application sharing during a remote session

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103379221A (zh) * 2012-04-23 2013-10-30 Lg电子株式会社 移动终端及其控制方法
US20160216850A1 (en) * 2015-01-26 2016-07-28 Hong Kong Applied Science and Technology Research Institute Company Limited Method for managing multiple windows on a screen for multiple users, and device and system using the same
CN106095237A (zh) * 2016-06-08 2016-11-09 联想(北京)有限公司 信息处理方法及电子设备
CN110515576A (zh) * 2019-07-08 2019-11-29 华为技术有限公司 显示控制方法及装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114501120A (zh) * 2022-01-11 2022-05-13 烽火通信科技股份有限公司 多终端无线投屏切换方法与电子设备
CN114501120B (zh) * 2022-01-11 2023-06-09 烽火通信科技股份有限公司 多终端无线投屏切换方法与电子设备

Also Published As

Publication number Publication date
CN113542825B (zh) 2022-10-11
EP4123437A4 (en) 2023-10-04
EP4123437A1 (en) 2023-01-25
US20230176806A1 (en) 2023-06-08
CN113542825A (zh) 2021-10-22
CN115623257A (zh) 2023-01-17

Similar Documents

Publication Publication Date Title
WO2021213379A1 (zh) 投屏显示方法、系统、终端设备和存储介质
WO2021098678A1 (zh) 投屏控制方法及电子设备
WO2019154181A1 (zh) 显示控制方法及移动终端
WO2021109926A1 (zh) 应用分享方法、电子设备及计算机可读存储介质
WO2021197263A1 (zh) 内容共享方法及电子设备
WO2021109907A1 (zh) 应用分享方法、第一电子设备及计算机可读存储介质
WO2021017763A1 (zh) 事件处理方法、终端设备及计算机可读存储介质
WO2021012931A1 (zh) 图标管理方法及终端
WO2019206036A1 (zh) 消息管理方法及终端
WO2020238449A1 (zh) 通知消息的处理方法及终端
WO2021017776A1 (zh) 信息处理方法及终端
WO2019196691A1 (zh) 一种键盘界面显示方法和移动终端
WO2021083087A1 (zh) 截屏方法及终端设备
WO2018133685A1 (zh) 界面显示方法、装置、终端及存储介质
WO2020151460A1 (zh) 对象处理方法及终端设备
WO2020042892A1 (zh) 通话模式切换方法及终端设备
WO2020001604A1 (zh) 显示方法及终端设备
WO2019114522A1 (zh) 屏幕控制方法、屏幕控制装置及移动终端
WO2021109959A1 (zh) 应用程序分享方法及电子设备
CN109407921A (zh) 一种应用程序的处理方法及终端设备
WO2021197265A1 (zh) 信息展示方法、电子设备及存储介质
WO2019228296A1 (zh) 显示处理方法及终端设备
WO2021175143A1 (zh) 图片获取方法及电子设备
WO2020215969A1 (zh) 内容输入方法及终端设备
WO2020181956A1 (zh) 应用标识的显示方法及终端设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21792145

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021792145

Country of ref document: EP

Effective date: 20221021

NENP Non-entry into the national phase

Ref country code: DE