WO2023273744A1 - 一种投屏控制方法与电子设备 - Google Patents

一种投屏控制方法与电子设备 Download PDF

Info

Publication number
WO2023273744A1
WO2023273744A1 PCT/CN2022/095705 CN2022095705W WO2023273744A1 WO 2023273744 A1 WO2023273744 A1 WO 2023273744A1 CN 2022095705 W CN2022095705 W CN 2022095705W WO 2023273744 A1 WO2023273744 A1 WO 2023273744A1
Authority
WO
WIPO (PCT)
Prior art keywords
electronic device
interface
window
application
display information
Prior art date
Application number
PCT/CN2022/095705
Other languages
English (en)
French (fr)
Inventor
刘成
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2023273744A1 publication Critical patent/WO2023273744A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • G06F3/1462—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay with means for detecting differences between the image stored in the host and the images displayed on the remote displays

Definitions

  • the present application relates to the field of electronic technology, and in particular to a screen projection control method and electronic equipment.
  • Screen projection technology is a technology that is emerging nowadays, which means that an electronic device can project the display interface on its display screen to another electronic device for display.
  • an electronic device can project the display interface on its display screen to another electronic device for display. For example, a movie on a mobile phone can be casted to be played on a TV, pictures on a tablet can be cast to be displayed on a TV, and so on.
  • the purpose of the present application is to provide a screen projection control method and an electronic device, which are used to improve the convenience of controlling a target device in a screen projection scene.
  • a screen projection control method is provided.
  • the method can be applied to a system including a first electronic device and a second electronic device.
  • the method includes: the first electronic device displays a first interface of a first application; in response to a screen projection operation, the first electronic device sends display information of the first interface to the second electronic device;
  • the second electronic device displays the first interface of the first application;
  • the first electronic device displays the second interface of the second application, and a first window, the first window includes a third interface
  • the third interface is the same as the current display interface of the first application projected onto the second electronic device; in response to a user operation, the first electronic device displays the third interface in the first window.
  • the interface is updated to the fourth interface; the first electronic device sends the display information of the fourth interface to the second electronic device; the display interface of the first application on the second electronic device is updated to the The fourth interface.
  • the first window is displayed on the display screen of the first electronic device.
  • the display information in the first window is the same as the display information projected onto the second electronic device. In this way, the user can control the display information projected to the second electronic device through the first window, which improves the convenience of controlling the projected information on the target device (ie, the second electronic device) in the screen projection scene.
  • the first window is suspended and displayed on the upper layer of the second interface, and the first window does not completely cover the second interface; or, the second interface and the first window Split screen display. That is to say, after the first electronic device launches the first application, the first electronic device displays the second interface and the first window of the second application.
  • the displaying the first window by the first electronic device includes: receiving an operation for calling out the first window; responding to the operation for calling out the first window , to display the first window. That is to say, after launching the first application, the first electronic device displays the second interface of the second application, and at this time, the first window may not be displayed. When it is detected that the user triggers the operation, the first window is invoked. In this way, the user can invoke the first window when needed (for example, when it is necessary to control the information displayed on the screen projected to the target device), and the user experience is better.
  • displaying the first window in response to the operation for calling out the first window, includes: displaying a first mark on the second interface; A marked operation calls out the first window.
  • the first mark may be in any form such as a button, an icon, or a text. That is, after the first electronic device launches the first application, the first electronic device displays the second interface of the second application and the first mark, and the first mark is used to call out the first window.
  • the display area of the first mark is smaller than the display area of the first window, which will not block the second interface too much, and the display of the first mark can play a certain role in guiding the user to call out the first window, and the user experience is better .
  • the method further includes: in response to the operation for hiding the first window, hiding the first window; or, in response to the operation for adjusting the display position of the first window and The operation of/or the display area adjusts the display position and/or display area of the first window. That is, after the first electronic device launches the first application, the first electronic device displays the second interface and the first window of the second application. If the user does not want to display the first window (for example, the first window blocks the second interface), the first window can be hidden, or the area and position of the first window can be adjusted, so that the user experience is better.
  • the first electronic device displaying the first window includes: the first electronic device creates a drawing surface, and display information on the drawing surface is displayed through a display screen of the first electronic device The first electronic device maps the display information in the first virtual display screen VD to the drawing surface; wherein, the first application runs in the first VD, and the first VD is used to The display information in the first VD is sent to the second electronic device.
  • the first electronic device when the first electronic device is about to launch the first application, it transfers the first application to run in the first VD, so that the second electronic device displays the first interface of the first application.
  • the display information of the first VD can also be mapped to the drawing surface corresponding to the local display screen (that is, the first window), so that the display information projected to the second electronic device is displayed in the first window on the local display screen.
  • the first application running in the first VD is not only sent to the second electronic device for display, but also sent to the local display screen (that is, the first window) for display, so as to realize the projection between the local display screen and the second electronic device. Synchronization of information is convenient for the user to control the display information on the second electronic device projected to the screen through the local display screen (that is, the first window).
  • the first electronic device maps display information in the first virtual display screen VD to the drawing surface, including: the first electronic device creates a second VD, and the second The VD corresponds to the drawing surface; the first electronic device maps the display information in the first VD to the second VD, and the display information in the second VD is displayed through the drawing surface.
  • the first electronic device has a built-in mapping (such as Mirror mapping) function, that is, the Mirror mapping function can be used directly without modifying the underlying code.
  • the mapping from the first VD to the second VD can use the mirror mapping that comes with the system, so that the modification of the underlying code is avoided.
  • the mapping displayed locally by the second VD to the mobile phone is a surface mapping, and does not need to be processed at the bottom layer. Therefore, this method is less difficult to implement and has higher processing efficiency.
  • the first electronic device in response to a user operation, updates the third interface in the first window to a fourth interface, including: in response to a user operation, displaying on the drawing interface The information is updated to the fourth interface; wherein, the display information of the fourth interface on the drawing interface is mapped to the second VD, and the second VD is used to convert the display information of the fourth interface Mapped to the first VD, the first VD is used to send the display information of the fourth interface to the second electronic device.
  • the bottom layer code does not need to be modified, and the realization difficulty is low and the efficiency is high.
  • the method further includes: the first electronic device sending display information of the fourth interface to the third electronic device; the third electronic device displaying the fourth interface ; The first electronic device transfers the control right to the display information projected onto the second electronic device to the third electronic device; the third electronic device controls the fourth electronic device on the local display screen When the interface is updated to the fifth interface, the second electronic device updates the fourth interface on the local display screen to the fifth interface.
  • the first electronic device can project the screen to the second electronic device and the third electronic device, and transfer the control right of the second electronic device to the third electronic device, releasing the processing pressure of the first electronic device, and the first electronic device Electronic devices can perform other operations without interruption.
  • the method further includes: canceling display of the first window on the first electronic device. Therefore, after the first electronic device transfers the control right over the second electronic device to the third electronic device, the first window may not be displayed on the first electronic device to avoid blocking the display screen.
  • the first electronic device transfers the control right of the display information projected on the second electronic device to a third electronic device, including: the first electronic device creates a third electronic device VD; the third VD is used to send the display information in the third VD to the third electronic device; the first electronic device maps the display information in the first VD to the third VD ; Wherein, the first application runs in the first VD, and the first VD is used to send the display information in the first VD to the second electronic device.
  • the transfer of the control right is realized through the information mapping between multiple VDs, without modifying the underlying code, which is less difficult to implement and has higher processing efficiency.
  • the second electronic device updates the fourth interface on the display screen of the local machine Updating to the fifth interface includes: when the third electronic device updates the fourth interface to the fifth interface, updating the display information in the third VD to the display information of the fifth interface, so The third VD is used to map the display information of the fifth interface to the first VD, and the first VD is used to send the display information of the fifth interface to the second electronic device. Therefore, after the third electronic device receives the right to control the screen projection information on the second electronic device (information projected by the first electronic device to the second electronic device), it realizes the control of the second electronic device through information mapping between multiple VDs. The screen projection information of the device is controlled. In this process, there is no need to modify the underlying code, which is less difficult to implement and has higher processing efficiency.
  • a screen projection control method including: a first electronic device displays a first interface of a first application; in response to a screen projection operation, the first electronic device sends display information of the first interface to the second electronic device;
  • the first electronic device displays a second interface of the second application, and a first window, and the first window includes a third interface, and the third interface is the same as the projected screen on the second electronic device.
  • the current display interface of the first application is the same;
  • the first electronic device updates the third interface in the first window to a fourth interface in response to a user operation
  • the first electronic device sends the display information of the fourth interface to the second electronic device, so that the display interface of the first application on the second electronic device is updated to the fourth interface.
  • the first window is suspended and displayed on the upper layer of the second interface, and the first window does not completely cover the second interface; or, the second interface and the first window Split screen display.
  • the first electronic device displays a first window, including:
  • the first window is displayed.
  • displaying the first window includes: displaying a first mark on the second interface;
  • the first window is invoked.
  • the method also includes:
  • the display position and/or display area of the first window is adjusted.
  • the first electronic device displays a first window, including:
  • the first electronic device creates a drawing surface, and the display information on the drawing surface is displayed through a display screen of the first electronic device;
  • the first electronic device maps the display information in the first virtual display screen VD to the drawing surface
  • the first application runs in the first VD, and the first VD is used to send the display information in the first VD to the second electronic device.
  • the first electronic device maps display information in the first virtual display screen VD to the drawing surface, including:
  • the first electronic device creates a second VD, the second VD corresponding to the drawing surface;
  • the first electronic device maps the display information in the first VD to the second VD, and the display information in the second VD is displayed through the drawing surface.
  • the first electronic device in response to a user operation, updates the third interface in the first window to a fourth interface, including:
  • the display information on the drawing interface is updated to the fourth interface; wherein, the display information of the fourth interface on the drawing interface is mapped to the second VD, and the second VD It is used for mapping the display information of the fourth interface to the first VD, and the first VD is used for sending the display information of the fourth interface to the second electronic device.
  • the method also includes:
  • the first electronic device sends display information of the fourth interface to a third electronic device, so that the third electronic device displays the fourth interface;
  • the first electronic device transfers the control right of the display information projected on the second electronic device to the third electronic device.
  • the method further includes: canceling display of the first window on the first electronic device.
  • the first electronic device transfers control over the display information projected on the second electronic device to a third electronic device, including:
  • the first electronic device creates a third VD; the third VD is used to send the display information in the third VD to the third electronic device;
  • the first electronic device maps the display information in the first VD to the third VD; wherein, the first application runs in the first VD, and the first VD is used to use the first VD
  • the display information in a VD is sent to the second electronic device.
  • the method further includes: the third VD is used to, when the fourth interface on the third electronic device is updated to the fifth interface, display the The information is updated to the display information of the fifth interface, and the display information of the fifth interface is mapped to the first VD, and the first VD is used to send the display information of the fifth interface to the Describe the second electronic device.
  • a communication system including: a first electronic device and a second electronic device;
  • the first electronic device includes: a processor; a memory; wherein the memory stores one or more computer programs, and the one or more computer programs include instructions, and when the instructions are executed by the processor, making the first electronic device perform the steps of the first electronic device in the method described in the first aspect above;
  • the second electronic device includes: a processor; a memory; wherein the memory stores one or more computer programs, and the one or more computer programs include instructions, and when the instructions are executed by the processor, Making the second electronic device execute the steps of the second electronic device in the method provided by the second aspect above.
  • an electronic device including:
  • processor memory, and, one or more programs
  • the one or more programs are stored in the memory, the one or more programs include instructions, and when the instructions are executed by the processor, the electronic device performs the above-mentioned second aspect The method steps provided.
  • a computer-readable storage medium the computer-readable storage medium is used to store a computer program, and when the computer program is run on a computer, the computer is made to perform the above-mentioned first or second aspect. provided method.
  • a computer program product including a computer program.
  • the computer program When the computer program is run on a computer, the computer is made to execute the method described in the first aspect or the second aspect above.
  • a graphical user interface on an electronic device the electronic device has a display screen, a memory, and a processor, and the processor is used to execute one or more computer programs stored in the memory,
  • the graphical user interface includes a graphical user interface displayed when the electronic device executes the method described in the second aspect above.
  • the embodiment of the present application further provides a chip, the chip is coupled with the memory in the electronic device, and is used to call the computer program stored in the memory and execute the technical solutions of the first aspect to the second aspect of the embodiment of the present application , "Coupling" in the embodiments of the present application means that two components are directly or indirectly combined with each other.
  • FIGS. 1A to 1B are schematic diagrams of mirror projection provided by an embodiment of the present application.
  • FIGS. 2A to 2B are schematic diagrams of application projection provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of switching from application projection to mirror projection according to an embodiment of the present application.
  • FIGS. 4A to 4B are schematic diagrams of an application scenario provided by an embodiment of the present application.
  • 5A to 5B are schematic diagrams of another application scenario provided by the embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of the main device application screen projection provided by the embodiment of the present application.
  • FIG. 8 is a schematic diagram of the background processing flow of the main device application screen projection provided by the embodiment of the present application.
  • FIG. 9 is a first schematic diagram of a master device displaying a first window provided in an embodiment of the present application.
  • 10 to 11 are the second schematic diagrams of the master device displaying the first window provided by the embodiment of the present application.
  • FIG. 14 is a schematic flowchart of a screen projection control method provided in an embodiment of the present application.
  • FIG. 15 is another schematic flow chart of the screen projection control method provided by the embodiment of the present application.
  • FIG. 16 is a schematic diagram of the software structure of the master device provided by the embodiment of the present application.
  • Fig. 17 is another schematic flow chart of the screen projection control method provided by the embodiment of the present application.
  • FIG. 18 to FIG. 19 are schematic diagrams of a master device projecting a screen to multiple target devices according to an embodiment of the present application.
  • FIG. 20 is a schematic flowchart of a screen projection control method provided in an embodiment of the present application.
  • FIG. 21 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • Application program is a software program that can realize one or more specific functions.
  • multiple applications can be installed in an electronic device, for example, an instant messaging application, a video application, an audio application, an image capture application, and the like.
  • instant messaging applications for example, can include SMS applications, Changlian, WhatsApp Photo sharing (Instagram), Kakao Wait.
  • the image capturing application may include, for example, a camera application (system camera or third-party camera application).
  • Video applications for example, can include wait.
  • Audio applications for example, may include Google Music, Kugou Music, Xiami, QQ Music and so on.
  • the applications mentioned in the following embodiments may be applications installed when the electronic device leaves the factory, or applications downloaded by the user from the network or obtained from other electronic devices during the use of the electronic device.
  • At least one of the embodiments of the present application involves one or more; wherein, a plurality means greater than or equal to two.
  • words such as “first” and “second” are only used for the purpose of distinguishing descriptions, and cannot be understood as express or implied relative importance, nor can they be understood as express or imply order.
  • the screen projection scene includes the sender and the receiver.
  • the sender sends the display information to the receiver for display.
  • this article refers to the sending end as the master device and the receiving end as the target device.
  • the main device may be a mobile phone, a tablet computer, a PC, a mobile phone, a watch, and the like.
  • the target device can be a large-screen device such as a tablet computer, a PC, or a TV.
  • the sending end and the receiving end can also have other names, for example, the sending end is an active screen projection device, and the receiving end is a passive screen projection device; or, the sending end is a master device , the receiving end is an auxiliary device; or, the sending end is a source device, and the receiving end is a target device; or, the sending end is a first electronic device, and the receiving end is a second electronic device, etc., which is not limited in this application.
  • the roles of the sending end and the receiving end can be interchanged, that is, the sending end can project a screen to the receiving end, and correspondingly, the receiving end can also project a screen to the sending end.
  • Screen projection technologies include mirror projection and application projection.
  • Mirroring screen projection means that the encoder at the sending end performs video encoding on the information currently displayed on the display screen at the sending end, and sends the encoded display information to the receiving end through the data channel between the receiving end and the receiving end through the decoder.
  • the received display information is decoded, and a Surface is created to carry and display the decoded display information.
  • the sending end and the receiving end can display the same screen, in other words, whatever content is currently displayed on the sending end can be projected to the receiving end.
  • FIG. 1A is a schematic diagram of a process of mirroring and projecting a screen.
  • the sending end projects the information displayed in the foreground (such as the interface A of the first application) to the receiving terminal (such as a PC), and the receiving end displays the displayed information (such as the interface A of the first application).
  • the receiving terminal such as a PC
  • the receiving end displays the displayed information (such as the interface A of the first application).
  • mirroring screen projection is that the current display information of the local display (local display) at the sending end is directly sent to the sending end after video encoding, and the sending end decodes and displays it. This method does not need to create a virtual display (virtual display, VD).
  • VD virtual display
  • Another implementation of mirroring screen projection is to create a VD. For example, as shown in FIG. 1B , the mobile phone creates a first VD, and maps the current display information of the mobile phone's local display screen to the first VD. Among them, the first VD is invisible to users, and belongs to the virtual display created in the background of the mobile phone, and the display information in the first VD is projected to the target device by default.
  • the first VD sends the display information to the encoder for encoding, and the encoded display information is sent to the receiving end (such as a PC or a TV set), so that the mobile phone projects the foreground display information to the receiving end for display.
  • the data source is the local display screen of the mobile phone.
  • the display information on the local display screen is mapped to the first VD in real time.
  • the display that needs to be encoded is the first VD.
  • the data sent to the receiving end After encoding, the data sent to the receiving end. Therefore, when the displayed information on the local display screen of the mobile phone changes, correspondingly, the displayed information on the receiving end also changes accordingly.
  • FIG. 2A is a schematic diagram of an application screen projection process.
  • the interface A of the first application is displayed on the main device.
  • the main device detects the screen projection operation of the application, the main device will project the first application to the target device for display, and the interface of the second application will be displayed in the foreground of the main device.
  • the master device casts the first application as a whole to the target device.
  • the granularity of screen projection is the application.
  • FIG. 2B Please refer to Figure 2B for the implementation principle of app projection.
  • the local display screen of the mobile phone is displaying the interface A of the first application.
  • the mobile phone detects the screen projection operation of the application, the mobile phone will transfer the first application to run on the first VD (for example, Move the first application from task stack 1 in the local display screen to task stack 2 in the first VD).
  • the interface of the second application is displayed on the local display screen.
  • the first VD sends the display information to the encoder for encoding, and the encoded display information is sent to the receiving end.
  • the display interface of the first application in the first VD changes, correspondingly, the display interface of the first application on the receiving end also changes, and the display interface on the local display screen
  • the display interface of the first VD and the display interface from the projection screen on the receiving end may be different. It can be simply understood that the source of data projected to the target device is no longer the local display screen of the mobile phone, but the first VD.
  • the following is an example of switching from app mirroring to mirroring.
  • FIG. 3 is a schematic diagram of switching from application projection to mirror projection.
  • the switching process includes: referring to (a) in FIG. 3 , returning the first application in the first VD (that is, the projected application) to the local display screen.
  • the rollback operation can be understood as a stack moving operation (for example, moving the first application in the task stack 2 in the first VD to the task stack 1 in the local display screen).
  • the first application in the first VD is released, that is, there is no first application in the first VD, and at this time, the screen projection of the application is terminated.
  • the local display screen is mirrored for screen projection. That is, the local display screen maps the first application to the first VD, and the first VD sends display information to the target device. That is to say, (b) in FIG. 3 is the same as the mirror projection process shown in FIG. 1B above. At this point, the switch from application projection to mirror projection is completed.
  • the projected application i.e. the first application
  • the interface of the first application needs to be displayed on the local display screen of the mobile phone.
  • One implementation manner is to switch from application screen projection to mirror projection, and then the user controls the first application on the local display screen, thereby realizing the control of the first application screen projection on the target device.
  • the user controls the first application on the local display screen so that interface A of the first application is switched to interface B, which is synchronized to the first VD, and then the first VD is synchronized to the target device , interface A of the first application is updated to interface B of the first application on the target device.
  • This method of controlling the screen projection to the application on the target device needs to be switched from application projection to mirror projection, which is cumbersome, time-consuming, and has a low user experience.
  • the embodiment of this application provides a control method for displaying information on a target device.
  • the first window can be displayed on the main device, and the screen displayed in the first window is projected to the target device.
  • the current display interface of the application on the screen; that is, the display information in the first window is synchronized with the current display information of the application being projected to the target device. Control the information displayed on the screen.
  • FIG. 4A to 4B are schematic diagrams of an application scenario provided by an embodiment of the present application.
  • the mobile phone is currently displaying the home page of the photo album application.
  • the mobile phone detects the screen-casting operation of the application, it casts the screen of the photo-album application to the target device, and the display interface of the photo-album application is displayed on the target device, as shown in (b) in FIG. 4A .
  • the mobile phone displays the display interface of the instant messaging application (such as the chat interface with contacts), while the target device still displays the interface of the photo album application, as shown in Figure 4A (d).
  • the first window 401 is displayed on the mobile phone, as shown in (a) in FIG. 4B .
  • the first window 401 may appear automatically after the mobile phone launches the photo album application. For example, after the mobile phone in (a) in FIG. 4A casts the photo album application to the target device, the mobile phone does not display the interface shown in (c) in FIG. 4A , but directly displays the interface shown in (a) in FIG. 4B .
  • the first window 401 can be displayed in a floating manner, and can completely or partially block the interface of the instant messaging application.
  • the first window 401 displays the current display interface of the photo album application projected to the target device (ie, the home page of the photo album application). That is to say, the first window 401 is the same as the current display interface of the screen casting application on the target device.
  • the user can control the photo album application projected to the target device through the first window 401.
  • the user can also perform other operations on the photo album application through the first window 401, such as sliding to the next image, zooming in and out of images, opening the home page of the photo album, and so on.
  • FIG. 5A to 5B are schematic diagrams of another application scenario provided by the embodiment of the present application.
  • the mobile phone is displaying an interface of a video playing application (for example, a playing interface of a TV series).
  • the video playback application is projected to the target device, as shown in (b) in FIG. 5A , and the interface of the video playback application is displayed on the target device.
  • the main page is displayed on the foreground of the mobile phone, as shown in (c) in Figure 5A, while the interface of the video playback application is still displayed on the target device, as in (d) in Figure 5A.
  • the first window 501 is displayed on the mobile phone, as shown in (a) in FIG. 5B .
  • the first window 501 may appear automatically after the mobile phone launches the video playing application. For example, after the mobile phone in (a) in Figure 5A casts the video playback application to the target device, the mobile phone does not display the interface shown in (c) in Figure 5A, and directly displays the interface shown in (a) in Figure 5B .
  • the first window 501 can be displayed in a floating state, and can completely or partially block the main interface.
  • the first window 501 displays the current display interface of the video playing application projected to the target device (ie, the playing interface of the TV series). That is to say, the first window 501 is the same as the current display interface of the screen casting application on the target device.
  • the user can control the video playback application projected to the target device through the first window 501, please refer to (c) in Figure 5B, the user opens the next episode of the TV series in the first window 501, corresponding , open the next episode of the TV series on the target device, as shown in (d) in Figure 5B.
  • the user can also perform other operations on the video playing application through the first window 501, such as speeding up playing, slowing down playing, selecting another TV, and so on.
  • the main device may be a portable electronic device, such as a mobile phone, a tablet computer, a portable computer, a wearable device with a wireless communication function (such as a smart watch, smart glasses, a smart bracelet, or a smart helmet, etc.), or a vehicle-mounted device.
  • portable electronic devices include, but are not limited to Or portable electronic devices with other operating systems.
  • the target device can be a large-screen device such as a tablet computer, a PC, or a TV.
  • Fig. 6 shows a schematic structural diagram of an electronic device.
  • the structure of FIG. 6 may be the structure of the master device or the structure of the target device.
  • the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, Antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, A display screen 194, and a subscriber identification module (subscriber identification module, SIM) card interface 195, etc.
  • SIM subscriber identification module
  • the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.
  • the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors. Wherein, the controller may be the nerve center and command center of the electronic equipment. The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.
  • a memory may also be provided in the processor 110 for storing instructions and data.
  • the memory in processor 110 is a cache memory.
  • the memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.
  • the USB interface 130 is an interface conforming to the USB standard specification, specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like.
  • the USB interface 130 can be used to connect the charger to charge the electronic equipment, and can also be used to transmit data between the electronic equipment and peripheral equipment.
  • the charging management module 140 is configured to receive a charging input from a charger.
  • the power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 .
  • the power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 .
  • the wireless communication function of the electronic device can be realized by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in an electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas.
  • Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
  • the antenna may be used in conjunction with a tuning switch.
  • the mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied to electronic devices.
  • the mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like.
  • the mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation.
  • the mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves through the antenna 1 for radiation.
  • at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 .
  • at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.
  • the wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite system, etc. (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.
  • the wireless communication module 160 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 .
  • the wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.
  • the antenna 1 of the electronic device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR techniques, etc.
  • GSM global system for mobile communications
  • general packet radio service general packet radio service
  • CDMA code division multiple access
  • WCDMA broadband Code division multiple access
  • time division code division multiple access time-division code division multiple access
  • TD-SCDMA time-division code division multiple access
  • LTE long term evolution
  • BT GNSS
  • the GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou navigation satellite system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • Beidou navigation satellite system beidou navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite based augmentation systems
  • the display screen 194 is used to display the display interface of the application and the like.
  • the display screen 194 includes a display panel.
  • the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc.
  • the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.
  • the electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.
  • the ISP is used for processing the data fed back by the camera 193 .
  • the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye.
  • ISP can also perform algorithm optimization on image noise, brightness, and skin color.
  • ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be located in the camera 193 .
  • Camera 193 is used to capture still images or video.
  • the object generates an optical image through the lens and projects it to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal.
  • the ISP outputs the digital image signal to the DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other image signals.
  • the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1.
  • the internal memory 121 may be used to store computer-executable program codes including instructions.
  • the processor 110 executes various functional applications and data processing of the electronic device by executing instructions stored in the internal memory 121 .
  • the internal memory 121 may include an area for storing programs and an area for storing data.
  • the storage program area can store the operating system and software codes of at least one application program (such as iQiyi application, WeChat application, etc.).
  • the data storage area can store data (such as images, videos, etc.) generated during the use of the electronic device.
  • the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.
  • the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device.
  • the external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, save pictures, videos and other files in the external memory card.
  • the electronic device can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.
  • the pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal.
  • pressure sensor 180A may be disposed on display screen 194 .
  • the gyro sensor 180B can be used to determine the motion posture of the electronic device.
  • the angular velocity of the electronic device about three axes ie, x, y, and z axes
  • the gyro sensor 180B can be used for image stabilization.
  • the air pressure sensor 180C is used to measure air pressure.
  • the electronic device calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.
  • the magnetic sensor 180D includes a Hall sensor.
  • the electronic device may detect opening and closing of the flip holster using the magnetic sensor 180D.
  • the electronic device when the electronic device is a flip machine, the electronic device can detect opening and closing of the flip according to the magnetic sensor 180D.
  • the acceleration sensor 180E can detect the acceleration of the electronic device in various directions (generally three axes). When the electronic device is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.
  • the distance sensor 180F is used to measure the distance.
  • Electronic devices can measure distance via infrared or laser light. In some embodiments, when shooting a scene, the electronic device can use the distance sensor 180F for distance measurement to achieve fast focusing.
  • Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes.
  • the light emitting diodes may be infrared light emitting diodes.
  • Electronic devices emit infrared light outwards through light-emitting diodes.
  • Electronic devices use photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object in the vicinity of the electronic device.
  • the electronic device may determine that there is no object in the vicinity of the electronic device.
  • the electronic device can use the proximity light sensor 180G to detect that the user holds the electronic device close to the ear to make a call, so as to automatically turn off the screen to save power.
  • the proximity light sensor 180G can also be used in leather case mode, automatic unlock and lock screen in pocket mode.
  • the ambient light sensor 180L is used for sensing ambient light brightness.
  • the electronic device can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness.
  • the ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.
  • the ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device is in the pocket to prevent accidental touch.
  • the fingerprint sensor 180H is used to collect fingerprints. Electronic devices can use the collected fingerprint features to unlock fingerprints, access application locks, take pictures with fingerprints, answer incoming calls with fingerprints, etc.
  • the temperature sensor 180J is used to detect temperature.
  • the electronic device uses the temperature detected by the temperature sensor 180J to implement a temperature treatment strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device may reduce the performance of a processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection.
  • the electronic device when the temperature is lower than another threshold, the electronic device heats the battery 142 to avoid abnormal shutdown of the electronic device caused by low temperature.
  • the electronic device boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
  • Touch sensor 180K also known as "touch panel”.
  • the touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”.
  • the touch sensor 180K is used to detect a touch operation on or near it.
  • the touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.
  • Visual output related to the touch operation can be provided through the display screen 194 .
  • the touch sensor 180K may also be disposed on the surface of the electronic device, which is different from the position of the display screen 194 .
  • the bone conduction sensor 180M can acquire vibration signals.
  • the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human voice.
  • the bone conduction sensor 180M can also contact the human pulse and receive the blood pressure beating signal.
  • the keys 190 include a power key, a volume key and the like.
  • the key 190 may be a mechanical key. It can also be a touch button.
  • the electronic device can receive key input and generate key signal input related to user settings and function control of the electronic device.
  • the motor 191 can generate a vibrating reminder.
  • the motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects.
  • the indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.
  • the SIM card interface 195 is used for connecting a SIM card. The SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to realize contact and separation with the electronic device.
  • FIG. 6 do not constitute a specific limitation on the electronic device.
  • the electronic device in the embodiment of the present invention may include more or less components than those shown in FIG. 6 .
  • the combination/connection relationship between the components in FIG. 6 can also be adjusted and modified.
  • the main device is a mobile phone and the target device is a TV set as an example.
  • FIG. 7 shows a kind of graphical user interface (graphical user interface, GUI) of mobile phone, and this GUI is the desktop of mobile phone.
  • the desktop includes icons for different applications.
  • the mobile phone detects the operation of the icon of the first application, it displays the interface of the first application as shown in (b) of FIG. 7 .
  • the first application may be any application, such as a video playing application, a browser, an instant messaging application, a photo album application, and the like.
  • the mobile phone detects a screen-casting operation, it displays a list of target devices, as shown in (c) in Figure 7.
  • the list of target devices includes the identification of target devices such as TVs and PCs. target device.
  • the screen projection button is displayed, and the screen projection operation refers to the click operation on the screen projection button; or, the screen projection operation can also be a voice command for instructing the screen projection, and the embodiment of the present application does not limit the type of the screen projection operation .
  • the target device list includes identifiers of target devices such as TV sets and PCs.
  • the foreground application that is, the first application
  • the TV displays the first application.
  • the interface of the second application is displayed in the foreground of the mobile phone, as shown in (e) in FIG. 7 .
  • the second application may be the last opened application before opening the first application, or the desktop of the mobile phone, or an interface of a pre-set application (set by default by the system or set by the user).
  • the mobile phone creates the first VD (optionally, the display parameters of the first VD are adapted to the target device, such as the same resolution), and the mobile phone transfers the first application on the local display screen to run in the first VD.
  • the first VD casts the screen of the first application to the target device.
  • the interface of the second application is displayed on the local display.
  • the area of the interface of the first application and the interface of the second application in Figure 8 is smaller than the area of the local display screen, in order to express the meaning that the local display screen can display the interface of the first application or the interface of the second application, It is not a limitation on the area size relationship between the application interface and the local display screen.
  • the user can perform other operations on the mobile phone, and of course the user can also control the screen projection on the mobile phone to the first application on the target device.
  • the mobile phone When the mobile phone casts the screen of the first application to the target device, the mobile phone displays the interface of the second application, and also displays the first window.
  • the first window displays the current display interface of the first application that is projected to the target device.
  • the interface displayed in the window is the same as the current interface displayed by the first application on the target device. The user can control the first application projected to the target device through the first window.
  • the mobile phone when it detects an operation for selecting a TV ID in the target device list, it casts the screen of the first application to the TV.
  • the interface of the first application is displayed on the television.
  • the interface of the second application is displayed on the mobile phone, and the first window 901 is also displayed.
  • the display interface in the first window 901 is the same as the current display interface of the first application projected to the target device.
  • the screen of the mobile phone displays the interface of the second application, and the first window 901 appears automatically, so that the user can control the screen projection to the target device through the first window 901 first application of .
  • the first window 901 can be displayed in various ways. For example, as shown in (c) of FIG. 9 , the first window 901 is displayed floatingly on the interface of the second application, and can completely cover or partially cover the interface of the second application.
  • the shape of the first window 901 is not limited in this embodiment of the application.
  • the display position and/or display area of the first window 901 may be set by default, or may be adjustable by the user.
  • the first window 901 may include a zoom button 902, and the zoom button 902 is used to zoom in or zoom out the first window.
  • the first window When the first window is enlarged, the first window can be displayed in full screen, and when the first window is reduced, it can be reduced to occupy the minimum area (for example, the minimum area set by the system).
  • the first window 901 can also be hidden, for example, when the mobile phone detects a hiding instruction for hiding the first window, the first window 901 is hidden.
  • the first window 901 includes a close button 903 , and when an operation on the close button 903 is detected, the first window 901 is hidden.
  • the mobile phone detects a voice instruction for instructing to hide the first window
  • the first window 901 is hidden.
  • the mobile phone detects that the user clicks on an area not covered by the first window 901 on the second interface, the first window 901 is hidden.
  • the first window 901 may also be called out.
  • the first window 901 can be called out through a calling instruction.
  • the mobile phone detects an operation on the close button 903 in the first window 901, the first window 901 is hidden, and an icon 1001 appears on the display screen of the mobile phone, please refer to (a) in FIG.
  • this icon 1001 is operated, the first window 901 is called.
  • the icon 1001 can also be displayed in other positions, for example, displayed in the status bar of (b) in Figure 10, or displayed in the sliding notification bar interface in (c) in Figure 10, or displayed Pull up the control center interface in (d) in FIG. 10 .
  • the first window is called out.
  • the first window 901 automatically appears on the display screen when the mobile phone projects the first window, which is convenient for the user to operate and has a high experience.
  • the mobile phone When the mobile phone projects the screen of the first application to the target device, the mobile phone displays the interface of the second application and displays a control button, and when an operation on the control button is detected, the first window is displayed.
  • the mobile phone when the mobile phone detects an operation for selecting a TV ID in the target device list, it casts the screen of the first application to the TV.
  • the interface of the first application is displayed on the television.
  • the mobile phone displays the interface of the second application, and also displays control buttons 1101 .
  • the mobile phone displays a first window 901, as shown in (d) in FIG. 11 .
  • control button 1101 can be displayed at any position, for example, the control button 1101 in (c) in FIG. interface, or displayed in the pull-up control center interface, etc., which are not limited in this embodiment of the present application.
  • the display position and/or display area of the first window 901 may be set by default or adjustable by the user.
  • the first window can also be hidden, and after hiding, the first window can also be called out.
  • the realization principle is the same as that of the first method above, and will not be repeated.
  • the difference between the second method (shown in FIG. 11 ) and the first method (shown in FIG. 9 ) is that in the first method, the mobile phone projects the first application to the target device while the mobile phone display screen The first window 901 appears automatically; in the second method, when the mobile phone projects the first application to the target device, the control button 1101 automatically appears on the display of the mobile phone, and when an operation on the control button 1101 is detected, the first window 901 appears .
  • the mobile phone may use the first method or the second method by default (for example, the system defaults), or the user may set the first method or the second method to be used, which is not limited in this embodiment of the application.
  • the user can control the screen projected to the application on the target device.
  • FIG. 13 shows the background processing flow of the mobile phone displaying the first window 901 .
  • the mobile phone is projecting the screen to the target device, so the first VD has been created, and the mobile phone has also created the second VD (optionally, the display parameters of the second VD can be adapted to the display parameters of the local display screen of the mobile phone, for example , same resolution).
  • the display information of the first application in the first VD may be mapped (for example, mirrored (Mirror) mapped) to the second VD.
  • the mobile phone also creates a drawing surface (Surface) for carrying the display information in the second VD. In other words, the display information in the second VD is displayed through the Surface.
  • the second VD is mapped to the local display screen ( That is, mapped to the first window for display).
  • the first VD sends the display information of the first application to the second VD and the target device respectively.
  • the first VD acts as a data source to provide the display information of the first application to the left and right respectively.
  • the first application always runs in the first VD, that is, the application screen projection is always in progress without termination, and there is no need to re-establish the mirror projection screen (that is, it is not necessary to switch the application screen projection to the mirror projection screen).
  • the establishment of the second VD and the first window that is, the surface corresponding to the second VD does not affect the process of application screen projection.
  • the process of switching from application projection to mirror projection in Figure 3 includes: rolling back the first application in the first VD to the local display screen. After the rollback, the first VD no longer includes the first application, and the application projection termination. Then, a mirrored screen projection is established, and the first application on the local display screen is mirrored to the first VD, and the display information in the first VD is synchronously displayed on the target device. To put it simply, you need to terminate the application screencasting first, and re-establish the mirroring screencasting to control the application screencasting to the target device.
  • the first VD serves as a data source and sends the display information of the first application to the second VD and the target device respectively. Therefore, the first application always runs in the first VD, so the application screen projection does not need to be terminated, and the mirror image projection does not need to be re-established, which is simpler than the process in Figure 3 .
  • FIG. 14 is a schematic flowchart of a method for controlling projecting a screen to a display interface on a target device provided by an embodiment of the present application. As shown in Figure 14, the process flow of the method includes:
  • the mobile phone displays the interface of the first application.
  • the mobile phone receives a screen projection operation, and determines whether it is a mirror image projection mode.
  • the mobile phone may output prompt information, and the prompt information is used to prompt the user to select a mirror image projection mode or a non-mirror screen projection mode (that is, an application projection screen mode).
  • the "moving” here can be understood as moving the running task (task) of the first application in the task stack of the local display screen to the task stack of the first VD, which can also be called “moving the stack”. After the first application is moved from the local display screen to run on the first VD, the local display screen does not have the running task of the first application.
  • the first VD projects the first application to the target device for display.
  • the input operation is an operation of clicking the control button 1101 in FIG. It is necessary to control the first application projected to the target device.
  • S6 may not be executed, so S6 is represented by a dotted line in the figure. If S6 is not executed, go directly from S5 to S7 to create the second VD. It can be understood that after the main device casts the first application, the first window will automatically appear in the foreground of the main device (that is, corresponding to the first method above).
  • S6 may be executed immediately after S5, or may be executed after S7 or S8, which is not limited in this embodiment of the present application.
  • a step may also be included: setting the first identifier of the first VD and the second identifier of the second VD.
  • the first identifier and/or the second identifier may be a VD name (name) or ID, which is not limited.
  • the first identifier of the first VD may be source VD
  • the second identifier of the second VD may be target VD, which is used to indicate that the first VD is mirrored to the second VD.
  • a step may also be included: setting a flag (flag), where the flag is used to indicate whether the first application projected to the target device needs to be controlled. For example, when the flag bit is 1, it is used to indicate that the first application that is projected to the target device needs to be controlled; when the flag bit is 0, it is used to indicate that the first application that is projected to the target device is not controlled. .
  • the method may further include a step of: creating a surface, where the surface may be a display area on the local display screen, such as the first window.
  • the surface can also exist all the time. For example, it exists by default after the mobile phone leaves the factory, so there is no need to create a surface before S9.
  • the update interface in the surface is mapped to the second VD, the second VD maps the update interface to the first VD, and the first VD projects the update interface to the target device.
  • the user controls the application projected to the target device through the surface corresponding to the second VD (that is, the first window).
  • controlling the display interface projected to the target device by creating the second VD has the following beneficial effects:
  • the mobile phone system will have a Mirror mapping function, that is, the Mirror mapping function can be used directly without modifying the underlying code.
  • the second VD is created, and the first VD can be mapped to the second VD using the mirror mapping provided by the system, so that the modification of the underlying code is avoided.
  • the mapping displayed locally by the second VD to the mobile phone is a surface mapping, and does not need to be processed at the bottom layer. Therefore, the technical solutions provided by the embodiments of the present application are less difficult to implement, and do not need to be processed at the bottom layer, so the processing efficiency is relatively high.
  • the mobile phone may not create the second VD.
  • the first VD directly maps the information displayed in the first VD to the first window of the local display screen for display. This method is more direct.
  • Mirror mapping may not be used, but through bottom layer processing, that is, the first VD draws the display information in the first VD through the bottom layer, synthesizes the picture, and passes the local display screen (such as the first VD a window) is displayed.
  • create a surface again for the first VD (this is because the first VD has created a surface to carry the display information of the target device when it is projected to the target device, and this surface is called an input surface), so The first VD corresponds to two surfaces. In this case, the display information of the input surface can be mapped to the recreated surface.
  • this method needs to modify the underlying code, it does not need to create a second VD, which saves the cost of VD creation.
  • the second embodiment can be understood as a refinement of the first embodiment.
  • the second embodiment introduces information interaction between different modules inside the main device, information interaction between different modules inside the target device, and information interaction process between the main device and the target device.
  • FIG. 15 is a schematic flowchart of a method for controlling an application that projects a screen to a target device in a screen projection scenario according to an embodiment of the present application.
  • This schematic flowchart can be understood as a refinement of the flowchart shown in FIG. 14 .
  • S1501 to S1508 in FIG. 15 introduce the implementation process of mirroring screen projection.
  • S1509 to S1517 introduce the implementation process of application screen projection, which can be understood as the refinement of S2 to S5 in FIG. 14 .
  • S1518 to S1531 introduce the process of controlling the display interface on the target device when the screen is projected by the application, which can be understood as the refinement process of S6 to S11 in FIG. 14 .
  • the process includes:
  • the screen mirroring module in the master device receives a screen mirroring command.
  • the screen mirroring module creates a first VD.
  • the screen mirroring module maps the display information on the local display screen of the master device to the first VD.
  • the screen mirroring module sends the display information in the first VD to the screen projection encoding module.
  • the screen projection encoding module encodes the display information.
  • the encoding manner includes H.264, MPEG-4, etc., which are not limited in this embodiment of the present application.
  • the screen projection encoding module sends the encoded display information to the target device.
  • the target device After receiving the encoded display information, the target device decodes it through the screen projection decoding module.
  • the screen projection decoding module in the target device sends the decoded display information to the screen projection surface module for display.
  • the above process is the process of mirroring screen projection, and the process of application screen projection includes the following steps:
  • the application screen projection module in the master device receives an application screen projection instruction.
  • the application screen projection module sends a VD creation request to the mirror image projection module.
  • the screen mirroring module creates a first VD.
  • the application projection module moves the first application to run in the first VD.
  • the "moving” here can be understood as moving the running task (task) of the first application in the task stack of the local display screen to the task stack of the first VD, which can also be called “moving the stack”. After the first application is moved from the local display screen to run on the first VD, the local display screen does not have the running task of the first application.
  • the screen mirroring module sends the display information in the first VD to the screen projection encoding module.
  • the screen projection encoding module encodes the display information.
  • the screen projection encoding module sends the encoded display information to the target device.
  • the target device After receiving the numbered display information, the target device decodes it through the screen projection decoding module.
  • the screen projection decoding module in the target device sends the decoded display information to the screen projection surface module for display.
  • the process of the master device controlling the display interface of the screen projection to the target device includes:
  • the application screen projection module sends the received control instruction to the application screen projection surface module, where the control instruction is used to instruct to control the display interface projected to the target device.
  • the application screen projection surface module sends a mirroring mapping instruction to the mirroring screen projection module, which is used to instruct the mirroring screen projection module to map the display information in the first VD to the second VD.
  • the screen mirroring module maps the display information in the first VD to the second VD.
  • the surface module of application screen projection receives an input operation, and the input operation is used to update the display information on the surface.
  • the first VD sends the updated display information to the screen projection encoding module.
  • the screen projection encoding module encodes the updated display information.
  • the screen projection coding module sends the coded display information to the target device.
  • the target device After receiving the numbered display information, the target device decodes it through the screen projection decoding module.
  • the screen projection decoding module in the target device sends the decoded display information (updated display information) to the screen projection surface module for display.
  • the third embodiment can be understood as a refinement of the first embodiment.
  • the third embodiment introduces the information interaction between different modules inside the main device, and the information interaction process between different modules in the main device.
  • FIG. 16 is a functional block diagram of the master device provided by the embodiment of the present application.
  • the main device includes display manager (display manager), display management service (display manager service) module, VD manager (VD manager), interface manager (surface manager) four modules. These four modules can be located in the Framework layer of the main device, and the Framework layer is located in the lower layer of the application layer.
  • the display manager is responsible for the connection between the upper layer (such as the application layer) and the ROM logic layer. It can be understood as a bridge between the upper layer and the ROM logic layer, and can do some logical processing for the upper layer data.
  • the display manager can provide an upper-layer interface, and the upper-layer interface can be invoked by applications in the application layer to implement dual VD mapping.
  • the display management service module is responsible for the mapping logic processing of dual VDs (for example, the establishment of the mapping relationship between the source VD and the target VD), and can also be responsible for the judgment of the DF of the VD, etc.
  • the display management service can be located in the Framework, and the solution for dual VD mapping is implemented in the Framework.
  • the VD manager is responsible for the management of the VD, such as the creation and deletion of the VD, and the setting of the MirrorFlag of the VD, etc., and can also receive calls from the application layer to create a VD for application projection.
  • the interface manager (surface manager) is responsible for presenting pictures on the surface, etc.
  • FIG. 17 is a schematic flowchart of a method for controlling the application of screen projection to a target device provided by an embodiment of the present application.
  • This figure can be understood as a schematic diagram of information interaction between different modules in the main device in FIG. 16 .
  • S1701 to S1704 in FIG. 17 introduce the application screen projection process, which can be understood as a refinement of S2 to S5 in FIG. 14 .
  • S1705 to S1715 introduce the process of controlling the display interface of the screen projected to the target device when the screen is projected by the application, which can be understood as the refinement process of S6 to S11 in Figure 14 .
  • the process includes:
  • the application layer After receiving the application screen projection instruction, the application layer sends a VD creation request to the VD manager.
  • the application layer can call the VD manager to create a VD.
  • the VD manager creates the first VD.
  • the display management service module moves the application to be mirrored (such as the first application) to the first VD to run.
  • the VD manager sets the first screen flag (DISPLAY_FLAG, DF) of the first VD.
  • the DF of a VD can be marked 1, for example, marked 1 can only display its own content (OWN_CONTENT_ONLY), the marked 1 means that only the content included in the VD is displayed, then, if the first application, then the interface of the first application is displayed (that is, the application screen projection), and if there is no application running in the VD, a black screen is displayed.
  • the DF of a VD may be tag 2, for example, tag 2 may be automatic mirroring (AUTO_MIRROR), and the tag 2 indicates that the display interface of the main screen can be mirrored by default when no application is running in the VD. Therefore, in S1703, the first DF of the first VD may be OWN_CONTENT_ONLY or AUTO_MIRROR.
  • the VD manager may use a VD flag setting (setVdFlag) method to set the first DF of the first VD.
  • setVdFlag VD flag setting
  • the master device completes the process of application projection (ie, application projection).
  • the control process of the application projected to the target device includes the following steps.
  • the application layer sends a dual VD instruction to the display manager.
  • the dual VD command includes the first identifier of the first VD and the second identifier of the second VD, and may also include other information, such as display parameters of the local display screen and the like.
  • the application layer can call the display manager through the setMirrorDisplay method to send a dual VD command to the display manager; or, it can also be understood that the system provides a setMirrorDisplay interface, and the application layer can call the setMirrorDisplay interface to send a dual VD command to the display manager .
  • the entry parameters of the setMirrorDisplay interface include (target Display Name, source Display Id), wherein, targetDisplayName refers to the name of the destination end of the dual-VD mapping, that is, the name of the second VD; source Display Id refers to the source end of the dual-VD mapping, namely The unique identifier of the first VD. Therefore, when the application layer calls the setMirrorDisplay interface, it needs to input the identifiers of the first VD and the second VD.
  • the display management service module acquires the first identifier of the first VD and the second identifier of the second VD.
  • the display management service module may acquire the first identifier of the first VD and the second identifier of the second VD in the dual VD mapping instruction.
  • the display manager may inject the target end (ie, the second ID of the second VD) into the display management service module through the setTartgetDisplayName method, and inject the source end (ie, the first ID of the first VD) ID into the display management service module through setSourceDisplayId module.
  • the first identifier of the first VD and the second identifier of the second VD are injected into the display management service module, they can be used by other modules in the Framework layer.
  • the display management service module sends a VD creation request to the VD manager, where the VD creation request carries the second identifier of the second VD.
  • the VD manager creates a second VD.
  • the VD manager sets the second DF of the second VD, where the second DF is used to indicate that the second VD is used to control the screen-casting application on the target device; or, is used to indicate that the source of the second VD is a specific VD (that is, the first VD).
  • mark 3 can be called a custom mirror display (MIRROR_CUSTOM_DISPLAY), indicating that the second VD is used to control the display information projected to the target device ; Or, it is used to indicate that the source terminal of the second VD is a specific VD (that is, the first VD).
  • MIRROR_CUSTOM_DISPLAY custom mirror display
  • the VD manager can set the second DF through the setVdFlag method.
  • the second DF will take effect at the piling in the display management service module, which will be described later in detail.
  • the display management service module sends a surface creation request to the interface manager, for requesting creation of a surface for bearing the second VD.
  • the interface manager creates a surface corresponding to the second VD.
  • the interface manager may create an independent drawing surface (surface) for carrying the display information of the second VD through the setVdSurface method, that is, draw the display information in the second VD on the surface for display.
  • the second VD can be created first and then the surface corresponding to the second VD can be created; or, the surface can be created first and then the surface corresponding to the surface can be created. Second VD.
  • the display management service module judges according to the second DF that it is necessary to control the application projected to the target device; or, it needs to map the display information in a specific VD to the second VD.
  • the display management service module can use the configure Display Locked method to perform piling, and insert another DF different from the above two DFs (mark 1 and mark 2), that is, mark 3, namely MIRROR_CUSTOM_DISPLAY.
  • mark 1 and mark 2 that is, mark 3
  • the display management service module will identify the piling point, and when it recognizes that the second DF is MIRROR_CUSTOM_DISPLAY, it is determined that it is necessary to control the application projected to the target device or to map the display information in a specific VD into the second VD.
  • the display management service module may run the isMirrorFlag method to judge the piling point.
  • the display management service module may insert the first identifier of the first VD at the piling point, indicating that the piling point is associated with the first VD, that is, the second VD is related to the first VD.
  • a VD establishes a mirror mapping relationship.
  • the display management service module may use the getMirrorDisplay method to establish a mirror mapping relationship at the staking point based on the first identifier of the first VD and the second identifier of the second VD.
  • the second VD is created in S1708, if no application is running in the second VD, the current display information on the main screen will be mapped by default.
  • the second DF of the second VD is set as MIRROR_CUSTOM_DISPLAY, and the mirror mapping relationship between the first VD and the second VD is established, the display information originally to be mapped to the second VD (that is, the main screen The current display information) is replaced by the display information in the first VD to realize the mapping from the first VD to the second VD.
  • S1714 can be completed jointly by the display management service module and the VD manager.
  • the display management service module sends an instruction to the VD manager to instruct the display information in the first VD to be mapped to the second VD, then the VD The manager only needs to execute the instruction.
  • the VD manager sends the display information in the second VD to the interface manager, so as to display it on the surface corresponding to the second VD created in the interface manager.
  • the display information in the second VD is displayed through the first window on the master device, so the master device controls the display information projected to the target device through the first window.
  • the display information in the second VD in the main device is displayed by other devices, then the display information projected to the target device can be controlled by other devices. It can be simply understood that the main device transfers the control of the display information projected to the target device to other devices, so that the main device can perform other operations without interruption, and control the main device to project the screen to the target device through other devices information on the display.
  • FIG. 18 is a schematic diagram of an application scenario adopted in the fourth embodiment.
  • the master device casts the screen of the first application to the target device 1 and the target device 2 , that is, the same screen projection interface is displayed on the target device 1 and the target device 2 .
  • the main device displays the interface of the second application, as shown in (b) in FIG. 18 .
  • the user can control the information displayed on the target device 2 from the main device on the target device 1 .
  • the screen projection interface on target device 1 is updated from interface A of the first application to interface B of the first application
  • the screen projection interface on target device 2 is also updated to interface B of the first application. Interface B.
  • the target device 1 and the target device 2 may be the same type of device or different types of devices, for example, the target device 1 is a portable device such as a mobile phone or a PC, and the target device 2 is a device such as a TV.
  • An exemplary scenario is that the main device is mobile phone A, the target device 1 is mobile phone B, and the target device 2 is a TV.
  • the first application is projected on mobile phone A to mobile phone B and the TV.
  • the user of mobile phone A can use mobile phone A to perform other operations, and the user of mobile phone B can use mobile phone B to control the information displayed on the TV screen projected by mobile phone A.
  • the main device creates the first VD and the second VD, moves the first application on the local display screen to run in the first VD, and the first VD projects the first application to the target device 2 for display.
  • the display information in the first VD is also mapped to the second VD, and the second VD projects the display information to the target device 1 for display. That is, the first VD corresponds to target device 2, and the second VD corresponds to target device 1.
  • the displayed information in the second VD is updated.
  • the second VD maps the updated display information to the first VD, and the first VD projects the updated display information to the target device 2 . That is, the target device 1 controls the display information projected by the master device to the target device 2 .
  • the second VD corresponds to the local display screen, so the display information projected to the target device can be controlled through the local display screen.
  • the second VD corresponds not to the local display screen, but to the target device 1, so the target device 1 can control the display information projected from the main device to the target device 2.
  • the main device modifies the original correspondence between the second VD and the local display screen to the correspondence between the second VD and the target device 1.
  • the main device does not need to display the
  • the display information means that the first window does not need to be displayed again, and the display information in the second VD is displayed on the target device 1 .
  • an embodiment of the present application provides a screen projection control method.
  • the method may be applicable to a communication system comprising a first electronic device and a second electronic device.
  • the first electronic device may be the above-mentioned master device
  • the second electronic device may be the above-mentioned target device.
  • the main device is the sender in the screen projection scenario
  • the target device is the receiver in the screen projection scenario.
  • the roles of the sending end and the receiving end can be interchanged, that is, the sending end can project a screen to the receiving end, and correspondingly, the receiving end can also project a screen to the sending end.
  • the first electronic device may be a device with a display screen, such as a mobile phone, a tablet computer, a PC, and a watch.
  • the target device can be a large-screen device such as a tablet computer, a PC, or a TV.
  • the process flow of the method includes:
  • the first electronic device displays a first interface of a first application.
  • the first electronic device detects an operation for opening the first application (for example, clicking the icon of the first application in the main interface operation), the first electronic device displays the first interface of the first application.
  • the first application may be any application, such as the photo album application in (a) in FIG. 4A , or the video playback application in (a) in FIG. 5A , and so on.
  • the first electronic device receives a screen projection operation.
  • the screen casting operation may include a sliding operation from the lower edge of the display screen and an operation of clicking a screen casting icon in the sliding menu (the sliding menu displayed in response to the sliding operation);
  • the sliding operation of the status bar at the top of the screen, and the operation of clicking the projection icon on the notification panel (notification panel displayed in response to the sliding operation), etc. can also be voice commands for instructing projection, which is not limited in this embodiment of the present application.
  • the first electronic device sends display information of the first interface to the second electronic device.
  • it may also include a process in which the first electronic device displays the device list and determines the second electronic device according to the user's selection operation in the device list, as shown in (c) in FIG. 7 .
  • the second electronic device displays the first interface of the first application.
  • the second electronic device displays the first interface of the first application.
  • the first electronic device displays the second interface and the first window of the second application, the first window includes a third interface, the third interface is the same as the first application projected to the second electronic device The current display interface of .
  • the first implementation manner is, taking FIG. 9 as an example, when the first electronic device launches the first application, the first electronic device displays the second interface of the second application, and the first window 901 automatically appears.
  • the second implementation is, taking FIG. 11 as an example, when the first electronic device launches the first application, the first electronic device displays the second interface of the second application and also displays the control button 1101 .
  • the first electronic device displays the first window 901 .
  • the display information in the first window 901 is the same as the display information projected to the second electronic device, and the user can control the display information projected to the second electronic device through the first window 901 . It should be noted that all the foregoing descriptions about the first window are applicable to this embodiment, and to save space, details are not repeated here.
  • the first electronic device receives a user operation.
  • the user operation refers to an operation for updating information displayed in the first window.
  • the user operation may be an operation of opening a certain image in the photo album application in the first window 401 .
  • the user operation may be an operation of opening the next TV episode in the video playing application in the first window 501 .
  • the first electronic device updates the third interface in the first window to a fourth interface.
  • interface A of the first application in the first window 901 is updated to interface B of the first application under the effect of user operations.
  • the first electronic device sends the display information of the fourth interface to the second electronic device.
  • the display interface of the first application on the second electronic device is updated to a fourth interface.
  • the first electronic device may also send the display information of the fourth interface to the third electronic device.
  • the third electronic device displays the fourth interface. That is to say, the first electronic device projects screens to the second electronic device and the third electronic device respectively.
  • the first electronic device may transfer the control right of the display information projected on the second electronic device to the third electronic device.
  • the master device casts the screen of the first application to the target device 1 and the target device 2, and transfers the control right of the display information projected to the target device 2 to the target device device1.
  • the first electronic device may cancel displaying the first window.
  • FIG. 21 is a schematic structural diagram of an electronic device 2100 provided by an embodiment of the present application.
  • the electronic device 2100 may be the above-mentioned master device or target device.
  • the electronic device 2100 may include: one or more processors 2101; one or more memories 2102; a communication interface 2103, and one or more computer programs 2104, and each of the above devices may communicate through one or more bus 2105 connection.
  • the one or more computer programs 2104 are stored in the memory 2102 and configured to be executed by the one or more processors 2101, the one or more computer programs 2104 include instructions.
  • the above instructions may be used to execute the relevant steps of the master device in the above corresponding embodiments.
  • the communication interface 2103 is used to realize the communication between the master device and other devices (such as the target device), for example, the communication interface may be a transceiver.
  • the embodiment of the present application also provides a communication system.
  • the communication system includes a first electronic device and a second electronic device.
  • the first electronic device may be the above-mentioned master device
  • the second electronic device may be the above-mentioned target device.
  • the first electronic device may be a device with a display screen, such as a mobile phone, a tablet computer, a PC, and a watch.
  • the second electronic device may be a large-screen device such as a tablet computer, a PC, or a television.
  • the structures of the first electronic device and the second electronic device can be referred to as shown in FIG. 21 . For example, when the electronic device 2100 shown in FIG.
  • the first electronic device when the instructions of one or more computer programs 2104 are executed by the processor, the first electronic device executes the first electronic device (That is, the steps of the master device).
  • the electronic device 2100 shown in FIG. 21 is a second electronic device, when the instructions of one or more computer programs 2104 are executed by the processor, the second electronic device executes the second electronic device ( i.e. the target device).
  • the methods provided in the embodiments of the present application are introduced from the perspective of an electronic device (such as a host device or a target device) as an execution subject.
  • the electronic device may include a hardware structure and/or a software module, and realize the above-mentioned functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above-mentioned functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.
  • the terms “when” or “after” may be interpreted to mean “if” or “after” or “in response to determining" or “in response to detecting ".
  • the phrases “in determining” or “if detected (a stated condition or event)” may be interpreted to mean “if determining" or “in response to determining" or “on detecting (a stated condition or event)” or “in response to detecting (a stated condition or event)”.
  • relational terms such as first and second are used to distinguish one entity from another, without limiting any actual relationship and order between these entities.
  • references to "one embodiment” or “some embodiments” or the like in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application.
  • appearances of the phrases “in one embodiment,” “in some embodiments,” “in other embodiments,” “in other embodiments,” etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean “one or more but not all embodiments” unless specifically stated otherwise.
  • the terms “including”, “comprising”, “having” and variations thereof mean “including but not limited to”, unless specifically stated otherwise.
  • all or part of them may be implemented by software, hardware, firmware or any combination thereof.
  • software When implemented using software, it may be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present invention will be generated in whole or in part.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.).
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media.
  • the available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, a Solid State Disk (SSD)).
  • a magnetic medium for example, a floppy disk, a hard disk, or a magnetic tape
  • an optical medium for example, DVD
  • a semiconductor medium for example, a Solid State Disk (SSD)

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

一种投屏控制方法与电子设备。第一电子设备显示第一应用的第一界面；响应于投屏操作，第一电子设备将第一界面的显示信息发送给第二电子设备；第二电子设备显示第一应用的第一界面；第一电子设备显示第二应用的第二界面，以及第一窗口，第一窗口中包括第三界面，第三界面与投屏到第二电子设备上的第一应用的当前显示界面相同；第一电子设备响应于用户操作，将第一窗口内第三界面更新为第四界面；第二电子设备上第一应用的显示界面更新为第四界面。通过这种方式，投屏场景中发送端可以便捷的控制投屏到接收端的显示信息。

Description

一种投屏控制方法与电子设备
相关申请的交叉引用
本申请要求在2021年06月28日提交中国专利局、申请号为202110717726.7、申请名称为“一种投屏控制方法与电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及电子技术领域，尤其涉及一种投屏控制方法与电子设备。
背景技术
投屏技术是一种时下正在兴起的技术，是指电子设备可以将其显示屏上的显示界面投屏到另一个电子设备上进行显示。比如，手机上的电影可以投屏在电视机上播放，平板电脑上的图片可以投屏在电视机上展示等等。
以手机向电视机投屏为例，一般来讲，手机向电视机投屏之后，用户可以在手机上进行其他操作，这种投屏方式被称为应用投屏。比如，手机将视频播放器的电视剧投屏到电视机上播放，用户可以在手机上使用即时通信应用聊天，当用户需要使用手机控制投屏到电视机上的内容(比如播放下一集)时，手机需要从应用投屏切换到镜像投屏模式(即手机与电视机显示相同界面)才能控制。这种需要应用投屏与镜像投屏之间不断切换的方式，操作繁琐，用户体验较低。
发明内容
本申请的目的在于提供了一种投屏控制方法与电子设备，用于提升投屏场景下对目标设备的控制的便捷性。
第一方面，提供一种投屏控制方法。该方法可以应用于包括第一电子设备和第二电子设备的系统。所述方法包括：所述第一电子设备显示第一应用的第一界面；响应于投屏操作，所述第一电子设备将所述第一界面的显示信息发送给所述第二电子设备；所述第二电子设备显示所述第一应用的所述第一界面；所述第一电子设备显示第二应用的第二界面，以及第一窗口，所述第一窗口中包括第三界面，所述第三界面与投屏到所述第二电子设备上的所述第一应用的当前显示界面相同；响应于用户操作，所述第一电子设备将所述第一窗口内所述第三界面更新为第四界面；所述第一电子设备将所述第四界面的显示信息发送给所述第二电子设备；所述第二电子设备上所述第一应用的显示界面更新为所述第四界面。
也就是说，第一电子设备向第二电子设备投屏(如应用投屏)时，第一电子设备的显示屏上显示第一窗口。第一窗口内的显示信息与投屏到第二电子设备上的显示信息相同。这样，用户可以通过第一窗口对投屏到第二电子设备上的显示信息进行控制，提升了投屏场景下对目标设备(即第二电子设备)上投屏信息进行控制的便捷性。
在一种可能的设计中，所述第一窗口悬浮显示于所述第二界面上层，所述第一窗口不完全覆盖所述第二界面；或者，所述第二界面与所述第一窗口分屏显示。也就是说，第一 电子设备投出第一应用之后，第一电子设备显示第二应用的第二界面与第一窗口。第一窗口和第二界面的显示方法可以有多种，例如分屏显示，或第一窗口位于上层，等等，本申请实施例不作限定。
在一种可能的设计中，所述第一电子设备显示第一窗口，包括：接收到用于唤出所述第一窗口的操作；响应于所述用于唤出所述第一窗口的操作，显示所述第一窗口。也就是说，第一电子设备投出第一应用之后，显示第二应用的第二界面，此时第一窗口可以不显示。当检测到用户触发所述操作后，唤出第一窗口。这样，用户可以在需要的时候(比如需要对投屏到目标设备上显示信息进行控制的时候)唤出第一窗口，用户体验较好。
在一种可能的设计中，响应于所述用于唤出所述第一窗口的操作，显示所述第一窗口，包括：所述第二界面上显示第一标记；响应于针对所述第一标记的操作，唤出所述第一窗口。示例性的，第一标记可以是按键、图标、文字等任何形式。也就是说，第一电子设备投出第一应用之后，第一电子设备显示第二应用的第二界面以及第一标记，该第一标记用于唤出第一窗口。第一标记的显示面积比第一窗口的显示面积小，不会太过遮挡第二界面，而且第一标记的显示可以对用户唤出第一窗口起到一定的操作指导作用，用户体验较好。
在一种可能的设计中，所述方法还包括：响应于用于隐藏所述第一窗口的操作，隐藏所述第一窗口；或者，响应于用于调整所述第一窗口的显示位置和/或显示面积的操作，调整所述第一窗口的显示位置和/或显示面积。也就是说，第一电子设备投出第一应用之后，第一电子设备显示第二应用的第二界面和第一窗口。如果用户不想要显示第一窗口(比如第一窗口遮挡住了第二界面)，可以隐藏第一窗口，或者调整第一窗口的面积、位置等，用户体验较好。
在一种可能的设计中，所述第一电子设备显示第一窗口，包括：所述第一电子设备创建绘制表面，所述绘制表面上的显示信息通过所述第一电子设备的显示屏显示；所述第一电子设备将第一虚拟显示屏VD中的显示信息映射到所述绘制表面上；其中，所述第一VD中运行所述第一应用，并且所述第一VD用于将所述第一VD内的显示信息发送给所述第二电子设备。
也就是说，第一电子设备要投出第一应用时，将第一应用转移到第一VD中运行，以使第二电子设备显示第一应用的第一界面。第一VD的显示信息还可以映射到与本地显示屏对应的绘制表面(即第一窗口)，以使本地显示屏上第一窗口中显示投屏到第二电子设备上的显示信息。简单来说，第一VD中运行的第一应用既发送给第二电子设备进行显示，也发送给本地显示屏(即第一窗口)进行显示，实现本地显示屏与第二电子设备上投屏信息的同步，以方便用户通过本地显示屏(即第一窗口)对投屏到的第二电子设备上的显示信息进行控制。
在一种可能的设计中，所述第一电子设备将第一虚拟显示屏VD中的显示信息映射到所述绘制表面上，包括：所述第一电子设备创建第二VD，所述第二VD与所述绘制表面对应；所述第一电子设备将所述第一VD内的显示信息映射到所述第二VD中，所述第二VD中的显示信息通过所述绘制表面显示。
需要说明的是，一般情况下，第一电子设备中会自带映射(如Mirror映射)功能，即Mirror映射功能可直接使用，不需要对底层代码进行修改。第一VD向第二VD映射可以使用系统自带的Mirror映射，那么就避免了对底层代码的修改。而且，第二VD向手机本地显示的映射为surface映射，也不需要经过底层处理。因此，这种方式实现难度较低，处 理效率较高。
在一种可能的设计中，响应于用户操作，所述第一电子设备将所述第一窗口内所述第三界面更新为第四界面，包括：响应于用户操作，所述绘制界面上显示信息更新为所述第四界面；其中，所述绘制界面上所述第四界面的显示信息被映射到所述第二VD中，所述第二VD用于将所述第四界面的显示信息映射到所述第一VD，所述第一VD用于将所述第四界面的显示信息发送给所述第二电子设备。这样的话，第一电子设备通过第一窗口控制第二电子设备上的投屏信息过程中，不需要对底层代码进行修改，实现难度低，效率高。
在一种可能的设计中，所述方法还包括：所述第一电子设备将所述第四界面的显示信息发送给所述第三电子设备；所述第三电子设备显示所述第四界面；所述第一电子设备将对投屏到所述第二电子设备上的显示信息的控制权转移给所述第三电子设备；所述第三电子设备控制本机显示屏上所述第四界面更新为第五界面时，所述第二电子设备将本机显示屏上的所述第四界面更新为所述第五界面。
也就是说，第一电子设备可以向第二电子设备和第三电子设备投屏，并且将对第二电子设备的控制权转移给第三电子设备，释放第一电子设备的处理压力，第一电子设备可以无打扰的进行其他操作。
在一种可能的设计中，所述方法还包括：所述第一电子设备上取消显示所述第一窗口。因此，第一电子设备将对第二电子设备的控制权转移给第三电子设备后，第一电子设备上可以不显示第一窗口，避免遮挡显示屏。
在一种可能的设计中，所述第一电子设备将对投屏到所述第二电子设备上的显示信息的控制权转移给第三电子设备，包括：所述第一电子设备创建第三VD；所述第三VD用于将所述第三VD内的显示信息发送给所述第三电子设备；所述第一电子设备将第一VD中的显示信息映射到所述第三VD上；其中，所述第一VD中运行所述第一应用，并且所述第一VD用于将所述第一VD内的显示信息发送给所述第二电子设备。在本申请实施例中，通过多个VD之间的信息映射实现控制权的转移，不需要对底层代码进行修改，这种方式实现难度较低，处理效率较高。
在一种可能的设计中，所述第三电子设备控制本机显示屏上所述第四界面更新为第五界面时，所述第二电子设备将本机显示屏上的所述第四界面更新为所述第五界面，包括：所述第三电子设备将所述第四界面更新为第五界面时，所述第三VD中的显示信息更新为所述第五界面的显示信息，所述第三VD用于将所述第五界面的显示信息映射到所述第一VD中，所述第一VD用于将所述第五界面的显示信息发送给所述第二电子设备。因此，第三电子设备接收对第二电子设备上投屏信息(第一电子设备向第二电子设备投屏的信息)的控制权后，通过多个VD之间的信息映射实现对第二电子设备的投屏信息进行控制，这个过程中不需要对底层代码进行修改，实现难度较低，处理效率较高。
第二方面，提供一种投屏控制方法，包括：第一电子设备显示第一应用的第一界面；响应于投屏操作，所述第一电子设备将所述第一界面的显示信息发送给第二电子设备；
所述第一电子设备显示第二应用的第二界面，以及第一窗口，所述第一窗口中包括第三界面，所述第三界面与投屏到所述第二电子设备上的所述第一应用的当前显示界面相同；
所述第一电子设备响应于用户操作，将所述第一窗口内所述第三界面更新为第四界面；
所述第一电子设备将所述第四界面的显示信息发送给所述第二电子设备，以使所述第二电子设备上所述第一应用的显示界面更新为所述第四界面。
在一种可能的设计中，所述第一窗口悬浮显示于所述第二界面上层，所述第一窗口不完全覆盖所述第二界面；或者，所述第二界面与所述第一窗口分屏显示。
在一种可能的设计中，所述第一电子设备显示第一窗口，包括：
接收到用于唤出所述第一窗口的操作；
响应于所述用于唤出所述第一窗口的操作，显示所述第一窗口。
在一种可能的设计中，响应于所述用于唤出所述第一窗口的操作，显示所述第一窗口，包括：所述第二界面上显示第一标记；
响应于针对所述第一标记的操作，唤出所述第一窗口。
在一种可能的设计中，所述方法还包括：
响应于用于隐藏所述第一窗口的操作，隐藏所述第一窗口；或者，
响应于用于调整所述第一窗口的显示位置和/或显示面积的操作，调整所述第一窗口的显示位置和/或显示面积。
在一种可能的设计中，所述第一电子设备显示第一窗口，包括：
所述第一电子设备创建绘制表面，所述绘制表面上的显示信息通过所述第一电子设备的显示屏显示；
所述第一电子设备将第一虚拟显示屏VD中的显示信息映射到所述绘制表面上；
其中，所述第一VD中运行所述第一应用，并且所述第一VD用于将所述第一VD内的显示信息发送给所述第二电子设备。
在一种可能的设计中，所述第一电子设备将第一虚拟显示屏VD中的显示信息映射到所述绘制表面上，包括：
所述第一电子设备创建第二VD，所述第二VD与所述绘制表面对应；
所述第一电子设备将所述第一VD内的显示信息映射到所述第二VD中，所述第二VD中的显示信息通过所述绘制表面显示。
在一种可能的设计中，响应于用户操作，所述第一电子设备将所述第一窗口内所述第三界面更新为第四界面，包括：
响应于用户操作，所述绘制界面上显示信息更新为所述第四界面；其中，所述绘制界面上所述第四界面的显示信息被映射到所述第二VD中，所述第二VD用于将所述第四界面的显示信息映射到所述第一VD，所述第一VD用于将所述第四界面的显示信息发送给所述第二电子设备。
在一种可能的设计中，所述方法还包括：
所述第一电子设备将所述第四界面的显示信息发送给第三电子设备，以使所述第三电子设备显示所述第四界面；
所述第一电子设备将对投屏到所述第二电子设备上的显示信息的控制权转移给第三电子设备。
在一种可能的设计中，所述方法还包括：所述第一电子设备上取消显示所述第一窗口。
在一种可能的设计中，所述第一电子设备将对投屏到所述第二电子设备上的显示信息的控制权转移给第三电子设备，包括：
所述第一电子设备创建第三VD；所述第三VD用于将所述第三VD内的显示信息发送给所述第三电子设备；
所述第一电子设备将第一VD中的显示信息映射到所述第三VD上；其中，所述第一 VD中运行所述第一应用，并且所述第一VD用于将所述第一VD内的显示信息发送给所述第二电子设备。
在一种可能的设计中，所述方法还包括：所述第三VD用于在所述第三电子设备上所述第四界面更新为第五界面时，将所述第三VD中的显示信息更新为所述第五界面的显示信息，并且将所述第五界面的显示信息映射到所述第一VD中，所述第一VD用于将所述第五界面的显示信息发送给所述第二电子设备。
第三方面，还提供一种通信系统，包括：第一电子设备和第二电子设备；
所述第一电子设备包括：处理器；存储器；其中，所述存储器存储有一个或多个计算机程序，所述一个或多个计算机程序包括指令，当所述指令被所述处理器执行时，使得所述第一电子设备执行如上述第一方面所述的方法中第一电子设备的步骤；
所述第二电子设备包括：处理器；存储器；其中，所述存储器存储有一个或多个计算机程序，所述一个或多个计算机程序包括指令，当所述指令被所述处理器执行时，使得所述第二电子设备执行如上述第二方面提供的方法中第二电子设备的步骤。
第四方面，还提供一种电子设备，包括：
处理器，存储器，以及，一个或多个程序；
其中，所述一个或多个程序被存储在所述存储器中，所述一个或多个程序包括指令，当所述指令被所述处理器执行时，使得所述电子设备执行如上述第二方面提供的方法步骤。
第五方面，还提供一种计算机可读存储介质，计算机可读存储介质用于存储计算机程序，当所述计算机程序在计算机上运行时，使得所述计算机执行如上述第一方面或第二方面提供的方法。
第六方面，还提供一种计算机程序产品，包括计算机程序，当所述计算机程序在计算机上运行时，使得所述计算机执行如上述第一方面或第二方面所述的方法。
第七方面，还提供一种电子设备上的图形用户界面，所述电子设备具有显示屏、存储器、以及处理器，所述处理器用于执行存储在所述存储器中的一个或多个计算机程序，所述图形用户界面包括所述电子设备执行上述第二方面所述的方法时显示的图形用户界面。
第八方面，本申请实施例还提供一种芯片，所述芯片与电子设备中的存储器耦合，用于调用存储器中存储的计算机程序并执行本申请实施例第一方面至第二方面的技术方案，本申请实施例中“耦合”是指两个部件彼此直接或间接地结合。
上述第二方面至第八方面的有益效果，参见第一方面的有益效果，不重复赘述。
附图说明
图1A至图1B为本申请一实施例提供的镜像投屏的示意图；
图2A至图2B为本申请一实施例提供的应用投屏的示意图；
图3为本申请一实施例提供的应用投屏向镜像投屏切换的示意图；
图4A至图4B为本申请实施例提供的一种应用场景的示意图；
图5A至图5B为本申请实施例提供的另一种应用场景的示意图；
图6为本申请实施例提供的电子设备的结构示意图；
图7为本申请实施例提供的主设备应用投屏的示意图；
图8为本申请实施例提供的主设备应用投屏的后台处理流程的示意图；
图9为本申请实施例提供的主设备显示第一窗口的第一种示意图；
图10至图11为本申请实施例提供的主设备显示第一窗口的第二种示意图；
图12至图13为本申请实施例提供的主设备通过第一窗口控制目标设备的示意图；
图14为本申请实施例提供的投屏控制方法的一种流程示意图；
图15为本申请实施例提供的投屏控制方法的另一种流程示意图；
图16为本申请实施例提供的主设备的软件结构的示意图；
图17为本申请实施例提供的投屏控制方法的又一种流程示意图；
图18至图19为本申请实施例提供的主设备向多个目标设备投屏的示意图；
图20为本申请实施例提供的投屏控制方法的流程示意图；
图21为本申请实施例提供的电子设备的结构示意图。
具体实施方式
以下，对本申请实施例中的部分用语进行解释说明，以便于本领域技术人员理解。
(1)应用程序(application，简称app)，简称应用，为能够实现某项或多项特定功能的软件程序。通常，电子设备中可以安装多个应用，例如，即时通讯类应用、视频类应用、音频类应用、图像拍摄类应用等等。其中，即时通信类应用，例如可以包括短信应用、畅连、
Figure PCTCN2022095705-appb-000001
WhatsApp
Figure PCTCN2022095705-appb-000002
照片分享(Instagram)、Kakao
Figure PCTCN2022095705-appb-000003
等。图像拍摄类应用，例如可以包括相机应用(系统相机或第三方相机应用)。视频类应用，例如可以包括
Figure PCTCN2022095705-appb-000004
Figure PCTCN2022095705-appb-000005
等等。音频类应用，例如可以包括Google Music、酷狗音乐、虾米、QQ音乐等等。以下实施例中提到的应用，可以是电子设备出厂时已安装的应用，也可以是用户在使用电子设备的过程中从网络下载或其他电子设备获取的应用。
(2)本申请实施例涉及的至少一个，包括一个或者多个；其中，多个是指大于或者等于两个。另外，需要理解的是，在本申请的描述中，“第一”、“第二”等词汇，仅用于区分描述的目的，而不能理解为明示或暗示相对重要性，也不能理解为明示或暗示顺序。
本申请实施例提供的技术方案适用于投屏场景。一般来说，投屏场景中包括发送端和接收端。发送端将显示信息发送给接收端来显示。为了方便描述，本文将发送端称为主设备，将接收端称为目标设备。其中，主设备可以是手机、平板电脑、PC、手机、手表等。目标设备可以是平板电脑、PC、电视机等大屏设备。可以理解的是，除了主设备和目标设备之外，发送端和接收端还可以有其它称呼，比如，发送端是主动投屏设备，接收端是被动投屏设备；或者，发送端是主设备，接收端是辅设备；或者，发送端是源设备，接收端是目标设备；或者，发送端是第一电子设备，接收端是第二电子设备等等，本申请对此不作限定。其中，发送端和接收端的角色可以互换，即，发送端可以向接收端投屏，对应的，所述接收端也可以向所述发送端投屏。
投屏技术包括镜像投屏和应用投屏。
镜像投屏，是指发送端的编码器将发送端显示屏上当前显示信息进行视频编码，将经过编码后的显示信息通过与接收端之间的数据通道发送给接收端，接收端通过解码器对接收到的显示信息进行解码，并创建一个Surface承载解码后的显示信息并展示。这样的话，发送端和接收端可展示相同的画面，换言之，发送端当前显示什么样的内容就向接收端投屏什么样的内容。示例性的，请参见图1A所示，为镜像投屏的过程的示意图。发送端(比如手机)将前台显示信息(比如第一应用的界面A)投屏到接收端(比如PC)，接收端显 示所述显示信息(如，第一应用的界面A)。当发送端前台显示信息发生变化，比如从界面A切换为界面B，对应的，接收端上的显示信息也变化，即从界面A切换为界面B。
镜像投屏的一种实现方式为，发送端的本地显示屏(local display)的当前显示信息直接经过视频编码后发送到发送端，由发送端进行解码并展示，这种方式不需要创建虚拟显示屏(virtual display，VD)。镜像投屏的另一种实现方式是通过创建VD来实现，比如，请参见图1B所示，手机创建第一VD，并将手机本地显示屏的当前显示信息向第一VD映射。其中，第一VD用户不可见，属于手机后台创建的虚拟显示器，而且第一VD中的显示信息被默认投屏到目标设备上。比如，第一VD将显示信息发送到编码器进行编码，经过编码之后的显示信息被发送给接收端(比如PC或电视机)，这样，手机将前台显示信息投屏到接收端显示。继续参见图1B，这种数据单向流动方式，数据源头是手机本地显示屏，本地显示屏上的显示信息实时的映射到第一VD，需要编码的display是第一VD，经过编码后的数据发送给接收端。因此，当手机本地显示屏上的显示信息发生变化时，对应的，接收端上的显示信息也相应的发生变化。
应用投屏与镜像投屏不同。简单来说，应用投屏场景中，接收端上由发送端投屏来的显示信息可以与发送端前台正在显示的显示信息不同。示例性的，请参见图2A所示，为应用投屏过程的示意图。主设备上显示第一应用的界面A，此时，若主设备检测到应用投屏操作，主设备将第一应用投屏到目标设备上显示，主设备前台显示第二应用的界面。可以理解为，主设备将第一应用整体投屏到目标设备，简单来说，投屏粒度是应用。
应用投屏的实现原理请参见图2B。如图2B中的(a)，手机本地显示屏正在显示第一应用的界面A，此时，如果手机检测到应用投屏操作，手机将第一应用转移到向第一VD上运行(比如，将第一应用从本地显示屏中的任务栈1中挪栈到第一VD中的任务栈2中)。如图2B中的(b)，第一应用被挪到第一VD后，本地显示屏上显示第二应用的界面。第一VD将显示信息发送到编码器进行编码，经过编码之后的显示信息被发送给接收端。由于第一应用被转移到第一VD中运行，所以当第一VD中第一应用的显示界面发生变化时，对应的，接收端上第一应用的显示界面也发生变化，而本地显示屏上的显示界面与第一VD以及接收端上的投屏来的显示界面可以不同。可以简单的理解为，向目标设备投屏的数据源头不再是手机本地显示屏，而是第一VD。
在一些实施例中，镜像投屏和应用投屏之间可以切换。比如，当前主设备与目标设备之间是镜像投屏，可以通过切换操作将镜像投屏切换为应用投屏。或者，当前主设备与目标设备之间是应用投屏，也可以通过切换操作将应用投屏切换为镜像投屏。
下面以应用投屏向镜像投屏切换为例进行介绍。
请参见图3所示，为应用投屏向镜像投屏切换的示意图。具体的，切换过程包括：参见图3中的(a)，将第一VD中的第一应用(即被投屏出去的应用)回退到本地显示屏中。所述回退操作可以理解为挪栈操作(比如将第一VD中任务栈2中的第一应用挪动到本地显示屏中的任务栈1中)。第一应用从第一VD回退到本地显示屏之后，第一VD中第一应用被释放，即第一VD中没有第一应用，此时，应用投屏终止。
请参见图3中的(b)，第一应用回退到本地显示屏之后，本地显示屏使用镜像投屏的方式进行投屏。即，本地显示屏将第一应用映射到第一VD中，第一VD将显示信息发送给目标设备。也就是说，图3中的(b)与前面的图1B所示的镜像投屏的过程相同。到此，完成应用投屏向镜像投屏的切换。
需要说明的是，应用投屏情况下，由于手机本地显示屏上不显示被投屏出去的应用(即第一应用)，此时，如果用户想要使用手机控制被投屏出去的第一应用，需要手机本地显示屏上显示第一应用的界面。一种实现方式为，从应用投屏切换到镜像投屏，然后，用户在本地显示屏上对第一应用进行控制，进而实现对目标设备上投屏的第一应用的控制。比如，参见图3中的(c)，用户在本地显示屏上对第一应用进行控制使得第一应用的界面A切换为界面B，同步到第一VD中，然后第一VD同步给目标设备，则目标设备上由第一应用的界面A更新为第一应用的界面B。
这种对投屏到目标设备上应用的控制方式，需要从应用投屏切换到镜像投屏，比较繁琐，耗时大，用户体验较低。
本申请实施例提供一种对投屏到目标设备上显示信息的控制方法，主设备向目标设备应用投屏时，主设备上可以显示第一窗口，第一窗口中显示被投屏到目标设备上的应用的当前显示界面；也就是说，第一窗口内的显示信息与被投屏到目标设备上的应用的当前显示信息是同步的，用户可以在第一窗口内对投屏到目标设备上的显示信息进行控制。
图4A至图4B为本申请实施例提供的一种应用场景的示意图。如图4A中的(a)，手机当前正在显示相册应用的首页。当手机检测到应用投屏操作时，将相册应用投屏到目标设备，目标设备上显示相册应用的显示界面，如图4A中的(b)。如图4A中的(c)，在投屏出相册应用之后，手机显示即时通信应用的显示界面(比如与联系人的聊天界面)，而目标设备上仍然显示相册应用的界面，如图4A中的(d)。
在一些实施例中，在显示图4A中的(c)所示的界面过程中，若手机检测到用于控制投屏到目标设备上的显示信息的操作(比如检测到用户点击图4A中的(c)界面中的标记400，或者接收到用于指示控制投屏到目标设备上的显示信息的语音指令)时，手机上显示第一窗口401，如图4B中的(a)。这种方式，用户在需要的时候可以通过触发标记400或语音指令唤出第一窗口401。在另一些实施例中，手机投出相册应用之后，第一窗口401可以自动出现。比如，图4A中的(a)中手机将相册应用投屏到目标设备后，手机不显示图4A中的(c)所示的界面，直接显示图4B中的(a)所示的界面。
第一窗口401可以悬浮显示，可以完全遮挡或部分遮挡即时通信应用的界面。第一窗口401中显示被投屏到目标设备上的相册应用的当前显示界面(即相册应用首页)。也就是说，第一窗口401与目标设备上投屏应用的当前显示界面相同。这样的话，用户可以通过第一窗口401对投屏到目标设备上的相册应用进行控制，比如，请参见图4B中的(c)，用户在第一窗口401中打开了相册应用中的某张图像，那么对应的，目标设备上打开相册应用的这张图像，如图4B中的(d)。当然，用户还可以通过第一窗口401对相册应用进行其他操作，比如滑动到下一张图像，放大缩小图像、打开相册首页等等。
图5A至图5B为本申请实施例提供的另一种应用场景的示意图。如图5A中的(a)，手机正在显示视频播放应用的界面(比如，某个电视剧的播放界面)。当手机检测到应用投屏操作时，将视频播放应用投屏到目标设备，如图5A中的(b)，目标设备上显示视频播放应用的界面。投出视频播放应用后，手机前台显示主页面，如图5A中的(c)，而目标设备上仍然显示视频播放应用的界面，如图5A中的(d)。
在一些实施例中，在显示图5A中的(c)所示的界面过程中，若手机检测到用于控制投屏到目标设备上的显示信息的操作(比如检测到用户点击图5A中的(c)界面中的标记500，或者接收到用于指示控制投屏到目标设备上的显示信息的语音指令)，手机上显示第 一窗口501，如图5B中的(a)。在另一些实施例中，手机投出视频播放应用之后，第一窗口501可以自动出现。比如，图5A中的(a)中手机将视频播放应用投屏到目标设备后，手机不显示图5A中的(c)所示的界面，直接显示图5B中的(a)所示的界面。
第一窗口501可以悬浮显示，可以完全遮挡或部分遮挡主界面。第一窗口501中显示被投屏到目标设备上的视频播放应用的当前显示界面(即，所述电视剧的播放界面)。也就是说，第一窗口501与目标设备上投屏应用的当前显示界面相同。这样的话，用户可以通过第一窗口501对投屏到目标设备上的视频播放应用进行控制，请参见图5B中的(c)，用户在第一窗口501中打开了电视剧的下一集，对应的，目标设备上打开所述电视剧的下一集，如图5B中的(d)。当然，用户还可以通过第一窗口501对视频播放应用进行其他操作，比如加速播放、减速播放、选择其它电视机等等。
以上通过示例介绍本申请实施例的两种应用场景，需要说明的是，对于其他应用场景本申请也是适用的。
下面介绍主设备。其中，主设备可以是便携式电子设备，诸如手机、平板电脑、便携计算机、具备无线通讯功能的可穿戴设备(如智能手表、智能眼镜、智能手环、或智能头盔等)、或车载设备等。便携式电子设备的示例性实施例包括但不限于搭载
Figure PCTCN2022095705-appb-000006
Figure PCTCN2022095705-appb-000007
或者其它操作系统的便携式电子设备。目标设备可以是平板电脑、PC、电视机等大屏设备。
图6示出了电子设备的结构示意图。图6的结构可以是主设备的结构或目标设备的结构。如图6所示，电子设备可以包括处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。其中传感器模块180可以包括压力传感器180A，陀螺仪传感器180B，气压传感器180C，磁传感器180D，加速度传感器180E，距离传感器180F，接近光传感器180G，指纹传感器180H，温度传感器180J，触摸传感器180K，环境光传感器180L，骨传导传感器180M等。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。其中，控制器可以是电子设备的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。
USB接口130是符合USB标准规范的接口，具体可以是Mini USB接口，Micro USB接口，USB Type C接口等。USB接口130可以用于连接充电器为电子设备充电，也可以 用于电子设备与外围设备之间传输数据。充电管理模块140用于从充电器接收充电输入。电源管理模块141用于连接电池142，充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入，为处理器110，内部存储器121，外部存储器，显示屏194，摄像头193，和无线通信模块160等供电。
电子设备的无线通信功能可以通过天线1，天线2，移动通信模块150，无线通信模块160，调制解调处理器以及基带处理器等实现。天线1和天线2用于发射和接收电磁波信号。电子设备中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。
移动通信模块150可以提供应用在电子设备上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块150可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块150还可以对经调制解调处理器调制后的信号放大，经天线1转为电磁波辐射出去。在一些实施例中，移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。
无线通信模块160可以提供应用在电子设备上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块160可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块160经由天线2接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。
在一些实施例中，电子设备的天线1和移动通信模块150耦合，天线2和无线通信模块160耦合，使得电子设备可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。所述GNSS可以包括全球卫星定位系统(global positioning system，GPS)，全球导航卫星系统(global navigation satellite system，GLONASS)，北斗卫星导航系统(beidou navigation satellite system，BDS)，准天顶卫星系统(quasi-zenith satellite system，QZSS)和/或星基增强系统(satellite based augmentation systems，SBAS)。
显示屏194用于显示应用的显示界面等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled， MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，电子设备可以包括1个或N个显示屏194，N为大于1的正整数。
电子设备100可以通过ISP，摄像头193，视频编解码器，GPU，显示屏194以及应用处理器等实现拍摄功能。
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头193中。
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，电子设备可以包括1个或N个摄像头193，N为大于1的正整数。
内部存储器121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器110通过运行存储在内部存储器121的指令，从而执行电子设备的各种功能应用以及数据处理。内部存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，以及至少一个应用程序(例如爱奇艺应用，微信应用等)的软件代码等。存储数据区可存储电子设备使用过程中所产生的数据(例如图像、视频等)等。此外，内部存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。
外部存储器接口120可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备的存储能力。外部存储卡通过外部存储器接口120与处理器110通信，实现数据存储功能。例如将图片，视频等文件保存在外部存储卡中。
电子设备可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音等。
压力传感器180A用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器180A可以设置于显示屏194。陀螺仪传感器180B可以用于确定电子设备的运动姿态。在一些实施例中，可以通过陀螺仪传感器180B确定电子设备围绕三个轴(即，x，y和z轴)的角速度。
陀螺仪传感器180B可以用于拍摄防抖。气压传感器180C用于测量气压。在一些实施例中，电子设备通过气压传感器180C测得的气压值计算海拔高度，辅助定位和导航。磁传感器180D包括霍尔传感器。电子设备可以利用磁传感器180D检测翻盖皮套的开合。在一些实施例中，当电子设备是翻盖机时，电子设备可以根据磁传感器180D检测翻盖的开合。进而根据检测到的皮套的开合状态或翻盖的开合状态，设置翻盖自动解锁等特性。加速度传感器180E可检测电子设备在各个方向上(一般为三轴)加速度的大小。当电子设备静止时可检测出重力的大小及方向。还可以用于识别电子设备姿态，应用于横竖屏切换，计步器等应用。
距离传感器180F，用于测量距离。电子设备可以通过红外或激光测量距离。在一些实 施例中，拍摄场景，电子设备可以利用距离传感器180F测距以实现快速对焦。接近光传感器180G可以包括例如发光二极管(LED)和光检测器，例如光电二极管。发光二极管可以是红外发光二极管。电子设备通过发光二极管向外发射红外光。电子设备使用光电二极管检测来自附近物体的红外反射光。当检测到充分的反射光时，可以确定电子设备附近有物体。当检测到不充分的反射光时，电子设备可以确定电子设备附近没有物体。电子设备可以利用接近光传感器180G检测用户手持电子设备贴近耳朵通话，以便自动熄灭屏幕达到省电的目的。接近光传感器180G也可用于皮套模式，口袋模式自动解锁与锁屏。
环境光传感器180L用于感知环境光亮度。电子设备可以根据感知的环境光亮度自适应调节显示屏194亮度。环境光传感器180L也可用于拍照时自动调节白平衡。环境光传感器180L还可以与接近光传感器180G配合，检测电子设备是否在口袋里，以防误触。指纹传感器180H用于采集指纹。电子设备可以利用采集的指纹特性实现指纹解锁，访问应用锁，指纹拍照，指纹接听来电等。
温度传感器180J用于检测温度。在一些实施例中，电子设备利用温度传感器180J检测的温度，执行温度处理策略。例如，当温度传感器180J上报的温度超过阈值，电子设备执行降低位于温度传感器180J附近的处理器的性能，以便降低功耗实施热保护。在另一些实施例中，当温度低于另一阈值时，电子设备对电池142加热，以避免低温导致电子设备异常关机。在其他一些实施例中，当温度低于又一阈值时，电子设备对电池142的输出电压执行升压，以避免低温导致的异常关机。
触摸传感器180K，也称“触控面板”。触摸传感器180K可以设置于显示屏194，由触摸传感器180K与显示屏194组成触摸屏，也称“触控屏”。触摸传感器180K用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器180K也可以设置于电子设备的表面，与显示屏194所处的位置不同。
骨传导传感器180M可以获取振动信号。在一些实施例中，骨传导传感器180M可以获取人体声部振动骨块的振动信号。骨传导传感器180M也可以接触人体脉搏，接收血压跳动信号。
按键190包括开机键，音量键等。按键190可以是机械按键。也可以是触摸式按键。电子设备可以接收按键输入，产生与电子设备的用户设置以及功能控制有关的键信号输入。马达191可以产生振动提示。马达191可以用于来电振动提示，也可以用于触摸振动反馈。例如，作用于不同应用(例如拍照，音频播放等)的触摸操作，可以对应不同的振动反馈效果。指示器192可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。SIM卡接口195用于连接SIM卡。SIM卡可以通过插入SIM卡接口195，或从SIM卡接口195拔出，实现和电子设备的接触和分离。
可以理解的是，图6所示的部件并不构成对电子设备的具体限定。本发明实施例中的电子设备可以包括比图6中更多或更少的部件。此外，图6中的部件之间的组合/连接关系也是可以调整修改的。
下面结合附图介绍本申请实施例提供的技术方案。
为了方便描述，本文以下的实施例以主设备是手机，目标设备是电视机为例进行介绍。
实施例一
图7中的(a)示出了手机的一种图形用户界面(graphical user interface，GUI)，该 GUI为手机的桌面。桌面中包括不同应用的图标。当手机检测到第一应用的图标的操作时，显示如图7中的(b)所示的第一应用的界面。第一应用可以是任意应用，比如视频播放应用、浏览器、即时通讯应用、相册应用等等。当手机检测到投屏操作时，显示目标设备列表，如图7中的(c)，目标设备列表中包括电视机、PC等目标设备的标识，用户可以在目标设备列表中选择要投屏的目标设备。其中，投屏操作可以有多种类型，比如，在下滑通知栏界面(从显示屏顶部向下滑动调出的界面)或者上拉控制中心界面(从显示屏底部向上滑动调出的界面)中显示投屏按键，投屏操作是指对所述投屏按键的点击操作；或者，投屏操作还可以是用于指示投屏的语音指令等，本申请实施例对投屏操作的类型不作限定。如图7中的(c)，目标设备列表中包括电视机、PC等目标设备的标识。假设手机检测到用户在目标设备列表中选择电视机标识的操作，响应于该操作，将前台应用(即第一应用)投屏到电视机上，如图7中的(d)，电视机上显示第一应用的界面。
手机向电视机投屏第一应用之后，手机前台显示第二应用的界面，如图7中的(e)。其中，第二应用可以是在打开第一应用之前上一个打开过的应用，或者，是手机桌面，或者，是预先设置好的应用(系统默认设置的或用户设置的)的界面。
以上是应用投屏的过程，手机将第一应用投出后前台显示第二应用的界面，该过程对应的手机后台处理流程请参见图8。如图8，手机创建第一VD(可选的，第一VD的显示参数与目标设备适配，比如分辨率相同)，手机将本地显示屏上的第一应用转移到第一VD中运行。第一VD将第一应用投屏到目标设备。第一应用从本地显示屏转移到第一VD之后，本地显示屏上显示第二应用的界面。需要说明的是，图8中第一应用的界面和第二应用的界面的面积小于本地显示屏的面积，是为了表达本地显示屏可以显示第一应用的界面或第二应用的界面的含义，并不是对应用界面与本地显示屏之间的面积大小关系的限定。
在应用投屏时，用户可以在手机上进行其他操作，当然用户还可以在手机上控制投屏到目标设备上的第一应用。
第一种方式
手机将第一应用投屏到目标设备时，手机显示第二应用的界面，还显示第一窗口，第一窗口中显示被投屏到目标设备上第一应用的当前显示界面，换言之，第一窗口中显示界面与目标设备上第一应用的当前显示界面相同。用户可以通过第一窗口控制投屏到目标设备上的第一应用。
示例性的，请参见图9中的(a)，当手机检测到用于选择目标设备列表中的电视机标识的操作时，向电视机投屏第一应用。如图9中的(b)，电视机上显示第一应用的界面。如图9中的(c)，在投出第一应用之后，手机上显示第二应用的界面，还显示第一窗口901。第一窗口901中的显示界面与投屏到目标设备上第一应用的当前显示界面相同。
也就是说，在手机使用应用投屏投出第一应用之后，手机显示屏显示第二应用的界面，且自动出现第一窗口901，以方便用户通过第一窗口901控制投屏到目标设备上的第一应用。
其中，第一窗口901可以有多种显示方式，比如，如图9中的(c)，第一窗口901悬浮显示于第二应用的界面上，可以完全覆盖或部分覆盖第二应用的界面。第一窗口901的形状本申请实施例不作限定。
第一窗口901的显示位置和/或显示面积可以是默认设置好的，或者，用户可调整的。比如，第一窗口901中可以包括缩放按键902，缩放按键902用于放大或缩小第一窗口。 在放大第一窗口时，第一窗口可以全屏显示，在缩小第一窗口时，可以缩小到占用最小面积(比如，系统设置的最小面积)。
第一窗口901还可以隐藏，比如，当手机检测到用于隐藏第一窗口的隐藏指令时，隐藏第一窗口901。示例性的，参见图9中的(c)，第一窗口901中包括关闭按键903，当检测到针对关闭按键903的操作时，隐藏第一窗口901。或者，当手机检测到用于指示隐藏第一窗口的语音指令时，隐藏第一窗口901。或者，当手机检测到用户点击第二界面上未被第一窗口901覆盖的区域时，隐藏第一窗口901。
第一窗口901隐藏之后，还可以唤出第一窗口901。比如，在第一窗口隐藏之后，通过唤出指令可以唤出第一窗口901。示例性的，当手机检测到针对第一窗口901中关闭按键903的操作时，第一窗口901隐藏，同时手机显示屏上出现图标1001，请参见图10中的(a)，当检测到针对该图标1001的操作时，唤出第一窗口901。当然，图标1001还可以显示在其它位置处，比如，显示在图10中的(b)的状态栏中，或者，显示在图10中的(c)中的下滑通知栏界面中，或者，显示在图10中的(d)中上拉控制中心界面中。或者，当检测到用于唤出第一窗口的语音指令时，唤出第一窗口。
因此，上面的第一种方式，手机在投出第一窗口的同时显示屏上自动出现第一窗口901，方便用户操作，体验较高。
第二种方式
手机将第一应用投屏到目标设备时，手机显示第二应用的界面，并且显示控制按键，当检测到针对该控制按键的操作时，显示第一窗口。
示例性的，请参见图11中的(a)，当手机检测到用于选择目标设备列表中电视机标识的操作时，向电视机投屏第一应用。如图11中的(b)，电视机上显示第一应用的界面。如图11中的(c)，在投出第一应用之后，手机上显示第二应用的界面，还显示控制按键1101。当检测到针对控制按键1101的操作时，手机显示第一窗口901，如图11中的(d)。
其中，控制按键1101可以显示在任意位置，比如图11中的(c)中控制按键1101悬浮显示在第二应用的界面上，或者，还可以显示在状态栏中，或者，显示在下滑通知栏界面中，或者，显示在上拉控制中心界面中，等等，本申请实施例不作限定。
第二种方式中第一窗口901的显示位置和/或显示面积可以是默认设置好的或者用户可调整的。而且，第一窗口也可以隐藏，且在隐藏之后，还可以唤出第一窗口，实现原理与前面第一种方式相同，不重复赘述。
因此，第二种方式(图11所示)与前面的第一种方式(图9所示)的区别在于：第一种方式中手机将第一应用投屏到目标设备的同时手机显示屏上自动出现第一窗口901；第二种方式中手机将第一应用投屏到目标设备的同时手机显示屏上自动出现控制按键1101，当检测到针对控制按键1101的操作时，出现第一窗口901。
可选的，手机可以默认(比如系统默认)使用第一种方式或第二种方式，或者，用户可以设置使用第一种方式或第二种方式，本申请实施例不作限定。
用户可以通过第一窗口901控制投屏到目标设备上的应用。
示例性的，请参见图12中的(a)和图12中的(b)，手机上第一窗口901中显示的界面与目标设备上投屏来的第一应用的显示界面相同，都是第一应用的界面A。如图12中的(c)和图12中的(d)，当用户在对第一窗口901内对第一应用进行控制导致第一应用更新为界面B时，对应的，目标设备上投屏来的第一应用也更新界面B。假设投屏到目标设 备的第一应用是相册应用，用户通过第一窗口控制投屏到目标设备上的相册应用的过程可以参见前文图4B。假设投屏到目标设备的第一应用是视频播放应用，用户通过第一窗口控制投屏到目标设备上的视频播放应用的过程请参见前文的图5B。
图13为手机显示第一窗口901的后台处理流程。如图13，手机向目标设备应用投屏，所以已创建第一VD，而且手机还创建第二VD(可选的，第二VD的显示参数可以与手机本地显示屏的显示参数适配，比如，分辨率相同)。第一VD中第一应用的显示信息可以映射(比如镜像(Mirror)映射)到第二VD中。手机还创建一个绘制表面(Surface)用于承载第二VD中的显示信息，换言之，第二VD中的显示信息通过所述Surface显示，简单来说，第二VD通过surface映射到本地显示屏(即映射到第一窗口进行显示)。
图13第一VD分别向第二VD和目标设备发送第一应用的显示信息，直观上看，第一VD作为数据源分别向左右两个方向提供第一应用的显示信息。这样的话，第一应用始终运行在第一VD中，即应用投屏一直在进行，没有终止，也就无需重新建立镜像投屏(即不需要将应用投屏切换为镜像投屏)。换句话说，第二VD以及第一窗口(即第二VD对应的surface)的建立并不影响应用投屏的进行。
图13中，当本地显示屏上第一窗口(即第二VD对应的surface)内第一应用的界面更新(比如，当用户在第一窗口中对第一应用进行控制导致第一应用的显示界面更新)，第二VD中第一应用的界面对应更新，相应的，第一VD中第一应用的界面更新，目标设备上的第一应用的界面也会更新。
需要说明的是，前面图3中介绍过目前在应用投屏时如果需要控制投屏到目标设备上的应用，需要将应用投屏切换为镜像投屏。下面结合图3与图13阐述目前的方案与本申请实施例提供的技术方案的区别。
图3中应用投屏切换为镜像投屏的流程包括：将第一VD中的第一应用回退到本地显示屏中，回退之后，第一VD中不再包括第一应用，应用投屏终止。然后，建立镜像投屏，将本地显示屏的第一应用镜像映射到第一VD中，第一VD中的显示信息同步到目标设备上显示。简单来说，需要先终止应用投屏，并重新建立镜像投屏才能控制投屏到目标设备上的应用。图13中，第一VD作为数据源分别向第二VD和目标设备发送第一应用的显示信息。因此，第一应用始终运行在第一VD中，所以应用投屏不需要终止，无需重新建立镜像投屏，比图3的过程简化。
图14为本申请实施例提供的对投屏到目标设备上显示界面的控制方法的流程示意图。如图14所示，所述方法的流程包括：
S1，手机显示第一应用的界面。
S2，手机接收到投屏操作，判断是否为镜像投屏模式。
比如，手机接收到投屏操作之后，可以输出提示信息，提示信息用于提示用户选择镜像投屏模式或者非镜像投屏模式(即应用投屏模式)。
S3，如果不是镜像投屏模式，创建第一VD。
S4，将本地显示屏上第一应用挪动到第一VD中运行。
这里的“挪动”可以理解为，将本地显示屏的任务栈中第一应用的运行任务(task)挪动到第一VD的任务栈中，也可以称为“挪栈”。第一应用从本地显示屏挪动到第一VD运行之后，本地显示屏没有第一应用的运行任务。
S5，第一VD将第一应用投屏到目标设备上显示。
S6，根据接收到的输入操作，判断是否要对投屏到目标设备上的第一应用进行控制。
比如，输入操作是点击图11中的控制按键1101的操作(即对应前面的第二种方式)；或者，用于指示对投屏到目标设备上的第一应用进行控制的语音指令时，确定要对投屏到目标设备上的第一应用进行控制。
可选的，S6也可以不执行，所以图中S6使用虚线表示。如果S6不执行，从S5直接进入S7创建第二VD，可以理解为在主设备投出第一应用之后，主设备前台自动出现第一窗口(即对应前面的第一种方式)。可选的，S6可以在S5后紧接着执行，也可以在S7或S8之后执行，本申请实施例不作限定。
S7，如果是，创建第二VD。
S8，建立第一VD和第二VD之间的镜像映射关系。所述镜像映射关系用于指示第一VD向第二VD镜像映射。
比如，在S8之前，还可以包括步骤：设置第一VD的第一标识、第二VD的第二标识。其中，第一标识和/或第二标识可以是VD名(name)或者ID，对此不限定。比如，第一VD的第一标识可以是source VD，第二VD的第二标识可以是target VD，用于指示第一VD向第二VD镜像映射。
可选的，在S8之前，还可以包括步骤：设置标记位(flag)，该标记位用于指示是否需要对投屏到目标设备上的第一应用进行控制。比如，该标记位为1时，用于指示需要对投屏到目标设备上的第一应用进行控制，当标记位为0时，用于指示不对投屏到目标设备上的第一应用进行控制。
S9，基于所述镜像映射关系，将第一VD中第一应用映射到第二VD中，第二VD将第一应用映射到与第二VD对应的surface上。
可以理解的是，在S9之前，还可以包括步骤：创建surface，该surface可以是在本地显示屏上的一个显示区域，比如第一窗口。当然，surface也可以是一直存在的，比如，手机出厂之后就默认存在的，那么在S9之前无需创建surface。
S10，根据用户操作，更新surface中第一应用的界面。
S11，surface中更新界面映射到第二VD中，第二VD将更新界面映射到第一VD，第一VD将更新界面投屏到目标设备。
通过以上步骤，用户通过第二VD对应的surface(即第一窗口)实现对投屏到目标设备上的应用进行控制。
本申请实施例中，通过创建第二VD实现对投屏到目标设备上的显示界面进行控制具有如下有益效果：
首先，需要说明的是，一般来说，手机系统会自带Mirror映射功能，即Mirror映射功能可直接使用，不需要对底层代码进行修改。本申请实施例中，创建第二VD，第一VD向第二VD映射可以使用系统自带的Mirror映射，那么就避免了对底层代码的修改。而且，第二VD向手机本地显示的映射为surface映射，也不需要经过底层处理。因此，本申请实施例提供的技术方案实现难度较低，而且不需要经过底层处理，处理效率较高。
可选的，手机也可以不创建第二VD，这样的话，第一VD直接将第一VD中显示信息映射到本地显示屏的第一窗口进行显示，这种方式较为直接。其中，第一VD直接向本地显示屏映射时可以不使用Mirror映射，而是经过底层处理，即第一VD将第一VD中的 显示信息经过底层绘制、合成画面之后通过本地显示屏(比如第一窗口)显示。比如，为第一VD再次创建一个surface(这是因为第一VD向目标设备投屏时已经创建过一个surface用于承载投屏到目标设备的显示信息，该surface被称为input surface)，所以第一VD对应了两个surface。这种情况下，可以将input surface的显示信息映射到再次创建的surface上。这种方式虽然需要对底层代码进行修改，但是不需要创建第二VD，节省VD创建成本。
实施例二
本实施例二可以理解为实施例一的细化。本实施例二介绍主设备内部不同模块之间的信息交互，还有目标设备内部不同模块之间的信息交互，以及主设备与目标设备之间的信息交互过程。
请参见图15，为本申请实施例提供在投屏场景下对投屏到目标设备的应用的控制方法的流程示意图。该流程示意图可以理解为图14所示的流程图的细化。具体来说，图15中S1501至S1508介绍镜像投屏的实现过程。S1509至S1517介绍应用投屏的实现过程，可以理解为对图14中S2至S5的细化。S1518至S1531介绍应用投屏时对投屏到目标设备上的显示界面进行控制的过程，可以理解为对图14中S6至S11的细化过程。
如图15，所述流程包括：
S1501，主设备中镜像投屏模块接收到镜像投屏指令。
S1502，镜像投屏模块创建第一VD。
S1503，镜像投屏模块将主设备本地显示屏上的显示信息映射到第一VD中。
S1504，镜像投屏模块将第一VD中的显示信息发送给投屏编码模块。
S1505，投屏编码模块对显示信息进行编码。
示例性的，编码方式包括H.264、MPEG-4等，本申请实施例不作限定。
S1506，投屏编码模块将编码后的显示信息发送给目标设备。
S1507，目标设备接收到编码后的显示信息之后，通过投屏解码模块进行解码。
S1508，目标设备中投屏解码模块将解码后的显示信息发送给投屏surface模块进行展示。
以上流程是镜像投屏过程，应用投屏的过程包括如下步骤：
S1509，主设备中应用投屏模块接收到应用投屏指令。
S1510，应用投屏模块向镜像投屏模块发送VD创建请求。
S1511，镜像投屏模块创建第一VD。
S1512，应用投屏模块将第一应用挪到第一VD中运行。
这里的“挪动”可以理解为，将本地显示屏的任务栈中第一应用的运行任务(task)挪动到第一VD的任务栈中，也可以称为“挪栈”。第一应用从本地显示屏挪动到第一VD运行之后，本地显示屏没有第一应用的运行任务。
S1513，镜像投屏模块将第一VD中的显示信息发送给投屏编码模块。
S1514，投屏编码模块对显示信息进行编码。
S1515，投屏编码模块将编码后的显示信息发送给目标设备。
S1516，目标设备接收到编号后的显示信息之后，通过投屏解码模块进行解码。
S1517，目标设备中投屏解码模块将解码后的显示信息发送给投屏surface模块进行展 示。
在应用投屏时，主设备对投屏到目标设备上的显示界面进行控制的流程包括：
S1518，应用投屏模块将接收到的控制指令发送给应用投屏surface模块，所述控制指令用于指示对投屏到目标设备上的显示界面进行控制。
S1519，应用投屏surface模块创建第二VD。
S1520，应用投屏surface模块向镜像投屏模块发送镜像映射指令，用于指示镜像投屏模块将第一VD中的显示信息映射到第二VD中。
S1521，镜像投屏模块将第一VD中的显示信息映射到第二VD中。
S1522，应用投屏surface模块创建surface用于承载第二VD中的显示信息。
S1523，应用投屏surface模块将第二VD中的显示信息映射到surface上显示。
S1524，应用投屏surface模块接收到输入操作，所述输入操作用于更新surface中的显示信息。
S1525，应用投屏surface模块更新第二VD中的显示信息。
S1526，应用投屏surface模块将第二VD中更新的显示信息同步到第一VD中。
S1527，第一VD将更新的显示信息发送给投屏编码模块。
S1528，投屏编码模块对更新的显示信息进行编码。
S1529，投屏编码模块将编码后的显示信息发送给目标设备。
S1530，目标设备接收到编号后的显示信息之后，通过投屏解码模块进行解码。
S1531，目标设备中投屏解码模块将解码后的显示信息(更新后的显示信息)发送给投屏surface模块进行展示。
实施例三
本实施例三可以理解为实施例一的细化。本实施例三介绍主设备内部不同模块之间的信息交互，以及主设备中不同模块之间的信息交互过程。
请参见图16，为本申请实施例提供的主设备的功能框图，该图也可以理解为主设备的软件架构图。如图16，主设备中包括显示管理器(display manager)、显示管理服务(display manager service)模块、VD管理器(VD manager)、界面管理器(surface manager)四个模块。这四个模块可以位于主设备的Framework层中，Framework层位于应用层的下层。
显示管理器，负责上层(比如应用层)和ROM逻辑层的连接，可以理解为上层与ROM逻辑层之间的桥梁，可以为上层数据作一些逻辑处理。比如，显示管理器可以提供上层接口，该上层接口可以供应用层中的应用调用，以实现双VD映射。
显示管理服务模块负责双Vd的映射逻辑处理(比如，源VD与目的VD之间的映射关系的建立)，还可以负责VD的DF的判断等。显示管理服务可以位于Framework中，负责双VD映射的方案在Framework中实现。
VD管理器负责VD的管理，比如，VD的创建、删除等，还可以负责VD的MirrorFlag的设置等，还可以接收应用层的调用以创建VD进行应用投屏等。
界面管理器(surface manager)，负责在surface上呈现画面等。
请参见图17，为本申请一实施例提供的对投屏到目标设备上应用的控制方法的流程示意图，该图可以理解为图16中主设备中不同模块之间的信息交互示意图。具体来说，图17中S1701至S1704介绍应用投屏过程，可以理解为对图14中S2至S5的细化。S1705至S1715介绍应用投屏时对投屏到目标设备上的显示界面进行控制的过程，可以理解为对 图14中S6至S11的细化过程。
如图17所示，所述流程包括：
S1701，应用层接收到应用投屏指令之后，向VD管理器发送VD创建请求。
可以理解为，应用层可以调用VD管理器创建VD。
S1702，VD管理器创建第一VD。
S1703，显示管理服务模块将要投屏应用(如第一应用)挪到第一VD中运行。
S1704，VD管理器设置第一VD的第一屏幕标记(DISPLAY_FLAG，DF)。
一般来说，一个VD的DF可以是标记1，比如，标记1可以是只显示自己内容(OWN_CONTENT_ONLY)，该标记1表示只显示该VD内包括的内容，那么，如果该VD上运行了第一应用，那么就显示第一应用的界面(即应用投屏)，如果该VD内没有任何应用运行时，显示黑屏。或者，一个VD的DF可以是标记2，比如，标记2可以是自动镜像(AUTO_MIRROR)，该标记2表示当该VD内没有任何应用运行时可以默认镜像主屏的显示界面。因此，S1703中，第一VD的第一DF可以是OWN_CONTENT_ONLY或者AUTO_MIRROR。
示例性的，VD管理器可以使用VD标识设置(setVdFlag)方法设置第一VD的第一DF。
以上过程主设备完成应用投屏(即应用投屏)过程。在应用投屏时对投屏到目标设备上的应用的控制流程包括如下步骤。
S1705，应用层向显示管理器发送双VD指令。
可选的，双VD指令中包括第一VD的第一标识和第二VD的第二标识，还可以包括其它信息，比如本地显示屏的显示参数等等。
示例性的，应用层可以通过setMirrorDisplay方法调用显示管理器以向显示管理器发送双VD指令；或者，也可以理解为系统提供setMirrorDisplay接口，应用层调用该setMirrorDisplay接口可以向显示管理器发送双VD指令。示例性的，setMirrorDisplay接口的入口参数包括(target Display Name，source Display Id)，其中，targetDisplayName指双VD映射的目的端即第二VD的名称；source Display Id指的是双VD映射的源端即第一VD的唯一标识。因此，当应用层调用setMirrorDisplay接口时需要输入第一VD和第二VD的标识。
S1706，显示管理服务模块获取第一VD的第一标识和第二VD的第二标识。
可选的，显示管理服务模块可以在双VD映射指令中获取第一VD的第一标识和第二VD的第二标识。或者，显示管理器可以通过setTartgetDisplayName方法将目的端(即第二VD的第二标识)注入到显示管理服务模块，通过setSourceDisplayId将源端(即第一VD的第一标识)标识注入到显示管理服务模块。当第一VD的第一标识和第二VD的第二标识被注入到显示管理服务模块后，可以供Framework层中其它模块使用。
S1707，显示管理服务模块向VD管理器发送VD创建请求，该VD创建请求中携带第二VD的第二标识。
S1708，VD管理器创建第二VD。
S1709，VD管理器设置第二VD的第二DF，第二DF用于指示第二VD用于对目标设备上投屏的应用进行控制；或者，用于指示第二VD的源端是一个特定的VD(即第一VD)。
DF除了上述标记1和标记2之外，本申请还提供一种标记3，标记3可称为自定义镜 像显示(MIRROR_CUSTOM_DISPLAY)，表征第二VD是用于控制投屏到目标设备上的显示信息；或者，用于指示第二VD的源端是一个特定的VD(即第一VD)。
示例性的，VD管理器可以通过setVdFlag方法设置第二DF。
示例性的，在设置第二DF之后，第二DF会在显示管理服务模块中的打桩处生效，具体将在后文介绍。
S1710，显示管理服务模块向界面管理器发送surface创建请求，用于请求创建用于承载第二VD的surface。
S1711，界面管理器创建第二VD对应的surface。
示例性的，界面管理器可以通过setVdSurface方法创建一块独立的绘制表面(surface)用于承载第二VD的显示信息，即将第二VD中显示信息绘制到surface上展示。
需要说明的是，上述S1707至S1709，与S1710至S1711的执行顺序，本申请实施例不作限定，可以先创建第二VD后创建第二VD对应的surface；或者，先创建surface再创建surface对应的第二VD。
S1712，显示管理服务模块根据第二DF判断需要对投屏到目标设备的应用进行控制；或者，需要将一个特定的VD中的显示信息映射到第二VD中。
示例性的，显示管理服务模块可以使用configure Display Locked方法进行打桩，插入与上面两种DF(标记1和标记2)不同的另一种DF，即标记3，即MIRROR_CUSTOM_DISPLAY。在前面S1710中VD管理器设置了第二VD的第二DF＝MIRROR_CUSTOM_DISPLAY之后，第二DF在打桩处生效。可以理解为，显示管理服务模块会对打桩点进行识别，当识别到第二DF是MIRROR_CUSTOM_DISPLAY时，确定需要对投屏到目标设备上的应用进行控制或者需要将一个特定的VD中的显示信息映射到第二VD中。示例性的，显示管理服务模块可以运行isMirrorFlag方法对打桩点进行判断。
S1713，建立第一VD与第二VD的镜像映射关系。
示例性的，显示管理服务模块判断出第二DF是MIRROR_CUSTOM_DISPLAY时，可以在打桩点处插入第一VD的第一标识，表征该打桩点处与第一VD存在关联关系，即第二VD与第一VD建立了镜像映射关系。示例性的，显示管理服务模块可以使用getMirrorDisplay方法在打桩点处基于第一VD的第一标识和第二VD的第二标识建立镜像映射关系。
S1714，将原本要映射到第二VD的显示信息替换为第一VD中的显示信息。
需要说明的是，S1708中创建第二VD之后，如果第二VD中不运行任何应用，将默认映射主屏上的当前显示信息。在本申请实施例中，由于设置了第二VD的第二DF是MIRROR_CUSTOM_DISPLAY，而且建立了第一VD与第二VD的镜像映射关系，所以将原本要映射到第二VD的显示信息(即主屏当前显示信息)替换为第一VD中的显示信息，实现第一VD向第二VD的映射。
示例性的，S1714可以由显示管理服务模块和VD管理器共同完成，比如，显示管理服务模块向VD管理器发送指令，用于指示第一VD中的显示信息映射到第二VD中，那么VD管理器执行该指令即可。
S1715，VD管理器将第二VD中的显示信息发送给界面管理器，以通过界面管理器中创建的第二VD对应的surface进行展示。
实施例四
在前面的实施例一中，第二VD中的显示信息通过主设备上第一窗口显示，所以由主设备通过第一窗口控制投屏到目标设备上的显示信息。本实施例四中考虑到如果主设备中第二VD中的显示信息通过其它设备显示的话，那么就可以通过其它设备控制投屏到目标设备上的显示信息。可以简单的理解为，主设备将对投屏到目标设备上的显示信息的控制权转移到其它设备，这样，主设备可以无打扰的进行其他操作，通过其它设备控制主设备投屏到目标设备上的显示信息。
示例性的，请参见图18，为本实施例四通过的一种应用场景的示意图。
如图18中的(a)，主设备将第一应用投屏到目标设备1和目标设备2上，即目标设备1和目标设备2上显示相同的投屏界面。在投出第一应用之后，主设备显示第二应用的界面，如图18中的(b)。用户可以在目标设备1上控制主设备投屏到目标设备2上的显示信息。比如，如图18中的(b)，当目标设备1上投屏界面由第一应用的界面A更新为第一应用的界面B时，目标设备2上投屏界面也更新为第一应用的界面B。
其中，目标设备1和目标设备2可以是相同类型的设备或不同类型的设备，比如，目标设备1是手机、PC等便捷式设备，目标设备2是电视机等设备。一种示例性的场景为，主设备是手机A，目标设备1是手机B，目标设备2是电视机。手机A上将第一应用投屏到手机B和电视机上，手机A的用户可以使用手机A进行其他操作，手机B的用户可以使用手机B控制手机A投屏到电视机上的显示信息。
图18所对应的主设备后台的处理流程请参见图19所示。如图19，主设备创建第一VD和第二VD，将本地显示屏上第一应用挪到第一VD中运行，第一VD将第一应用投屏到目标设备2上显示。第一VD中的显示信息还映射到第二VD中，第二VD将显示信息投屏到目标设备1上显示。也就是说，第一VD对应目标设备2，第二VD对应目标设备1。
当目标设备1上第一应用的界面更新时，第二VD中的显示信息更新。第二VD将更新的显示信息映射到第一VD中，第一VD将更新的显示信息投屏到目标设备2上。即，目标设备1控制主设备投屏到目标设备2上的显示信息。
请结合图13和图19，可知，图13中，第二VD对应的是本地显示屏，所以通过本地显示屏可以控制投屏到目标设备上的显示信息。图19中，第二VD对应的不是本地显示屏，而是目标设备1，所以通过目标设备1可以控制主设备投屏到目标设备2上的显示信息。
一种可实现方式为，主设备将原来的第二VD与本地显示屏的对应关系修改为第二VD与目标设备1的对应关系，这样的话，主设备上可以无需再显示第二VD中的显示信息即无需再显示第一窗口，目标设备1上显示第二VD中的显示信息。
基于同一发明构思，本申请实施例提供一种投屏控制方法。该方法可以适用于通信系统，所述通信系统包括第一电子设备和第二电子设备。其中，第一电子设备可以是前文中的主设备，第二电子设备可以是前文中的目标设备。主设备是投屏场景中的发送端，目标设备是投屏场景中的接收端。当然，发送端和接收端的角色可以互换，即，发送端可以向接收端投屏，对应的，所述接收端也可以向所述发送端投屏。示例性的，第一电子设备可以是手机、平板电脑、PC、手表等具有显示屏的设备。目标设备可以是平板电脑、PC、电视机等大屏设备。
如图20，所述方法的流程包括：
S2001，第一电子设备显示第一应用的第一界面。
示例性的，以图7中的(a)和图7中的(b)为例，第一电子设备检测到用于打开第一应用的操作(如，在主界面中点击第一应用的图标的操作)时，第一电子设备显示第一应用的第一界面。其中，第一应用可以是任意应用，比如可以是图4A中的(a)的相册应用，或者，是图5A中的(a)的视频播放应用等等。
S2002，第一电子设备接收到投屏操作。
示例性的，所述投屏操作可以包括从显示屏下边缘的上滑操作以及在上滑菜单(响应于上滑操作显示的上滑菜单)中点击投屏图标的操作；或者，包括从手机屏幕顶部状态栏的下滑操作，以及点击通知面板(响应于下滑操作显示的通知面板)的投屏图标的操作等等，还可以是用于指示投屏的语音指令，本申请实施例不作限定。
S2003，响应于投屏操作，第一电子设备将第一界面的显示信息发送给第二电子设备。
可选的，在S2003之前，还可以包括第一电子设备显示设备列表，以及根据用户在设备列表中的选择操作，确定第二电子设备的过程，如图7中的(c)所示。
S2004，第二电子设备显示第一应用的第一界面。
示例性的，如图7中的(d)，第二电子设备显示第一应用的第一界面。
S2005，第一电子设备显示第二应用的第二界面以及第一窗口，第一窗口中包括第三界面，所述第三界面与投屏到所述第二电子设备上的所述第一应用的当前显示界面相同。
第一种实现方式为，以图9为例，第一电子设备投出第一应用时，第一电子设备显示第二应用的第二界面，并且自动出现第一窗口901。第二种实现方式为，以图11为例，第一电子设备投出第一应用时，第一电子设备显示第二应用的第二界面，还显示控制按键1101。当检测到针对控制按键1101的操作时，第一电子设备显示第一窗口901。第一窗口901中的显示信息与投屏到第二电子设备上的显示信息相同，用户可以通过第一窗口901对投屏到第二电子设备上的显示信息进行控制。需要说明的是，前文关于第一窗口的所有描述均适用于此实施例中，为了节省篇幅，此处不重复赘述。
第一电子设备显示第一窗口的后台实现过程请参见图13，不重复赘述。
S2006，第一电子设备接收到用户操作。
示例性的，所述用户操作是指用于更新第一窗口内显示信息的操作。比如，如图4B，所述用户操作可以是在第一窗口401中打开相册应用中某张图像的操作。或者，如图5B，所述用户操作可以是在第一窗口501中打开视频播放应用中电视机下一集的操作。
S2007，响应于用户操作，第一电子设备将第一窗口内所述第三界面更新为第四界面。
示例性的，以图12为例，在用户操作的作用下，第一窗口901中的第一应用的界面A更新为第一应用的界面B。
S2008，第一电子设备将所述第四界面的显示信息发送给第二电子设备。
S2009，第二电子设备上第一应用的显示界面更新为第四界面。
示例性的，继续以图12为例，在第一窗口901内的第一应用的界面A更新为第一应用的界面B时，对应的，第二电子设备上投屏来的第一应用的界面A也更新为界面B。其中，通过第一窗口对投屏到第二电子设备上显示信息进行控制的后台处理过程请参见前文对图13的相关描述，此处不重复赘述。
在一些实施例中，第一电子设备还可以将第四界面的显示信息发送给第三电子设备。第三电子设备显示所述第四界面。也就是说，第一电子设备向第二电子设备和第三电子设 备分别投屏。第一电子设备可以将对投屏到第二电子设备上的显示信息的控制权转移给第三电子设备。这样，第三电子设备控制本机显示屏上第四界面更新为第五界面时，对应的，第二电子设备将本机显示屏上的所述第四界面更新为所述第五界面。示例性的，以图18中的(a)为例，主设备将第一应用投屏到目标设备1和目标设备2，并且将对投屏到目标设备2的显示信息的控制权转移给目标设备1。因此，图18中的(b)中，目标设备1更新投屏来的第一应用的界面为界面B时，对应的，目标设备2上对投屏来的第一应用的界面更新为界面B。其中，主设备将对投屏到目标设备1上的显示信息的控制权转移给目标设备2的后台实现过程请参见图19，此处不重复赘述。
可选的，第一电子设备将对投屏到第二电子设备上的显示信息的控制权转移给第三电子设备之后，第一电子设备可以取消显示所述第一窗口。
基于相同的构思，图21为本申请实施例提供的电子设备2100的结构示意图。电子设备2100可以是前文中的主设备或目标设备。如图21所示，电子设备2100可以包括：一个或多个处理器2101；一个或多个存储器2102；通信接口2103，以及一个或多个计算机程序2104，上述各器件可以通过一个或多个通信总线2105连接。其中该一个或多个计算机程序2104被存储在上述存储器2102中并被配置为被该一个或多个处理器2101执行，该一个或多个计算机程序2104包括指令。比如，上述指令可以用于执行如上面相应实施例中主设备的相关步骤。通信接口2103用于实现主设备与其他设备(比如目标设备)的通信，比如通信接口可以是收发器。
基于相同的构思，本申请实施例还提供一种通信系统。所述通信系统中包括第一电子设备和第二电子设备。其中，第一电子设备可以是前文中的主设备，第二电子设备可以是前文中的目标设备。示例性的，第一电子设备可以是手机、平板电脑、PC、手表等具有显示屏的设备。第二电子设备可以是平板电脑、PC、电视机等大屏设备。其中，第一电子设备和第二电子设备的结构可以参见图21所示。比如，当图21所示的电子设备2100是第一电子设备时，当一个或多个计算机程序2104的指令被处理器执行时，使得所述第一电子设备执行如前文中第一电子设备(即主设备)的步骤。当图21所示的电子设备2100是第二电子设备时，当一个或多个计算机程序2104的指令被所述处理器执行时，使得所述第二电子设备执行如前文中第二电子设备(即目标设备)的步骤。
上述本申请提供的实施例中，从电子设备(例如主设备或目标设备)作为执行主体的角度对本申请实施例提供的方法进行了介绍。为了实现上述本申请实施例提供的方法中的各功能，电子设备可以包括硬件结构和/或软件模块，以硬件结构、软件模块、或硬件结构加软件模块的形式来实现上述各功能。上述各功能中的某个功能以硬件结构、软件模块、还是硬件结构加软件模块的方式来执行，取决于技术方案的特定应用和设计约束条件。
以上实施例中所用，根据上下文，术语“当…时”或“当…后”可以被解释为意思是“如果…”或“在…后”或“响应于确定…”或“响应于检测到…”。类似地，根据上下文，短语“在确定…时”或“如果检测到(所陈述的条件或事件)”可以被解释为意思是“如果确定…”或“响应于确定…”或“在检测到(所陈述的条件或事件)时”或“响应于检测到(所陈述的条件或事件)”。另外，在上述实施例中，使用诸如第一、第二之类的关系术语来区份一个实体和另一个实体，而并不限制这些实体之间的任何实际的关系和顺序。
在本说明书中描述的参考“一个实施例”或“一些实施例”等意味着在本申请的一个或多个实施例中包括结合该实施例描述的特定特征、结构或特点。由此，在本说明书中的不同 之处出现的语句“在一个实施例中”、“在一些实施例中”、“在其他一些实施例中”、“在另外一些实施例中”等不是必然都参考相同的实施例，而是意味着“一个或多个但不是所有的实施例”，除非是以其他方式另外特别强调。术语“包括”、“包含”、“具有”及它们的变形都意味着“包括但不限于”，除非是以其他方式另外特别强调。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本发明实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘Solid State Disk(SSD))等。在不冲突的情况下，以上各实施例的方案都可以组合使用。
需要指出的是，本专利申请文件的一部分包含受著作权保护的内容。除了对专利局的专利文件或记录的专利文档内容制作副本以外，著作权人保留著作权。

Claims (27)

  1. 一种投屏控制方法，应用于包括第一电子设备和第二电子设备的系统，其特征在于，所述方法包括：
    所述第一电子设备显示第一应用的第一界面；
    响应于投屏操作，所述第一电子设备将所述第一界面的显示信息发送给所述第二电子设备；
    所述第二电子设备显示所述第一应用的所述第一界面；
    所述第一电子设备显示第二应用的第二界面，以及第一窗口，所述第一窗口中包括第三界面，所述第三界面与投屏到所述第二电子设备上的所述第一应用的当前显示界面相同；
    响应于用户操作，所述第一电子设备将所述第一窗口内所述第三界面更新为第四界面；
    所述第一电子设备将所述第四界面的显示信息发送给所述第二电子设备；
    所述第二电子设备上所述第一应用的显示界面更新为所述第四界面。
  2. 根据权利要求1所述的方法，其特征在于，
    所述第一窗口悬浮显示于所述第二界面上层，所述第一窗口不完全覆盖所述第二界面；或者，所述第二界面与所述第一窗口分屏显示。
  3. 根据权利要求1或2所述的方法，其特征在于，所述第一电子设备显示第一窗口，包括：
    接收到用于唤出所述第一窗口的操作；
    响应于所述用于唤出所述第一窗口的操作，显示所述第一窗口。
  4. 根据权利要求3所述的方法，其特征在于，响应于所述用于唤出所述第一窗口的操作，显示所述第一窗口，包括：
    所述第二界面上显示第一标记；
    响应于针对所述第一标记的操作，唤出所述第一窗口。
  5. 根据权利要求1-4任一所述的方法，其特征在于，所述方法还包括：
    响应于用于隐藏所述第一窗口的操作，隐藏所述第一窗口；或者，
    响应于用于调整所述第一窗口的显示位置和/或显示面积的操作，调整所述第一窗口的显示位置和/或显示面积。
  6. 根据权利要求1-5任一所述的方法，其特征在于，所述第一电子设备显示第一窗口，包括：
    所述第一电子设备创建绘制表面，所述绘制表面上的显示信息通过所述第一电子设备的显示屏显示；
    所述第一电子设备将第一虚拟显示屏VD中的显示信息映射到所述绘制表面上；其中，所述第一VD中运行所述第一应用，并且所述第一VD用于将所述第一VD内的显示信息发送给所述第二电子设备。
  7. 根据权利要求6所述的方法，其特征在于，所述第一电子设备将第一虚拟显示屏VD中的显示信息映射到所述绘制表面上，包括：
    所述第一电子设备创建第二VD，所述第二VD与所述绘制表面对应；
    所述第一电子设备将所述第一VD内的显示信息映射到所述第二VD中，所述第二VD中的显示信息通过所述绘制表面显示。
  8. 根据权利要求7所述的方法，其特征在于，响应于用户操作，所述第一电子设备将所述第一窗口内所述第三界面更新为第四界面，包括：
    响应于用户操作，所述绘制界面上显示信息更新为所述第四界面；其中，所述绘制界面上所述第四界面的显示信息被映射到所述第二VD中，所述第二VD用于将所述第四界面的显示信息映射到所述第一VD，所述第一VD用于将所述第四界面的显示信息发送给所述第二电子设备。
  9. 根据权利要求1-8任一所述的方法，其特征在于，所述方法还包括：
    所述第一电子设备将所述第四界面的显示信息发送给所述第三电子设备；
    所述第三电子设备显示所述第四界面；
    所述第一电子设备将对投屏到所述第二电子设备上的显示信息的控制权转移给所述第三电子设备；
    所述第三电子设备控制本机显示屏上所述第四界面更新为第五界面时，所述第二电子设备将本机显示屏上的所述第四界面更新为所述第五界面。
  10. 根据权利要求9所述的方法，其特征在于，所述方法还包括：
    所述第一电子设备上取消显示所述第一窗口。
  11. 根据权利要求9或10所述的方法，其特征在于，所述第一电子设备将对投屏到所述第二电子设备上的显示信息的控制权转移给第三电子设备，包括：
    所述第一电子设备创建第三VD；所述第三VD用于将所述第三VD内的显示信息发送给所述第三电子设备；
    所述第一电子设备将第一VD中的显示信息映射到所述第三VD上；其中，所述第一VD中运行所述第一应用，并且所述第一VD用于将所述第一VD内的显示信息发送给所述第二电子设备。
  12. 根据权利要求11所述的方法，其特征在于，所述第三电子设备控制本机显示屏上所述第四界面更新为第五界面时，所述第二电子设备将本机显示屏上的所述第四界面更新为所述第五界面，包括：
    所述第三电子设备将所述第四界面更新为第五界面时，所述第三VD中的显示信息更新为所述第五界面的显示信息，所述第三VD用于将所述第五界面的显示信息映射到所述第一VD中，所述第一VD用于将所述第五界面的显示信息发送给所述第二电子设备。
  13. 一种投屏控制方法，其特征在于，包括：
    第一电子设备显示第一应用的第一界面；
    响应于投屏操作，所述第一电子设备将所述第一界面的显示信息发送给第二电子设备；
    所述第一电子设备显示第二应用的第二界面，以及第一窗口，所述第一窗口中包括第三界面，所述第三界面与投屏到所述第二电子设备上的所述第一应用的当前显示界面相同；
    所述第一电子设备响应于用户操作，将所述第一窗口内所述第三界面更新为第四界面；
    所述第一电子设备将所述第四界面的显示信息发送给所述第二电子设备，以使所述第二电子设备上所述第一应用的显示界面更新为所述第四界面。
  14. 根据权利要求13所述的方法，其特征在于，
    所述第一窗口悬浮显示于所述第二界面上层，所述第一窗口不完全覆盖所述第二界面；或者，所述第二界面与所述第一窗口分屏显示。
  15. 根据权利要求13或14所述的方法，其特征在于，所述第一电子设备显示第一窗口， 包括：
    接收到用于唤出所述第一窗口的操作；
    响应于所述用于唤出所述第一窗口的操作，显示所述第一窗口。
  16. 根据权利要求15所述的方法，其特征在于，响应于所述用于唤出所述第一窗口的操作，显示所述第一窗口，包括：
    所述第二界面上显示第一标记；
    响应于针对所述第一标记的操作，唤出所述第一窗口。
  17. 根据权利要求13-16任一所述的方法，其特征在于，所述方法还包括：
    响应于用于隐藏所述第一窗口的操作，隐藏所述第一窗口；或者，
    响应于用于调整所述第一窗口的显示位置和/或显示面积的操作，调整所述第一窗口的显示位置和/或显示面积。
  18. 根据权利要求13-17任一所述的方法，其特征在于，所述第一电子设备显示第一窗口，包括：
    所述第一电子设备创建绘制表面，所述绘制表面上的显示信息通过所述第一电子设备的显示屏显示；
    所述第一电子设备将第一虚拟显示屏VD中的显示信息映射到所述绘制表面上；
    其中，所述第一VD中运行所述第一应用，并且所述第一VD用于将所述第一VD内的显示信息发送给所述第二电子设备。
  19. 根据权利要求18所述的方法，其特征在于，所述第一电子设备将第一虚拟显示屏VD中的显示信息映射到所述绘制表面上，包括：
    所述第一电子设备创建第二VD，所述第二VD与所述绘制表面对应；
    所述第一电子设备将所述第一VD内的显示信息映射到所述第二VD中，所述第二VD中的显示信息通过所述绘制表面显示。
  20. 根据权利要求19所述的方法，其特征在于，响应于用户操作，所述第一电子设备将所述第一窗口内所述第三界面更新为第四界面，包括：
    响应于用户操作，所述绘制界面上显示信息更新为所述第四界面；其中，所述绘制界面上所述第四界面的显示信息被映射到所述第二VD中，所述第二VD用于将所述第四界面的显示信息映射到所述第一VD，所述第一VD用于将所述第四界面的显示信息发送给所述第二电子设备。
  21. 根据权利要求13-20任一所述的方法，其特征在于，所述方法还包括：
    所述第一电子设备将所述第四界面的显示信息发送给第三电子设备，以使所述第三电子设备显示所述第四界面；
    所述第一电子设备将对投屏到所述第二电子设备上的显示信息的控制权转移给第三电子设备。
  22. 根据权利要求21所述的方法，其特征在于，所述方法还包括：
    所述第一电子设备上取消显示所述第一窗口。
  23. 根据权利要求21或22所述的方法，其特征在于，所述第一电子设备将对投屏到所述第二电子设备上的显示信息的控制权转移给第三电子设备，包括：
    所述第一电子设备创建第三VD；所述第三VD用于将所述第三VD内的显示信息发送给所述第三电子设备；
    所述第一电子设备将第一VD中的显示信息映射到所述第三VD上；其中，所述第一VD中运行所述第一应用，并且所述第一VD用于将所述第一VD内的显示信息发送给所述第二电子设备。
  24. 根据权利要求23所述的方法，其特征在于，所述方法还包括：
    所述第三VD用于在所述第三电子设备上所述第四界面更新为第五界面时，将所述第三VD中的显示信息更新为所述第五界面的显示信息，并且将所述第五界面的显示信息映射到所述第一VD中，所述第一VD用于将所述第五界面的显示信息发送给所述第二电子设备。
  25. 一种通信系统，其特征在于，包括：第一电子设备和第二电子设备；
    所述第一电子设备包括：处理器；存储器；其中，所述存储器存储有一个或多个计算机程序，所述一个或多个计算机程序包括指令，当所述指令被所述处理器执行时，使得所述第一电子设备执行如权利要求1至12任一项所述的方法中第一电子设备的步骤；
    所述第二电子设备包括：处理器；存储器；其中，所述存储器存储有一个或多个计算机程序，所述一个或多个计算机程序包括指令，当所述指令被所述处理器执行时，使得所述第二电子设备执行如权利要求1至12任一项所述的方法中第二电子设备的步骤。
  26. 一种电子设备，其特征在于，包括：
    处理器，存储器，以及，一个或多个程序；
    其中，所述一个或多个程序被存储在所述存储器中，所述一个或多个程序包括指令，当所述指令被所述处理器执行时，使得所述电子设备执行如权利要求13至24中任意一项所述的方法步骤。
  27. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质用于存储计算机程序，当所述计算机程序在计算机上运行时，使得所述计算机执行如权利要求1至24中任意一项所述的方法。
PCT/CN2022/095705 2021-06-28 2022-05-27 一种投屏控制方法与电子设备 WO2023273744A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110717726.7 2021-06-28
CN202110717726.7A CN115599325A (zh) 2021-06-28 2021-06-28 一种投屏控制方法与电子设备

Publications (1)

Publication Number Publication Date
WO2023273744A1 true WO2023273744A1 (zh) 2023-01-05

Family

ID=84692511

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/095705 WO2023273744A1 (zh) 2021-06-28 2022-05-27 一种投屏控制方法与电子设备

Country Status (2)

Country Link
CN (1) CN115599325A (zh)
WO (1) WO2023273744A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160179456A1 (en) * 2014-12-23 2016-06-23 Sivapathalingham Sivavakeesar Spontaneous Collaboration Apparatus, System and Methods thereof
CN106897038A (zh) * 2015-12-17 2017-06-27 北京传送科技有限公司 一种投屏系统
CN110381195A (zh) * 2019-06-05 2019-10-25 华为技术有限公司 一种投屏显示方法及电子设备
CN111314768A (zh) * 2020-02-24 2020-06-19 北京小米移动软件有限公司 投屏方法、投屏装置、电子设备以及计算机可读存储介质
CN112988102A (zh) * 2021-05-11 2021-06-18 荣耀终端有限公司 投屏方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160179456A1 (en) * 2014-12-23 2016-06-23 Sivapathalingham Sivavakeesar Spontaneous Collaboration Apparatus, System and Methods thereof
CN106897038A (zh) * 2015-12-17 2017-06-27 北京传送科技有限公司 一种投屏系统
CN110381195A (zh) * 2019-06-05 2019-10-25 华为技术有限公司 一种投屏显示方法及电子设备
CN111314768A (zh) * 2020-02-24 2020-06-19 北京小米移动软件有限公司 投屏方法、投屏装置、电子设备以及计算机可读存储介质
CN112988102A (zh) * 2021-05-11 2021-06-18 荣耀终端有限公司 投屏方法和装置

Also Published As

Publication number Publication date
CN115599325A (zh) 2023-01-13

Similar Documents

Publication Publication Date Title
US20220147228A1 (en) Display Method and Related Apparatus
CN111316598B (zh) 一种多屏互动方法及设备
CN109766066B (zh) 一种消息处理的方法、相关装置及系统
WO2022257977A1 (zh) 电子设备的投屏方法和电子设备
WO2021036651A1 (zh) 一种显示方法及电子设备
CN115297199A (zh) 一种具有折叠屏的设备的触控方法与折叠屏设备
WO2022100610A1 (zh) 投屏方法、装置、电子设备及计算机可读存储介质
US20230422154A1 (en) Method for using cellular communication function, and related apparatus and system
CN112527174B (zh) 一种信息处理方法及电子设备
WO2021036898A1 (zh) 折叠屏设备中应用打开方法及相关装置
WO2022179405A1 (zh) 一种投屏显示方法及电子设备
WO2022089122A1 (zh) 一种应用窗口的投屏方法与电子设备
WO2022042769A2 (zh) 多屏交互的系统、方法、装置和介质
CN112527222A (zh) 一种信息处理方法及电子设备
WO2022135163A1 (zh) 一种投屏显示方法及电子设备
WO2022028537A1 (zh) 一种设备识别方法及相关装置
WO2021238967A1 (zh) 一种内容分享的方法、装置及系统
US11895713B2 (en) Data sharing and instruction operation control method and system
WO2023045712A1 (zh) 投屏异常处理方法及电子设备
CN114065706A (zh) 一种多设备数据协作的方法及电子设备
WO2021052388A1 (zh) 一种视频通信方法及视频通信装置
WO2023045597A1 (zh) 大屏业务的跨设备流转操控方法和装置
WO2023273744A1 (zh) 一种投屏控制方法与电子设备
CN115729431A (zh) 一种控件内容的拖拽方法、电子设备及系统
WO2023160224A1 (zh) 一种拍摄方法及相关设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22831566

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE