WO2023020025A1

WO2023020025A1 - Screen projection method and electronic device

Info

Publication number: WO2023020025A1
Application number: PCT/CN2022/091554
Authority: WO
Inventors: 刘诗聪
Original assignee: 荣耀终端有限公司
Priority date: 2021-08-20
Filing date: 2022-05-07
Publication date: 2023-02-23
Also published as: CN113778360A; CN113778360B

Abstract

Embodiments of the present application provide a screen projection method and an electronic device. A first electronic device identifies control information of a first control located in a first layer and control information of a second control located in a second layer in a first application, and the layer types of the layers where the controls are located are determined respectively according to the control information of the first control and the control information of the second control; and in the case that the layer types of the two layers are determined to be different, display pictures comprising different controls are generated according to the determined layer types and respectively sent to different electronic devices for displaying. Therefore, separation of different layers in an interface corresponding to a same application is achieved, and the display pictures comprising different controls can be displayed on different electronic devices, so that the first electronic device initiating screen projection and the second electronic device receiving screen projection content can display different pictures, thereby facilitating better adaptation to different screen projection scenes.

Description

Screen projection method and electronic device

This application claims the priority of the Chinese patent application with the application number 202110958660.0 and the title of the invention "screen projection method and electronic equipment" filed with the China Patent Office on August 20, 2021, the entire contents of which are incorporated by reference in this application.

technical field

The embodiments of the present application relate to the field of terminals, and in particular, to a method for projecting a screen and an electronic device.

Background technique

With the development of terminal technology, more and more terminals have screen projection functions. For example, in family, work, teaching, and game competition scenarios, terminals can greatly facilitate people to watch by projecting the currently displayed screen onto a large screen. screen content.

However, the content of the screen projected on the large screen by the current screen projection method is consistent with that displayed on the terminal. For the conference application (software), the process of the conference often involves adding people, setting the chairman, mute people, and sharing content. Selection and other operations, the existing screen projection method on the terminal interface to operate the conference control button will also be projected to the large screen, which will cause interference to the video stream content displayed on the large screen, thus affecting the ability to watch the conference content through the large screen user experience.

Contents of the invention

In order to solve the above technical problems, the present application proposes a screen projection method and electronic equipment. In this method, by identifying the layers in the interface corresponding to the first application, and then generating pictures including different controls according to the identified layer types and requirements, and transmitting the generated pictures to different electronic devices respectively , so that the image finally displayed on the electronic device that initiates the screen projection is different from the image displayed on the electronic device that accepts the screen projection content, so that it can better adapt to different screen projection scenarios.

In the first aspect, a method for screen projection is provided, which is applied to a first electronic device that initiates screen projection, the first electronic device is installed with a first application, the first application includes a first control and a second control, and the The first control is located on the first layer, and the second control is located on the second layer. The screen projection method includes: obtaining the control information of the first control and the control information of the second control; determining the layer type of the first layer according to the control information of the first control; The control information of the second control determines the layer type of the second layer, and the layer type of the first layer is different from the layer type of the second layer; according to the layer type of the first layer type and the layer type of the second layer, generate a first display picture and a second display picture, and display the first picture on the screen of the first electronic device, and project the second display picture to The screen of the second electronic device displays, the first display screen includes the first control and the second control, and the second display screen includes the first control and does not include the second control. In this way, the first electronic device that initiates screen projection recognizes the control information of the first control located on the first layer and the control information of the second control located on the second layer in the first application, and respectively according to the control information of the first control Information and the control information of the second control determine the layer type of the layer where each control is located. In the case that the layer types of the two layers are determined to be different, the display screens including different controls are generated according to the determined layer type and sent to the Different electronic devices display, such as sending the first display screen including the first control and the second control to the first electronic device, and sending the second display screen including the first control but not including the second control to the second electronic device. Device display, so that the separation of different layers in the interface corresponding to the same application is realized, and the display screen including different controls can be displayed on different electronic devices, so that the first electronic device that initiates screen projection and the first electronic device that receives the content of the screen projection Second, the electronic device can display different images, and thus can better adapt to different screen projection scenarios.

Exemplarily, the electronic device that initiates screen projection is a mobile phone.

Exemplarily, the first application is a conference application.

Exemplarily, when the first application is a conference application, the interface includes at least a video stream playback layer and a conference control button layer.

Exemplarily, the video stream playback layer includes at least a video stream playback control, and the conference control button layer includes at least a conference control button control.

Exemplarily, when the first application is a meeting application, the first display picture displayed on the screen of the first electronic device includes controls included in layers of all layer types, such as the first control in the first layer and the second control in the second layer, the second display picture displayed on the screen of the second electronic device only includes the controls included in the video stream playback layer, such as including the first control in the first layer, not Include the second control in the second layer.

According to the first aspect, the determining the layer type of the first layer according to the control information of the first control includes: extracting the control name and the name of the first control from the control information of the first control The size information of the first control; the control name is analyzed, and the control type of the first control, the package name of the first application and the interface where the first control is located are analyzed from the control name information, determine the layer type of the first layer according to the control type, the package name, the interface information and the size information; after parsing the first control from the control name control type and the interface information where the first control is located, and if the package name of the first application is not resolved, obtain the process identification number PID of the process that draws the first control, and determine the second control according to the PID A source of the control. Determine the layer type of the first layer according to the control type, the source, the interface information, and the size information, and the source includes the package name of the first application. In this way, the purpose of the first control can be determined according to the control type of the first control, the application type of the first application can be determined according to the package name of the first application, and then according to the purpose of the first control, the application type of the first application , combined with the size information of the first control and the specific interface information in the first application, it is possible to accurately identify the layer type of the layer where the first control is included in most applications on the market.

Exemplarily, when the package name of the first application is not parsed from the control name of the first control, by obtaining the PID of the process drawing the first control, the source of the process corresponding to the PID can be determined according to the unique PID , that is, the first application, and then obtain the package name of the first application. In this way, no matter whether there is a package name of the first application in the control name of the first control, the image of the first layer can be accurately determined according to the control information of the first control. layer type.

According to the first aspect, or any implementation manner of the above first aspect, before parsing the control name, the method further includes: using the control name and the size information as retrieval keywords; according to the keyword, searching for a control matching the keyword in the layer identification record library; when finding a control matching the keyword, determining the layer type corresponding to the control as the first layer layer type; when no control matching the keyword is found, the step of parsing the name of the control is performed. In this way, before determining the layer type of the first layer according to the obtained control information of the first control, the table lookup operation is performed according to the control name and size information, and when the matched control is operated, the corresponding control of the found control is directly The layer type is determined as the layer type of the first layer, so that there is no need to analyze and process according to the control information of the first control, which speeds up the processing speed. It not only takes into account the speed, avoids resource occupation, but also determines the appropriate layer type.

According to the first aspect, or any implementation manner of the above first aspect, when the layer type of the first layer cannot be determined according to the control information of the first control, obtain the current display of the first application A screen, the currently displayed screen includes the first control; and the layer type of the first layer is determined according to the content displayed by the first control in the currently displayed screen. In this way, when the layer type of the first layer cannot be determined according to the control information of the first control, the image of the first layer where the first control is located can be accurately determined by analyzing the screen currently displayed by the first application. Layer type, thereby ensuring subsequent separate drawing of layers based on layer type.

According to the first aspect, or any implementation manner of the above first aspect, the determining the layer type of the second layer according to the control information of the second control includes: from the control of the second control Extract the control name of the second control and the size information of the second control from the information; analyze the control name, and analyze the control type of the second control and the first control name from the control name When using the package name of the application and the interface information where the second control is located, determine the layer type of the second layer according to the control type, the package name, the interface information and the size information; The control type of the second control and the interface information of the second control are analyzed from the control name, and when the package name of the first application is not resolved, obtain the process of drawing the second control Identification number PID, determine the source of the second control according to the PID, determine the layer type of the second layer according to the control type, the source, the interface information and the size information, so The source includes a package name of the first application. In this way, the purpose of the second control can be determined according to the control type of the second control, the application type of the first application can be determined according to the package name of the first application, and then according to the purpose of the second control, the application type of the first application , combined with the size information of the second control and the specific interface information in the first application, it is possible to accurately identify the layer type of the layer where the second control is included in most applications on the market.

According to the first aspect, or any implementation manner of the above first aspect, when the layer type of the second layer cannot be determined according to the control information of the second control, obtain the current display of the first application The currently displayed screen includes the second control; and the layer type of the second layer is determined according to the content displayed by the second control in the currently displayed screen. In this way, when the layer type of the second layer cannot be determined according to the control information of the second control, the image of the second layer where the second control is located can be accurately determined by analyzing the screen currently displayed by the first application. Layer type, thereby ensuring subsequent separate drawing of layers based on layer type.

According to the first aspect, or any implementation manner of the above first aspect, the first display screen and the second layer are generated according to the layer type of the first layer and the layer type of the second displaying a picture, and displaying the first picture on the screen of the first electronic device, and projecting the second display picture to the screen of the second electronic device for display, including: according to the layer of the first layer type and the layer type of the second layer, generate a first display picture and a second display picture, and cache the first display picture to the first display cache, and cache the second display picture to the second display cache; according to the cache order, take the first display picture from the first display cache, and display the first display picture on the screen of the first electronic device; Record the second display screen in the screen to obtain the screen projection content; send the screen projection content to the second electronic device for the second electronic device to decode the screen projection content to obtain the second screen projection content Displaying the picture, and displaying it on the screen of the second electronic device. In this way, by buffering the display screens that need to be displayed on the screens of different electronic devices into different display buffers, and then taking out the display screens from the corresponding display buffers for display, it is possible to realize batch processing of cached content and avoid Thread congestion ensures the smoothness of the transmitted display screen.

According to the first aspect, or any implementation manner of the above first aspect, the first display screen and the second layer are generated according to the layer type of the first layer and the layer type of the second displaying a picture, and caching the first display picture into a first display cache, and caching the second display picture into a second display cache, including: determining a first layer filter rule corresponding to the first display cache and A second layer filtering rule corresponding to the second display cache; according to the first layer filtering rule, the layer type of the first layer, and the layer type of the second layer, determine the The first display screen includes the first layer and the second layer; obtaining the resource of the first control in the first layer and the resource of the second control in the second layer, Generate the first display picture according to the resources of the first control and the resources of the second control, and cache the first display picture in the first display cache; according to the filtering rule of the second layer, The layer type of the first layer and the layer type of the second layer, determine that the second display screen includes the first layer; obtain the first control in the first layer resources, generate the second display screen according to the resources of the first control, and cache the second display screen in the second display cache. In this way, by setting different layer filtering rules for different display buffers, when generating a display screen, the layer filtering rules corresponding to the display buffer and the determined layer types of each layer can be used to determine the cached layer in the display buffer. The control that needs to be included in the display screen is obtained, and the resources of the control are obtained to draw the display screen, so as to obtain a display screen suitable for display by different electronic devices.

According to the first aspect, or any implementation manner of the above first aspect, determining the first layer filter rule corresponding to the first display cache and the second layer filter rule corresponding to the second display cache, The method includes: obtaining the first device identifier of the first electronic device and the second device identifier of the second electronic device; searching the layer filter rule table that matches the first device identifier in the layer filter rule table, and The found layer filter rule is determined as the first layer filter rule corresponding to the first display cache; and the layer filter rule matching the second device identifier is searched in the layer filter rule table, Determining the found layer filtering rule as the second layer filtering rule corresponding to the second display cache. In this way, by pre-determining and storing the layer filtering rules, when layer filtering is required, the existing layer filtering rules can be obtained directly, which is convenient and fast.

According to the first aspect, or any implementation manner of the above first aspect, determining the first layer filter rule corresponding to the first display cache and the second layer filter rule corresponding to the second display cache, The method includes: displaying on the screen of the first electronic device a user-operable layer filtering rule decision interface, where the layer filtering rule decision interface includes the first control and the first control where the first control is located. A layer type of a layer, the second control and the layer type of the second layer where the second control is located; in response to the user setting the first layer filter for the first display cache The operation behavior of the rule is to generate the first layer filtering rule; in response to the user's operation behavior of setting the second layer filtering rule for the second display cache, the second layer filtering rule is generated. In this way, by providing a user operation entry, the user decides the layer filtering rules of the current first application, which not only improves user participation, but also makes the screen projection scene better adapt to different user needs.

According to the first aspect, or any implementation manner of the above first aspect, the recording the second display screen in the second display cache to obtain the projected screen content includes: obtaining the first electronic The first screen aspect ratio of the device and the second screen aspect ratio of the second electronic device; when the first screen aspect ratio is different from the second screen aspect ratio, all the Perform black border removal processing on the second display screen, and record the second display screen after black border removal to obtain the screen projection content; when the aspect ratio of the first screen is the same as that of the second screen, Recording the second display picture in the second display cache to obtain screen projection content. In this way, black border removal processing is introduced, so as to ensure that the second display screen displayed on the second electronic device has no black borders, or reduce the black borders as much as possible, thereby improving the viewing experience of the user watching the projected screen.

According to the first aspect, or any implementation manner of the above first aspect, the recording the second display screen in the second display cache to obtain the projected screen content includes: obtaining the second electronic The display capability of the device; determining the refresh frame rate of the video stream according to the display capability; recording the second display picture in the second display buffer according to the refresh frame rate of the video stream to obtain screen projection content. In this way, when recording the projected content, the first electronic device first negotiates with the second electronic device on the refresh frame rate of the video stream, so as to ensure that the transmitted video stream can be displayed normally on the second electronic device, and avoid bandwidth transition occupy.

According to the first aspect, or any implementation manner of the above first aspect, the first application is a conference application, the first control is a video control, and the layer type of the first layer is video streaming playback layer; the second control is a button control, and the layer type of the second layer is a conference control button layer. In this way, the first display picture displayed on the screen of the first electronic device not only includes the video picture of the participants displayed through the video stream playback control, but also includes the conference control button control that can be operated by the user, while the second electronic device The second display picture displayed on the screen of the device only includes the video pictures of the participants displayed through the video stream playback control, and does not include the conference control button controls that can be operated by the user, so that users who participate in the conference through the first electronic device can either Viewing the conference screen, you can also operate the conference control buttons, and the user watching the conference through the second electronic device will not be disturbed by the operation performed by the first electronic device.

According to the first aspect, or any implementation manner of the above first aspect, the first application is a conference application, the first control is a whiteboard annotation control, and the layer type of the first layer is a whiteboard annotation map layer; the second control is a button control, and the layer type of the second layer is a conference control button layer. In this way, the first display picture displayed on the screen of the first electronic device includes not only the whiteboard content, but also the conference control button controls that can be operated by the user, while the second display picture displayed on the screen of the second electronic device It only includes the content of the whiteboard, and does not include the conference control button controls that can be operated by users, so that users who participate in the conference through the first electronic device can not only watch the drawing on the whiteboard, but also operate the conference control buttons, and watch through the second electronic device Users in the conference will not be disturbed by operations performed on the side of the first electronic device.

In a second aspect, an electronic device is provided. The electronic device is a first electronic device, the first electronic device is installed with a first application, the first application includes a first control and a second control, the first control is located on the first layer, and the first Two controls are located on the second layer, the electronic device includes: one or more processors; memory; and one or more computer programs, wherein the one or more computer programs are stored on the memory, when the When the computer program is executed by the one or more processors, the electronic device executes the following steps: acquiring the control information of the first control and the control information of the second control; information to determine the layer type of the first layer; determine the layer type of the second layer according to the control information of the second control, and the layer type of the first layer is the same as that of the second control The layer types of the layers are different; according to the layer type of the first layer and the layer type of the second layer, a first display screen and a second display screen are generated, and displayed on the first electronic device The screen displays the first picture, and projects the second display picture to the screen of the second electronic device for display, the first display picture includes the first control and the second control, and the second display The screen includes the first control and does not include the second control.

According to the second aspect, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps: extracting the control name and the control name of the first control from the control information of the first control The size information of the first control; parsing the control name, and parsing the control type of the first control, the package name of the first application and the location of the first control from the control name interface information, determine the layer type of the first layer according to the control type, the package name, the interface information and the size information; after parsing the first layer from the control name When the control type of the control and the interface information where the first control is located, if the package name of the first application is not resolved, obtain the process identification number PID of the process that draws the first control, and determine the PID according to the PID The source of the first control is to determine the layer type of the first layer according to the control type, the source, the interface information and the size information, and the source includes the package name of the first application .

According to the second aspect, or any implementation manner of the above second aspect, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps: combining the control name and the Dimension information is used as a retrieval keyword; according to the keyword, search for a control that matches the keyword in the layer identification record library; when finding a control that matches the keyword, the image corresponding to the control The layer type is determined as the layer type of the first layer; when no control matching the keyword is found, the step of parsing the control name is performed.

According to the second aspect, or any implementation manner of the above second aspect, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps: When the control information cannot determine the layer type of the first layer, obtain the screen currently displayed by the first application, and the currently displayed screen includes the first control; according to the currently displayed screen The content displayed by the first control determines the layer type of the first layer.

According to the second aspect, or any implementation manner of the above second aspect, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps: from the second control Extract the control name of the second control and the size information of the second control from the control information; analyze the control name, and analyze the control type of the second control, the control type of the second control from the control name When the package name of an application and the interface information where the second control is located, determine the layer type of the second layer according to the control type, the package name, the interface information and the size information; Analyzing the control type of the second control and the interface information of the second control from the control name, and obtaining the process of drawing the second control when the package name of the first application is not parsed A process identification number PID, determining the source of the second control according to the PID, determining the layer type of the second layer according to the control type, the source, the interface information and the size information, The source includes a package name of the first application.

According to the second aspect, or any implementation manner of the above second aspect, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps: When the control information cannot determine the layer type of the second layer, obtain the currently displayed screen of the first application, and the currently displayed screen includes the second control; according to the currently displayed screen The content displayed by the second control determines the layer type of the second layer.

According to the second aspect, or any implementation manner of the above second aspect, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps: according to the first layer The layer type of the layer type and the layer type of the second layer, generate a first display screen and a second display screen, and cache the first display screen in the first display buffer, and cache the second display screen to the second display cache; according to the cache order, take the first display picture from the first display cache, and display the first display picture on the screen of the first electronic device; The second display screen in the second display buffer is recorded to obtain screen projection content; the screen projection content is sent to the second electronic device for the second electronic device to decode the screen projection content to obtain the screen projection content The second display screen is displayed on the screen of the second electronic device.

According to the second aspect, or any implementation manner of the above second aspect, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps: determine the first display cache The corresponding first layer filter rule and the second layer filter rule corresponding to the second display cache; according to the first layer filter rule, the layer type of the first layer and the second image The layer type of the layer, determine that the first display screen includes the first layer and the second layer; obtain the resources of the first control in the first layer and the second layer According to the resource of the second control, generate the first display picture according to the resource of the first control and the resource of the second control, and cache the first display picture in the first display cache; according to The second layer filter rule, the layer type of the first layer, and the layer type of the second layer determine that the second display screen includes the first layer; obtain the second layer The resource of the first control in a layer is used to generate the second display frame according to the resource of the first control, and cache the second display frame in the second display cache.

According to the second aspect, or any implementation manner of the above second aspect, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps: acquiring the first electronic device The first device identifier of the second electronic device and the second device identifier of the second electronic device; the layer filter rule matching the first device identifier is searched in the layer filter rule table, and the found layer filter rule is determined as The first layer filter rule corresponding to the first display cache; look up the layer filter rule matching the second device identifier in the layer filter rule table, and determine the found layer filter rule Cache the second layer filter rule corresponding to the second display.

According to the second aspect, or any implementation manner of the above second aspect, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps: on the first electronic device A layer filter rule decision interface that can be operated by the user is displayed on the screen, and the layer filter rule decision interface includes the first control and the layer type of the first layer where the first control is located, so The second control and the layer type of the second layer where the second control is located; in response to the user's operation behavior of setting the filter rule of the first layer for the first display cache, generating the second control A layer filtering rule; generating the second layer filtering rule in response to the user's operation behavior of setting the second layer filtering rule for the second display cache.

According to the second aspect, or any implementation manner of the above second aspect, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps: acquiring the first electronic device The first screen aspect ratio of the second electronic device and the second screen aspect ratio of the second electronic device; when the first screen aspect ratio is different from the second screen aspect ratio, the Perform black border removal processing on the second display picture, and record the second display picture after black border removal to obtain the screen projection content; when the first screen aspect ratio is the same as the second screen aspect ratio, The second display picture in the second display cache is recorded to obtain screen projection content.

According to the second aspect, or any implementation manner of the above second aspect, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps: acquiring the second electronic device The display capability; determine the video stream refresh frame rate according to the display capability; according to the video stream refresh frame rate, record the second display picture in the second display buffer to obtain screen projection content.

The second aspect and any implementation manner of the second aspect correspond to the first aspect and any implementation manner of the first aspect respectively. For technical effects corresponding to the second aspect and any implementation manner of the second aspect, reference may be made to the technical effects corresponding to the above-mentioned first aspect and any implementation manner of the first aspect, and details are not repeated here.

In a third aspect, a computer-readable storage medium is provided. The medium includes a computer program, and when the computer program is run on the electronic device, the electronic device is made to execute the first aspect and the screen projection method in any one of the first aspect. Exemplarily, the electronic device may be a mobile phone.

The third aspect and any implementation manner of the third aspect correspond to the first aspect and any implementation manner of the first aspect respectively. For technical effects corresponding to the third aspect and any implementation manner of the third aspect, reference may be made to the technical effects corresponding to the above-mentioned first aspect and any implementation manner of the first aspect, and details are not repeated here.

In a fourth aspect, an embodiment of the present application provides a computer program, where the computer program includes instructions for executing the method in the first aspect and any possible implementation manner of the first aspect.

The fourth aspect and any implementation manner of the fourth aspect correspond to the first aspect and any implementation manner of the first aspect respectively. For the technical effects corresponding to the fourth aspect and any one of the implementation manners of the fourth aspect, refer to the above-mentioned first aspect and the technical effects corresponding to any one of the implementation manners of the first aspect, and details are not repeated here.

In a fifth aspect, the embodiment of the present application provides a chip, and the chip includes a processing circuit and sending and receiving pins. Wherein, the transceiving pin and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the method in the second aspect or any possible implementation of the second aspect to control the receiving pin to receive signals, so as to Control the send pin to send signal. Exemplarily, the chip is a chip of an electronic device, and the electronic device may be a mobile phone.

The fifth aspect and any implementation manner of the fifth aspect correspond to the first aspect and any implementation manner of the first aspect respectively. For the technical effects corresponding to the fifth aspect and any one of the implementation manners of the fifth aspect, refer to the technical effects corresponding to the above-mentioned first aspect and any one of the implementation manners of the first aspect, and details are not repeated here.

Description of drawings

Fig. 1 is one of the schematic diagrams of scenarios where the screen projection function is enabled;

Fig. 2 is the second schematic diagram of the scenario where the screen projection function is enabled;

Fig. 3 is one of the schematic diagrams of the terminal and the large-screen display content after screen projection using the screen projection method provided by the embodiment of the present application;

Fig. 4 is a schematic diagram of the software structure of the mobile phone shown exemplarily;

Fig. 5 is a schematic diagram of the layer structure included in the picture displayed by the conference application;

Fig. 6 is a schematic diagram of the modules included in the mobile phone and the large screen;

Fig. 7 is a schematic flow chart of a screen projection method provided by an embodiment of the present application;

Fig. 8 is a schematic diagram of control information acquired in the screen projection method provided by the embodiment of the present application;

FIG. 9 is a schematic diagram of an interface of filtering rules determined by a user decision layer provided by an embodiment of the present application;

Fig. 10 is an exemplary illustration of one of the module interaction schematic diagrams for displaying different screens on the mobile phone and the large screen using the screen projection method provided by the embodiment of the present application;

Fig. 11 is one of the sequence diagrams schematically showing the screen projection method provided by the embodiment of the present application to draw the screen to be displayed;

Fig. 12 is an exemplary diagram showing the second sequence diagram of drawing a picture to be displayed using the screen projection method provided by the embodiment of the present application;

Fig. 13 is the second schematic diagram of the scene of the terminal and the content displayed on the large screen after screen projection using the screen projection method provided by the embodiment of the present application;

Fig. 14a and Fig. 14b are schematic diagrams showing the third scene of the content displayed on the terminal and the large screen after screen projection using the screen projection method provided by the embodiment of the present application;

Fig. 15 is an exemplary illustration of one of the module interaction schematic diagrams for displaying different screens on the mobile phone and the large screen using the screen projection method provided by the embodiment of the present application;

Fig. 16a is a schematic diagram showing the layer relationship between the physical screen and the virtual screen of the mobile phone in the embodiment of the present application;

Fig. 16b is a schematic diagram of a method for establishing a screen coordinate system in an exemplary embodiment of the present application;

Fig. 17 is a schematic structural diagram of a device provided by an embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations.

The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of objects. For example, the first target object, the second target object, etc. are used to distinguish different target objects, rather than describing a specific order of the target objects.

In the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

Before describing the technical solutions of the embodiments of the present application, firstly, the application scenarios of the embodiments of the present application will be described with reference to the accompanying drawings. For ease of description, the embodiment of the present application takes the content of the screen to be projected as an example of a conference using a conference application program, and uses a mobile phone as an electronic device to project the screen of the conference process, and uses a TV as a display A large screen for projecting images.

The process of enabling the screen projection function of the mobile phone will be described below with reference to FIG. 1 .

Referring to FIG. 1 , for example, the display interface of the mobile phone 100 displays a setting page 10a of the mobile phone, and the setting page 10a includes one or more controls, such as sound and vibration setting options, notification setting options, device connection setting options, application Settings options, battery settings options, storage settings options, security settings options, etc.

Exemplarily, after the user clicks the device connection 10a-1 on the setting page 10a, the mobile phone jumps from the setting page 10a to the device connection page 10b in response to the user's operation behavior.

Exemplarily, the device connection page 10b includes one or more controls, such as Bluetooth setting options, NFC (Near Field Communication, near field communication) setting options, mobile phone screen projection setting options, USB (Universal Serial Bus, Universal Serial Bus ) setting options and print setting options, etc.

Exemplarily, after the user clicks the mobile phone screen projection 10b-1 in the device connection page 10b, the mobile phone jumps from the device connection page 10b to the mobile phone screen projection page 10c in response to the user's operation behavior.

Exemplarily, the mobile phone screen projection page 10c includes a control for enabling the mobile phone screen projection function, such as the wireless screen projection setting option 10c-1 shown in FIG. 1 .

Understandably, in addition to being named "wireless screen projection" in Figure 1, the setting options for enabling the screen projection function of the mobile phone displayed on the screen projection page 10c of the mobile phone, in actual application scenarios, vary depending on the model of the mobile phone and the version of the system. Different, it may also be named "multi-screen interaction", "screen mirroring", etc., which will not be listed here, and there is no limitation in this implementation.

Exemplarily, when the user clicks the wireless screen projection setting option 10c-1 in the screen projection page 10c of the mobile phone, the mobile phone will display a list of available devices in the blank area of the screen projection page 10c in response to the user's operation, and use Control 10c-2 displays the content "Searching for available devices... Please ensure that the wireless projection function of the large-screen device is turned on" in the display area of the available device list.

It can be understood that what is shown in FIG. 1 is only a specific display style of the available device list when searching for available large-screen devices. It is an example for better understanding the technical solution of this embodiment. as the only limitation on this example. In an actual application scenario, after the user clicks the wireless screen projection setting option 10c-1 in the mobile phone screen projection page 10c, the mobile phone responds to the user's operation behavior and specifically jumps from the mobile phone screen projection page 10c to a dedicated The Available Devices List page is displayed.

Exemplarily, after searching for available large-screen devices, the control 10c-3 will be used to display the searched large-screen devices, such as large-screen 1 and large-screen 2, in the display area of the available device list.

Understandably, in an actual application scenario, the searched large-screen devices may be TVs, projectors, etc., which are not listed here, and are not limited in this implementation.

Exemplarily, in an actual application scenario, when the large-screen device is a TV, the presented screen may be a TV screen, or a large screen composed of multiple TV screens, which is not limited in this application .

Exemplarily, when the user clicks on the large screen 1 (10c-3-1) in the screen projection page 10c of the mobile phone, the mobile phone responds to the user's operation behavior, initiates a pairing request to the large screen 1, and establishes a network connection, and then the mobile phone The content displayed on the display interface will be projected onto the large screen 1 .

In this way, the operation of turning on the screen mirroring function on the mobile phone by using the setting page as the entry is completed.

In addition, the embodiment of the present application also provides another method for starting the screen projection function. The process of enabling the screen projection function of the mobile phone will be described below with reference to FIG. 2 .

Referring to FIG. 2 , for example, the display interface of the mobile phone 100 shows a picture 20 during a conference using a conference application program. When the user slides down from the upper edge of the mobile phone in the direction of the arrow, the mobile phone responds to the user's operation Behavior, the drop-down notification bar 30 is displayed on the upper edge area of the display interface.

Exemplarily, the drop-down notification bar 30 includes one or more controls, such as a time bar, Wi-Fi setting options, Bluetooth setting options, mobile data setting options, automatic rotation setting options, and screen mirroring setting options.

Exemplarily, after the user clicks the screen image 30-1 in the drop-down notification bar 30, the mobile phone may pop up an interface for searching available devices on the display interface in response to the user's operation behavior, and after searching for available large-screen devices , displayed on this interface, for the user to select the large-screen device that needs to be paired and establish a network connection.

Understandably, the interface for searching available devices that pops up on the display interface may cover the entire display interface in full screen, or may only cover a partial area, and the specific implementation manner is not limited in this application.

In this way, the operation of turning on the screen mirroring function on the mobile phone is completed by using the pull-down notification bar as the entry point.

Exemplarily, after the method of enabling the screen projection function shown in Figure 1 or Figure 2 is used to establish a network connection between the mobile phone and the large screen, the screen projection method provided by the embodiment of the present application is used to project the screen of the current meeting, The operating system of the mobile phone will determine the layer where each control in the screen to be displayed is located, and then filter the screen that needs to be displayed on the screen of the mobile phone and the screen that needs to be displayed on a large screen (such as a TV screen) according to the preset filtering rules , and then perform composite rendering on the filtered layers, and finally send the obtained pictures to the display separately, so that the mobile phone and the large screen can display different content during the conference respectively.

Referring to FIG. 3 , for example, the display interface of the mobile phone 100 displays a screen 20 during a conference, and the screen 20 includes a video stream playback layer 20-1 and a conference control button layer 20-2.

Understandably, the video stream playback layer 20-1 includes one or more video stream playback controls, and these video stream playback controls are used to display the video stream acquired during the conference.

In addition, it should be noted that in an example, multiple video stream playback controls can be integrated into one video stream playback layer. For this application scenario, when the server corresponding to the conference application transmits the video stream to the mobile phone , the video streams corresponding to multiple video stream playback controls can be combined into one for transmission.

Correspondingly, in another example, each video stream playback control can be set to correspond to a video stream playback layer. For this application scenario, when the server corresponding to the conference application transmits the video stream to the mobile phone, it needs to send the video stream to different The video streaming playback control streams the video.

It should be understood that the above description is only an example for better understanding of the technical solution of this embodiment, and is not the only limitation to this embodiment.

In addition, referring to FIG. 3 , the conference control button layer 20-2 may include one or more controls, such as mute setting options, video setting options, sharing setting options, participant setting options and more setting options, etc., which are no longer described here. List them one by one, and this application does not limit them.

For the screen 20 displayed on the mobile phone 100 in FIG. 3 , it is assumed that the preset filtering rule is that the content of the screen projected on the large screen 200 only includes video stream playback controls for displaying video streams, that is, the screen on the large screen 200 includes the graphics The layer only has the video stream playback layer. Based on this, for the picture 20 displayed on the display interface of the mobile phone 100, after the processing of the screen projection method provided by the embodiment of the present application, the picture 20' finally projected on the display interface of the large screen 200 only includes the video stream playback layer 20 The mirror content of -1 is 20-1'. In this way, even if the user operates the conference control button in the conference control button layer 20-2 on the mobile phone, such as muting, adding participants, etc., the display interface of the large screen always displays the mirror image of the video stream playback control 20-1 Content 20-1', the operation process of the conference control button in the conference control button layer 20-2 will not be projected to the large screen, so as not to interfere with the user watching the video stream displayed on the large screen, and realize the separation of control and display .

In addition, it should be noted that the descriptions of Figures 1 to 3 and subsequent embodiments relate to the names and quantities of the controls displayed on the display interface of the mobile phone involved in the drawings, as well as the names and quantities of the controls in the drop-down notification bar It is only an illustrative example, and the present application does not limit it.

In addition, it should be noted that the screen projection method provided in the embodiment of the present application can not only be applied to the application scenario of one-to-one projection, but also applicable to the application scenario of one-to-many projection. As long as the mobile phone with the screen projection function is turned on, Or other electronic devices support one-to-one screen projection, and one-to-many screen projection is sufficient. The specific implementation details of one-to-one screen projection and one-to-many screen projection are not described in this application.

In addition, it can be understood that, in the description of the embodiments of the present application, a mobile phone is taken as an example for illustration, and in other embodiments, the present application is also applicable to laptop computers, desktop computers, palmtop computers (such as tablet computer) and other electronic devices that support screen mirroring.

In order to better describe the screen projection method provided by the embodiment of the present application, the conference application program is installed, and the electronic device for screen projection is a mobile phone. The software structure of the mobile phone is described in conjunction with FIG. 4 .

Referring to FIG. 4 , FIG. 4 is a software structural block diagram of the mobile phone 100 according to the embodiment of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces.

For ease of description, the embodiment of the present application takes the Android system as an example to describe the software structure of the mobile phone 100 with the Android system.

Specifically, in some embodiments, the Android system is divided into five layers, from top to bottom are application program layer, application program framework layer (also known as: system framework layer), system library and Android runtime layer, hardware Abstraction layer (hardware abstraction layer, HAL) and kernel layer.

The application layer may include application programs such as camera, gallery, calendar, WLAN, meeting, music, and video (hereinafter referred to as applications for short). It should be noted that the applications included in the application program layer shown in FIG. 4 are only illustrative, and the present application does not limit this. Understandably, the applications included in the application layer do not constitute a specific limitation on the mobile phone 100 . In other embodiments of the present application, the mobile phone 100 may include more or fewer applications than the applications contained in the application program layer shown in FIG. Applications.

The application framework layer provides an application programming interface (Application Programming Interface, API) and a programming framework for the application of the application layer, including various components and services to support the developer's Android development. The application framework layer also includes some predefined functions. As shown in Figure 4, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, a camera service, and the like.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.

Content providers are used to store and retrieve data and make it accessible to applications. Data can include videos, images, audio, calls made and received, browsing history and bookmarks, phonebook, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the download completion, message reminder, etc. The notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, prompting text information in the status bar, issuing a prompt sound, vibrating the electronic device, and flashing the indicator light, etc.

The camera service is used to call a camera (including a front camera and/or a rear camera) in response to an application request.

In addition, in order to implement the screen projection method provided by the embodiment of the present application, the application framework layer further includes a display management framework and a display rendering framework. Among them, the display management framework is used to identify the layers to which the controls included in the conference application belong, and mark and record them; the display rendering framework is used to filter the layers identified by the display management framework according to preset filtering rules, and Filtered layers for composite rendering.

Exemplarily, for the screen 20 displayed on the display interface of the mobile phone 100 in FIG. 3, in one example, there are two layers, the video stream playback layer 20-1 and the conference control button layer 20-2 shown in FIG. 5 Composite rendering.

Specifically, the display management framework located in the application framework layer identifies the controls drawn by the conference application installed in the application layer, and then determines the layer type of each control, and marks them, for example, by identifying Stream controls, such as the layer type corresponding to the SurfaceView control is the video stream playback layer, that is, 20-1 in Figure 5, for the mute option setting control, video option setting control, sharing option setting control, participant option setting control, etc. The layer type corresponding to the conference control button is the conference control button layer, that is, 20-2 in FIG. 5 .

System library and Android runtime layer includes system library and Android runtime (Android Runtime). A system library can include multiple function modules. For example: surface manager, 2D graphics engine, 3D graphics processing library (eg: OpenGL ES), media library, font library, etc. Among them, the browser kernel is responsible for interpreting the syntax of the webpage (such as an application HTML and JavaScript under the standard general markup language) and rendering (displaying) the webpage; layer processing, etc.; the 3D graphics processing library is used to realize 3D graphics drawing, image rendering, synthesis and layer processing, etc.; the media library is used to realize the input of different streaming media; the font library is used to realize the input of different fonts.

The Android runtime is responsible for the scheduling and management of the Android system, including the core library and virtual machine. Wherein, the core library includes two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android; the virtual machine is used to run the Android application developed using the java language.

In addition, it should be noted that in order to enable the Android application to run in the virtual machine, both the application program layer and the application program framework layer need to run in the virtual machine. When running an Android application, the virtual machine executes the java files of the application program layer and the application program framework layer as binary files.

In addition, it should be noted that in practical applications, virtual machines are used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

Understandably, the components contained in the application framework layer, system library and runtime layer shown in FIG. 4 do not constitute a specific limitation on the mobile phone 100 . In practical applications, the mobile phone 100 may include more or fewer components than shown, or combine some components, or separate some components, or arrange different components.

The HAL layer is an interface layer between the operating system kernel and the hardware circuit. The HAL layer includes but is not limited to: Audio Hardware Abstraction Layer (Audio HAL) and Camera Hardware Abstraction Layer (Camera HAL). Among them, the Audio HAL is used to process the audio stream, for example, to perform noise reduction and directional enhancement on the audio stream, and the Camera HAL is used to process the image stream.

The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver. Wherein, the hardware may include devices such as a camera, a display screen, a microphone, a processor, and a memory.

In the embodiment of the present application, the display screen in the hardware can display pictures during the meeting, the camera in the hardware can be used to collect images, and the microphone in the hardware can be used to collect sound signals and generate analog audio electrical signals.

In addition, it should be noted that in an actual application scenario, in order to implement the screen projection method provided by the embodiment of the present application, an electronic device (such as a mobile phone) used to project content and a large screen used to display the content projected by the electronic device (such as TV) need to include at least the content shown in Figure 6.

Take the content that needs to be projected as an example during the meeting, see Figure 6. The mobile phone used for projecting content needs to install the conference application at least in the application layer, and introduce a map for recording layer identification information in the application framework layer. The layer identification record library, the layer identification module used to identify the layer corresponding to the control, the layer filter module used to filter the layer identified by the layer identification module, and the image filtered by the layer filter module Composite rendering module for compositing rendering at the layer, as well as collaborative assistant and network communication modules at the system library and Android runtime layer.

Wherein, the layer identification record library and the layer identification module are specifically located in the display management framework of the application program framework layer, and the layer filtering module and the composite rendering module are specifically located in the display rendering framework of the application program framework layer.

Since the large screen used to display the content projected by the mobile phone only needs to be used to display the projected content, and does not need to perform layer identification and composite rendering, the large screen must at least include a projection display module located in the application framework layer and a Synthetic rendering module for synthesizing and rendering the content transmitted from the mobile phone, and a collaborative assistant and network communication module located in the system library and Android runtime layer.

Based on the above structure, in the actual application scenario, the user triggers the operation of enabling the screen projection function through the method shown in Figure 1 or Figure 2. The collaborative assistant at the time layer is matched to a large screen that also has a collaborative assistant, that is, supports the screen projection function, and establishes a communication connection with the matched large screen through the network communication module.

When the user starts the meeting application and joins the meeting, the meeting application at the application layer will interact with the layer identification record library and the layer identification module in the display management framework of the application framework layer. After the layer is identified, it will be filtered by the layer filtering module in the display rendering framework of the application framework layer, and the filtered layer will be composited and rendered by the composite rendering module, so as to obtain the picture that needs to be displayed on the mobile phone screen (such as Screen A) and the screen that needs to be displayed on the large screen (such as screen B).

Exemplarily, after the screen A and the screen B are obtained, the screen A can be directly sent for display through the display driver of the mobile phone, and then the screen A is displayed on the screen of the mobile phone.

In addition, it is understandable that for the screencasting function, the mirroring process is actually performed based on the mirroring protocol, that is, the screen B is recorded in the system library of the mobile phone and the screencasting recording module of the Android runtime layer, and after the recording is completed, with the help of The communication connection established by the network communication module is transmitted to the large screen, and the large screen performs display processing through the projection display module, and finally presents picture B on the large screen.

In addition, it should be noted that in an actual application scenario, the application to be projected is not limited to a meeting application, and the above description is only for illustration.

In order to have a systematic understanding of the screen projection method provided in this embodiment, a specific description will be made below in conjunction with FIG. 7 .

First of all, it should be noted that the screen projection method provided in this embodiment is specifically applied to the first electronic device that initiates screen projection, for example, a mobile phone.

Exemplarily, the first electronic device is installed with the first application, and for ease of understanding, the meeting application is still taken as an example below.

Exemplarily, the first application includes a first control and a second control, and the first control is located on the first layer, and the second control is located on the second layer.

Exemplarily, the first control and the second control are different types of controls.

Exemplarily, in an implementation scenario, the first control is a video control, such as SurfaceView, and correspondingly, the layer type of the first layer is a video stream playback layer; the second control is a button control, For example, it may be Button. Correspondingly, the layer type of the second layer is a conference control button layer.

Exemplarily, in another implementation scenario, the first control is a whiteboard annotation control, such as BlankWindow, and correspondingly, the layer type of the first layer is a whiteboard annotation layer; the second control is a button control, For example, it may be Button. Correspondingly, the layer type of the second layer is a conference control button layer.

Referring to Figure 7, the screen projection method provided by this application specifically includes the following steps:

Step S1: Obtain control information of the first control and control information of the second control.

For ease of description, this embodiment still takes the first application as a meeting application as an example, and the premise of executing step S1 is that the user has completed the mobile phone 100 and TV 200 using the method of enabling the screen projection function shown in Figure 1 or Figure 2 above. communication link between.

In addition, it should be noted that the interface of any application is usually rendered and synthesized by multiple layers, and there are at least two types of layers included. Taking the conference application as an example, the included layers include at least video stream Playback layer and conference control button layer.

Exemplarily, each type of layer includes at least one control. That is, the number of first controls in the first layer may be one or more, and the number of second controls in the second layer may also be one or more. For example, in a conference application, the video stream playback layer includes one or more video stream playback controls, and the conference control button layer includes one or more conference control button controls. See the description in Figure 3 for details. This time No longer.

Exemplarily, the operation of obtaining the control information of the first control and the control information of the second control may be obtained by calling a preset control information grabbing program by the display processing module located in the application framework layer in the mobile phone 100 .

Step S2: Determine the layer type of the first layer according to the control information of the first control.

Exemplarily, taking the Android system as an example of the operating system of the mobile phone, the format of the obtained control information of each control in the conference application may be as shown in FIG. 8 .

Referring to FIG. 8, the obtained control information includes but is not limited to the control name (Name) and size information (disp frame), such as the window type, which is not limited in this application.

Assume that the first control has two for displaying video images of participants, such as control a and control b, and has a second control for users to operate, such as control c. For example, "SurfaceView-com.huawei.welink/com.[...]on.view.activity.InMeetingActivity#0rel-2" in Figure 8 is the control name of control a, and "0 0 283 283" is Size information of control a; "SurfaceView-com.huawei.welink/com.[...]on.view.activity.InMeetingActivity#1rel-1" is the control name of control b, and "0 283 2288 1080" is control b size information; "Button-com.huawei.welink/com.[...]on.view.activity.InMeetingActivity#0rel-0" is the control name of control c, and "413 2130 625 2200" is the size of control c information.

Among them, the control name usually includes three parts. Taking the control name of control a as an example, "SurfaceView" indicates the control type, which indicates that control a is used to display video stream content, and "com.huawei.welink" indicates the location of control a. The application package name of the application. The application type can be determined according to the application package name. For example, welink is a meeting application, and "on.view.activity.InMeetingActivity" indicates the current display interface meeting interface, that is, the interface information is the meeting interface.

To sum up, when determining the layer type of the first layer according to the control information of the first control, the process is as follows:

Firstly, the control name of the first control and the size information of the first control are extracted from the control information of the first control. For example, the control name of control a is "SurfaceView-com.huawei.welink/com.[...]on.view.activity.InMeetingActivity#0rel-2", and the size information is "0 0 283 283".

Then, analyze the control name.

Specifically, when parsing the control type of the first control, the package name of the first application, and the interface information of the first control from the control name, according to the control type, the package name , the interface information and the size information determine the layer type of the first layer. In this way, the purpose of the first control can be determined according to the control type of the first control, the application type of the first application can be determined according to the package name of the first application, and then according to the purpose of the first control, the application type of the first application , combined with the size information of the first control and the specific interface information in the first application, it is possible to accurately identify the layer type of the layer where the first control is included in most applications on the market.

When the control type of the first control and the interface information of the first control are parsed from the control name, and the package name of the first application is not parsed, obtain the process of drawing the first control The process identification number PID, determine the source of the first control according to the PID, determine the layer type of the first layer according to the control type, the source, the interface information and the size information , the source includes the package name of the first application. In this way, when the package name of the first application is not parsed from the control name of the first control, by obtaining the PID of the process drawing the first control, the source of the process corresponding to the PID can be determined according to the unique PID, that is, The first application, and then obtain the package name of the first application, so that regardless of whether the control name of the first control contains the package name of the first application, the layer type of the first layer can be accurately determined according to the control information of the first control .

In addition, in order to improve the overall efficiency, the layer identification record library can be set in advance, and the known controls of various sizes and positions in various applications can be associated with the layer type of the layer where they are located. Analyze the control name of a control. Before determining the layer type according to the analysis result and position information, first use the control name and size information as the search keywords, and then search for the control matching the keyword in the layer identification record library according to the keywords. . Correspondingly, when a control matching the keyword is found, the layer type corresponding to the control is determined as the layer type of the first layer; when no control matching the keyword is found , after parsing the control name, determine the layer type according to the parsing result and location information. In this way, not only the layer type can be determined, but also the processing speed and the consumption of device resources can be taken into consideration.

In addition, if in the actual application scenario, the layer type of the first layer cannot be determined by searching the layer identification record library and determining the layer type of the first layer according to the control information of the first control, A screen currently displayed by the first application may also be acquired.

It should be noted that the currently displayed screen includes the first control.

Correspondingly, the layer type of the first layer is determined according to the content displayed by the first control in the currently displayed screen.

Specifically, the method may be manually determined by technicians through captured page data and then updated to the layer identification library, or it may be determined based on analysis and analysis based on a preset algorithm. For example, by analyzing and determining the content displayed in each control in the screen and the text corresponding to the icon of the control, the specific analysis process will not be repeated in this application.

It should be understood that the above are only several specific ways to determine the layer type of the control to be drawn, which are examples for better understanding the technical solution of this embodiment, and are not the only limitation to this embodiment .

Step S3: Determine the layer type of the second layer according to the control information of the second control.

Specifically, when determining the layer type of the second layer according to the control information of the second control, the process is as follows:

extracting the control name of the second control and the size information of the second control from the control information of the second control;

Parsing the control name, when parsing the control type of the second control, the package name of the first application, and the interface information of the second control from the control name, according to the control type , the package name, the interface information and the size information, determine the layer type of the second layer;

When the control type of the second control and the interface information of the second control are parsed from the control name, and the package name of the first application is not parsed, obtain the process of drawing the second control The process identification number PID, determine the source of the second control according to the PID, determine the layer type of the second layer according to the control type, the source, the interface information and the size information , the source includes the package name of the first application.

Similarly, before analyzing the control name of the second control and determining the layer type according to the analysis result and position information, first use the control name and size information as the retrieval keywords, and then search for the layer identification record library according to the keywords. Keyword matching controls. Correspondingly, when a control matching the keyword is found, the layer type corresponding to the control is determined as the layer type of the second layer; when no control matching the keyword is found , after parsing the control name, determine the layer type according to the parsing result and location information. In this way, not only the layer type can be determined, but also the processing speed and the consumption of device resources can be taken into consideration.

Similarly, when the layer type of the second layer cannot be determined according to the control information of the second control, the currently displayed screen of the first application is acquired, and the currently displayed screen includes the second control ; Determine the layer type of the second layer according to the content displayed by the second control in the currently displayed screen.

It is not difficult to find that the process of determining the layer type of the second layer according to the control information of the second control is roughly the same as determining the layer type of the first layer according to the control information of the first control in step S2, and will not be repeated this time.

In addition, it is understandable that the layer type of the first layer is different from the layer type of the second layer. For example, when the first control is a video control, such as SurfaceView, the layer type of the first layer is a video stream playback layer; when the first control is a whiteboard annotation control, the layer type of the first layer is whiteboard Annotation layer; when the second control is a button control, such as button, the layer type of the second layer is a conference control button layer.

Step S4: Generate a first display screen and a second display screen according to the layer type of the first layer and the layer type of the second layer, and display the The first frame, projecting the second display frame to the screen of the second electronic device for display.

Understandably, when the first application only includes the first control and the second control, the first display screen includes the first control and the second control, and the second display screen includes the first control , excluding the second control.

Exemplarily, in the application scenario of yes, the first application may also include controls located in other layers, and the correspondingly generated first display screen and second display screen may also include other controls according to business requirements. No restrictions.

Exemplarily, in an actual application scenario, a first display cache for caching the first display picture and a second display cache for caching the second display picture may be allocated in the first electronic device.

Correspondingly, after the first display picture and the second display picture are generated according to the layer type of the first layer and the layer type of the second layer, the first display picture may be cached in the The first display cache caches the second display picture in the second display cache; then, according to the cache order, fetches the first display picture from the first display cache, and stores the first display picture in the first display cache. displaying the screen of the first electronic device; recording the second display screen in the second display buffer to obtain screen projection content; sending the screen projection content to the second electronic device for The second electronic device decodes the projected screen content to obtain the second display picture, and displays it on the screen of the second electronic device. In this way, by buffering the display screens that need to be displayed on the screens of different electronic devices into different display buffers, and then taking out the display screens from the corresponding display buffers for display, it is possible to realize batch processing of cached content and avoid Thread congestion ensures the smoothness of the transmitted display screen.

In addition, it is understandable that in actual application scenarios, different layer filtering rules are set for different display caches, so that when generating a display screen, the layer filtering rules corresponding to the display cache and the determined layers of each layer can be The layer type determines the control to be included in the display screen cached in the display buffer, and then obtains the resources of the control to draw the display screen, and obtains a display screen suitable for display by different electronic devices.

Specifically, the process of generating the first display screen and the second display screen according to the layer type of the first layer and the layer type of the second layer, and caching the generated display screens to the corresponding display cache is as follows : First, determine the first layer filter rule corresponding to the first display cache and the second layer filter rule corresponding to the second display cache; then, according to the first layer filter rule, the first image The layer type of the layer and the layer type of the second layer, determine that the first display screen includes the first layer and the second layer; then, obtain all the layers in the first layer resources of the first control and resources of the second control in the second layer, generate the first display screen according to the resources of the first control and the resources of the second control, and convert the The first display screen is cached to the first display cache; then, according to the second layer filtering rule, the layer type of the first layer, and the layer type of the second layer, determine the second layer The second display screen includes the first layer; then, acquire the resource of the first control in the first layer, generate the second display screen according to the resource of the first control, and convert the first control The second display frame is buffered into the second display buffer.

Exemplarily, this embodiment provides two ways of determining layer filtering rules, which will be described respectively below.

Method 1: Select the first layer filter rule and the second layer filter rule from the predetermined layer filter rule table

Obtain the first device identifier of the first electronic device and the second device identifier of the second electronic device; search the layer filter rule matching the first device identifier in the layer filter rule table, and find The layer filter rule is determined as the first layer filter rule corresponding to the first display cache; the layer filter rule matching the second device identifier is searched in the layer filter rule table, and the search is performed The obtained layer filtering rule is determined as the second layer filtering rule corresponding to the second display cache. In this way, by pre-determining and storing the layer filtering rules, when layer filtering is required, the existing layer filtering rules can be obtained directly, which is convenient and fast.

For example, see Table 1 for a preset layer filtering rule table.

Table 1 Layer filtering rules Table 1

Based on Table 1, the layer type of the first layer is a video stream playback layer, the type of the second layer is a conference control button layer, the device ID of the first electronic device is D_01, and the device ID of the second electronic device is When it is D_02, the first layer filter rule suitable for the first display cache found in layer filter rule table 1 according to the device identifier of the first electronic device is "the display layer type is video stream playback layer and conference Control button layer", according to the device identification of the second electronic device, the second layer filter rule suitable for the second display cache found in the layer filter rule table 1 is "the display layer type is a video stream playback layer, Do not show meeting control button layer".

Understandably, since the first control of the video category is located at the video stream playback layer, and the second control of the button category is located at the conference control button layer, the first display screen generated according to the filtering rules of the first layer and each layer type will include the first control on the video stream playback layer and the second control on the conference control button layer.

Correspondingly, the second display screen generated according to the filtering rule of the second layer and each layer type will include the first control located at the video stream playback layer, but not include the second control located at the conference control button layer.

For example, see Table 2 for another preset layer filtering rule table.

Table 2 Layer filtering rules Table 2

Based on Table 2, the layer type of the first layer is a whiteboard annotation layer, the type of the second layer is a conference control button layer, the device ID of the first electronic device is D_01, and the device ID of the second electronic device is D_02 , then the first layer filter rule suitable for the first display cache found in layer filter rule table 2 according to the device identifier of the first electronic device is "display layer type is whiteboard annotation layer and conference control button map Layer", according to the device ID of the second electronic device, the filter rule for the second layer suitable for the second display cache found in the layer filter rule table 2 is "The display layer type is a whiteboard annotation layer, and conference control is not displayed button layer".

Understandably, since the first control of the whiteboard annotation class is located in the whiteboard annotation layer, and the second control of the button class is located in the conference control button layer, the first display screen generated according to the filter rules of the first layer and each layer type will include a first control on the whiteboard annotation layer and a second control on the meeting control button layer.

Correspondingly, the second display screen generated according to the filtering rule of the second layer and each layer type will include the first control located at the whiteboard annotation layer, but not include the second control located at the conference control button layer.

Method 2: Provide user entry, let the user decide the filtering rules of the first layer and the filtering rules of the second layer

In order to improve user participation, a user operation entry can also be provided, and the user decides layer filtering rules. Specifically: first, a layer filtering rule decision interface that can be operated by the user is displayed on the screen of the first electronic device, so The layer filtering rule decision interface includes the first control and the layer type of the first layer where the first control is located, the second control and the second map where the second control is located The layer type of the layer; then, in response to the operation behavior of the user setting the first layer filter rule for the first display cache, generate the first layer filter rule; in response to the user setting the first layer filter rule for the second display caching the operation behavior of setting the filtering rules of the second layer, and generating the filtering rules of the second layer.

Referring to FIG. 9 , it shows a schematic diagram of an interface for users to decide the filtering rules of the first layer and the filtering rules of the second layer.

Exemplarily, the display interface of the mobile phone 100 displays an operation interface for setting the filtering rules of the first layer and the filtering rules of the second layer. In this operation interface, it is divided into an area for setting filter rules for the first layer and an area for setting filter rules for the second layer.

Still taking the premise that the first application includes the first control and the second control, the first control is located on the first layer, and the second control is located on the second layer as an example. When the layer type of the first layer is determined to be the video stream playback layer and the second layer is the conference control button layer according to the above method, continue to refer to FIG. 9 . The video stream playback layer and the conference control button layer are displayed in the area for setting the filtering rules of the first layer, and corresponding check boxes are set after each layer option; in the area for setting the filtering rules of the second layer The video stream playback layer and the conference control button layer are also displayed in , and check boxes are set correspondingly after each layer option.

Exemplarily, when the user selects the checkboxes of the video stream playback layer and the conference control button layer in the setting area of the first layer filter rule, and clicks the save button, the mobile phone responds to According to the user's operation behavior, the first layer filter rule of the first display cache is generated according to the user's selection, specifically "the display layer type is a video stream playback layer and a conference control button layer".

Exemplarily, when the user selects the check box of the video stream playback layer in the setting area of the second layer filter rule, and clicks the save button, the mobile phone responds to the user's operation behavior, according to the user's Select to generate the filter rule for the second layer of the second display cache, specifically "the display layer type is a video stream playback layer, and the conference control button layer is not displayed".

In order to better illustrate the process of projecting content from the first electronic device that initiates projection to the second device (hereinafter referred to as the large-screen device) that accepts projection, the embodiment of this application assumes that the first electronic device that initiates projection is a mobile phone, and the large-screen The device is a TV, and the first application is a conference application as an example. The following three scenarios are used in combination with the screen projection method provided in the embodiment of the present application to describe.

scene one:

The specific implementation manner of the embodiment of the present application will be described in detail below with reference to FIG. 3 and FIG. 10 to FIG. 12 . Specifically, the screen projection process can be divided into three parts. The first part is the layer identification process. The layer identification process is mainly the process of identifying the controls that need to be drawn by the conference application and determining the layer type. The second part is the filtering and rendering synthesis process. The filtering and rendering synthesis process is mainly to filter different types of layers according to the identified layer identification information, and then render and synthesize the filtered layers. The third part is the display process, which is to render and synthesize different images, and transmit them to the corresponding screens (mobile phone screen and TV screen) for display.

The following is a schematic diagram of the interaction process between each module in the mobile phone and each module in the TV (large screen) shown in Figure 10, the sequence diagram of the interaction of each module in the mobile phone shown in Figure 11, and the modules in the mobile phone shown in Figure 12 The timing diagram for interacting with each module in the TV provides a detailed description of the entire screen projection process.

See Figure 11, specifically including:

101. Send a drawing request for drawing control 1 and control 2.

Referring to FIG. 10 , for example, after the conference application (subsequently referred to as: conference application) installed in the application layer of the mobile phone is started, assuming that the conference application is used to join the meeting, the screen that needs to be presented on the screen of the mobile phone by default includes Control 1 (assumed to be a video stream playback control) and control 2 (assumed to be a conference control button control), the conference application will send an application for drawing the control to the display processing module in the display management framework located in the application framework layer for determining control information of draw requests.

Optionally, the drawing request includes, but is not limited to: the application ID of the meeting application (for example, it may be the name of the application program package), the ID of the control to be drawn (for example, it may be the name of the control), and the like.

102. Determine the control information of the control 1 and the control information of the control 2 respectively.

Referring to FIG. 10 , for example, after receiving the drawing request sent by the conference application for drawing control 1 and control 2 , the display processing module respectively determines the control information corresponding to the control to be drawn according to the information carried in the drawing request.

Exemplarily, the display processing module determines the application type according to the captured application ID. For example, according to the above-mentioned initiating drawing request is a conference application, the application type determined according to the ID of the conference application may be a conference type.

Understandably, regarding the determination of the application type, the corresponding relationship between the application ID and the application type can be preset, and then when the application ID is captured, the corresponding application type can be directly determined according to the preset corresponding relationship.

Exemplarily, the display processing module determines the control type according to the captured control name, for example, when the captured control name is SurfaceView, the determined control type is a video stream playback control or a 3D picture display control.

It is understandable that the determination of the control type can also be determined through the preset corresponding relationship, that is, the corresponding relationship between different control names and control types is determined in advance, and then when the control name is captured, directly according to the preset corresponding relationship Determine the corresponding control type.

Thus, the display processing module can determine the control information of the control to be drawn according to the received drawing request.

103. Send the control information of the control 1 and the control information of the control 2.

After the display processing module determines the control information of the control 1 and the control information of the control 2 to be drawn, it sends the determined drawing information of each control to the layer identification module, and the layer identification module identifies the layer to which the control belongs.

104. Send drawing requests for control 1 and control 2.

While sending the control information to the layer identification blur, the display processing module can send drawing requests for the control 1 and the control 2 to the layer filtering module in the display rendering framework.

Exemplarily, in an actual application scenario, the display processing module may obtain configuration information corresponding to the conference application according to the determined conference type and control type, for example, it may be the resolution corresponding to the screen displayed by the conference application (for example, 1080*720) , and the configuration information of the control, such as the size and position of the control, and then generate a drawing request that needs to be sent to the layer filtering module according to the determined information, and send the drawing request to the layer filtering module.

In addition, the display processing module may also send the determined configuration information as control information corresponding to each control to the layer identification module for processing.

Exemplarily, in an actual application scenario, the operation of the display processing module sending a drawing request to the layer filtering module may be performed synchronously with the operation of sending the determined control information to the layer identification module, or it may be performed while sending the control It may be performed before sending the control information, or after sending the control information.

Exemplarily, in an actual application scenario, if the display processing module sends a drawing request to the layer filtering module after sending the control information to the layer recognition module, then the drawing request can be It is sent to the layer filter module only after the layer to which the drawn control belongs.

105. Determine the layer type of the control 1 according to the control information of the control 1, and determine the layer type of the control 2 according to the control information of the control 2.

It should be noted that in an actual application scenario, the layer identification module can be divided into two parts: a layer identification module and a layer identification record library.

Wherein, the layer identification module specifically identifies and analyzes the control information sent by the display processing module according to the preset layer identification algorithm, and then determines the type of the layer to which the control belongs, and identifies the layer to which the control belongs; the layer identification The record library is used to record the relationship between known controls and layers.

Exemplarily, in an actual application scenario, after the layer identification module receives the control information of each control delivered by the display processing module, it can first search in the layer identification record library according to the control information. Corresponding layer and identification information, then directly determine the found layer as the layer type corresponding to the control, if not found, the layer identification module will send it to the display processing module according to the preset layer identification algorithm Identify and analyze the control information, and then determine the layer type to which the control belongs.

For example, after receiving the control information of the control 1 and the control information of the control 2 sent by the display processing module, the layer identification module first searches the layer identification record library according to the control information of the control 1 and the control information of the control 2. Assuming that the content matching the control information of control 1 is found in the layer identification record library, if the layer type corresponding to the control information of control 1 is a video stream playback layer, then the layer type of control 1 is determined as video Stream playback layer. Assuming that no content matching the control information of control 2 is found in the layer identification record library, the layer identification module identifies and analyzes the control information of control 2 according to the preset layer identification algorithm, and then determines the control information of control 2 The layer type to which it belongs, for example, according to the layer identification algorithm, it is determined that the layer type of the control 2 is the conference control button layer.

Regarding the process of identifying and analyzing control information based on the preset layer identification algorithm, the embodiment of this application provides several specific implementation methods, as follows:

Exemplarily, in a possible implementation manner, the layer identification module may determine the layer type to which the control belongs according to the size, position, splicing condition of the control, and in combination with the application type and control type in the control information.

For example, for conference applications, during a video conference, usually the video stream playback controls of the current speaker will be located in the middle area of the entire screen, and the video stream playback controls of other participants will be located in the top area of the mobile phone screen. The conference control button control operated by the user is located at the bottom area of the mobile phone screen, and the specific style is shown in FIG. 3 .

For another example, the sizes of the video stream playback controls used to display the participants (including the presenter) are the same, and the video stream playback controls of multiple participants are spliced together to form a complete video stream playback layer. The conference control button case for the user to operate is displayed in a conference control button layer, which is located at the bottom area, or the top area, or the left and right sides of the screen of the mobile phone.

Based on the predicted information, the layer identification module can determine the layer type to which the control belongs according to the size, position, splicing condition of the control, combined with the application type and control type in the control information.

For example, in Fig. 3, the video stream playback controls used to display the pictures of the participants correspond to a video playback layer 20-1 together, for the user to operate mute setting options, video setting options, sharing setting options, participants setting options, more Multiple setting options together correspond to a conference control button layer 20-2.

Optionally, in an actual application scenario, it is also possible to set a video stream playback control to correspond to a video stream playback layer, and a conference control button control to correspond to a conference control button layer, which is not limited in this application.

Exemplarily, in another possible implementation, for some non-video stream playback controls, that is, not used to display the video stream of participants, whether it needs to be classified into the video stream playback layer, so as to achieve In the screen projection mode, when the large screen only displays the content in the video stream playback layer, it can be projected to the large screen for display. The layer identification module can monitor whether such non-video stream playback controls can accept touch, And the touch duration (for example, the duration of 10 frames), whether there is information such as cursor changes, prompts, etc. to determine whether such controls need to be classified into the video stream playback layer.

For example, for the whiteboard annotation mode under the sharing settings option, the video stream playback controls of the participants are usually not displayed in this mode, but the whiteboard annotation controls need to be displayed. If the control is not classified into the video stream playback layer, then The large screen cannot display the whiteboard annotation screen synchronously with the mobile phone screen. Therefore, for this type of control, by monitoring the changes of the whiteboard brush/cursor, as well as the movement of the position, prompt information, etc., when it is determined that it is a whiteboard annotation control, it is also classified into the video stream playback layer, that is, it is displayed on the large screen. show.

Exemplarily, in another possible implementation manner, the drawing request sent by the conference application to the display processing module does not carry a specific package name feature, and the display processing module cannot determine the specific application type and control type, as shown in Fig. The layer identification module can determine the layer type by judging the source information of the drawing request, and then perform layer identification on the control.

Optionally, when determining the layer type according to the source information, it is possible to determine the corresponding information of the control according to these related information by judging related information such as a virtual IP address (Virtual IP Address, VIP) and a process identification number (Process Identification, PID). The layer type for .

Understandably, in actual application scenarios, the correspondence between the source information of various known controls and the control and the corresponding layer type can be established, and then the drawing request does not carry specific package name features, so the control cannot be determined When the control information is displayed, the layer identification module can determine the layer type to which the control belongs according to the source information corresponding to the initiated drawing request and the predetermined corresponding relationship.

Exemplarily, in another possible implementation, for a scenario where the layer type of the control cannot be determined by any of the above implementations, the layer identification module can intercept the data stream uploaded from the HAL layer, for example, Intercept the first 5 frames, and then analyze the 5 frames of data, and use the pre-trained recognition model to analyze and identify whether the current screen contains controls whose layer type is the video stream playback layer, or the layer type conference Controls that control the button layer, which in turn determines the type of layer each control belongs to.

106. Send the layer type of control 1 and the layer type of control 2.

After the layer recognition module determines the layer type to which each control belongs, it sends the control and the corresponding layer type to the layer filter module, so that the layer filter module can compare the map according to the identification information added in the identified layer. layer to filter.

It should be noted that in actual application scenarios, in order to ensure the validity of the content of the records in the layer identification record library, after each layer identification module identifies the layer type of the control, the control and the corresponding The layer type is updated to the layer identification record library.

107. Filter out the control 2 according to a preset filtering rule.

For example, referring to FIG. 10 , in an actual application scenario, multiple filtering rules may be pre-installed in the layer filtering module.

Specifically, the filtering rules can be divided according to the type and model of the large-screen device. For example, for a large-screen device such as a TV, the filtering rule can be to only project a video stream to play a layer on a large screen; for a large-screen device such as a projector , filter rules can be displayed on large-screen devices for all layers.

Optionally, in an actual application scenario, the filtering rule may also be that the large-screen device displays all content in the video stream playback layer, and displays some controls in the conference control button layer, for example, only displaying the mute setting option.

In addition, for the filter rule corresponding to the picture displayed on the screen of the mobile phone, all layers may be displayed, that is, no layer is filtered.

In addition, in the actual application scenario, if other large-screen devices are used as the main control device, that is, the device for users to operate, the mobile phone screen can directly not display any content according to business needs, and display on the set main control device. All layers.

Exemplarily, referring to FIG. 10 and FIG. 11 , control 1 is a video stream playback control, and control 2 is a conference control button control. When the control 1 and control 2 of the layer type are determined to reach the layer filtering module, assuming that the filtering rule (rule 1 in Figure 10) corresponding to the mobile phone screen is to send and display the contents of all layers, after filtering out according to rule 1 , in essence, control 1 and control 2 are not filtered out, but all resources of control 1 and control 2 are sent to the composite rendering module for picture drawing.

Correspondingly, assuming that the filtering rule (rule 2 in FIG. 10 ) corresponding to the TV screen (big screen) is to only display the content in the video stream playback layer, then according to rule 2, the layer filtering module will filter out the control 2, Only the resources of control 1 are sent to the composite rendering module for picture drawing.

In actual application scenarios, filtering rules can also be decided by users.

Exemplarily, after identifying the layer type included in the picture that the current application needs to draw by setting the layer identification module, the identified layer type is sent to the application layer respectively, such as the conference application in Figure 11, and Shows the layer filter module in the render frame.

Correspondingly, after the identified layer type reaches the application layer, the mobile phone responds, and an interface that displays the layer corresponding to each control can be popped up on the current interface, allowing the user to choose which layers the controls are located in and put them on the TV screen. , which are displayed on the screen of the mobile phone, that is, the filtering rules corresponding to the TV screen and the filtering rules of the mobile phone screen are obtained. Next, the filtering rules determined by the user are sent to the layer filtering module in the display rendering framework, so that the layer filtering module can filter the layers according to the filtering rules.

By providing the entrance of user decision-making and filtering rules, users can participate in the selection of screen projection content, so that the application scenarios of screen projection can better meet the needs of different users, thereby improving the user experience.

108. Send the resource of control 1.

After the layer filtering module filters the controls according to the preset filtering rules, it sends the control resources satisfying the filtering rules to the composite rendering module for picture drawing.

It should be noted that after the controls are filtered according to the preset filtering rules, the resources to be sent to the controls can be determined according to the information written in the drawing request issued by the display processing module. The specific determination method is not limited in this application. This is also not described.

109. Draw the control 1 according to the resources of the control 1 to obtain the screen A.

Specifically, the composition rendering module first draws each control to be displayed according to the received control resources, for example, drawing logic, and then synthesizes the layers where each control is located to obtain a complete picture.

Exemplarily, for the process of drawing the control 1 according to the resources of the control 1 in the embodiment of the present application, and then obtaining the picture A, the essence is to draw each video stream playback control in the video playback stream layer 20-1 shown in FIG. 3 , and The process of synthesizing the layers where these controls are located into one screen, that is, the final screen A obtained is 20-1 in FIG. 3 .

Correspondingly, when the content obtained after filtering according to rule 1 is control 1 and control 2, the composite rendering module will draw control 1 according to the resources of control 1, draw control 2 according to the resources of control 2, and then draw the image where control 1 is located Layer and the layer where control 2 is located are synthesized, and finally a complete picture B is obtained.

Thus, layer separation, filtering and composite rendering are realized.

After obtaining the picture A that needs to be displayed on the TV screen, the picture A will be cached in the display cache A, and in the subsequent process, the picture A can be obtained directly from the display cache A.

Correspondingly, after obtaining the screen B that needs to be displayed on the screen of the mobile phone, the screen B will be buffered into the display buffer B, and in the subsequent process, the screen B can be obtained directly from the display buffer B.

In addition, it should be noted that in an actual application scenario, display cache A and display cache B may be the same cache area. For this scenario, when caching different images into the cache area, carry specific identifications (such as It can be specific to which device), so that there is no need to allocate multiple cache areas inside the mobile phone, thereby avoiding redundancy problems.

In addition, after obtaining screen A and screen B, how to send and display the cached screens from the display cache of the mobile phone to different screens for display will be described in detail below with reference to FIG. 10 and FIG. 12 .

See Figure 12, specifically including:

201. Draw the control 1 according to the resources of the control 1 to obtain a screen A.

The synthetic rendering module draws the control according to the resources of the control, and then obtains the screen that meets the filtering requirements. For details, see the description of step 109 in FIG. 11 , which will not be repeated here.

202. Send picture A.

Exemplarily, referring to FIG. 10 , after synthesizing and drawing the picture to be sent to the display, the composite rendering module temporarily buffers the obtained picture into the corresponding display buffer, and then sends the buffered picture to the corresponding module by the display buffer.

Exemplarily, referring to Figure 10, the picture A that needs to be projected to the TV screen (large screen) for display will be cached in the display buffer A, and when it needs to be projected to the TV screen, the screen projection recording module will take out the picture A from the display cache A , to record the screen A, that is, to display that the cached screen A in the cache A is sent to the screen projection recording module.

203. Record screen A to obtain screen projection content.

After the screen projection and recording module obtains the screen A, it records the screen A, and after the recording is completed, performs video encoding on the recorded content, and then obtains the screen projection content.

It is understandable that in actual application scenarios, each frame of video stream corresponds to a picture, so the obtained picture A can be at different times, so when the screencasting recording module is recording, it can record according to the preset recording requirements. Multi-frame images are recorded, such as 30 frames per second. Therefore, the obtained screen projection content is essentially a dynamic video stream.

It should be noted that the screen projection and recording module is provided by the electronic device system with the screen projection function. This application does not describe the specific recording process, and this application does not limit the video encoding performed after recording.

204. Send the screen projection content.

After the screen projection and recording module completes the recording and video encoding of screen A, it will transmit the obtained screen projection content to the video decoding module in the large-screen device (a TV set in this embodiment) through a pre-established communication connection.

205. Decode the projected screen content to obtain screen A.

After the video decoding module in the TV receives the projected screen content transmitted by the mobile phone, it will decode according to the predetermined method, and then analyze the picture A.

It can be understood that the screen projection content recorded by the screen projection recording module is essentially a video stream, so the additional decoding operation is to decode the video stream to obtain the video content to be displayed.

It can be understood that the video content at different times is actually composed of multiple frames, so what is displayed on the TV screen is a continuously changing frame for each frame.

206. Send picture A.

After the video decoding module parses out the picture A, it will send the picture A to the TV screen, specifically to the projection display module of the TV.

207. Display screen A.

The screen projection display module displays the received picture A on the display screen 2, that is, the TV screen.

Understandably, since screen A only includes controls whose layer type is a video stream playback layer, the content displayed on the TV screen is specifically the mirror image of 20-1 in FIG. 3 , that is, 20-1'.

208. Draw control 1 according to the resource of control 1, draw control 2 according to the resource of control 2, and obtain screen B.

209. Send picture B.

Exemplarily, referring to FIG. 10 , the picture B that needs to be displayed on the screen of the mobile phone will be cached in the display buffer B. The cached picture B is sent to the display driver of the mobile phone.

210. Send picture B.

After receiving the screen B, the display driver of the mobile phone will upload the screen B to the conference application at the application layer, and display it on the conference interface corresponding to the conference application.

211. Display screen B.

It is understandable that since the screen B includes all the layers, not only 20-1 in FIG. 3 but also 20-2 will be displayed on the screen of the mobile phone.

From the above description, it is not difficult to find that by adding a layer identification module and a layer filtering module to the electronic device projecting content, such as the application framework layer on the mobile phone side, the layer identification module can identify the currently joined conference application. Among the conference layers, which layer is the video stream playback layer and which layer is the conference control button layer. Then, the layer filtering process is performed in the layer filtering module, so as to filter out which layers need to be displayed on the mobile phone screen and which layers need to be displayed on the large screen according to the preset filtering rules. For example, for the mobile phone screen, set all layers to be displayed, so that users can not only see the video stream content involved in the entire conference process, but also see the conference control buttons that can be operated by the user, so that the user can control the conference through the conference control button to operate. For the large screen, only the video stream playback layer is set to be displayed, so that other users who use the large screen to watch the conference can only see the content of the video stream involved in the conference process. When the user operates the conference control button displayed on the mobile phone screen, the entire operation Users who use a large screen to watch the meeting will not see it, so it will not affect the viewing of the large screen.

In addition, since the large screen only displays the content of the video stream involved in the meeting process, other content related to private information displayed on the mobile phone screen will not be projected on the large screen, thereby ensuring user privacy.

Scene two:

In the embodiment described in scenario 1, the screen of the mobile phone displays images of all participants and conference control buttons, and the screen of the TV only displays images of participants. In the following, in combination with Figure 13, the mobile phone screen displays the pictures of all participants, as well as the marked content and conference control buttons added by the participants to the picture, and the scene where the TV screen displays the pictures of the participants and the marked content added by the participants to the picture will be described in detail illustrate.

Referring to FIG. 13 , for example, the display interface of the mobile phone 100 displays a screen 20 after joining a conference through a conference application, and the screen 20 includes a video stream playback layer 20-1, a conference control button layer 20-2 and other conference participants Annotated content 20-3 added by personnel, and annotated content 20-4 added by the user using the mobile phone 100 for himself.

Understandably, for the identification and filtering methods of layers 20-1 and 20-2, the specific process can refer to the description in scene 1, which will not be repeated here.

For layers 20-3 and 20-4, in the actual application scenario, it is not collected by the camera of the mobile phone used by the participants, but added by the participants themselves on the screen presented, such as some current live broadcasts During the interaction, the live broadcaster can add the annotation content drawn based on Augmented Reality (AR) technology, see Figure 13. The AR annotation content added by the peer user is 20-3, and the AR annotation added by the local user The content is 20-4. For the controls of such content, when determining whether they should be classified into the video stream playback layer, it may specifically be determined according to the positional relationship between these controls and the video playback stream control. Therefore, when the layer identification module determines the layer type according to the control information of the control, it will determine whether such controls need to be classified into the video stream playback map according to the positional relationship between the controls of the known layer type and such controls layer.

Correspondingly, since the AR annotation content added by the local user is intended to be viewed by other participants, the content of the marked AR logo needs to be transmitted to other participants through the server, that is, on other participants' mobile phone screens and big screens. All screens need to be displayed, but on the large screen matched with the local machine, there is essentially no need for delivery. Therefore, in one implementation scenario, the AR annotation content added by the local user is not displayed on the large screen, while it is displayed on other users’ It is displayed on the matching large screen of the mobile phone.

See Figure 13, 20-4 is added by the local user, so the content of this layer is only displayed on the mobile phone screen, not displayed on the matching large screen 200, and 20-3 is added by other participants on their mobile phones , so 20-4 needs to be displayed on the screen 20 of the mobile phone 100, and 20-3 also needs to be displayed when the screen is projected onto the large screen 200, so the mirror image content 20-1 of 20-1 will be displayed on the large screen 200 1' and 20-3' mirror content 20-3'.

In addition, in another implementation scenario, in order to allow users watching the large screen 200 to see the AR annotation content 20-3 added by users who join the conference using the mobile phone 100, it can also be classified into the video stream playback layer , so that the screen projected on the large screen 200 not only includes the mirror image content of the AR annotation content 20-3 added by the peer user, but also displays the mirror image content of the AR annotation content 20-4 added by the local user.

Therefore, by classifying the controls that do not belong to the video stream playback layer into the video stream playback layer according to business needs, the content displayed by the non-video stream playback controls can also be projected to the large screen for display, further enriching the projection screen scene.

Scene three:

In the embodiment described in scenario 1, the screen of the mobile phone displays images of all participants and conference control buttons, and the screen of the TV only displays images of participants. 14a, 14b and 15, when the screen of the mobile phone is switched from the scene 1 to the whiteboard annotation mode, the changes of the screens displayed on the mobile phone screen and the TV screen will be described in detail.

Referring to FIG. 14a, for example, the display interface of the mobile phone 100 displays a screen 20 after joining a conference through a conference application, and the screen 20 includes a video stream playing layer 20-1 and a conference control button layer 20-2. The video stream playback layer 20-1 includes one or more video stream playback controls. For a specific description, refer to Scene 1, which will not be repeated here. The picture 20' projected onto the TV 200 through the mobile phone 100 only includes the mirror image 20-1' of the video stream playback layer 20-1.

Exemplarily, the video stream playback layer 20-1' includes one or more video stream playback controls, and the displayed video stream playback controls are in one-to-one correspondence with the video stream playback controls displayed on the screen 20 of the mobile phone 100, and the video The content displayed in the stream playback controls is the same.

In addition, referring to FIG. 14a, exemplary, the conference control button layer 20-2 displayed on the display interface of the mobile phone 100 includes one or more conference control buttons, such as mute setting options, video setting options, sharing Setup options, Attendee setup options, and more setup options.

Exemplarily, after the user clicks the sharing setting option in the conference control button layer 20-2, the mobile phone 100 responds to the user's operation behavior, and displays a prompt box 40 for selecting shared content on the display interface, as shown in FIG. 14a . Wherein, the prompt box 40 includes one or more controls, such as desktop setting option 40-1, whiteboard setting option 40-2, cancel setting option 40-3 and start sharing 40-4 shown in FIG. 14a.

It should be noted that the names and quantities of the controls displayed on the display interface of the mobile phone 100 involved in FIG. 14a and FIG. 14b, and the names and quantities of the controls displayed in the prompt box 40 are only illustrative examples, and are not limited in this application. .

Continuing to refer to FIG. 14a, for example, when the user clicks the whiteboard setting option 40-2 in the prompt box 40, the mobile phone 100 marks the state of the whiteboard setting option 40-2 as selected in response to the user's operation behavior, and then if The user clicks the start sharing setting option 40-4, and the mobile phone 100 will switch from the current interface to the whiteboard annotation mode interface in response to the user's operation behavior, as shown in FIG. 14b.

Referring to FIG. 14b, for example, after switching to the whiteboard annotation mode, the screen 20 displayed on the display interface of the mobile phone 100 includes but is not limited to the conference control button layer 20-2 shown in FIG. 14a, switching to the whiteboard annotation mode Then the whiteboard annotation layer 20-5 drawn by the whiteboard annotation control, and the conference control button layer 20-6 for stopping sharing. Wherein, the conference control button layer 20-6 includes one or more controls, such as a prompt control in FIG. 14b for prompting the user that the whiteboard is currently being shared, and a control for the user to stop sharing.

Continuing to refer to FIG. 14b, for example, since the controls in the conference control button layer 20-6 are still for users to operate and do not involve the content of the whiteboard, the layer recognition module will use such controls when casting a screen. identified and filtered by the layer filtering module to ensure that the image 20' projected on the TV screen only includes the content drawn by the user in the whiteboard annotation layer 20-5'.

In addition, it should be understood that after the user clicks the whiteboard setting option 40-2 in the prompt box 40, the mobile phone 100 marks the state of the whiteboard setting option 40-2 as selected in response to the user's operation behavior, and then the user clicks After clicking the cancel setting option 40-3, the mobile phone 100 responds to the user's operation behavior, and the prompt box 40 will disappear from the display interface, and return to the content displayed on the mobile phone 100 display interface shown in the lower left corner of Figure 14a.

It should be noted that in the actual application scenario, when switching to the whiteboard annotation mode, the brush that draws content on the whiteboard will be directly used as a separate layer. The picture displayed on the screen usually does not display the brush that can draw on the whiteboard, but directly displays the content drawn on the whiteboard. However, on the mobile phone side, the brush is displayed so that the user who draws the content knows where to draw.

In addition, in another implementation scenario, for example, when displaying content in a shared document, the cursor displayed on the mobile phone screen can be projected onto the large screen, so that users watching the large screen can know the currently selected content.

The entire screen projection process involved in scenario 3 will be described in detail below in conjunction with the schematic diagram of the interaction process between each module in the mobile phone and each module in the TV (big screen) shown in FIG. 15 .

Referring to FIG. 15 , for example, after the conference application (subsequently referred to as: conference application) installed on the application layer of the mobile phone is started, assuming that the conference application is used to join the conference, the screen that needs to be presented on the screen of the mobile phone by default includes Control 1 (assumed to be a video stream playback control) and control 2 (assumed to be a conference control button control), the conference application will send an application for drawing the control to the display processing module in the display management framework located in the application framework layer for determining control information of draw requests. At this time, if the user clicks the sharing setting option displayed in the conference control button layer and selects the provided whiteboard mode, the conference application will also send the display management framework located in the application framework layer to determine the control information. The processing module sends a drawing request for drawing the whiteboard annotation control. For the process of performing the operation of switching the whiteboard mode on the mobile phone interface, refer to the interface diagrams shown in Fig. 14a and Fig. 14b for details.

Continuing to refer to FIG. 15 , for example, after the display processing module receives the drawing request for drawing control 1 and control 2 sent by the conference application, according to the information carried in the drawing request, respectively determine the control information corresponding to the control to be drawn, and The layer identification module identifies the operation of the layer type of control 1 and control 2. For a specific description, please refer to scene 1, which will not be repeated here.

In addition, it should be understood that the setting of the whiteboard annotation control is also a kind of control, but the specific control properties and control information are different from the video stream playback control and the conference control button control, but the process of determining the control information of the whiteboard annotation control is the same as that of the video stream playback control. The process is similar for controls and conference control button controls. The display processing module can also determine the application type by the captured application ID. For example, according to the above-mentioned initiating drawing request is a meeting application, then the application type determined according to the meeting application ID can be the meeting type, and determined according to the captured control name Control type, for example, when the captured control is named BlankWindow, the determined control type is a whiteboard annotation control.

Correspondingly, after the display processing module determines the control information corresponding to the whiteboard annotation control, it will also send the determined control information to the layer identification module, and the layer identification module will identify the layer type, and then the layer identification module will The identified layer types are sent to the layer filtering module. Then, the layer filtering module filters according to the preset filtering rules, and sends the filtered resources of the controls that need to be displayed on different devices to the composite rendering module, and the composite rendering module synthesizes images that need to be sent to different devices for display .

Continuing to refer to Figure 15, for example, assuming that the preset rule 1 in the filtering module is a filtering rule corresponding to the screen of the mobile phone, rule 1 specifies that the contents of all layers are displayed on the screen of the mobile phone, then the content filtered out according to rule 1 Whiteboard annotation controls displayed for switching to whiteboard mode, and meeting control button controls for user actions.

Correspondingly, assuming that the preset rules in the filter module are the filter rules corresponding to the TV screen, rule 2 stipulates that in the whiteboard mode, the TV screen only displays the content in the whiteboard annotation control, then according to the filter rule 2, the layer filter module The video stream playback control and conference control button control will be filtered out, and only the resources of the whiteboard annotation control will be sent to the composite rendering module for picture drawing.

Continuing to refer to FIG. 15 , for example, after receiving the resource of the control filtered by the layer filtering module according to different filtering rules, the composite rendering module draws the control according to the corresponding resource, and then obtains images to be displayed on different devices.

Similarly, the picture drawn by the composite rendering module can be cached in the corresponding display cache first. For example, the picture A drawn according to the resource of the whiteboard annotation control and the resource of the conference control button control needs to be cached in the display cache A. When the picture needs to be displayed At this time, the picture A is taken out from the display buffer A and sent to the display driver of the mobile phone, and then sent to the screen of the mobile phone for display by the display driver of the mobile phone, thereby obtaining the picture 20 displayed on the display interface of the mobile phone 100 in FIG. 14b.

Correspondingly, the screen B drawn according to the resources of the whiteboard annotation control needs to be cached in the display buffer B, and when the screen projection is required, the screen B is taken out from the display buffer B and sent to the screen projection recording module for recording, and then the screen projection content is obtained , and the screen projection recording module sends the obtained screen projection content to the TV with a pre-established communication connection, and the video decoding module in the TV performs decoding processing, and then parses out the screen B, and then transmits the screen B to the screen projection The display module finally displays picture B on the TV screen.

Therefore, by separating the layers of the conference screen, the electronic device that projects the content, such as a mobile phone, displays the image synthesized by all the layers, and on the large-screen device that displays the projected content, such as a TV, only the video stream playback image is displayed Layer, so that when the user uses the operation behavior of the conference empty button control displayed on the conference control button layer displayed on the mobile phone side, when the current conference screen is switched to the whiteboard mode, the entire switching process will not be displayed on the TV screen, and the TV screen will still display The content in the video stream plays the layer. After switching to the whiteboard mode on the mobile phone, the TV screen will switch the currently displayed screen to the whiteboard annotation control, which will not affect the visual experience of users who use the TV screen to watch the conference, and can be displayed in time The switched whiteboard screen realizes non-constant switching of different interfaces.

In addition, it should be noted that, in an actual application scenario, the resolution, screen size, and ratio of the electronic device projecting the content and the large-screen device displaying the projected content will be different, so in one implementation, you can The frame rate of the recorded projected screen content is pre-determined according to the resolution. In another implementation, black border processing can be performed on the screen projected on the large-screen device according to information such as screen size and ratio.

Regarding the processing method of removing black borders during screen projection, the details are as follows:

In the process of recording the second display picture in the second display buffer to obtain the projected screen content, the first screen aspect ratio of the first electronic device and the first screen aspect ratio of the second electronic device may be obtained first. second screen aspect ratio; then, when the first screen aspect ratio is different from the second screen aspect ratio, perform black border removal processing on the second display picture in the second display buffer, and record The second display screen after removing black borders obtains the screen projection content; when the aspect ratio of the first screen is the same as that of the second screen, the second screen in the second display buffer Display the screen to record and get the screencast content. In this way, black border removal processing is introduced, so as to ensure that the second display screen displayed on the second electronic device has no black borders, or reduce the black borders as much as possible, thereby improving the viewing experience of the user watching the projected screen.

Understandably, the second display frame is drawn on a virtual screen in the first electronic device.

Exemplarily, when the aspect ratio of the first screen is different from that of the second screen, in order to remove the black border of the second display image, the visible area of the virtual screen may be set first.

In order to distinguish it from the virtual screen, the screen of the mobile phone is called a physical screen. The physical screen of a mobile phone is generally rectangular or approximately rectangular. The virtual screen corresponds to the physical screen and is generally rectangular. In the embodiment of the present application, the visible area of the virtual screen is part or all of the display area of the physical screen.

Referring to Figure 16a, when the physical screen of a mobile phone displays an interface, it includes a display content layer (DisplayRect) and an observation window layer (Viewport). Similarly, a virtual screen can also include a display content layer and an observation window layer, whether it is a physical screen or a virtual screen. The screen, the visible area of the human eye of the screen is related to the area setting information of the observation window layer. For example, in Figure 16a, the observation window layer of the physical screen is 2340*1080, then the visible area of the human eye is 2340*1080, and the observation window layer of the virtual screen is 1920*1080, then the visible area of the human eye is also 1920*1080. It should be noted that the interface displayed on the virtual screen is not displayed to the user during screen projection, and its visible area to the human eye refers to the area that can be recorded during screen recording.

Regardless of whether it is a physical screen or a virtual screen, the setting information of its area is related to the resolution of the physical screen. For example, as shown in Figure 16b, the origin O can be set at the top left corner of the physical screen, the x-axis passes horizontally to the right through the origin O, and the y-axis passes vertically downwards through the origin O to establish a coordinate system. Each pixel of can be identified by coordinates (x, y). Based on this coordinate system, referring to Figure 16a, taking the resolution of the physical screen as 2340*1080 as an example, the display area of the physical screen display content layer can be set to (0, 0, 234, 1080), and the display area of the observation window layer is (0, 0, 2340, 1080), the display area of the display content layer of the virtual screen can be set to (0, 0, 2340, 1080), and the display area of the observation window layer can be set to (210, 0, 2130, 1080) . In this embodiment of the present application, the visible area of the virtual screen may be the display area of the observation window layer of the virtual screen. By setting the display area of the observation window layer, the visible area of the virtual screen can be changed, that is, the aspect ratio and content of the video frame sent to the second electronic device can be changed. For example, if the visible area is set to (210, 0, 2130, 1080), the aspect ratio of the visible area becomes 16:9, that is, the aspect ratio of the second display image obtained by recording the virtual screen with this ratio is 16:9, the content in the actual video frame is the content in the rectangle ABCD in Figure 16b.

In a possible implementation manner, setting the visible area of the virtual screen on the mobile phone may include:

The mobile phone obtains the size of the second display picture drawn in the virtual screen and the size of the display control in the large-screen device;

If it is determined that the size of the display control is smaller than the size of the second display image, the visible area of the virtual screen is set according to the display area of the display control;

If it is determined that the size of the display control is not smaller than the size of the second display image, the visible area of the virtual screen is set to be the same as the display area of the screen of the large-screen device.

In another possible implementation manner, the visible area of the virtual screen in the mobile phone may include:

The mobile phone performs screenshots of the second display screen drawn for the first time according to the filtering rules of the second layer and the layer types of each layer, and performs black edge detection on the pictures obtained from the screen captures;

If a black border is detected, the visible area of the virtual screen is set according to the position of the non-black border area in the second display frame;

If black borders are not detected, set the visible area of the virtual screen to be the same as the display area of the screen of the large-screen device.

When detecting whether there are black edges in the picture, it can be realized by detecting whether the pixel color in the specified area of the picture is black. For example, if the screen resolution of the mobile phone is 2340*1080, that is, the screen aspect ratio is 19.5:9, and the video Or the aspect ratio of full-screen playback such as PPT is generally 16:9. Then, it can be detected whether the RGB values of pixels in the area (0, 0, 210, 1080) of the picture and the area (2130, 0, 2340, 1080) are uniform. is (0, 0, 0), if yes, the above area is a black border area, otherwise, it is not a black border area.

In yet another possible implementation manner, setting the visible area of the virtual screen on the mobile phone may include:

The mobile phone obtains the size of the second display screen and the size of the screen display control of the large-screen device;

If it is judged that the size of the display control is not smaller than the size of the second display screen, take a screenshot of the virtual screen, and perform black border detection on the picture obtained by the screenshot;

If a black border is detected, set the visible area of the virtual screen according to the non-black border area;

In order to ensure the accuracy of the black border detection result, the mobile phone can take multiple screenshots of the virtual screen, and set the visible area of the virtual screen according to the black border detection results of multiple pictures obtained from multiple screenshots.

Through the above several methods, the black border removal process can be completed, and finally by recording the virtual screen image after the black border removal processing, the second display screen that needs to be displayed on the large screen device can be obtained.

The above description is only an example for better understanding of the technical solution of this embodiment, and is not intended as the only limitation to this embodiment.

Regarding the negotiation method of the end-to-end frame rate during screen projection, the details are as follows:

In the process of recording the second display picture in the second display buffer to obtain the projected screen content, the display capability of the second electronic device may be obtained first; and the refresh frame of the video stream is determined according to the display capability rate; according to the refresh frame rate of the video stream, record the second display picture in the second display buffer to obtain screen projection content. In this way, when recording the projected content, the first electronic device first negotiates with the second electronic device on the refresh frame rate of the video stream, so as to ensure that the transmitted video stream can be displayed normally on the second electronic device, and avoid bandwidth transition occupy.

That is to say, firstly, according to the display capability of the large-screen device, determine the refresh frame rate of the video stream required by the electronic device for projecting content to the large-screen device during the screen projection process.

It is understandable that in actual application scenarios, the hardware capabilities of the electronic device for projecting content and the large-screen device may not be equal. When the device is playing, it needs to be discarded, so the video stream transmitted at a high frame rate actually wastes bandwidth. Therefore, when performing screen projection, the refresh frame rate of the video stream is determined according to the display capability of the large-screen device, which not only ensures that the transmitted video stream can be displayed normally on the large-screen device, but also avoids excessive bandwidth occupation.

Then, on the side of the electronic device where the content is projected, according to the negotiated refresh frame rate, directly record the projected screen content corresponding to the refresh frame rate, so as to ensure that the refresh frame rate of the video stream transmitted to the large-screen device side is supported by the large-screen device. Therefore, large-screen devices do not need to discard videos.

In addition, in another implementation, the frame processing of the video stream can be performed in the rendering and composition module, that is, for the picture that needs to be sent to a large-screen device for display, the rendering and composition module directly follows the large-scale Refresh the frame rate of the video stream of the screen device to process the frame and synthesize the video stream that meets the requirements, so that the video stream that needs to be transmitted to the large-screen device for display has been framed according to the display capability of the large-screen device, which can reduce the network transmission process The bandwidth occupation in the middle can reduce the power consumption when the large-screen device renders the received video stream.

In addition, it can be understood that, in specific implementation, in order to realize the above-mentioned functions, the electronic device for projecting content includes corresponding hardware and/or software modules for performing various functions. Combining the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions in combination with the embodiments for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In this embodiment, the functional modules of the electronic device may be divided according to the above method example. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic, and is only a logical function division, and there may be other division methods in actual implementation.

Exemplarily, FIG. 17 shows a schematic block diagram of an apparatus 300 according to an embodiment of the present application. The device 300 may include: a processor 301 , a transceiver/transceiving pin 302 , and optionally a memory 303 .

Various components of the device 300 are coupled together through a bus 304, wherein the bus 304 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for clarity of illustration, the various buses are referred to as bus 304 in the figure.

Optionally, the memory 303 may be used for the instructions in the foregoing method embodiments. The processor 301 can be used to execute instructions in the memory 303, and control the receiving pin to receive signals, and control the sending pin to send signals.

In addition, it should be noted that in an actual application scenario, the apparatus 300 may be an electronic device for projecting a picture to a large screen for display in the above method embodiment, such as a mobile phone.

Specifically, the device 300 is the first electronic device that initiates screen projection, and the first electronic device is installed with a first application, the first application includes a first control and a second control, and the first control is located at the first One layer, when the second control is located on the second layer, one or more computer programs in the electronic device are stored on the memory, and when the computer programs are executed by the one or more processors, causing the electronic device to perform the following steps:

acquiring control information of the first control and control information of the second control;

determining the layer type of the first layer according to the control information of the first control;

Determine the layer type of the second layer according to the control information of the second control, and the layer type of the first layer is different from the layer type of the second layer;

Generate a first display picture and a second display picture according to the layer type of the first layer and the layer type of the second layer, and display the first picture on the screen of the first electronic device , projecting the second display picture to the screen of the second electronic device for display, the first display picture includes the first control and the second control, the second display picture includes the first control , excluding the second control.

Exemplarily, in an example, when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps:

extracting the control name of the first control and the size information of the first control from the control information of the first control;

Parsing the control name, when parsing the control type of the first control, the package name of the first application, and the interface information of the first control from the control name, according to the control type , the package name, the interface information and the size information, determine the layer type of the first layer;

When the control type of the first control and the interface information of the first control are parsed from the control name, and the package name of the first application is not parsed, obtain the process of drawing the first control The process identification number PID, determine the source of the first control according to the PID, determine the layer type of the first layer according to the control type, the source, the interface information and the size information , the source includes the package name of the first application.

Using the control name and the size information as retrieval keywords;

According to the keyword, search for a control matching the keyword in the layer identification record library;

When a control matching the keyword is found, the layer type corresponding to the control is determined as the layer type of the first layer;

When no control matching the keyword is found, the step of parsing the control name is performed.

When the layer type of the first layer cannot be determined according to the control information of the first control, acquire a screen currently displayed by the first application, where the screen currently displayed includes the first control;

The layer type of the first layer is determined according to the content displayed by the first control in the currently displayed screen.

When the layer type of the second layer cannot be determined according to the control information of the second control, acquire the currently displayed screen of the first application, the currently displayed screen includes the second control;

The layer type of the second layer is determined according to the content displayed by the second control in the currently displayed screen.

Generate a first display picture and a second display picture according to the layer type of the first layer and the layer type of the second layer, and cache the first display picture into the first display cache, and store the first display picture in the first display buffer. The second display frame is buffered into the second display buffer;

Taking out the first display frame from the first display buffer according to the buffer sequence, and displaying the first display frame on the screen of the first electronic device;

Recording the second display picture in the second display cache to obtain screen projection content;

Sending the screen projection content to the second electronic device for the second electronic device to decode the screen projection content to obtain the second display picture and display it on the screen of the second electronic device.

determining a first layer filtering rule corresponding to the first display cache and a second layer filtering rule corresponding to the second display cache;

According to the filter rule of the first layer, the layer type of the first layer, and the layer type of the second layer, it is determined that the first display picture includes the first layer and the second layer. second layer;

Acquire the resource of the first control in the first layer and the resource of the second control in the second layer, and generate the resource according to the resource of the first control and the resource of the second control the first display frame, and buffer the first display frame into the first display buffer;

determining that the second display screen includes the first layer according to the second layer filtering rule, the layer type of the first layer, and the layer type of the second layer;

Acquiring the resource of the first control in the first layer, generating the second display frame according to the resource of the first control, and caching the second display frame into the second display cache.

acquiring a first device identifier of the first electronic device and a second device identifier of the second electronic device;

Searching for a layer filtering rule matching the first device identifier in the layer filtering rule table, and determining the found layer filtering rule as the first layer filtering rule corresponding to the first display cache;

Searching for a layer filtering rule matching the second device identifier in the layer filtering rule table, and determining the found layer filtering rule as the second layer filtering rule corresponding to the second display cache .

Displaying on the screen of the first electronic device a layer filter rule decision interface that can be operated by the user, the layer filter rule decision interface includes the first control and the first image where the first control is located The layer type of the layer, the layer type of the second control and the second layer where the second control is located;

generating the first layer filter rule in response to the user's operation behavior of setting the first layer filter rule for the first display cache;

The second layer filter rule is generated in response to the user's operation behavior of setting the second layer filter rule for the second display cache.

Obtaining a first screen aspect ratio of the first electronic device and a second screen aspect ratio of the second electronic device;

When the aspect ratio of the first screen is different from the aspect ratio of the second screen, performing black border removal processing on the second display picture in the second display buffer, and recording the second screen image after black border removal 2. Displaying the screen to obtain the screen projection content;

When the aspect ratio of the first screen is the same as that of the second screen, recording the second display picture in the second display buffer to obtain screen projection content.

acquiring the display capability of the second electronic device;

Determine the refresh frame rate of the video stream according to the display capability;

Recording the second display picture in the second display buffer according to the refresh frame rate of the video stream to obtain screen projection content.

Wherein, all relevant content of each step involved in the above-mentioned method embodiment can be referred to the function description of the corresponding function module, and will not be repeated here.

This embodiment also provides a computer-readable storage medium, in which computer instructions are stored, and when the computer instructions are run on an electronic device/network device (such as an OTA server, a CABE server), the electronic device/network The device executes the above related method steps to implement the screen projection method in the above embodiment.

This embodiment also provides a computer program product, which, when running on a computer, causes the computer to execute the above related steps, so as to implement the screen projection method in the above embodiment.

In addition, embodiments of the present application also provide a chip (which may also be a component or module), which may include one or more processing circuits and one or more transceiver pins; wherein, the transceiver pins and the The processing circuits communicate with each other through internal connection paths, and the processing circuits execute the above-mentioned relevant method steps to implement the screen projection method in the above-mentioned embodiment, to control the receiving pin to receive signals, and to control the sending pin to send signals.

Wherein, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment is all used to execute the corresponding method provided above, therefore, the beneficial effects it can achieve can refer to the above-mentioned The beneficial effects of the corresponding method will not be repeated here.

Through the description of the above embodiments, those skilled in the art can understand that for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be assigned by different Completion of functional modules means that the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or It may be integrated into another device, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

A unit described as a separate component may or may not be physically separated, and a component shown as a unit may be one physical unit or multiple physical units, which may be located in one place or distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above contents is within the scope of the present application.

If an integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium Among them, several instructions are included to make a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the methods in various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes such as U disk, mobile hard disk, read only memory (ROM), random access memory (random access memory, RAM), magnetic disk or optical disk.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Under the inspiration of this application, without departing from the purpose of this application and the scope of protection of the claims, many forms can also be made, all of which belong to the protection of this application.

Claims

A screen projection method, characterized in that it is applied to a first electronic device that initiates screen projection, the first electronic device is installed with a first application, the first application includes a first control and a second control, and the first A control is located on the first layer, and the second control is located on the second layer, and the method includes:

acquiring control information of the first control and control information of the second control;

determining the layer type of the first layer according to the control information of the first control;

Determine the layer type of the second layer according to the control information of the second control, and the layer type of the first layer is different from the layer type of the second layer;

Generate a first display picture and a second display picture according to the layer type of the first layer and the layer type of the second layer, and display the first picture on the screen of the first electronic device , projecting the second display picture to the screen of the second electronic device for display, the first display picture includes the first control and the second control, the second display picture includes the first control , excluding the second control.
The method according to claim 1, wherein the determining the layer type of the first layer according to the control information of the first control comprises:

extracting the control name of the first control and the size information of the first control from the control information of the first control;

Parsing the control name, when parsing the control type of the first control, the package name of the first application, and the interface information of the first control from the control name, according to the control type , the package name, the interface information and the size information, determine the layer type of the first layer;

When the control type of the first control and the interface information of the first control are parsed from the control name, and the package name of the first application is not parsed, obtain the process of drawing the first control The process identification number PID, determine the source of the first control according to the PID, determine the layer type of the first layer according to the control type, the source, the interface information and the size information , the source includes the package name of the first application.
The method according to claim 2, wherein, before parsing the control name, the method further comprises:

Using the control name and the size information as retrieval keywords;

According to the keyword, search for a control matching the keyword in the layer identification record library;

When a control matching the keyword is found, the layer type corresponding to the control is determined as the layer type of the first layer;

When no control matching the keyword is found, the step of parsing the control name is performed.
The method according to claim 3, further comprising:

When the layer type of the first layer cannot be determined according to the control information of the first control, acquire a screen currently displayed by the first application, where the screen currently displayed includes the first control;

The layer type of the first layer is determined according to the content displayed by the first control in the currently displayed screen.
The method according to claim 1, wherein the determining the layer type of the second layer according to the control information of the second control comprises:

extracting the control name of the second control and the size information of the second control from the control information of the second control;

Parsing the control name, when parsing the control type of the second control, the package name of the first application, and the interface information of the second control from the control name, according to the control type , the package name, the interface information and the size information, determine the layer type of the second layer;

When the control type of the second control and the interface information of the second control are parsed from the control name, and the package name of the first application is not parsed, obtain the process of drawing the second control The process identification number PID, determine the source of the second control according to the PID, determine the layer type of the second layer according to the control type, the source, the interface information and the size information , the source includes the package name of the first application.
The method according to claim 5, wherein the method further comprises:

When the layer type of the second layer cannot be determined according to the control information of the second control, acquire the currently displayed screen of the first application, the currently displayed screen includes the second control;

The layer type of the second layer is determined according to the content displayed by the second control in the currently displayed screen.
The method according to claim 1, wherein a first display picture and a second display picture are generated according to the layer type of the first layer and the layer type of the second layer, and Displaying the first picture on the screen of the first electronic device, and projecting the second display picture to the screen of the second electronic device for display includes:

Generate a first display picture and a second display picture according to the layer type of the first layer and the layer type of the second layer, and cache the first display picture into the first display cache, and store the first display picture in the first display buffer. The second display frame is buffered into the second display buffer;

Taking out the first display frame from the first display buffer according to the buffer sequence, and displaying the first display frame on the screen of the first electronic device;

Recording the second display picture in the second display cache to obtain screen projection content;

Sending the screen projection content to the second electronic device for the second electronic device to decode the screen projection content to obtain the second display picture and display it on the screen of the second electronic device.
The method according to claim 7, wherein the first display picture and the second display picture are generated according to the layer type of the first layer and the layer type of the second layer, and Caching the first display frame into the first display buffer, and caching the second display frame into the second display buffer includes:

determining a first layer filtering rule corresponding to the first display cache and a second layer filtering rule corresponding to the second display cache;

According to the filter rule of the first layer, the layer type of the first layer, and the layer type of the second layer, it is determined that the first display picture includes the first layer and the second layer. second layer;

Acquire the resource of the first control in the first layer and the resource of the second control in the second layer, and generate the resource according to the resource of the first control and the resource of the second control the first display frame, and buffer the first display frame into the first display buffer;

determining that the second display screen includes the first layer according to the second layer filtering rule, the layer type of the first layer, and the layer type of the second layer;

Acquiring the resource of the first control in the first layer, generating the second display frame according to the resource of the first control, and caching the second display frame into the second display cache.
The method according to claim 8, wherein the determining the first layer filter rule corresponding to the first display cache and the second layer filter rule corresponding to the second display cache comprises:

acquiring a first device identifier of the first electronic device and a second device identifier of the second electronic device;

Searching for a layer filtering rule matching the first device identifier in the layer filtering rule table, and determining the found layer filtering rule as the first layer filtering rule corresponding to the first display cache;

Searching for a layer filtering rule matching the second device identifier in the layer filtering rule table, and determining the found layer filtering rule as the second layer filtering rule corresponding to the second display cache .
The method according to claim 8, wherein the determining the first layer filter rule corresponding to the first display cache and the second layer filter rule corresponding to the second display cache comprises:

Displaying on the screen of the first electronic device a layer filter rule decision interface that can be operated by the user, the layer filter rule decision interface includes the first control and the first image where the first control is located The layer type of the layer, the layer type of the second control and the second layer where the second control is located;

generating the first layer filter rule in response to the user's operation behavior of setting the first layer filter rule for the first display cache;

The second layer filter rule is generated in response to the user's operation behavior of setting the second layer filter rule for the second display cache.
The method according to claim 7, wherein the recording the second display screen in the second display buffer to obtain screen projection content includes:

Obtaining a first screen aspect ratio of the first electronic device and a second screen aspect ratio of the second electronic device;

When the aspect ratio of the first screen is different from the aspect ratio of the second screen, performing black border removal processing on the second display picture in the second display buffer, and recording the second screen image after black border removal 2. Displaying the screen to obtain the screen projection content;

When the aspect ratio of the first screen is the same as that of the second screen, recording the second display picture in the second display buffer to obtain screen projection content.
The method according to claim 7, wherein the recording the second display screen in the second display buffer to obtain screen projection content includes:

acquiring the display capability of the second electronic device;

Determine the refresh frame rate of the video stream according to the display capability;

Recording the second display picture in the second display buffer according to the refresh frame rate of the video stream to obtain screen projection content.
The method according to any one of claims 1 to 12, wherein the first application is a conference application, the first control is a video control, and the layer type of the first layer is video stream A layer is played; the second control is a button control, and the layer type of the second layer is a conference control button layer.
The method according to any one of claims 1 to 12, wherein the first application is a conference application, the first control is a whiteboard annotation control, and the layer type of the first layer is whiteboard annotation layer; the second control is a button control, and the layer type of the second layer is a conference control button layer.
An electronic device, characterized in that the electronic device is a first electronic device, the first electronic device is installed with a first application, the first application includes a first control and a second control, and the first control Located on the first layer, the second control is located on the second layer, and the electronic equipment includes:

one or more processors;

memory;

and one or more computer programs, wherein said one or more computer programs are stored on said memory, and said computer programs, when executed by said one or more processors, cause said electronic device to perform the following steps:

acquiring control information of the first control and control information of the second control;

determining the layer type of the first layer according to the control information of the first control;

Determine the layer type of the second layer according to the control information of the second control, and the layer type of the first layer is different from the layer type of the second layer;

Generate a first display picture and a second display picture according to the layer type of the first layer and the layer type of the second layer, and display the first picture on the screen of the first electronic device , projecting the second display picture to the screen of the second electronic device for display, the first display picture includes the first control and the second control, the second display picture includes the first control , excluding the second control.
The electronic device according to claim 15, wherein when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps:

extracting the control name of the first control and the size information of the first control from the control information of the first control;

Parsing the control name, when parsing the control type of the first control, the package name of the first application, and the interface information of the first control from the control name, according to the control type , the package name, the interface information and the size information, determine the layer type of the first layer;

When the control type of the first control and the interface information of the first control are parsed from the control name, and the package name of the first application is not parsed, obtain the process of drawing the first control The process identification number PID, determine the source of the first control according to the PID, determine the layer type of the first layer according to the control type, the source, the interface information and the size information , the source includes the package name of the first application.
The electronic device according to claim 16, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

Using the control name and the size information as retrieval keywords;

According to the keyword, search for a control matching the keyword in the layer identification record library;

When a control matching the keyword is found, the layer type corresponding to the control is determined as the layer type of the first layer;

When no control matching the keyword is found, the step of parsing the control name is performed.
The electronic device according to claim 17, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

When the layer type of the first layer cannot be determined according to the control information of the first control, acquire a screen currently displayed by the first application, where the screen currently displayed includes the first control;

The layer type of the first layer is determined according to the content displayed by the first control in the currently displayed screen.
The electronic device according to claim 15, wherein when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps:

extracting the control name of the second control and the size information of the second control from the control information of the second control;

Parsing the control name, when parsing the control type of the second control, the package name of the first application, and the interface information of the second control from the control name, according to the control type , the package name, the interface information and the size information, determine the layer type of the second layer;

When the control type of the second control and the interface information of the second control are parsed from the control name, and the package name of the first application is not parsed, obtain the process of drawing the second control The process identification number PID, determine the source of the second control according to the PID, determine the layer type of the second layer according to the control type, the source, the interface information and the size information , the source includes the package name of the first application.
The electronic device according to claim 19, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

When the layer type of the second layer cannot be determined according to the control information of the second control, acquire the currently displayed screen of the first application, the currently displayed screen includes the second control;

The layer type of the second layer is determined according to the content displayed by the second control in the currently displayed screen.
The electronic device according to claim 15, wherein when the computer program is executed by the one or more processors, the electronic device is made to perform the following steps:

Generate a first display picture and a second display picture according to the layer type of the first layer and the layer type of the second layer, and cache the first display picture into the first display cache, and store the first display picture in the first display buffer. The second display frame is buffered into the second display buffer;

Taking out the first display frame from the first display buffer according to the buffer sequence, and displaying the first display frame on the screen of the first electronic device;

Recording the second display picture in the second display cache to obtain screen projection content;

Sending the screen projection content to the second electronic device for the second electronic device to decode the screen projection content to obtain the second display picture and display it on the screen of the second electronic device.
The electronic device according to claim 21, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

determining a first layer filtering rule corresponding to the first display cache and a second layer filtering rule corresponding to the second display cache;

According to the filter rule of the first layer, the layer type of the first layer, and the layer type of the second layer, it is determined that the first display picture includes the first layer and the second layer. second layer;

Acquire the resource of the first control in the first layer and the resource of the second control in the second layer, and generate the resource according to the resource of the first control and the resource of the second control the first display frame, and buffer the first display frame into the first display buffer;

determining that the second display screen includes the first layer according to the second layer filtering rule, the layer type of the first layer, and the layer type of the second layer;

Acquiring the resource of the first control in the first layer, generating the second display frame according to the resource of the first control, and caching the second display frame into the second display cache.
The electronic device according to claim 21, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

acquiring a first device identifier of the first electronic device and a second device identifier of the second electronic device;

Searching for a layer filtering rule matching the first device identifier in the layer filtering rule table, and determining the found layer filtering rule as the first layer filtering rule corresponding to the first display cache;

Searching for a layer filtering rule matching the second device identifier in the layer filtering rule table, and determining the found layer filtering rule as the second layer filtering rule corresponding to the second display cache .
The electronic device according to claim 21, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

Displaying on the screen of the first electronic device a layer filter rule decision interface that can be operated by the user, the layer filter rule decision interface includes the first control and the first image where the first control is located The layer type of the layer, the layer type of the second control and the second layer where the second control is located;

generating the first layer filter rule in response to the user's operation behavior of setting the first layer filter rule for the first display cache;

The second layer filter rule is generated in response to the user's operation behavior of setting the second layer filter rule for the second display cache.
The electronic device according to claim 20, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

Obtaining a first screen aspect ratio of the first electronic device and a second screen aspect ratio of the second electronic device;

When the aspect ratio of the first screen is different from the aspect ratio of the second screen, performing black border removal processing on the second display picture in the second display buffer, and recording the second screen image after black border removal 2. Displaying the screen to obtain the screen projection content;

When the aspect ratio of the first screen is the same as that of the second screen, recording the second display picture in the second display buffer to obtain screen projection content.
The electronic device according to claim 20, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

acquiring the display capability of the second electronic device;

Determine the refresh frame rate of the video stream according to the display capability;

Recording the second display picture in the second display buffer according to the refresh frame rate of the video stream to obtain screen projection content.
A screen projection method, characterized in that it is applied to a first electronic device that initiates screen projection, the first electronic device is installed with a first application, the first application includes a first control and a second control, and the first A control is located on the first layer, and the second control is located on the second layer, and the method includes:

acquiring control information of the first control and control information of the second control;

determining the layer type of the first layer according to the control information of the first control;

Determine the layer type of the second layer according to the control information of the second control, and the layer type of the first layer is different from the layer type of the second layer;

Generate a first display picture and a second display picture according to the layer type of the first layer and the layer type of the second layer, and display the first picture on the screen of the first electronic device , projecting the second display picture to the screen of the second electronic device for display, the first display picture includes the first control and the second control, the second display picture includes the first control , excluding the second control;

Wherein, the determining the layer type of the first layer according to the control information of the first control includes:

extracting the control name of the first control and the size information of the first control from the control information of the first control;

Analyzing the control name of the first control, and analyzing the control type of the first control, the package name of the first application, and the interface where the first control is located from the control name of the first control information, determine the layer type of the first layer according to the control type of the first control, the package name, the interface information where the first control is located, and the size information of the first control;

When the control type of the first control and the interface information of the first control are parsed from the control name of the first control, and the package name of the first application is not parsed, obtain and draw the first control The process identification number PID of the process of a control, the source of the first control is determined according to the PID for drawing the first control, the control type of the first control, the source of the first control, the first control The interface information of the control and the size information of the first control determine the layer type of the first layer, and the source of the first control includes the package name of the first application;

The determining the layer type of the second layer according to the control information of the second control includes:

extracting the control name of the second control and the size information of the second control from the control information of the second control;

Parse the control name of the second control, and analyze the control type of the second control, the package name of the first application, and the interface where the second control is located from the control name of the second control information, determine the layer type of the second layer according to the control type of the second control, the package name, the interface information where the second control is located, and the size information of the second control;

When the control type of the second control and the interface information of the second control are parsed from the control name of the second control, and the package name of the first application is not parsed, obtain and draw the second control. The process identification number PID of the process of the second control, determine the source of the second control according to the PID for drawing the second control, and determine the source of the second control according to the control type of the second control, the source of the second control, and the location of the second control The interface information of the second control and the size information of the second control determine the layer type of the second layer, and the source of the second control includes the package name of the first application.
The method according to claim 27, wherein, before parsing the control name, the method further comprises:

Using the control name and the size information as retrieval keywords;

According to the keyword, search for a control matching the keyword in the layer identification record library;

When a control matching the keyword is found, the layer type corresponding to the control is determined as the layer type of the first layer;

When no control matching the keyword is found, the step of parsing the control name is performed.
The method according to claim 28, further comprising:

When the layer type of the first layer cannot be determined according to the control information of the first control, acquire a screen currently displayed by the first application, where the screen currently displayed includes the first control;

The layer type of the first layer is determined according to the content displayed by the first control in the currently displayed screen.
The method according to claim 27, further comprising:

When the layer type of the second layer cannot be determined according to the control information of the second control, acquire the currently displayed screen of the first application, the currently displayed screen includes the second control;

The layer type of the second layer is determined according to the content displayed by the second control in the currently displayed screen.
The method according to claim 27, wherein the first display picture and the second display picture are generated according to the layer type of the first layer and the layer type of the second layer, and Displaying the first picture on the screen of the first electronic device, and projecting the second display picture to the screen of the second electronic device for display includes:

Generate a first display picture and a second display picture according to the layer type of the first layer and the layer type of the second layer, and cache the first display picture into the first display cache, and store the first display picture in the first display buffer. The second display frame is buffered into the second display buffer;

Taking out the first display frame from the first display buffer according to the buffer sequence, and displaying the first display frame on the screen of the first electronic device;

Recording the second display picture in the second display cache to obtain screen projection content;

Sending the screen projection content to the second electronic device for the second electronic device to decode the screen projection content to obtain the second display picture and display it on the screen of the second electronic device.
The method according to claim 31, characterized in that, according to the layer type of the first layer and the layer type of the second layer, a first display picture and a second display picture are generated, and Caching the first display frame into the first display buffer, and caching the second display frame into the second display buffer includes:

determining a first layer filtering rule corresponding to the first display cache and a second layer filtering rule corresponding to the second display cache;

According to the filter rule of the first layer, the layer type of the first layer, and the layer type of the second layer, it is determined that the first display picture includes the first layer and the second layer. second layer;

Acquire the resource of the first control in the first layer and the resource of the second control in the second layer, and generate the resource according to the resource of the first control and the resource of the second control the first display frame, and buffer the first display frame into the first display buffer;

determining that the second display screen includes the first layer according to the second layer filtering rule, the layer type of the first layer, and the layer type of the second layer;

Acquiring the resource of the first control in the first layer, generating the second display frame according to the resource of the first control, and caching the second display frame into the second display cache.
The method according to claim 32, wherein the determining the first layer filter rule corresponding to the first display cache and the second layer filter rule corresponding to the second display cache comprises:

acquiring a first device identifier of the first electronic device and a second device identifier of the second electronic device;

Searching for a layer filtering rule matching the first device identifier in the layer filtering rule table, and determining the found layer filtering rule as the first layer filtering rule corresponding to the first display cache;

Searching for a layer filtering rule matching the second device identifier in the layer filtering rule table, and determining the found layer filtering rule as the second layer filtering rule corresponding to the second display cache .
The method according to claim 32, wherein the determining the first layer filter rule corresponding to the first display cache and the second layer filter rule corresponding to the second display cache comprises:

Displaying on the screen of the first electronic device a layer filter rule decision interface that can be operated by the user, the layer filter rule decision interface includes the first control and the first image where the first control is located The layer type of the layer, the layer type of the second control and the second layer where the second control is located;

generating the first layer filter rule in response to the user's operation behavior of setting the first layer filter rule for the first display cache;

The second layer filter rule is generated in response to the user's operation behavior of setting the second layer filter rule for the second display cache.
The method according to claim 31, wherein the recording of the second display screen in the second display buffer to obtain screen projection content includes:

Obtaining a first screen aspect ratio of the first electronic device and a second screen aspect ratio of the second electronic device;

When the aspect ratio of the first screen is different from the aspect ratio of the second screen, performing black border removal processing on the second display picture in the second display buffer, and recording the second screen image after black border removal 2. Displaying the screen to obtain the screen projection content;

When the aspect ratio of the first screen is the same as that of the second screen, recording the second display picture in the second display buffer to obtain screen projection content.
The method according to claim 31, wherein the recording of the second display screen in the second display buffer to obtain screen projection content includes:

acquiring the display capability of the second electronic device;

Determine the refresh frame rate of the video stream according to the display capability;

Recording the second display picture in the second display buffer according to the refresh frame rate of the video stream to obtain screen projection content.
The method according to any one of claims 27 to 36, wherein the first application is a conference application, the first control is a video control, and the layer type of the first layer is video stream A layer is played; the second control is a button control, and the layer type of the second layer is a conference control button layer.
The method according to any one of claims 27 to 36, wherein the first application is a conference application, the first control is a whiteboard annotation control, and the layer type of the first layer is whiteboard annotation layer; the second control is a button control, and the layer type of the second layer is a conference control button layer.
An electronic device, characterized in that the electronic device is a first electronic device, the first electronic device is installed with a first application, the first application includes a first control and a second control, and the first control Located on the first layer, the second control is located on the second layer, and the electronic equipment includes:

one or more processors;

memory;

and one or more computer programs, wherein said one or more computer programs are stored on said memory, and said computer programs, when executed by said one or more processors, cause said electronic device to perform the following steps:

acquiring control information of the first control and control information of the second control;

determining the layer type of the first layer according to the control information of the first control;

Determine the layer type of the second layer according to the control information of the second control, and the layer type of the first layer is different from the layer type of the second layer;

Generate a first display picture and a second display picture according to the layer type of the first layer and the layer type of the second layer, and display the first picture on the screen of the first electronic device , projecting the second display picture to the screen of the second electronic device for display, the first display picture includes the first control and the second control, the second display picture includes the first control , excluding the second control;

Wherein, the determining the layer type of the first layer according to the control information of the first control includes:

extracting the control name of the first control and the size information of the first control from the control information of the first control;

Analyzing the control name of the first control, and analyzing the control type of the first control, the package name of the first application, and the interface where the first control is located from the control name of the first control information, determine the layer type of the first layer according to the control type of the first control, the package name, the interface information where the first control is located, and the size information of the first control;

When the control type of the first control and the interface information of the first control are parsed from the control name of the first control, and the package name of the first application is not parsed, obtain and draw the first control The process identification number PID of the process of a control, the source of the first control is determined according to the PID for drawing the first control, the control type of the first control, the source of the first control, the first control The interface information of the control and the size information of the first control determine the layer type of the first layer, and the source of the first control includes the package name of the first application;

The determining the layer type of the second layer according to the control information of the second control includes:

extracting the control name of the second control and the size information of the second control from the control information of the second control;

Parse the control name of the second control, and analyze the control type of the second control, the package name of the first application, and the interface where the second control is located from the control name of the second control information, determine the layer type of the second layer according to the control type of the second control, the package name, the interface information where the second control is located, and the size information of the second control;

When the control type of the second control and the interface information of the second control are parsed from the control name of the second control, and the package name of the first application is not parsed, obtain and draw the second control. The process identification number PID of the process of the second control, determine the source of the second control according to the PID for drawing the second control, and determine the source of the second control according to the control type of the second control, the source of the second control, and the location of the second control The interface information of the second control and the size information of the second control determine the layer type of the second layer, and the source of the second control includes the package name of the first application.
The electronic device according to claim 39, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

Using the control name and the size information as retrieval keywords;

According to the keyword, search for a control matching the keyword in the layer identification record library;

When a control matching the keyword is found, the layer type corresponding to the control is determined as the layer type of the first layer;

When no control matching the keyword is found, the step of parsing the control name is performed.
The electronic device according to claim 40, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

When the layer type of the first layer cannot be determined according to the control information of the first control, acquire a screen currently displayed by the first application, where the screen currently displayed includes the first control;

The layer type of the first layer is determined according to the content displayed by the first control in the currently displayed screen.
The electronic device according to claim 39, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

When the layer type of the second layer cannot be determined according to the control information of the second control, acquire the currently displayed screen of the first application, the currently displayed screen includes the second control;

The layer type of the second layer is determined according to the content displayed by the second control in the currently displayed screen.
The electronic device according to claim 39, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

Generate a first display picture and a second display picture according to the layer type of the first layer and the layer type of the second layer, and cache the first display picture into the first display cache, and store the first display picture in the first display buffer. The second display frame is buffered into the second display buffer;

Taking out the first display frame from the first display buffer according to the buffer sequence, and displaying the first display frame on the screen of the first electronic device;

Recording the second display picture in the second display cache to obtain screen projection content;

Sending the screen projection content to the second electronic device for the second electronic device to decode the screen projection content to obtain the second display picture and display it on the screen of the second electronic device.
The electronic device according to claim 43, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

determining a first layer filtering rule corresponding to the first display cache and a second layer filtering rule corresponding to the second display cache;

According to the filter rule of the first layer, the layer type of the first layer, and the layer type of the second layer, it is determined that the first display picture includes the first layer and the second layer. second layer;

Acquire the resource of the first control in the first layer and the resource of the second control in the second layer, and generate the resource according to the resource of the first control and the resource of the second control the first display frame, and buffer the first display frame into the first display buffer;

determining that the second display screen includes the first layer according to the second layer filtering rule, the layer type of the first layer, and the layer type of the second layer;

Acquiring the resource of the first control in the first layer, generating the second display frame according to the resource of the first control, and caching the second display frame into the second display cache.
The electronic device according to claim 43, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

acquiring a first device identifier of the first electronic device and a second device identifier of the second electronic device;

Searching for a layer filtering rule matching the first device identifier in the layer filtering rule table, and determining the found layer filtering rule as the first layer filtering rule corresponding to the first display cache;

Searching for a layer filtering rule matching the second device identifier in the layer filtering rule table, and determining the found layer filtering rule as the second layer filtering rule corresponding to the second display cache .
The electronic device according to claim 43, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

Displaying on the screen of the first electronic device a layer filter rule decision interface that can be operated by the user, the layer filter rule decision interface includes the first control and the first image where the first control is located The layer type of the layer, the layer type of the second control and the second layer where the second control is located;

generating the first layer filter rule in response to the user's operation behavior of setting the first layer filter rule for the first display cache;

The second layer filter rule is generated in response to the user's operation behavior of setting the second layer filter rule for the second display cache.
The electronic device according to claim 42, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

Obtaining a first screen aspect ratio of the first electronic device and a second screen aspect ratio of the second electronic device;

When the aspect ratio of the first screen is different from the aspect ratio of the second screen, performing black border removal processing on the second display picture in the second display buffer, and recording the second screen image after black border removal 2. Displaying the screen to obtain the screen projection content;

When the aspect ratio of the first screen is the same as that of the second screen, recording the second display picture in the second display buffer to obtain screen projection content.
The electronic device according to claim 42, wherein when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps:

acquiring the display capability of the second electronic device;

Determine the refresh frame rate of the video stream according to the display capability;

Recording the second display picture in the second display buffer according to the refresh frame rate of the video stream to obtain screen projection content.
A computer-readable storage medium, comprising a computer program, characterized in that, when the computer program is run on an electronic device, the electronic device is made to execute the screen projection method according to any one of claims 1 to 14, Or execute the screen projection method according to any one of claims 27 to 38.
A chip, characterized in that it includes: one or more processing circuits and one or more transceiver pins; wherein, the transceiver pins and the processing circuits communicate with each other through an internal connection path, and the processing circuits implement rights The screen projection method described in any one of claims 1 to 14 is used to control the receiving pin to receive signals, to control the sending pin to send signals, or to perform the screen projection method described in any one of claims 27 to 38.