WO2023273905A1

WO2023273905A1 - Screen synchronization method for information, transmitter, receiver, and computer-readable storage medium

Info

Publication number: WO2023273905A1
Application number: PCT/CN2022/099222
Authority: WO
Inventors: 张忠威
Original assignee: 中兴通讯股份有限公司
Priority date: 2021-06-28
Filing date: 2022-06-16
Publication date: 2023-01-05
Also published as: CN115599326A

Abstract

A screen synchronization method for information, a transmitter, a receiver, and a computer-readable storage medium. The screen synchronization method for information comprises: a transmitter obtaining image information of a target image layer where target display information is located (S100); and transmitting the image information to a receiver, so that the receiver displays the image information on a display end (S200).

Description

Information sharing screen method, sending end, receiving end and computer-readable storage medium

Cross References to Related Applications

This application is based on a Chinese patent application with application number 202110722446.5 and a filing date of June 28, 2021, and claims the priority of this Chinese patent application. The entire content of this Chinese patent application is hereby incorporated by reference into this application.

technical field

The embodiments of the present application relate to the technical field of communication, for example, to a method for sharing information on the same screen, a sending end, a receiving end, and a computer-readable storage medium.

Background technique

In today's society, with the popularity of smart mobile terminals (such as mobile phones, tablet computers, etc.) Smartphones and tablets have become indispensable devices for work, life and entertainment. However, due to the limitation of the screen size of the mobile phone and the flat panel, multimedia resources cannot be viewed well on the mobile phone and the flat panel, which affects people's life and work to a certain extent. In this regard, Miracast (mirror technology), airplay (airplay technology) and other screen technologies have been introduced in some cases to synchronize the screen of a mobile phone or tablet to a large-screen smart device.

These technologies can meet most of the usage scenarios of ordinary users, but now people are very sensitive to private information, and the same-screen technology will display the screen of the mobile phone or tablet on the large-screen device without reservation, so that the user's personal information is completely exposed. under the public. In particular, when organizing meetings and discussing project documents, the same-screen technology will display all the news reminders on the user's mobile phone or tablet on the large screen, such as displaying personal privacy information such as mobile phone message pop-up boxes and new message reminders. Displayed on a large screen is extremely unfriendly to personal privacy protection.

Contents of the invention

Embodiments of the present application provide a method for sharing information on the same screen, a sending end, a receiving end, and a computer-readable storage medium.

In the first aspect, the embodiment of the present application provides a method for sharing information on the same screen, which is applied to the sending end. The method for sharing information on the same screen includes: obtaining the image information of the target layer where the target display information is located; sending the image information to the receiving end, so that the receiving end displays the image information on the display end.

In the second aspect, the embodiment of the present application provides a method for sharing information on the same screen, which is applied to the receiving end. The method for sharing information on the same screen includes: receiving image information from the sending end, wherein the image information is the The image information of the target layer where the target display information is located; displaying the image information on the display terminal.

In a third aspect, an embodiment of the present application provides a sending end, including: a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor executes the computer program At the same time, the method for sharing information on the same screen as described in the first aspect of the present application is realized.

In a fourth aspect, an embodiment of the present application provides a receiving end, including: a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor executes the computer program At the same time, the method for sharing information on the same screen as described in the second aspect of the present application is realized.

In a fifth aspect, the embodiment of the present application provides a computer-readable storage medium, storing computer-executable instructions, and the computer-executable instructions are used to execute the information screen sharing method as described above.

Additional features and advantages of the application will be set forth in the description which follows, and, in part, will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Description of drawings

The accompanying drawings are used to provide a further understanding of the technical solution of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical solution of the present application, and do not constitute a limitation to the technical solution of the present application.

Fig. 1 is a schematic structural diagram of an information screen system provided by an embodiment of the present application;

FIG. 2 is a schematic flowchart of a method for sharing information on the same screen provided by an embodiment of the present application;

FIG. 3 is a schematic flowchart of a method for sharing information on the same screen provided by another embodiment of the present application;

FIG. 4 is a schematic flowchart of a method for sharing information on the same screen according to another embodiment of the present application;

FIG. 5 is a schematic structural diagram of a terminal provided by an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a terminal provided by another embodiment of the present application;

FIG. 7 is a schematic flowchart of a method for sharing information on the same screen provided by another embodiment of the present application;

FIG. 8 is a schematic diagram of a network distribution process provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of a same-screen preparation process provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of the same-screen encoding process provided by an embodiment of the present application;

Fig. 11 is a schematic diagram of an on-screen decoding process provided by an embodiment of the present application.

detailed description

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, not to limit the present application.

It should be noted that although the functional modules are divided in the schematic diagram of the device, and the logical sequence is shown in the flowchart, in some cases, it can be executed in a different order than the module division in the device or the flowchart in the flowchart. steps shown or described. The terms "first", "second" and the like in the specification, claims or the above drawings are used to distinguish similar objects, and not necessarily used to describe a specific order or sequence.

Referring to Fig. 1 to Fig. 3, an embodiment of the present application provides an information same-screen system, as shown in Fig. 1, the system includes a sending end and a receiving end, wherein the sending end includes but not limited to mobile phones, computers, tablet computers and TVs and other devices with screen projection functions; the receiving end includes but not limited to screen projection boxes, projectors, TVs and other devices with screen projection functions.

For ease of description, in the embodiments of this document, a mobile phone is used as an example, and a screen projection box is used as an example at the receiving end. The matching process between the mobile phone and the screen projection box is as follows: First, the screen projection box cannot operate independently to access the network, and needs to use the mobile phone to configure the network for it, so the screen projection box generates Bluetooth name, MAC (Media Access Control, Media Access Control) A two-dimensional code of address and other information; then, the screen-casting APP (Application, application) on the mobile phone scans the two-dimensional code to obtain the information contained in the code, and realizes the Bluetooth pairing of the two; then, the screen-casting APP Send the SSID (Service Set Identifier) and password of wifi (Wireless Fidelity) to the screen projection box, and the screen projection box will realize the LAN connection between the two according to the received wifi SSID and password.

Referring to Fig. 2, the system implements a method for sharing information on the same screen, and the method includes the following steps:

Step S100, the sending end obtains the image information of the target layer where the target display information is located.

In this step S100, the target display information refers to the information on the mobile phone that needs to be on the same screen as the projection box. It is worth noting that the target display information includes one of the following: document information, or picture information, or video information, wherein document information is such as a word document executed by a mobile phone, picture information is such as a photo viewed on a mobile phone, and video information is such as a video played on a mobile phone. video. If the target display information is a video, the video needs to be on the same screen as the projection box, so that the user can see the corresponding screen on the corresponding display through the projection box. If the target display information is a word document, the document needs to be on the same screen as the projection box, so that the user can see the corresponding document on the corresponding display through the projection box.

Due to the different levels of different applications on the mobile phone, for example, photo applications and video applications are at different levels, photo applications and chat software such as WeChat are at different levels, and chat software such as WeChat and QQ are at different levels. Here For existing public knowledge, no longer describe in detail. Therefore, the target layer where the target display information is located refers to the layer where the target display information needs to be located on the mobile phone. Image information refers to bitmap (raster image or bitmap) data. Obtaining image information is also called screenshot. In order to realize screen projection, it is necessary to collect image information first, and then send the collected image information to the receiving end, so that the receiving end can Play on the same screen.

In this embodiment, the sending end collects image information at a set frequency, such as but not limited to, collecting image information at a frequency of 20 times per second, so as to maintain the clarity of the collected image information and ensure the projection effect of the image . In this step, the process for the sender to obtain image information is, for example, for a word document on the same screen, the screen-casting APP on the mobile phone will set a thread task, through which the API interface tool at the bottom of the mobile phone will be controlled at 20 times per second. The acquisition of the image information corresponding to the layer occupied by the photo application is completed at a frequency of times.

Step S200, the sending end sends the image information to the receiving end, so that the receiving end displays the image information on the display end.

In this step S200, when the receiving end is a screen projection box, since the screen projection box does not have a display function, the display end refers to a display that establishes a connection with the screen projection box. When the receiving end is a projector, the display end refers to a projection module on the projector.

It is worth noting that the mobile phone will encode and encapsulate the image information and send it to the receiving end. For example: after the sender performs H264 encoding on the image information, and encapsulates it according to a self-defined data format, to obtain the encapsulated data. It should be noted that the H264 encoding performed by the sending end and custom encapsulation are common knowledge of those skilled in the art, and will not be described in detail here. After the screen projection box receives the data, it will first decapsulate and decode it to obtain the image information, render the image information, and then send the rendered image information to the rendering area of the screen projection box, and then connect the screen projection box to the display Afterwards, the data in the rendered area will be directly transferred to the display, and the display will be displayed.

In this embodiment, the sending end obtains the image information of the target layer where the target display information is located, and sends the image information to the corresponding receiving end, so as to realize the same-screen display of the target display information. Unlike some situations where the mobile terminal directly uses the system’s native screen capture function to obtain screen pictures, this embodiment directly obtains image information from the corresponding layer of the target display information that needs to be displayed on the same screen, and can accurately share the image information that needs to be displayed on the same screen, avoiding Image information on unrelated layers on the same screen can effectively protect user privacy.

It should be noted that, based on the above-mentioned embodiment, before step S100, further steps are included:

Step S101 , acquiring target display information, and displaying the target display information on a target layer.

Step S102 , acquiring a same-screen instruction for displaying target display information on the same screen, and determining a target layer where the target display information is located according to the same-screen instruction.

In step S102, the same-screen instruction can be an instruction sent by the same-screen box, or an instruction created by the screen-casting App on the mobile phone. After the mobile phone obtains the instruction, it determines the target display information that needs to be displayed on the same screen according to the instruction. And the target layer corresponding to the target display information.

Referring to Fig. 3, it is worth noting that, based on the above-mentioned embodiment, further steps are included:

Step S300, the sending end obtains non-target display information, and displays the non-target display information on a non-target layer.

It is worth noting that the non-target display information includes one of the following: pop-up message or reminder message. The pop-up message or reminder message contains the personal privacy of the mobile phone user. For example, the mobile phone will receive a WeChat message pop-up window or a QQ message pop-up window. If the target display information at this time is a photo, then according to some situations, the mobile phone terminal directly Using the system's native screen capture function to obtain screen pictures will transmit information and data related to personal privacy, such as WeChat message pop-up windows, QQ message pop-up windows, etc., to the screen projection box, which leaks personal privacy. The purpose of this implementation is that the same screen is required The target display information is displayed on the target layer; the non-target display information that does not need to be on the same screen is displayed on the non-target layer; in this way, the data that needs to be on the same screen can be accurately displayed on the same screen, and image information on irrelevant layers on the same screen can be avoided. Avoid divulging personal privacy.

It should be noted that, based on the above embodiments, the sending end and the receiving end are connected to the same local area network, and the communication connection is realized through the TCP protocol. The implementation process includes (taking the target display information as a word document as an example): the sending end sends the first request and page parameters to the receiving end, so that the receiving end starts the corresponding same-screen page according to the first request and page parameters, and makes the receiving end The end feeds back the corresponding second request; wherein, the page parameters include page width and height and resolution; the sending end receives the second request, performs same-screen initialization, and sends a third request to the receiving end. Complete the three-way handshake process.

In this embodiment, the sending end and the receiving end are connected through the TCP protocol, and the connection is recommended through the three-way handshake mechanism, so that the communication link has the advantages of reliability and stability.

An embodiment of the present application provides an information screen-sharing system. The system includes a sending end and a receiving end, wherein the sending end includes but not limited to mobile phones, computers, tablet computers, and TVs and other devices with screen projection functions; the receiving end Including but not limited to screen projection boxes, projectors, TVs and other devices with screen projection functions.

Referring to Fig. 4, the system implements a method for sharing information on the same screen, and the method includes the following steps:

Step S400, the receiving end receives the image information from the sending end, wherein the image information is the image information of the target layer where the target presentation information in the sending end is located.

Step S500, the receiving end displays the image information on the display end.

It should be noted that this implementation is based on the same inventive concept as steps S100 to S200 in the above-mentioned embodiment, so the content of the above-mentioned embodiment is also applicable to this embodiment, and will not be repeated here.

Referring to FIG. 5, an embodiment of the present application provides a sending end, including: a data acquisition unit 100 and a data sending unit 200, wherein:

The data acquiring unit 100 acquires the image information of the target layer where the target presentation information is located.

The data sending unit 200 sends the image information to the receiving end, so that the receiving end displays the image information on the display end.

It should be noted that this implementation is based on the same inventive concept as steps S100 to S200 in the above method embodiment, so the content of the above embodiment is also applicable to this embodiment, and will not be repeated here.

Referring to FIG. 6, an embodiment of the present application provides a receiving end, including: a data receiving unit 300 and a same-screen display unit 300, wherein:

The data receiving unit 300 is configured to receive image information from the sender, wherein the image information is image information of the target layer where the target presentation information in the sender is located.

The on-screen display unit 400 is used to display image information on a display terminal.

It should be noted that this implementation is based on the same inventive concept as steps S400 to S500 in the above method embodiment, so the content of the above embodiment is also applicable to this embodiment, and will not be repeated here.

Referring to Fig. 7 to Fig. 11, an embodiment of the present application provides an information same-screen system, the system includes: same-screen sender, sender APP, same-screen receiver, receiver APP and same-screen device inter-link road. This system takes the document page as an example.

Same-screen sender: iOS/Android mobile phone/tablet; sender APP: used to take screenshots of mobile phones/tablets, and encode the screenshots in H264, and then encapsulate them into the agreed data format; same-screen receiver: used to decode And display the screen projection box of the Android smart system; the receiving end APP: used to decode the encoded stream and display the image on the large screen; the link between the devices on the same screen: TCP long chain, used to connect the sending end The encoded data is transmitted to the receiving end.

Referring to Fig. 7, the system can implement a method for sharing information on the same screen, which includes the following steps:

First, the mobile phone and the projection box are connected to the same local area network and run the APPs at both ends.

Second, the mobile APP discovers and connects to the projection box in the LAN.

Third, open the document on the mobile phone and click the "same screen" button, and the TV will automatically open the same screen page.

Fourth, the document page on the mobile phone is displayed on the same screen as the TV.

Examples are as follows:

Referring to Figure 8, the process of connecting the mobile phone and the projection box to the same LAN is as follows:

Since the projection box does not have a remote control, it cannot operate independently to access the network, so it needs to use a mobile phone to configure the network for it. The operation steps are as follows:

1. Connect the projection box to the TV, connect it to the power and start it up, and a QR code will be generated on the homepage, which contains the Bluetooth name of the box, MAC address and other information;

2. Turn on the "scan code" function of the mobile APP, scan the QR code on the TV, and you can get the information in the QR code;

3. Scan the Bluetooth device with the same name on the mobile phone to achieve pairing, and transmit the SSID and password of the WiFi that needs to be connected to the box through Bluetooth;

4. After receiving the Bluetooth message, the box end obtains the SSID and password for networking;

5. So far, the mobile phone and the projection box have been connected to the same LAN.

Referring to Figure 9, the preparation process for the same screen function is as follows:

To use the same screen function, the user needs to click the "Same Screen" button of the mobile APP. At this time, the mobile APP and the box APP will carry out the preparatory work before the same screen, as follows:

1. The user clicks the "same screen" button of the mobile APP;

2. The mobile phone sends a request for the same screen to the box-side APP through the link, and carries the data of the width, height and resolution of the document display page on the mobile phone;

3. After receiving the message, the box end starts the same-screen page, and replies the message to the mobile phone, indicating that it has received the same-screen message and is ready;

4. When the mobile phone receives the reply from the box, it immediately initiates the same-screen initialization and starts the same-screen function.

Referring to Figure 10, the encoding process of the mobile phone on the same screen is as follows:

In order to protect user privacy, the same screen function of this application will not intercept user message notifications, so the mobile phone cannot directly use the system’s native screen capture function to obtain screen images, but obtain screenshots of the page where the document is displayed for encoding. The principle is as follows:

1. The mobile APP obtains the page of the current display document at a frequency of 20 times per second, and sends it to the encoder in sequence as frame data;

2. The encoder performs H264 encoding on the page data and outputs them sequentially;

3. The encoded data is packaged into a custom data format, and sent to the projection box APP through the link connected to the box end in turn.

Referring to Figure 11, the decoding and display process of the same screen box is as follows:

The box end continuously receives the frame data sent from the mobile phone end, which needs to be processed and rendered before it can be displayed in the form of video. The principle is as follows:

1. The Screencasting Box APP receives the data, and obtains the coded data by analyzing the custom data;

2. The encoded data is sent to the decoder for decoding, and the decoded frame data is rendered on the display area of the box end;

3. The above process is continuously executed to realize the function of sharing the document page on the mobile terminal to the TV terminal on the same screen.

An embodiment of the present application provides a terminal, and the terminal includes: a memory, a processor, and a computer program stored in the memory and operable on the processor.

The processor and memory can be connected by a bus or other means.

As a non-transitory computer-readable storage medium, memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory includes memory located remotely from the processor, and these remote memories may be connected to the processor through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

It should be noted that the terminal in this embodiment can be applied as the sending end or the receiving end in the embodiment shown in FIG. 1 , and the terminal in this embodiment can constitute a part of the system architecture in the embodiment shown in FIG. 1 , these embodiments all belong to the same inventive concept, so these embodiments have the same implementation principle and technical effect, and will not be described in detail here.

The non-transitory software programs and instructions required to realize the information screen-on-screen method of the above-mentioned embodiment are stored in the memory, and when executed by the processor, the information screen-on-screen method in the above-mentioned embodiment is executed, for example, executing the above-described Figure 2 The method steps S100 to S200 in FIG. 4 and the method steps S400 to S500 in FIG. 4 .

The terminal embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, the eleventh embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by a processor or a controller, for example, Executed by a processor in the above-mentioned terminal embodiment, the above-mentioned processor can be made to execute the information screen-sharing method in the above-mentioned embodiment, for example, execute the method steps S100 to S200 in FIG. 2 described above and the method steps in FIG. 4 S400 to S500.

The embodiment of the present application includes: the sending end of the embodiment of the present application obtains the image information of the target layer where the target display information is located, and sends the image information to the corresponding receiving end, so as to realize the same-screen display of the target display information. Unlike some situations where the mobile terminal directly uses the system’s native screen capture function to obtain screen pictures, the embodiment of this application directly obtains image information from the corresponding layer of the target display information that needs to be displayed on the same screen, which can accurately synchronize the data that needs to be displayed on the same screen, avoiding The image information on the irrelevant layer of the same screen can effectively protect the user's personal privacy.

Those skilled in the art can understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware and an appropriate combination thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit . Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. permanent, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

The above is the description of the embodiments of the present application, but the present application is not limited to the above-mentioned embodiments, and those skilled in the art can also make various equivalent deformations or replacements without violating the sharing conditions of the spirit of the present application. Any modification or substitution is included within the scope defined by the claims of the present application.

Claims

A method for sharing information on the same screen, which is applied to the sending end, and the method for sharing information on the same screen includes:

Obtain the image information of the target layer where the target display information is located;

sending the image information to a receiving end, so that the receiving end displays the image information on a display end.
The information co-screening method according to claim 1, wherein, before the acquisition of the image information of the target layer where the target display information is located, the information co-screening method further comprises:

Obtain target display information, and display the target display information on the target layer;

Acquiring a same-screen instruction for displaying the target display information on the same screen, and determining the target layer where the target display information is located according to the same-screen instruction.
The method for sharing information on the same screen according to claim 1, wherein the method for sharing information on the same screen further comprises:

Obtain non-target display information, and display the non-target display information on a non-target layer.
The method according to claim 1, wherein said acquiring the image information of the target layer where the target display information is located comprises:

The image information of the target layer where the target display information is located is collected at a set frequency.
The method according to claim 1, wherein the sending the image information to the receiving end comprises:

After the image information is encoded and encapsulated, it is sent to the receiving end.
The method according to claim 3, wherein the non-target display information includes at least one of the following: pop-up message or reminder message.
The method according to any one of claims 1 to 6, wherein the target presentation information includes at least one of the following: document information, or picture information, or video information.
A method for sharing information on the same screen, applied to a receiving end, the method for sharing information on the same screen includes:

receiving image information from the sending end, wherein the image information is the image information of the target layer where the target presentation information in the sending end is located;

The image information is displayed on the display terminal.
The method according to claim 8, wherein when the image information is coded and packaged image information, displaying the image information on the display terminal includes:

The encoded and encapsulated image information is decapsulated and decoded, and the decapsulated and decoded image information is displayed on the display terminal.
The method according to claim 8 or 9, wherein the target display information includes at least one of the following: document information, or picture information, or video information.
A sending end, comprising: a memory, a processor, and a computer program stored on the memory and operable on the processor, when the processor executes the computer program, any A method for sharing information on one screen.
A receiving end, comprising: a memory, a processor, and a computer program stored on the memory and operable on the processor, when the processor executes the computer program, any of the following claims 8 to 10 are realized. A method for sharing information on one screen.
A computer-readable storage medium, storing computer-executable instructions, the computer-executable instructions are used to execute the information screen-sharing method according to any one of claims 1 to 7 or any one of claims 8 to 10 The information screen method described in one item.