CN111190564B

CN111190564B - Screen interaction method and system

Info

Publication number: CN111190564B
Application number: CN202010035935.9A
Authority: CN
Inventors: 周清会; 柳荣; 魏贤杰
Original assignee: Shanghai Manheng Digital Technology Co ltd
Current assignee: Shanghai Manheng Digital Technology Co ltd
Priority date: 2020-01-14
Filing date: 2020-01-14
Publication date: 2024-02-13
Anticipated expiration: 2040-01-14
Also published as: CN111190564A

Abstract

The application relates to a screen interaction method and system, and relates to the technical field of Internet. The method comprises the following steps: inputting client information into a server queue; requesting a server queue, and inputting client information to a system through a control terminal; when the client is confirmed to enter the system, screen information is sent to the server through the client; the server is requested to acquire the client IP list, and the screen information of the client is loaded through the control terminal. According to the screen interaction method and system, the HTTP service is started to be accessed by other terminal equipment, the CMD command is utilized to newly name the firewall rule and the port is started, the firewall rule can be accessed by other terminal equipment in the local area network, the screen of the same terminal equipment is monitored by multiple screens, the problems of screen sharing delay, blocking, white screen and the like are solved, and the user experience is comprehensively improved.

Description

Screen interaction method and system

Technical Field

The application relates to the technical field of Internet, in particular to a screen interaction method and system.

Background

With the development of internet application, remote control or sharing and the like realized by a network are utilized as new interaction modes, so that different users can interact with each other. Screen interaction is one of important applications of data sharing technology, and is widely applied to the fields of document display or operation color, collaborative office or teaching and the like. The screen interaction technology is based on a graphical user interface (Graphical User Interface, GUI) and can include various interaction forms such as monitoring the screen of other terminals through one terminal device, sharing the screen of one of the terminal devices through the other terminal device, and sharing the screen between different types of terminals.

In the Windows self-contained remote monitoring scenario, the monitored screen is closed to the login interface. The current remote monitoring software can realize monitoring and the monitored end can also operate, but only one terminal device can be monitored. In a scene using a Winform plug-in monitor screen, a white screen phenomenon occurs when the number of terminal devices increases, and the waiting time is long, and the jam is easy.

Therefore, it is desirable to provide a screen interaction method and system, which can be accessed by other terminal devices in a local area network by starting HTTP service for other terminal devices to access, using CMD command to name firewall rules and starting ports, so as to realize multi-screen monitoring of the screen of the same terminal device, solve the problems of screen sharing delay, blocking, white screen and the like, and comprehensively improve user experience.

Disclosure of Invention

According to a first aspect of some embodiments of the present application, there is provided a screen interaction method applied in a terminal (e.g., an electronic device, etc.), the method may include: inputting client information into a server queue; requesting a server queue, and inputting client information to a system through a control terminal; when the client is confirmed to enter the system, screen information is sent to the server through the client; the server is requested to acquire the client IP list, and the screen information of the client is loaded through the control terminal.

In some embodiments, the method may further comprise: client monitoring is performed by the control end.

In some embodiments, the method may further comprise: and executing screen sharing of the client to other clients through the control terminal.

In some embodiments, the method may further comprise: according to the IP list, selecting a client to be monitored through a control terminal and sending UDP information; according to the UDP information, saving screen information through a client and starting HTTP service; executing CMD command by the client to open the accessible port and returning the service address to the control end.

In some embodiments, the opening HTTP service further comprises: according to the UDP information, saving the picture stream of the screen information to the local by the client at a frame rate of 25 frames per second; when the HTTP service is started, the HTTP service points to the local image stream and can be accessed by clients in the local area network.

In some embodiments, the performing, by the control side, client monitoring further comprises: and opening a window by the control end according to the service address to monitor the client screen.

In some embodiments, the performing, by the control terminal, screen sharing of the client to the other client further includes: using UDP, sending the service address of the client to other clients through the control terminal; according to the service address, windows are opened by other clients to share the client screen.

In some embodiments, the entering the client information into the server queue further comprises: according to the HTTP protocol, sending information to a server through a client; client information is entered into a Redis queue maintained by the server.

In some embodiments, the validating the client entry system further comprises: the client sends request information to the system, wherein the request information comprises a request for the system to feed back whether the client enters the system or not; and when the system feeds back that the client enters the system, the system confirms that the client enters the system.

According to a second aspect of some embodiments of the present application, there is provided a system comprising: a memory configured to store data and instructions; a processor in communication with a memory, wherein, when executing instructions in the memory, the processor is configured to: inputting client information into a server queue; requesting a server queue, and inputting client information to a system through a control terminal; when the client is confirmed to enter the system, screen information is sent to the server through the client; the server is requested to acquire the client IP list, and the screen information of the client is loaded through the control terminal.

Therefore, according to the screen interaction method and system of some embodiments of the present application, by opening HTTP service for other terminal devices to access, using CMD commands to newly name firewall rules and opening ports, the firewall rules and the ports can be accessed by other terminal devices in the local area network, so as to realize multi-screen monitoring of the screen of the same terminal device, solve the problems of screen sharing delay, blocking, white screen, and the like, and comprehensively improve user experience.

Drawings

For a better understanding and to set forth of some embodiments of the present application, reference will now be made to the description of embodiments taken in conjunction with the accompanying drawings in which like reference numerals identify corresponding parts throughout.

Fig. 1 is an exemplary schematic diagram of a screen interaction system provided in accordance with some embodiments of the present application.

Fig. 2 is an exemplary flow chart of a screen interaction method provided in accordance with some embodiments of the present application.

Fig. 3 is an exemplary flow chart of a particular screen interaction method provided in accordance with some embodiments of the present application.

Fig. 4 is an exemplary flow chart of client monitoring of screen interaction methods provided in accordance with some embodiments of the present application.

Fig. 5 is a screen sharing exemplary flowchart of a screen interaction method provided according to some embodiments of the present application.

Detailed Description

The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of the various embodiments of the present application defined by the claims and their equivalents. These embodiments include various specific details for ease of understanding, but these are to be considered exemplary only. Accordingly, those skilled in the art will appreciate that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present application. In addition, descriptions of well-known functions and constructions will be omitted herein for brevity and clarity of description.

The terms and phrases used in the following specification and claims are not limited to a literal sense, but rather are only used for the purpose of clarity and consistency in understanding the present application. Thus, it will be appreciated by those skilled in the art that the descriptions of the various embodiments of the present application are provided for illustration only and not for the purpose of limiting the application as defined by the appended claims and their equivalents.

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which embodiments of the present application are shown, it being apparent that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

It is noted that the terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in this application refers to and encompasses any or all possible combinations of one or more of the associated listed items. The expressions "first", "second", "said first" and "said second" are used for modifying the respective elements irrespective of order or importance, and are used merely for distinguishing one element from another element without limiting the respective elements.

A terminal according to some embodiments of the present application may be an electronic device that may include one or a combination of several of a personal computer (PC, e.g., tablet, desktop, notebook, netbook, palmtop PDA), a client device, a virtual reality device (VR), a renderer, a smart phone, a mobile phone, an electronic book reader, a Portable Multimedia Player (PMP), an audio/video player (MP 3/MP 4), a camera, a wearable device, and the like. According to some embodiments of the present application, the wearable device may include an accessory type (e.g., a watch, a ring, a bracelet, glasses, or a Head Mounted Device (HMD)), an integrated type (e.g., an electronic garment), a decorative type (e.g., a skin pad, a tattoo, or an in-built electronic device), etc., or a combination of several. In some embodiments of the present application, the electronic device may be flexible, not limited to the devices described above, or may be a combination of one or more of the various devices described above. In this application, the term "user" may indicate a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

The embodiment of the application provides a screen interaction method and a system. In order to facilitate understanding of the embodiments of the present application, the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

Fig. 1 is an exemplary schematic diagram of a screen interaction system provided in accordance with some embodiments of the present application. As shown in fig. 1, the screen interaction system 100 may include a network 110, a control terminal 120, a client 130, a server 140, and the like. Specifically, the control terminal 120 and the client 130 establish communication through a network, for example, the control terminal 120 and the client 130 may communicate in the same local area network (e.g., a network environment of the same router, etc.). Further, the control end 120 may be connected to the network 110 by a wired (e.g., a network cable, etc.) or wireless, and the client 130 may establish a communication connection with the network 110 by a wired or wireless (e.g., WIFI, etc.) method. In some embodiments, the client 130 may send request information, etc., to the control 120, server 140, etc. Further, the control end 120, the server 140 may feed back information, etc. to the client 130. Based on the feedback information, the client 130 may determine to enter the system and send the information to the server 140. In some embodiments, the control terminal 120 may send request information to the server 140 to obtain information sent by the client 130, and so on.

According to some embodiments of the present application, the control end 120 and the client 130 may be the same or different terminal devices. The terminal device may include, but is not limited to, an intelligent terminal, a mobile terminal, a computer, a renderer, etc. In a teaching scenario, the control end 120 may include a teacher end, the client end 130 may include a student end, and so on. In a conference scenario, the control end 120 may include a host end, the client end 130 may include a report end, and so on. In some embodiments, server 140 is one of the computers, with the advantages of faster operation, higher load, etc., than a conventional computer, and the corresponding price is higher. In a network environment, a server may provide computing or application services for other clients (e.g., terminals such as PCs, smartphones, ATMs, and the like, as well as large devices such as transportation systems). The server has high-speed CPU operation capability, long-time reliable operation, strong I/O external data throughput capability and better expansibility. Services that the server may provide include, but are not limited to, assuming the ability to respond to service requests, assuming services, securing services, and the like. The server has an extremely complex internal structure including an internal structure similar to a general computer, etc., as an electronic device, and the internal structure of the server may include a central processing unit (Central Processing Unit, CPU), a hard disk, a memory, a system bus, etc., as an example.

In some embodiments of the present application, the screen interaction system 100 may omit one or more elements or may further include one or more other elements. By way of example, the screen interaction system 100 may include a plurality of clients 130, such as a plurality of client-side all-in-one machines, and the like. For another example, the screen interaction system 100 may include a plurality of control terminals 120. For another example, the screen interaction system 100 may include a plurality of servers 140 or the like. The network 110 may be any type of communication network that may include a computer network (e.g., a local area network (LAN, local Area Network) or wide area network (WAN, wide Area Network)), the internet, and/or a telephone network, among others, or a combination of several. In some embodiments, network 110 may be other types of wireless communication networks. The wireless communication may include microwave communication and/or satellite communication, etc. The wireless communication may include cellular communication, such as global system for mobile communications (GSM, global System for Mobile Communications), code division multiple access (CDMA, code Division Multiple Access), third generation mobile communications (3G,The 3rd Generation Telecommunication), fourth generation mobile communications (4G), fifth generation mobile communications (5G), sixth generation mobile communications (6G), long term evolution technology (LTE, long Term Evolution), long term evolution technology upgrades (LTE-a, LTE-Advanced), wideband code division multiple access (WCDMA, wideband Code Division Multiple Access), universal mobile telecommunications system (UMTS, universal Mobile Telecommunications System), wireless broadband (WiBro, wireless Broadband), and the like, or a combination of several. In some embodiments, the client 130 may be other electronic devices with equivalent functional modules, which may include one or a combination of several of a virtual reality device (VR), a renderer, a personal computer (PC, e.g., tablet, desktop, notebook, netbook, palmtop PDA), a smart phone, a mobile phone, an e-book reader, a Portable Multimedia Player (PMP), an audio/video player (MP 3/MP 4), a camera, a wearable device, and so on.

In some embodiments, the WIFI may be other types of wireless communication technologies. According to some embodiments of the present application, the wireless communication may include wireless local area network (WiFi, wireless Fidelity), bluetooth low energy (BLE, bluetooth Low Energy), zigBee, near field communication (NFC, near Field Communication), magnetic security transmission, radio frequency and body area network (BAN, body Area Network), etc., or a combination of several. According to some embodiments of the present application, the wired communication may include a global navigation satellite system (Glonass/GNSS, global Navigation Satellite System), a global positioning system (GPS, global Position System), a beidou navigation satellite system or galileo (european global satellite navigation system), or the like. The wired communication may include universal serial bus (USB, universal Serial Bus), high-definition multimedia interface (HDMI, high-Definition Multimedia Interface), recommended standard 232 (RS-232,Recommend Standard 232), plain old telephone service (POTS, plain Old Telephone Service), etc., or a combination of several.

It should be noted that the above description of the screen interaction system 100 is for convenience of description only, and is not intended to limit the application to the scope of the illustrated embodiments. It will be understood by those skilled in the art that various changes in form and details may be made to the application areas of implementing the above-described methods and systems based on the principles of the present system without departing from such principles, and any combination of individual elements or connection of constituent subsystems with other elements may be possible. Such variations are within the scope of the present application.

Fig. 2 is an exemplary flow chart of a screen interaction method provided in accordance with some embodiments of the present application. As shown in fig. 2, the process 200 may be implemented by the screen interaction system 100. In some embodiments, the screen interaction method 200 may be initiated automatically or by instruction. The instructions may include system instructions, device instructions, user instructions, action instructions, etc., or a combination of the several. At 201, client information is entered into a server queue. Operation 201 may be implemented by the client 130, the server 140 of the screen interaction system 100. In some embodiments, the client 130 may send information to the server according to the HTTP protocol. Further, client 130 information is entered into a Redis queue maintained by server 140. The Redis (Remote Dictionary Server) remote dictionary service is a high-performance Key-Value storage system or database. Redis may support storing more Value types, which may include String, list, set, zset, and Hash.

At 202, a server queue is requested, and client information is entered into the system through a control terminal. Operation 202 may be implemented by the control end 120, the server 140 of the screen interaction system 100. In some embodiments, the control terminal 120 may request the server queue and in turn enter the client 130 information into the system.

At 203, when the client is confirmed to enter the system, screen information is sent to the server through the client. Operation 203 may be implemented by the client 130, the server 140 of the screen interaction system 100. In some embodiments, the client 130 may send request information to the system, including requesting the system to feedback whether the client 130 entered the system. When the system feeds back that client entered the system, the client 130 was confirmed to enter the system. Further, the client 130 may transmit screen information to the server 140. The screen information may include a screenshot of the client 130, etc., for example, the screenshot may be a JEPG stream (available as a video picture), etc.

At 204, the server is requested to obtain a client IP list, and the screen information of the client is loaded through the control terminal. Operation 204 may be implemented by the control side 120, the server 140 of the screen interaction system 100. In some embodiments, the control terminal 120 may request the server 140 to obtain an IP list of the client 130. Further, the control terminal 120 may load screen information of the client 130, such as a screenshot (JEPG stream), or the like.

According to some embodiments of the present application, the process 200 may further include client monitoring and/or screen sharing, etc. As shown in fig. 3, fig. 3 is an exemplary flowchart of a particular screen interaction method provided in accordance with some embodiments of the present application. The process 300 is a specific screen interaction method, and operations 301 to 304 are consistent with operations 201 to 204 in the process 200, and are not described herein.

In some embodiments, operation 304 may further proceed to operation 305, where client monitoring is performed by the control terminal at 305. Operation 305 may be implemented by the control side 120, the server 140 of the screen interaction system 100. As an example, the control terminal 120 may perform monitoring or the like on one or more clients 130, which may include viewing a screen or the like of the client 130 in real time. The specific flow of client monitoring performed by the control end is shown in fig. 4, and fig. 4 is an exemplary flow chart of client monitoring of the screen interaction method according to some embodiments of the present application. The process 400 is a specific process of client monitoring, and may be implemented by the screen interaction system 100.

In 401, according to the IP list, the client to be monitored is selected by the control end and UDP information is sent. In some embodiments, the control 120 may click on a client 130 in the IP list, select the client 130 to be monitored and send UDP (User Datagram Protocol) information to the client.

At 402, screen information is saved by the client and HTTP service is started according to UDP information. In some embodiments, the client 130 may receive the UDP information, save screen information, and turn on the HTTP service. The saving of the screen information may include saving a picture stream of the screen information to the local at a Frame rate (Frame rate) of 25 frames per second. The frame rate is generally represented by FPS (Frames Per Second), and may be any integer number of frames per second, e.g., 15 frames per second, 24 frames per second, etc. As an example, the frame rate of a two-dimensional animation is typically 15 frames per second, and the base frame rate of a movie is typically 24 frames per second.

In some embodiments, the client 130 may turn on the HTTP service and the client within the local area network reads the graphics stream local to the shared client. As an example, when an HTTP service is turned on, the HTTP service points to a local image stream that is accessible to clients within the local area network. The picture stream may include a JEPG stream, which may be a video picture. By reading the JPG stream, all frames at the beginning and the following of the first frame are JPG pictures, and the problem of white screen caused by the fact that a plug-in cannot read data at the beginning is avoided. The HTTP service is very light, the transmission of the picture stream is rapid, and the problem of sharing delay in the screen interaction process is solved.

At 403, CMD commands are executed by the client to open the accessible port and return the service address to the control. In some embodiments, the client 130 may execute the CMD command to open its own port for access and return the service address to the control 120. As an example, when the client 130 opens the HTTP service for other clients to access, and when other clients cannot access due to the firewall, the CMD command is used to newly name the firewall rule and open the port, and the lan authority is set so that the port is exposed to the lan and can be accessed by other clients in the lan. The CMD command comprises a Screen Task, and the Screen Task is the name of the firewall rule.

At 404, a window is opened by the control end to monitor the client screen based on the service address. In some embodiments, according to the service address returned by the client, the control end 120 may browse the service address by opening an embedded browser window, and may view the screen of the client.

In some embodiments, operation 304 may further proceed to operation 306, where screen sharing of the client is performed by the control terminal to other clients at 306. Operation 306 may be implemented by the control end 120, the server 140 of the screen interaction system 100. As an example, the control terminal 120 may share the screen of one client 130 to other clients 130, etc., which may include the multiple clients 130 viewing the screen of a certain client 130, etc., in real time. The specific flow of the screen sharing of the client performed by the control terminal to other clients is shown in fig. 5, and fig. 5 is an exemplary flow chart of screen sharing of the screen interaction method according to some embodiments of the present application. The process 500 is a specific process of client screen sharing, and may be implemented by the screen interaction system 100, where operations 501 to 503 are consistent with operations 401 to 403 in the process 400, and are not described herein.

In some embodiments, operation 503 may further proceed to operation 504, where the service address of the client is sent to other clients through the control terminal using UDP at 504. As an example, the control terminal 120 may transmit the service address of the client to other clients through UDP.

At 505, a window is opened by other clients to share the client screen according to the service address. According to the service address returned by the client, other clients 130 can browse the service address by opening an embedded browser window, and can share and view the screen of the client.

It should be noted that, the operations 305 and 306 may be performed simultaneously, or the operations 306 may be performed first and then the operations 305 may be performed, which is not limited in this application, that is, the operations may be performed sequentially in no execution timing, and the above description of the operations of the flowchart 300 is merely illustrative of embodiments of the present application.

It should be noted that the descriptions of the flow 200, the flow 300, the flow 400, and the flow 500 are merely for convenience of description, and are not intended to limit the application to the scope of the embodiments. It will be understood by those skilled in the art that various modifications and changes in form and detail of the functions implementing the above-described processes and operations may be made based on the principles of the present system by any combination of the individual operations or by constituting sub-processes in combination with other operations without departing from such principles. For example, the process 200 may further include operations such as client monitoring and/or screen sharing. Such variations are within the scope of the present application.

In summary, according to the screen interaction method and system of the embodiment of the application, by opening the HTTP service for other terminal devices to access, using the CMD command to newly name the firewall rule and opening the port, the firewall rule can be accessed by other terminal devices in the local area network, so as to realize multi-screen monitoring of the screen of the same terminal device, solve the problems of screen sharing delay, blocking, white screen and the like, and comprehensively improve user experience.

It should be noted that the above-described embodiments are merely examples, and the present application is not limited to such examples, but various changes may be made.

It should be noted that in this specification the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

Finally, it is also to be noted that the above-described series of processes includes not only processes performed in time series in the order described herein, but also processes performed in parallel or separately, not in time series.

Those skilled in the art will appreciate that all or part of the processes in the methods of the embodiments described above may be implemented by hardware associated with computer program instructions, where the program may be stored on a computer readable storage medium, where the program, when executed, may include processes in embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), or the like.

The foregoing disclosure is only illustrative of some of the preferred embodiments of the present application and is not intended to limit the scope of the claims hereof, as persons of ordinary skill in the art will understand that all or part of the processes for accomplishing the foregoing embodiments may be practiced with equivalent changes which may be made by the claims herein and which fall within the scope of the invention.

Claims

1. A method of screen interaction, comprising:

inputting client information into a server queue;

requesting a server queue, and inputting client information to a system through a control terminal;

when the client is confirmed to enter the system, screen information is sent to the server through the client;

requesting a server to acquire an IP list of a client, and loading screen information of the client through a control terminal;

client monitoring is executed through the control end;

executing screen sharing of the client to other clients through the control terminal;

according to the IP list, selecting a client to be monitored through a control terminal and sending UDP information;

according to the UDP information, saving screen information through a client and starting HTTP service;

executing CMD command by the client to open the accessible port and returning the service address to the control end.

2. The method of claim 1, wherein opening the HTTP service further comprises:

according to the UDP information, saving the picture stream of the screen information to the local by the client at a frame rate of 25 frames per second;

when the HTTP service is turned on, the HTTP service points to the local graphics stream.

3. The method of claim 2, wherein the performing, by the control side, client monitoring further comprises:

and opening a window by the control end according to the service address to monitor the client screen.

4. The method of claim 3, wherein the performing, by the control terminal, screen sharing of the client to the other clients further comprises:

using UDP, sending the service address of the client to other clients through the control terminal;

according to the service address, windows are opened by other clients to share the client screen.

5. The method of claim 1, wherein entering client information into a server queue further comprises:

according to the HTTP protocol, sending information to a server through a client;

client information is entered into a Redis queue maintained by the server.

6. The method of claim 1, wherein the validating that the client entered the system further comprises:

the client sends request information to the system, wherein the request information comprises a request for the system to feed back whether the client enters the system or not;

and when the system feeds back that the client enters the system, the system confirms that the client enters the system.

7. A system, comprising:

a memory configured to store data and instructions;

a processor in communication with a memory, wherein, when executing instructions in the memory, the processor is configured to:

inputting client information into a server queue;

client monitoring is executed through the control end;