CN117827357A

CN117827357A - Remote operation method, device and equipment of terminal equipment

Info

Publication number: CN117827357A
Application number: CN202410245130.5A
Authority: CN
Inventors: 刘明明; 陈冠诚; 徐琨
Original assignee: Beijing Testin Information Technology Co Ltd
Current assignee: Beijing Testin Information Technology Co Ltd
Priority date: 2024-03-04
Filing date: 2024-03-04
Publication date: 2024-04-05

Abstract

The embodiment of the specification discloses a remote operation method, a device and equipment of terminal equipment, wherein the method comprises the following steps: when detecting that the target page displayed by the terminal equipment cannot be normally displayed remotely, sending a page element grabbing instruction of the target page to the terminal equipment; receiving control tree information constructed by the page controls grabbed in the target page by the terminal equipment based on the page element grabbing instruction; rendering the control tree in the page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page; based on the operation behavior of the user in the remote rendering page, the remote control terminal device executes an operation corresponding to the operation behavior in the target page. By the method, development efficiency of developers of financial and securities application programs can be improved, the developers can debug and test more conveniently, and powerful guarantee is provided for stability and user experience of the application programs.

Description

Remote operation method, device and equipment of terminal equipment

Technical Field

The present document relates to the field of computer technologies, and in particular, to a method, an apparatus, and a device for remote operation of a terminal device.

Background

In a terminal device, in order to protect information of a user, a page containing specified privacy data of the user often does not allow operations such as screenshot (or screen capturing) or screen recording, for example, a login page or a financial transaction page of some financial application programs does not allow operations such as screenshot (or screen capturing) or screen recording, so in a scene such as a remote real machine test of the terminal device, when the terminal device (such as a mobile phone) needs to be remotely controlled to perform certain operations related to the page, the content of the page is normally displayed on the terminal device, but cannot be normally displayed (such as a black screen page displayed on a control terminal) at a control terminal (such as a server).

In general, during the remote real machine test of the terminal device, only some operations related to the above pages can be performed by manually operating the above pages, so that the remote real machine test efficiency of the terminal device is low. Therefore, a technical scheme capable of improving the testing efficiency of the remote true machine of the terminal equipment is needed.

Disclosure of Invention

The embodiment of the specification aims to provide a technical scheme capable of improving the remote true machine testing efficiency of terminal equipment.

In order to achieve the above technical solution, the embodiments of the present specification are implemented as follows:

in a first aspect, an embodiment of the present disclosure provides a method for remotely operating a terminal device, where the method includes: when detecting that a target page which is not normally displayed at present by the remote display terminal equipment is detected, sending a page element grabbing instruction of the target page to the terminal equipment; receiving control tree information constructed by the page controls grabbed in the target page by the terminal equipment based on the page element grabbing instruction; rendering the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page; based on the operation behaviors of the user in the remote rendering page, remotely controlling the terminal equipment to execute operations corresponding to the operation behaviors in the target page.

In a second aspect, an embodiment of the present disclosure provides a remote operation system for a terminal device, where the system includes a server and a terminal device, where: the server is configured to send a page element grabbing instruction of a target page to the terminal equipment when detecting that the target page currently displayed by the terminal equipment cannot be normally and remotely displayed; the terminal equipment is configured to send the control tree information to the server based on the control tree information constructed by the page controls grabbed in the target page by the page element grabbing instruction; the server is configured to render the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page, and remotely control the terminal equipment to execute operations corresponding to the operation behaviors in the target page based on the operation behaviors of a user in the remote rendering page.

In a third aspect, an embodiment of the present disclosure provides a remote operation method of a terminal device, applied to a front end server of a platform, where the method includes: when detecting that a target page which cannot be displayed at present by the normal remote display terminal equipment is detected, sending a page element grabbing instruction of the target page to an upper computer; receiving control tree information constructed by the page controls grabbed in the target page of the terminal equipment by the upper computer based on the page element grabbing instruction; rendering the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page; based on the operation behaviors of the user in the remote rendering page, the terminal equipment is remotely controlled by the upper computer to execute the operation corresponding to the operation behaviors in the target page.

In a fourth aspect, a method for remotely operating a terminal device provided in an embodiment of the present disclosure is applied to an upper computer, where the method includes: when a platform front-end server detects a target page which cannot be normally displayed at present by a remote display terminal device, receiving a page element grabbing instruction of the target page sent by the platform front-end server; information of a control tree constructed by page controls grabbed in a target page of the terminal equipment based on the page element grabbing instruction; and the information of the control tree is used for triggering the platform front-end server to render the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page, and based on the operation behavior of a user in the remote rendering page, the terminal equipment is remotely controlled by the upper computer to execute the operation corresponding to the operation behavior in the target page.

In a fifth aspect, an embodiment of the present disclosure provides a remote operation system of a terminal device, where the method includes: the system comprises a platform front-end server, an upper computer and terminal equipment, wherein: the platform front-end server is configured to send a page element grabbing instruction of a target page to an upper computer when detecting that the target page currently displayed by the terminal equipment cannot be normally displayed remotely; the upper computer is configured to send an instruction for grabbing a control tree constructed by page controls in the target page to the terminal equipment based on the page element grabbing instruction; the terminal equipment is configured to send the information of the control tree constructed by the page control in the grabbed target page to the upper computer; the upper computer is configured to send the control tree information to the platform front-end server; the platform front-end server is configured to render the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree, obtain a remote rendering page corresponding to the target page, and remotely control the terminal equipment to execute operation corresponding to the operation behavior in the target page through the upper computer based on the operation behavior of a user in the remote rendering page.

In a sixth aspect, an embodiment of the present disclosure provides a remote operation device for a terminal device, where the device includes: the instruction sending module is used for sending a page element grabbing instruction of the target page to the terminal equipment when detecting that the target page which is not normally displayed at present by the terminal equipment can not be displayed remotely; the information receiving module is used for receiving information of a control tree constructed by the terminal equipment based on the page control grabbed in the target page by the page element grabbing instruction; the rendering module is used for rendering the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page; and the remote control module is used for remotely controlling the terminal equipment to execute the operation corresponding to the operation behavior in the target page based on the operation behavior of the user in the remote rendering page.

In a seventh aspect, an embodiment of the present disclosure provides a remote operation device for a terminal device, where the device includes: the system comprises a grabbing instruction sending module, a grabbing instruction processing module and a display module, wherein the grabbing instruction sending module is used for sending a page element grabbing instruction of a target page to an upper computer when detecting that the target page which is not normally displayed at present by a remote display terminal device is detected; the information receiving module is used for receiving the information of a control tree constructed by the page control grabbed in the target page of the terminal equipment by the upper computer based on the page element grabbing instruction; the page rendering module is used for rendering the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page; and the remote control module is used for remotely controlling the terminal equipment to execute the operation corresponding to the operation behavior in the target page through the upper computer based on the operation behavior of the user in the remote rendering page.

In an eighth aspect, an embodiment of the present disclosure provides a remote operation device for a terminal device, where the device includes: the instruction receiving module is used for receiving a page element grabbing instruction of a target page sent by the platform front-end server when the platform front-end server detects that the target page which is not normally displayed at present by the remote display terminal equipment; the page element grabbing module is used for grabbing information of a control tree constructed by page controls grabbed in a target page of the terminal equipment based on the page element grabbing instruction; the information sending module is used for sending the information of the control tree to the platform front-end server, the information of the control tree is used for triggering the platform front-end server to render the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree, a remote rendering page corresponding to the target page is obtained, and based on the operation behavior of a user in the remote rendering page, the terminal equipment is remotely controlled to execute the operation corresponding to the operation behavior in the target page through the device.

In a ninth aspect, embodiments of the present disclosure provide a remote operation device for a terminal device, where the remote operation device for a terminal device includes: a processor; and a memory arranged to store computer executable instructions which when executed implement the steps of the method of remote operation of a terminal device as described in the first, third or fourth aspects above.

In a tenth aspect, the embodiments of the present specification also provide a storage medium for storing computer executable instructions which, when executed by a processor, implement the steps of the method of remote operation of a terminal device as described in the first, third or fourth aspects above.

In an eleventh aspect, embodiments of the present specification further provide a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of remote operation of a terminal device as described in the first, third or fourth aspects above.

Drawings

For a clearer description of embodiments of the present description or of the solutions of the prior art, the drawings that are required to be used in the description of the embodiments or of the prior art will be briefly described, it being obvious that the drawings in the description below are only some of the embodiments described in the description, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art;

fig. 1 is a schematic structural diagram of a remote operation system of a terminal device according to the present disclosure;

Fig. 2 is a schematic diagram illustrating an embodiment of a remote operation method of a terminal device in the present specification;

fig. 3 is a schematic diagram of another embodiment of a remote operation method of a terminal device in the present specification;

fig. 4 is a schematic diagram of a target page displayed by a terminal device in the present specification;

fig. 5 is a target page currently displayed by a terminal device displayed by a server side in the present specification;

FIG. 6 is a schematic diagram of a remote rendered page of the present specification;

fig. 7 is a schematic diagram of another embodiment of a remote operation method of a terminal device in the present specification;

fig. 8 is a schematic structural diagram of a remote operation system of another terminal device according to the present disclosure;

fig. 9 is a schematic diagram illustrating another embodiment of a remote operation method of a terminal device according to the present disclosure;

fig. 10 is a schematic diagram illustrating another embodiment of a remote operation method of a terminal device according to the present disclosure;

fig. 11 is a schematic diagram illustrating another embodiment of a remote operation method of a terminal device according to the present disclosure;

fig. 12 is a schematic diagram of a remote operation procedure of a terminal device according to the present specification;

fig. 13 is a diagram showing an embodiment of a remote operation device of a terminal apparatus according to the present specification;

fig. 14 is a diagram showing another embodiment of a remote operation device of the terminal apparatus according to the present specification;

fig. 15 is a diagram showing still another embodiment of a remote operation device of the terminal apparatus according to the present specification;

Fig. 16 is a remote operation device embodiment of a terminal device of the present specification.

Detailed Description

The embodiment of the specification provides a remote operation method, a device and equipment of terminal equipment.

In order to make the technical solutions in the present specification better understood by those skilled in the art, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.

In practical application, the server may include not only a front end program of the platform controlled by the remote real machine, but also an upper computer program for establishing a bridge for information interaction between the front end program of the platform and the terminal device, through which the front end program and the terminal device can perform information interaction, and the upper computer program can directly establish communication connection with one or more different terminal devices. The terminal device may be a mobile terminal device such as a mobile phone, a tablet computer, a computer device such as a notebook computer or a desktop computer, or an IoT device (specifically, a smart watch, a vehicle-mounted device, etc.), etc.

In the embodiment of the specification, through an automation tool, the information of a control tree of a page which is normally displayed in the terminal equipment but is abnormal in remote display is obtained, the server re-renders the page in the terminal equipment at the server side through the information of the control tree, and the terminal equipment is remotely controlled to execute the operation in the current page based on the page. Specific processing can be seen from the details in the following examples.

As shown in fig. 2, the embodiment of the present disclosure provides a remote operation method of a terminal device, and an execution subject of the method may be a server, and the server may be the server shown in fig. 1. The method specifically comprises the following steps:

in step S202, when it is detected that the target page currently displayed by the terminal device cannot be normally displayed remotely, a page element grabbing instruction of the target page is sent to the terminal device.

The target page may be a designated page in a designated application program in the terminal device, for example, a page related to fund security and personal privacy information of a user in an APP such as finance class, securities class, etc. (e.g., a login page of the APP, a transfer page of the APP, etc., where the APP may be an APP in any type of operating system, specifically, an APP in an Android operating system, an APP in an iOS operating system, etc.), or may be any page that is not allowed to perform screen capturing (or screenshot) or screen recording, etc., which may be specifically set according to practical situations, and embodiments of the present disclosure do not limit the present disclosure. The page element grabbing instruction may be used to grab various different elements contained in the specified page, where the grabbing includes not only information such as identification of each element, but also attribute information of each element and information such as hierarchical structure relationship between different elements, and in practical application, the page element grabbing instruction may be constructed in various different manners, for example, may be written in Java, or may be written in Python, and may be specifically set according to practical situations, which is not limited in this embodiment of the present disclosure.

In practice, applications such as finance and securities are used as main tools for performing finance trade and securities investment, and fund security and personal privacy information of users are involved, so that protection of information security and privacy interests is particularly important, and in order to achieve the objective, different applications usually take a plurality of different security measures, including prohibiting screen capturing or screen recording operations on sensitive pages.

However, during remote real machine debugging of the terminal device, there are not small challenges due to the existence of the above security measures. Because the terminal equipment can be limited with the functions of screen capturing and screen recording in the process of using the appointed application program, the appointed page displayed on the remote real machine of the terminal equipment can be presented in a black screen form, so that a tester (or a user) cannot view the related information of the appointed page, and further processes such as debugging and verification cannot be performed.

When the problems are solved, a user of the terminal equipment usually needs to communicate with a tester and provide an account number, a password and the like, then customer service cooperates with a machine room personnel to manually operate each mobile phone, the account number and the password are input one by one, and the result is recorded in a photographing mode and then fed back to the user of the terminal equipment. However, the above manual operation method has potential safety hazards, such as that account number and password data may be illegally intercepted or leaked in the transmission process, so that loss is brought to the user of the terminal device, and the communication cost and time cost of the above method are quite high due to the fact that more people are involved.

In order to cope with the above challenges, a new function, namely a control tree, is introduced into a remote real machine debugging interface of a terminal device, and the function can clearly render text information frames (such as an account number input box, a password input box, a privacy protocol choosing box and the like) on a designated page, so that a tester can smoothly log in an application program through the function, and further, the next operation can be completed. The solution provides a high-efficiency and practical way for developers of financial and securities application programs, ensures the information security of the developers, simultaneously can carry out debugging and testing work more conveniently, and provides powerful guarantee for the stability and user experience of the application programs. Specifically, when a user of the terminal device needs to use a certain application program, the user may start the application program using the terminal device, may acquire data of a designated page through the application program, and may display a corresponding page (i.e., a target page) on a screen of the terminal device. In the process of remote real machine testing of the terminal equipment, because the terminal equipment can be limited with the functions of screen capturing and screen recording in the process of displaying the target page by using the specified application program, the server cannot normally and remotely display (such as black screen display) the target page currently displayed by the terminal equipment, at this time, related information such as identification information (such as the name and corresponding website of the target page) of the target page is obtained, a page element grabbing instruction of the target page can be generated based on the related information, and the page element grabbing instruction of the target page can be sent to the terminal equipment.

The triggering manner of the page element grabbing instruction for generating the target page may include various manners, for example, the server may preset one or more function keys in the currently displayed page, where the function keys may include a triggering key for grabbing the page element, and may also include other related keys, for example, a system setting key, a network connection key, a sound control, an image acquisition key, etc., and when a user (i.e., a tester) clicks the triggering key for grabbing the page element, the server may execute the processing of the page element grabbing instruction for generating the target page. In practical applications, in addition to the process of triggering the server to execute the process of generating the page element grabbing instruction in the above manner, the process may be implemented in other manners, for example, the process of generating the page element grabbing instruction may be triggered by a designated physical input key (such as an Enter key, a space key or a plurality of key combinations) in an input device (such as a keyboard), and for example, the process of generating the page element grabbing instruction may be triggered by a preset designated input operation (such as drawing a designated shape (such as a circle, a rectangle or a straight line) in a current interface or a gesture of a user (such as a sliding operation from left to right, a circular gesture or the like) acquired by an image capturing component of the server, which may be specifically set according to practical situations.

In step S204, information of a control tree constructed by the page control grabbed in the target page by the terminal device based on the page element grabbing instruction is received.

The page control can be an encapsulation of data and methods, and the page control can comprise attributes and methods, wherein the attributes are simple visitors of the control data, and the methods are simple and visible functions of the control. The page control may be used to develop a control for constructing a User Interface (UI) to help complete development of page elements such as windows, text boxes, buttons, drop-down menus, etc. in software development, and may specifically be set according to actual situations, which is not limited in the embodiments of the present specification. The control tree may be a tree structure constructed from a plurality of page controls and in a specified hierarchical structure.

In implementation, after receiving a page element grabbing instruction sent by a server, the terminal equipment can analyze the page element grabbing instruction, if an analysis result indicates that the page element grabbing instruction can be directly used for grabbing a page element, the terminal equipment can directly execute the page element grabbing instruction to grab a page control in a target page, and a corresponding control tree is constructed through the grabbed page control and hierarchical structure information corresponding to the page control, so that information of the control tree can be obtained. If the analysis result indicates that the page element grabbing instruction cannot be directly used for grabbing the page element, the terminal equipment can generate an instruction with the same function as the page element grabbing instruction based on semantic information of the page element grabbing instruction, so that the page element grabbing instruction is converted into an instruction which can be directly used by the terminal equipment to grab the page element, at the moment, the terminal equipment can grab a page control in a target page based on the converted instruction, and construct a corresponding control tree through hierarchical structure information corresponding to the grabbed page control and the page control, so that information of the control tree can be obtained.

In practical applications, since the page control in the target page may include a lot of information, some information (such as color, brightness, corresponding links, etc. of the page control) may not play any role on the server side, the above information may be removed from the control tree, and only information having a certain role on the server side (such as identification, position, text information in the page control, etc. of the page control) is reserved, so as to simplify and reduce the data size of the control tree.

In step S206, based on the information of the control tree, rendering the control tree in the page that cannot be normally and remotely displayed at present to obtain a remote rendering page corresponding to the target page.

In implementation, after obtaining the information of the control tree, the server may set the page control in the control tree to the page that cannot be normally and remotely displayed (such as the above-mentioned black screen display) according to the hierarchical structure relationship between the page controls presented in the control tree, and then, the server may start the page rendering mechanism to sequentially perform rendering processing on the page control set to the page that cannot be normally and remotely displayed, so as to complete rendering processing on the control tree in the page that cannot be normally and remotely displayed, and finally, the rendered page, that is, the remote rendering page corresponding to the target page, may be obtained. The remote rendering page is the same as the target page in the page style and the layout of the page control, namely, the remote rendering page is used for replacing the target page displayed in the terminal equipment in the server.

In step S208, based on the operation behavior of the user in the remote rendering page, the remote control terminal device performs an operation corresponding to the operation behavior in the target page.

In implementation, after the server displays the remote rendering page, because the remote rendering page includes the page control in the target page, the user can control the terminal device to execute corresponding operation in the target page by operating the page control in the remote rendering page, specifically, the user can execute operation in the remote rendering page, based on the operation, the remote control terminal device executes operation corresponding to the operation in the target page, for example, the target page includes an account number input box and a password input box, and page controls such as a login button and a cancel button, the remote rendering page also includes an account number input box and a password input box, and page controls such as a login button and a cancel button, the user can input a user account number in the account number input box and input a login password in the password input box, then the login button can be clicked, at this time, the server can acquire the login password of the user in the account number input box and the password input box, and the login operation information can be sent to the terminal device, the terminal device can input the received user account number input box and the password input in the target page, and the login password input box can be received in the target page, the login request can be generated by the terminal device after the terminal device receives the corresponding request based on the login password input in the account number input box and the corresponding operation in the target page, and the login page can be clicked on the corresponding request.

According to the remote operation method of the terminal equipment, when the target page which cannot be normally displayed by the terminal equipment at present is detected, a page element grabbing instruction of the target page is sent to the terminal equipment, then, information of a control tree constructed by page controls grabbed in the target page by the terminal equipment based on the page element grabbing instruction is received, then, based on the information of the control tree, the control tree is rendered in the page which cannot be normally displayed at present, a remote rendering page corresponding to the target page is obtained, finally, based on the operation behavior of a user in the remote rendering page, the terminal equipment can be remotely controlled to execute the operation corresponding to the operation behavior in the target page, so that information of the control tree of the page which is normally displayed in the terminal equipment but is abnormal in the remote display is obtained through an automation tool, the page in the terminal equipment is re-rendered through the information of the control tree, and the operation in the current page is executed by the terminal equipment is remotely controlled based on the page, so that development efficiency of developers of financial and securities application programs is improved, and meanwhile, the developers can conduct and test work more conveniently and guarantee stability and stability of the application programs are guaranteed and user experience is provided.

In practical application, for the case that the operating system installed in the terminal device is an Android operating system, the page element grabbing instruction may include a uiaautometer dump command, where uiaautometer is a test framework provided by the Android operating system and is used for testing and analyzing a user interface of an application program in the Android operating system, and the UI command is used for obtaining a UI hierarchy of a current page of the terminal device and displaying the UI hierarchy in an XML format. Based on this, the specific processing manner of the above step S204 may be various, and the following provides an alternative processing manner, which may specifically include the following: and receiving XML information of a control tree constructed by the terminal equipment based on the UIAutometer dump command in the page element grabbing instruction and grabbing the page control in the target page.

In implementation, the server may send the uiaautometer dump command in the page element grabbing instruction to the terminal device, and the terminal device may grab the UI hierarchical structure of the currently displayed target page of the terminal device using the uiaautometer dump command, so that the page control may be grabbed in the target page, and using the grabbed page control, construct a corresponding control tree through the UI hierarchical structure of the target page, and display the control tree using XML format information.

In practical application, in order to simplify the processing procedure, the server may also use the uiaautometer dump command in the page element grabbing instruction to actively grab the page control in the target page from the target page currently displayed by the terminal device, and use the UI hierarchical structure included in the page control to construct a corresponding control tree through the UI hierarchical structure of the target page, and use the XML format information to display the control tree. The specific implementation mode can be set according to actual conditions.

The specific processing manner of step S206 may be varied, and the following provides an alternative processing manner, which may specifically include the following: and converting the information of the control tree into control tree data in a preset data exchange format, and rendering the control tree in a page which cannot be normally and remotely displayed at present based on the converted control tree data to obtain a remote rendering page corresponding to the target page, wherein the preset data exchange format can comprise one or more of a JSON format, a protocol buffer Protobuf format, a BSON format and a message packet message pack format.

The JSON data structure mainly includes an object and an array, the object is an unordered set of key-Value pairs, the key is a character string, the Value can be of any type, including the object and the array, the array is an ordered set of values (Value), and each Value can be of any type, including the object and the array. Data in JSON format can be exchanged between multiple programming languages.

In implementation, considering that the information of the control tree grabbed by the server may not be easy to machine analyze or cannot be completely independent of the text format of the programming language to store and represent the data, so that the efficiency of subsequent processing is reduced, for this purpose, the information of the control tree may be converted into control tree data in a preset data exchange format, such as JSON format, which is easy to machine analyze and is completely independent of the text format of the programming language to store and represent the data, for example, for the case that the information of the control tree is in XML format, the information of the control tree in XML format may be converted into control tree data in JSON format. Then, according to the relation of UI hierarchical structures among the page controls presented in the control tree data in the preset data exchange format, the page controls in the control tree data in the preset data exchange format are set to the page which cannot be normally and remotely displayed at present (the black screen display is described above), and the server can start the page rendering mechanism to sequentially render the page controls set to the page which cannot be normally and remotely displayed at present, so that the control tree is rendered in the page which cannot be normally and remotely displayed at present, and finally the remote rendering page corresponding to the target page can be obtained.

It should be noted that, the preset data exchange format may include, in addition to the above-described format, an Apache Avro format, for example.

The specific processing manner of step S206 may be varied, and an alternative processing manner is provided below, and as shown in fig. 3, the following processing of step S2062 and step S2064 may be specifically included.

In step S2062, the control tree is screened based on the preset data screening rule and the information of the control tree, so as to obtain the screened information of the control tree, where the data screening rule includes a rule of screening out data except the preset attribute information of the operable component in the page control in the control tree and the text data used for displaying in the inoperable component.

The operable components may include a plurality of types, for example, the operable components may include a click component in a page control (such as a click component in a "login" page control in fig. 4, a click component in an "acquire verification code" page control in fig. 4, etc.), an input component (such as a mobile phone number input component in a "mobile phone number input box" page control in fig. 4, a short message verification code input component in a "short message verification code input box" page control in fig. 4, etc.), and so on. The preset attribute information may include various kinds, for example, the preset attribute information may include identification, location information, size information, text data (such as text data of "please input a mobile phone number" of the mobile phone number input component in fig. 4), and the like, for presenting the user with attribute information related to a page control at the same location of the target page at the server side. The non-operable components may include a variety of, for example, the non-operable components may include components for prompting (e.g., "please read and opt out agreeing to user privacy protection terms" in fig. 4, etc.), icon components for presentation only (e.g., a cell phone icon component in the "cell phone number input box" page control in fig. 4, a text message icon component in the "text message authentication code input box" page control in fig. 4, etc.), etc.

In an implementation, the server may analyze the information of the control tree to determine related information of different page components, and may use the data filtering rule to reserve preset attribute information of an operable component in a page control in the control tree and text data for display in an inoperable component, where data except the reserved data may be deleted from the information of the control tree, so as to obtain filtered information of the control tree. For example, larger icons displayed in the upper half of the target page in fig. 4, "<" icons displayed in the upper left corner of the target page, a mobile phone icon component, a short message icon component, text data of "the page does not allow screenshot", and the like can be deleted from the control tree information, and finally, the screened control tree information can be obtained.

In step S2064, based on the information of the screened control tree, rendering the control tree in the page which cannot be normally and remotely displayed at present, to obtain a remote rendering page corresponding to the target page.

In implementation, according to the relation of the UI hierarchical structure between the page controls presented in the screened control tree information, the page controls in the screened control tree information are set to the page which cannot be normally and remotely displayed at present, and a page rendering mechanism is started to sequentially perform rendering processing on the page controls set to the page which cannot be normally and remotely displayed at present, so that a remote rendering page corresponding to the target page is obtained, and specific processing can be referred to the related content and is not described herein.

In practical applications, the specific processing manner of step S2064 may be varied, and the following provides an alternative processing manner, which may specifically include the following: and highlighting an operable component in a page which cannot be normally and remotely displayed at present based on the information of the screened control tree, and displaying text data used for displaying in the inoperable component so as to perform rendering processing of the control tree and obtain a remote rendering page corresponding to the target page.

The highlighting may be performed by thickening an edge line, or may be performed by setting a special color (such as white, red, or green), or the like, and may be specifically set according to practical situations.

In implementation, as shown in fig. 5, taking a page which cannot be normally and remotely displayed at present as a black screen page as an example, according to the UI hierarchical structure relationship between page controls presented in the information of the screened control tree, setting the page controls in the information of the screened control tree into the black screen page, starting a page rendering mechanism to sequentially perform rendering processing on the page controls set into the black screen page, in the rendering process, according to the identification, the position information, the size information and the text data of the operable component, rendering the operable component into the black screen page, and highlighting the operable component outside the operable component in a manner of rendering a white rectangular frame (as shown in fig. 6), and simultaneously displaying the text data for display in the inoperable component, thereby implementing rendering processing on the control tree to obtain a remote rendering page corresponding to the target page, for example, taking the page shown in fig. 4 as the target page currently displayed in the terminal device, then fig. 6 is the remote page displayed at the server side, wherein the target page and the remote rendering page are displayed in the terminal device, one is displayed in the same manner that the two pages are in the same functional content and the two pages are in the same position and the same. In addition, as shown in fig. 5 or fig. 6, the server may preset one or more function buttons in the currently displayed black screen page, where the function buttons may include a triggering button for capturing a page element, and may also include other related buttons, such as a system setting button, a network connection button, a sound control, an image capturing button, and so on.

It should be noted that, the foregoing is described by taking a page that the server cannot normally and remotely display as a black screen page as an example, in practical application, the page that the server cannot normally and remotely display may also be presented in other forms, for example, a page that the server cannot normally and remotely display may also be presented in a blank page form, or a page that the server cannot normally and remotely display may also be presented in a blue screen page form, which may be specifically set according to practical situations, and the embodiment of the present disclosure does not limit this.

In practical application, the target page includes a page that is not allowed to perform screenshot operation or screen recording operation, and the target page that the server cannot normally display at present on the remote display terminal device includes that the target page that the server remotely displays at present on the remote display terminal device is a black screen page.

In practical applications, the specific processing manner of the step S208 may be varied, and an alternative processing manner is provided below, as shown in fig. 7, and may specifically include the following processing of the step S2082 and the step S2084.

In step S2082, data of the operation behavior of the user in the remote rendering page is acquired.

In an implementation, taking the page shown in fig. 4 as a target page currently displayed in the terminal device, and the page shown in fig. 6 as a remote rendering page currently displayed in the server, for example, the user may input a mobile phone number in a mobile phone number input box in the remote rendering page currently displayed in the server, click "acquire a verification code", then input the acquired verification code into a short message verification code input box, and then click "i read and agree with" user privacy protection clause ", and finally click a login key, at this time, the server may acquire data of operation behaviors including the mobile phone number in the mobile phone number input box and the verification code in the short message verification code input box, and related information of the object of the user's click or clicking operation.

In step S2084, the data of the operation behavior is sent to the terminal device, where the data of the operation behavior is used to trigger the terminal device to execute an operation corresponding to the operation behavior in the target page.

In implementation, the server may send the above information to the terminal device, where the terminal device may input the received mobile phone number into a mobile phone number input box in the target page, and input the received verification code into a short message verification code input box in the target page, and after the input is completed, may sequentially perform the clicking and clicking operations in the target page based on the clicking and clicking operations of the received user and related information of the object of the clicking or clicking operation of the user, where the terminal device may generate a login request based on the mobile phone number and the verification code, and send the login request to a corresponding service server to log in a corresponding application program or service platform, so as to implement remote real machine operation on the remote display abnormal page in the terminal device.

The embodiment of the specification provides a remote operation method of terminal equipment, which is characterized in that information of a control tree of a page which is normally displayed in the terminal equipment but is abnormal in remote display is obtained through an automation tool, the page in the terminal equipment is re-rendered at a server side through the information of the control tree, and the terminal equipment is remotely controlled to execute operation in the current page based on the page, so that development efficiency of developers of financial and securities application programs is improved, the developers can debug and test the application programs more conveniently while ensuring information safety, and powerful guarantee is provided for stability and user experience of the application programs.

Based on the remote operation method of the terminal device provided in the foregoing embodiment, the embodiment of the present disclosure further provides a remote operation system of the terminal device, as shown in fig. 1, where the system includes a server 110 and a terminal device 120, where:

the server 110 is configured to send a page element grabbing instruction of the target page to the terminal device 120 when it is detected that the target page currently displayed by the terminal device 120 cannot be normally displayed remotely.

The terminal device 120 is configured to send the control tree information to the server 110 based on the control tree information constructed by the page controls grabbed in the target page by the page element grabbing instruction.

The server 110 is configured to render the control tree in a page that cannot be normally and remotely displayed at present based on the information of the control tree, obtain a remote rendering page corresponding to the target page, and based on an operation behavior of the user in the remote rendering page, execute an operation corresponding to the operation behavior in the target page by the remote control terminal device 120.

The specific processing procedures of the server 110 and the terminal device 120 may refer to the relevant content in the remote operation method of the terminal device provided in the foregoing embodiment, which is not described herein.

The embodiment of the specification also provides another remote operation system of the terminal equipment, which comprises all functional units of the remote operation system of the terminal equipment shown in fig. 1, and is improved on the basis of the functional units, wherein the improvement comprises the following steps:

for the case where the Android operating system is installed in the terminal device 120, the page element grabbing instruction may include an uiaautometer dump command, and the terminal device 120 is configured to send XML information of the control tree to the server 110 based on XML information of the control tree constructed by the page control grabbed in the target page by the uiaautometer dump command in the page element grabbing instruction.

In addition, the server 110 is configured to convert the information of the control tree into control tree data in a preset data exchange format, and render the control tree in a page that cannot be normally and remotely displayed at present based on the converted control tree data to obtain a remote rendering page corresponding to the target page, where the preset data exchange format includes one or more of JSON format, protocol buffer Protobuf format, BSON format, and packet message pack format.

The specific processing procedure of the server 110 may refer to the relevant content in the remote operation method of the terminal device provided in the foregoing embodiment, which is not described herein.

In addition, the server 110 is configured to perform screening processing on the control tree based on a preset data screening rule and information of the control tree to obtain screened information of the control tree, where the data screening rule includes a rule of screening out data except preset attribute information of an operable component in a page control in the control tree and text data used for displaying in an inoperable component; and based on the information of the screened control tree, rendering the control tree in the page which cannot be normally and remotely displayed at present, and obtaining a remote rendering page corresponding to the target page.

In practical application, the server 110 is configured to highlight the operable component in the page that cannot be normally and remotely displayed at present based on the information of the screened control tree, and display text data for display in the inoperable component, so as to perform rendering processing on the control tree, and obtain a remote rendering page corresponding to the target page.

The specific processing procedure of the server 110 may refer to the relevant content in the remote operation method of the terminal device 120 provided in the foregoing embodiment, which is not described herein.

In practical applications, the target page includes a page that is not allowed to perform a screenshot operation or a screen recording operation, and the target page that cannot be normally displayed by the remote display terminal device 120 currently includes that the target page that is currently displayed by the remote display terminal device 120 is a black screen page.

Further, a server 110 configured to obtain data of the user's operational behavior in the remote rendered page; transmitting the data of the operation behavior to the terminal device 120;

the terminal device 120 is configured to execute an operation corresponding to the operation behavior in the target page based on the data of the operation behavior.

According to the remote operation system of the terminal equipment, when the target page which cannot be normally and remotely displayed by the terminal equipment is detected, a server sends a page element grabbing instruction of the target page to the terminal equipment, then receives information of a control tree constructed by a page control which is grabbed in the target page by the terminal equipment based on the page element grabbing instruction, then renders the control tree in the page which cannot be normally and remotely displayed at present based on the information of the control tree, a remote rendering page corresponding to the target page is obtained, finally, based on operation behaviors of a user in the remote rendering page, the remote control terminal equipment can execute operations corresponding to the operation behaviors in the target page, and therefore, information of the control tree of the page which is normally and abnormally displayed in the terminal equipment is obtained through an automation tool, the page in the terminal equipment is re-rendered through the information of the control tree, and the operation in the current page is executed by the terminal equipment is remotely controlled based on the page, so that development efficiency of developers of financial class and securities application programs is improved, and meanwhile, the developers can conduct and test work more conveniently and guarantee the stability and experience of the application programs are provided for the user.

The remote operation method of the terminal device and the remote operation system of the corresponding terminal device provided in the foregoing embodiments are a system and a corresponding execution method that integrate the front end and the upper computer of the platform into a server, where in practical application, the front end and the upper computer of the platform may be set independently, without being integrated into the same device, so as to form a new system, as shown in fig. 8, the system may include a front end server of the platform, the upper computer, and the terminal device, where the front end server of the platform may be a server of the platform for performing remote real machine control on the terminal device, and the server may be an independent server, or may be a server cluster constructed by multiple different servers. The upper computer can be used as a bridge for establishing information interaction between the front-end program of the platform and the terminal equipment, the upper computer can be arranged between the front-end server of the platform and the terminal equipment, can be used as an intermediate server or intermediate equipment, the front-end server of the platform can perform information interaction with the terminal equipment through the upper computer, the upper computer can be a server or a designated terminal equipment and the like, and in practical application, the upper computer can comprise a plurality of upper computers, and each upper computer can directly establish communication connection with one or a plurality of different terminal equipment. The terminal device may be a mobile terminal device such as a mobile phone, a tablet computer, a computer device such as a notebook computer or a desktop computer, or an IoT device (specifically, a smart watch, a vehicle-mounted device, etc.), etc.

Based on the system architecture shown in fig. 8, the embodiment of the present disclosure provides a remote operation method of a terminal device, as shown in fig. 9, where an execution body of the method may be a platform front-end server. The method specifically comprises the following steps:

in step S902, when it is detected that the target page currently displayed by the remote display terminal device cannot be normally displayed, a page element grabbing instruction of the target page is sent to the upper computer.

In step S904, information of a control tree constructed by a page control grabbed in a target page of the terminal device by the upper computer based on the page element grabbing instruction is received.

In implementation, after the upper computer receives the page element grabbing instruction sent by the platform front end server, the page element grabbing instruction may be sent to the terminal device, and the terminal device may obtain the information of the control tree through the specific processing manner in step S204, and then may send the information of the control tree to the upper computer, where the upper computer may send the information of the control tree to the platform front end server. Or after the upper computer receives the page element grabbing instruction sent by the front-end server of the platform, the upper computer can grab the page control in the target page currently displayed by the terminal equipment directly through the page element grabbing instruction, and construct a corresponding control tree through the grabbed page control and the hierarchical structure information corresponding to the page control, so that the information of the control tree can be obtained, and the upper computer can send the information of the control tree to the front-end server of the platform. Or, the upper computer can generate an instruction with the same function as the page element grabbing instruction based on the semantic information of the page element grabbing instruction, so that the page element grabbing instruction is converted into an instruction which can be directly used by the upper computer, the page control can be grabbed in a target page currently displayed by the terminal equipment based on the converted instruction, a corresponding control tree is constructed through the grabbed page control and the hierarchical structure information corresponding to the page control, and therefore information of the control tree can be obtained, and the upper computer can send the information of the control tree to a front-end server of the platform.

In step S906, based on the information of the control tree, rendering the control tree in the page which cannot be normally and remotely displayed at present to obtain a remote rendering page corresponding to the target page.

In step S908, based on the operation behavior of the user in the remote rendering page, an operation corresponding to the operation behavior is performed in the target page by the host computer remote control terminal device.

The specific processing procedures of the above steps S902, S906 and S908 may be referred to the relevant contents of the steps S202, S206 and S208 in the above embodiments, and are not described herein.

In practical applications, the specific processing manner of the step S902 may be various, and the following provides an alternative processing manner, which may specifically include the following: when detecting that the target page displayed by the terminal equipment cannot be normally displayed remotely, sending a page element grabbing instruction of the target page to an upper computer through a preset communication protocol, wherein the communication protocol comprises a WebSocket protocol.

It should be noted that the above communication protocol may include, in addition to WebSocket protocol, other communication protocols, for example, gppc protocol, socket protocol, etc., which may be specifically set according to actual situations, and the embodiment of the present disclosure is not limited to this.

For the case that the Android operating system is installed in the terminal device, the page element grabbing instruction may include a uiaautometer dump command, and the specific processing manner of the step S904 may be various, and the following provides an optional processing manner, which may specifically include the following: and receiving XML information of a control tree constructed by the upper computer based on the UIAutometer dump command in the page element grabbing instruction and grabbing the page control in the target page of the terminal equipment.

In implementation, after the upper computer receives the page element grabbing instruction, the page element grabbing instruction can be sent to the terminal equipment, the terminal equipment can send the XML information of the control tree to the upper computer through the XML information of the control tree constructed by the page control grabbed in the target page by the UIIAutometer dump command in the page element grabbing instruction, and the upper computer can send the XML information of the control tree to the front-end server of the platform. Or, the upper computer can also directly use the UIAutometer dump command in the page element grabbing command to grab the page control in the target page of the terminal device, so that a corresponding control tree can be constructed through the grabbed page control, the control tree can be described by using XML format information, XML information of the control tree is obtained, and the upper computer can send the XML information of the control tree to the front-end server of the platform.

In addition, the specific processing manner of the step S906 may be varied, and the following provides an alternative processing manner, which may specifically include the following: and converting the information of the control tree into control tree data in a preset data exchange format, and rendering the control tree in a page which cannot be normally and remotely displayed at present based on the converted control tree data to obtain a remote rendering page corresponding to the target page, wherein the preset data exchange format can comprise one or more of a JSON format, a protocol buffer Protobuf format, a BSON format and a message packet message pack format.

In addition to the above processing, the processing may also be performed by specifically receiving control tree data in a preset data exchange format based on information conversion of the control tree sent by the upper computer, rendering the control tree in a page that cannot be normally and remotely displayed at present based on the converted control tree data to obtain a remote rendering page corresponding to the target page, where the preset data exchange format may include one or more of JSON format, protocol buffer Protobuf format, BSON format, and packet message pack format

The specific processing procedures of the foregoing may be referred to the relevant content in the foregoing embodiments, and are not repeated herein.

In addition, the specific processing manner of the step S906 may be varied, and the following provides an alternative processing manner, which may specifically include the following: and highlighting an operable component in the page which cannot be normally and remotely displayed at present based on the information of the control tree, and displaying text data used for displaying in the inoperable component so as to perform rendering processing of the control tree and obtain a remote rendering page corresponding to the target page.

In addition, the specific processing manner of the step S908 may be varied, and the following provides an alternative processing manner, which may specifically include the following: acquiring data of operation behaviors of a user in the remote rendering page; and sending the data of the operation behavior to the upper computer, wherein the data of the operation behavior is used for triggering the upper computer to control the terminal equipment to execute the operation corresponding to the operation behavior in the target page.

According to the remote operation method of the terminal equipment, when the target page which cannot be normally and remotely displayed in the terminal equipment is detected through the front-end server of the platform, a page element grabbing instruction of the target page is sent to the upper computer, then, information of a control tree constructed by page controls grabbed in the target page of the terminal equipment by the upper computer based on the page element grabbing instruction is received, then, based on the information of the control tree, the control tree is rendered in the page which cannot be normally and remotely displayed at present, a remote rendering page corresponding to the target page is obtained, finally, operation corresponding to the operation behavior can be executed in the target page through the upper computer by remotely controlling the terminal equipment, and therefore, through an automatic tool, the front-end server of the platform obtains information of the control tree of the page which is normally and abnormally displayed in the terminal equipment in the remote display process, the page in the terminal equipment is re-rendered through the information of the control tree, and the operation in the current process is executed based on the page remote control terminal equipment, and therefore, development efficiency of a developer of financial and securities application programs is improved, the developer can be guaranteed to safely and safely debug the page, the user can be tested and conveniently, and the user can be tested and stably.

Based on the system architecture shown in fig. 8, the embodiment of the present disclosure provides a remote operation method of a terminal device, and as shown in fig. 10, an execution subject of the method may be an upper computer. The method specifically comprises the following steps:

in step S1002, when the platform front-end server detects that the target page currently displayed by the terminal device cannot be normally displayed remotely, a page element grabbing instruction of the target page sent by the platform front-end server is received.

In step S1004, information of a control tree constructed by a page control grabbed in a target page of the terminal device is grabbed based on the page element grabbing instruction.

The specific processing procedure of the step S1004 may be referred to the relevant content in the step S904 in the foregoing embodiment, and will not be described herein.

In step S1006, the information of the control tree is sent to the platform front end server, where the information of the control tree is used to trigger the platform front end server to render the control tree in the page that cannot be normally and remotely displayed at present based on the information of the control tree, to obtain a remote rendering page corresponding to the target page, and based on the operation behavior of the user in the remote rendering page, the operation corresponding to the operation behavior is executed in the target page by the upper computer remote control terminal device.

The specific processing procedures of the step S1002 and the step S1006 may be referred to the relevant content in the above embodiment, and are not described herein.

In practical applications, the filtering process may also be performed on the obtained control tree information, as shown in fig. 11, and specifically may include the process of step S1008 described below.

In step S1008, screening the control tree based on a preset data screening rule and information of the control tree, to obtain screened information of the control tree, where the data screening rule includes a rule of screening data except for preset attribute information of an operable component in a page control in the control tree and text data used for presentation in an inoperable component.

Based on the processing of step S1008, the processing of step S1006 includes: and transmitting the information of the screened control tree to a front-end server of the platform, wherein the information of the screened control tree is used for triggering the front-end server of the platform to render the control tree in a page which cannot be normally and remotely displayed at present based on the information of the screened control tree, so as to obtain a remote rendering page corresponding to the target page, and executing operation corresponding to the operation behavior in the target page through an upper computer remote control terminal device based on the operation behavior of a user in the remote rendering page.

In practical application, if an Android operating system is installed in the terminal device and the page element grabbing instruction includes a uiaautometer dump command, the specific processing manner of the step S1004 may be various, and the following provides an optional processing manner, which may specifically include the following: XML information of a control tree constructed based on a UIAutometer dump command in a page element grabbing instruction and a page control grabbed in a target page of terminal equipment.

The embodiment of the specification provides a remote operation method of terminal equipment, a target page which cannot be normally and remotely displayed at present is detected through a platform front-end server, a page element grabbing instruction of the target page sent by the platform front-end server is received, then, based on the page element grabbing instruction, the information of a control tree constructed by page controls grabbed in the target page of the terminal equipment is finally sent to the platform front-end server, the information of the control tree is used for triggering the platform front-end server to render the control tree in the page which cannot be normally and remotely displayed at present based on the information of the control tree, a remote rendering page corresponding to the target page is obtained, and based on the operation behavior of a user in the remote rendering page, the operation corresponding to the operation behavior is executed in the target page through an upper computer, so that the platform front-end server obtains the information of the control tree of the page which is normally displayed in the terminal equipment but is abnormal in the remote display process, and the page of the control tree is finally, the control tree is re-displayed in the terminal equipment at the server side, and the current page is executed based on the remote control terminal equipment, so that the operation class of the control tree is improved, the application program is more convenient and safe to the user experience is guaranteed, the application program is provided for a developer.

The following provides a detailed description of a remote operation method of a terminal device in this embodiment of the present disclosure in connection with a specific application scenario, as shown in fig. 12, where the application scenario may be a remote real machine debugging scenario of the terminal device, the terminal device may be a mobile phone, the user may be a tester or a user, the target page may be a login page of a financial APP, the target page is not allowed to perform screen capturing and screen recording, the front-end server of the platform may not normally remotely display the target page currently displayed by the terminal device, and in this embodiment, the front-end server of the platform displays the target page currently displayed by the terminal device with a black screen.

In order to make the processing procedure simpler, a new function, namely a control tree button, can be introduced into the remote real machine debugging interface of the terminal equipment, and the last icon at the right lower corner of fig. 5 or the icon with a black frame at the right lower corner of fig. 6 can be seen. The function can clearly render the text information frames (such as an account number input frame, a password input frame, a privacy protocol choosing frame and the like) on the appointed page, and through the function, a tester can smoothly log in an application program so as to finish the next operation. The solution provides a high-efficiency and practical way for developers of financial application programs, ensures the safety of information, simultaneously enables the developers to debug and test more conveniently, and provides powerful guarantee for the stability and user experience of the application programs.

In step A2, the user encounters a black screen page, and the user clicks a control tree button, or the platform front-end server detects that the target page currently displayed by the terminal equipment cannot be normally displayed remotely.

In step A4, the platform front-end server sends a page element grabbing instruction of the target page to the upper computer through a WebSocket protocol.

In step A6, the upper computer grasps XML information of a control tree constructed by the page control grasped in the target page of the mobile phone based on the uiacomputer dump command in the page element grasping instruction.

In step A8, the mobile phone may return XML information of the control tree to the upper computer.

In step a10, the upper computer performs screening processing on the control tree based on a preset data screening rule and XML information of the control tree to obtain information of the screened control tree, where the data screening rule includes a rule of screening out data except preset attribute information of an operable component in a page control in the control tree and text data used for displaying in an inoperable component.

In step a12, the upper computer converts the information of the screened control tree into control tree data in a preset data exchange format, and sends the converted control tree data to the front-end server of the platform.

In step a14, the platform front-end server highlights the operable component in the page which cannot be normally and remotely displayed at present based on the converted control tree data, and displays text data for display in the inoperable component, so as to perform rendering processing of the control tree, and obtain a remote rendering page corresponding to the target page.

In step A16, the platform front-end server obtains data of the user's operational behavior in the remote rendered page.

In step a18, the platform front-end server transmits the data of the operation behavior to the upper computer.

In step a20, the host computer transmits the data of the operation behavior to the mobile phone.

In step a22, the mobile phone performs an operation corresponding to the operation behavior in the target page based on the data of the operation behavior.

Based on the remote operation method of the terminal device provided in the foregoing embodiments corresponding to fig. 9 to fig. 12, the embodiment of the present disclosure further provides a remote operation system of the terminal device, as shown in fig. 8, where the system includes a platform front end server 810, an upper computer 820, and a terminal device 830, where:

the platform front-end server 810 is configured to send a page element grabbing instruction of a target page to the upper computer 820 when detecting that the target page currently displayed by the remote display terminal device 830 cannot be normally displayed;

The upper computer 820 is configured to send an instruction for grabbing a control tree constructed by page controls in a target page to the terminal device 830 based on the page element grabbing instruction;

the terminal device 830 is configured to send the control tree information constructed by the page controls in the grabbed target page to the upper computer 820;

the upper computer 820 is configured to send the control tree information to the platform front-end server 810;

the platform front-end server 810 is configured to render the control tree in the page that cannot be normally and remotely displayed at present based on the information of the control tree, obtain a remote rendering page corresponding to the target page, and remotely control the terminal device 830 to execute an operation corresponding to the operation behavior in the target page through the upper computer 820 based on the operation behavior of the user in the remote rendering page.

The specific processing procedures of the platform front-end server 810, the upper computer 820 and the terminal device 830 may refer to the relevant content in the remote operation method of the terminal device provided in the foregoing embodiment, which is not described herein again.

The embodiment of the specification also provides another remote operation system of the terminal equipment, which comprises all functional units of the remote operation system of the terminal equipment shown in fig. 8, and is improved on the basis of the functional units, wherein the improvement comprises the following steps:

In practical application, the platform front-end server 810 is configured to send a page element grabbing instruction of a target page to the upper computer 820 through a preset communication protocol when detecting that the target page displayed by the remote display terminal device 830 cannot be normally displayed, where the communication protocol includes WebSocket protocol.

For the case where the Android operating system is installed in the terminal device 830, the above page element grabbing instruction may include an uialutator dump command, and the platform front-end server 810 is configured to receive XML information of a control tree constructed by the upper computer 820 based on the uialutator dump command in the page element grabbing instruction and grabbing the control in the target page of the terminal device 830.

In addition, the upper computer 820 is configured to convert the information of the control tree into control tree data in a preset data exchange format, and send the converted control tree data to the platform front-end server 810, where the preset data exchange format may include one or more of JSON format, protocol buffer Protobuf format, BSON format, and packet message pack format; the platform front-end server 810 is configured to render the control tree in the page which cannot be normally and remotely displayed at present based on the converted control tree data, so as to obtain a remote rendering page corresponding to the target page.

In addition, the platform front-end server 810 is configured to highlight the operable component in the page that cannot be normally and remotely displayed at present based on the information of the control tree, and display text data for display in the inoperable component, so as to perform rendering processing of the control tree, and obtain a remote rendering page corresponding to the target page.

Further, platform front-end server 810 configured to obtain data of user's operational behavior in the remote rendered page; transmitting the data of the operation behavior to the upper computer 820;

the upper computer 820 is configured to send the data of the operation behavior to the terminal device 830;

the terminal device 830 is configured to perform an operation corresponding to the operation behavior in the target page based on the data of the operation behavior.

In practical application, the upper computer 820 is configured to perform screening processing on the control tree based on a preset data screening rule and information of the control tree to obtain screened information of the control tree, where the data screening rule includes a rule of screening out data except preset attribute information of an operable component in a page control in the control tree and text data used for displaying in an inoperable component; the information of the screened control tree is sent to a platform front-end server 810;

The platform front-end server 810 is configured to render the control tree in the page that cannot be normally and remotely displayed at present based on the screened information of the control tree, obtain a remote rendering page corresponding to the target page, and remotely control the terminal device 830 to execute an operation corresponding to the operation behavior in the target page through the upper computer 820 based on the operation behavior of the user in the remote rendering page.

In practical application, if the Android operating system is installed in the terminal device 830, and the page element grabbing instruction includes the uiaautometer dump command, the upper computer 820 is configured to construct XML information of the control tree based on the page control grabbed in the target page of the terminal device 830 by the uiaautometer dump command in the page element grabbing instruction.

The embodiment of the specification provides a remote operation system of terminal equipment, when a target page which cannot be normally and remotely displayed in the terminal equipment is detected through a platform front-end server, a page element grabbing instruction of the target page is sent to an upper computer, then, information of a control tree constructed by page controls grabbed in the target page of the terminal equipment by the upper computer based on the page element grabbing instruction is received, then, based on the information of the control tree, the control tree is rendered in the page which cannot be normally and remotely displayed at present, a remote rendering page corresponding to the target page is obtained, finally, operation corresponding to the operation behavior can be executed in the target page through the upper computer remote control terminal equipment, and therefore, through an automation tool, the platform front-end server obtains information of the control tree of the page which is normally and abnormally displayed in the terminal equipment in the remote display process, and executes current operation on the basis of the page remote control terminal equipment, thereby improving the development efficiency of a developer of financial application program and securities application program, ensuring the development performance of the page, and ensuring the safety and the debugging performance of the application program.

Based on the foregoing remote operation method of the terminal device provided in the embodiments corresponding to fig. 2 to fig. 7, based on the same concept, the embodiment of the present disclosure further provides a remote operation device of the terminal device, as shown in fig. 13.

The remote operation device of the terminal equipment comprises: an instruction sending module 1301, an information receiving module 1302, a rendering module 1303, and a remote control module 1304, wherein:

the instruction sending module 1301 is configured to send a page element grabbing instruction of a target page to a terminal device when detecting that the target page currently displayed by the terminal device cannot be normally displayed remotely;

the information receiving module 1302 receives information of a control tree constructed by the terminal device based on the page control grabbed in the target page by the page element grabbing instruction;

the rendering module 1303 renders the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page;

and a remote control module 1304 for remotely controlling the terminal device to execute an operation corresponding to the operation behavior in the target page based on the operation behavior of the user in the remote rendering page.

In this embodiment of the present disclosure, the page element grabbing instruction includes an uiacoexists dump command, and the information receiving module 1302 receives XML information of a control tree constructed by the terminal device based on the page control grabbed in the target page by the uiacoexists dump command in the page element grabbing instruction.

In this embodiment of the present disclosure, the rendering module 1303 converts the information of the control tree into control tree data with a preset data exchange format, and renders the control tree in a page that cannot be normally and remotely displayed at present based on the converted control tree data to obtain a remote rendering page corresponding to the target page, where the preset data exchange format includes one or more of JSON format, protocol buffer Protobuf format, BSON format, and packet message pack format.

In the embodiment of the present disclosure, the rendering module 1303 includes:

the screening unit is used for screening the control tree based on a preset data screening rule and the information of the control tree to obtain the information of the screened control tree, wherein the data screening rule comprises a rule for screening out data except the preset attribute information of the operable components in the page control in the control tree and the text data used for displaying in the inoperable components;

And the rendering unit is used for performing rendering processing on the control tree in the page which cannot be normally and remotely displayed at present based on the screened information of the control tree, so as to obtain a remote rendering page corresponding to the target page.

In this embodiment of the present disclosure, the rendering unit highlights the operable component in a page that cannot be normally and remotely displayed at present based on the information of the screened control tree, and displays text data for display in the inoperable component, so as to perform rendering processing of the control tree, and obtain a remote rendered page corresponding to the target page.

In this embodiment of the present disclosure, the target page includes a page that is not allowed to perform a screenshot operation or a screen recording operation, and the target page that cannot be displayed at present by the normal remote display terminal device includes a target page that is displayed at present by the remote display terminal device as a black screen page.

In the embodiment of the present disclosure, the remote control module 1304 includes:

an operation behavior acquisition unit that acquires data of operation behaviors of a user in the remote rendering page;

and the remote control unit is used for sending the data of the operation behaviors to the terminal equipment, wherein the data of the operation behaviors are used for triggering the terminal equipment to execute the operation corresponding to the operation behaviors in the target page.

According to the remote operation device of the terminal equipment, when the target page which cannot be normally displayed by the terminal equipment at present is detected, a page element grabbing instruction of the target page is sent to the terminal equipment, then, information of a control tree constructed by a page control which is grabbed in the target page by the terminal equipment based on the page element grabbing instruction is received, then, the control tree is rendered in the page which cannot be normally displayed at present based on the information of the control tree, a remote rendering page corresponding to the target page is obtained, finally, the terminal equipment can be remotely controlled to execute an operation corresponding to the operation behavior in the remote rendering page based on the operation behavior of a user, and therefore, information of the control tree of the page which is normally displayed in the terminal equipment but is abnormal in remote display is obtained through an automation tool, the page in the terminal equipment is re-rendered at a server end through the information of the control tree, and the operation in the current page is executed by the terminal equipment is remotely controlled based on the page, so that development efficiency of developers of financial and securities application programs is improved, and meanwhile, the developers can conduct and test work more conveniently and guarantee stability and user experience are provided.

Based on the foregoing remote operation method of the terminal device provided in the embodiments corresponding to fig. 9 and fig. 12, based on the same concept, the embodiment of the present disclosure further provides a remote operation device of the terminal device, as shown in fig. 14.

The remote operation device of the terminal equipment comprises: a fetching instruction sending module 1401, an information receiving module 1402, a page rendering module 1403, and a remote control module 1404, wherein:

the capturing instruction sending module 1401 is used for sending a page element capturing instruction of a target page to an upper computer when detecting that the target page which is not normally displayed at present by the remote display terminal equipment cannot be displayed;

an information receiving module 1402, configured to receive information of a control tree constructed by a page control grabbed in a target page of the terminal device by the upper computer based on the page element grabbing instruction;

the page rendering module 1403 is used for rendering the control tree in the page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page;

and a remote control module 1404 for remotely controlling the terminal device to execute an operation corresponding to the operation behavior in the target page through the upper computer based on the operation behavior of the user in the remote rendering page.

In this embodiment of the present disclosure, the capturing instruction sending module 1401 sends, when detecting that a target page that cannot be displayed by a normal remote display terminal device currently, a page element capturing instruction of the target page to an upper computer through a preset communication protocol, where the communication protocol includes WebSocket protocol.

In this embodiment of the present disclosure, the page element grabbing instruction includes an uiacoexists dump command, and the information receiving module 1402 receives XML information of a control tree constructed by the upper computer based on the page control grabbed in the target page of the terminal device by the uiacoexists dump command in the page element grabbing instruction.

According to the remote operation device of the terminal equipment, when the target page which cannot be normally and remotely displayed in the terminal equipment is detected through the front-end server of the platform, a page element grabbing instruction of the target page is sent to the upper computer, then, information of a control tree constructed by page controls grabbed in the target page of the terminal equipment by the upper computer based on the page element grabbing instruction is received, then, based on the information of the control tree, the control tree is rendered in the page which cannot be normally and remotely displayed at present, a remote rendering page corresponding to the target page is obtained, finally, operation corresponding to the operation behavior can be executed in the target page through the upper computer by remotely controlling the terminal equipment, and therefore, through an automation tool, the front-end server of the platform obtains information of the control tree of the page which is normally and abnormally displayed in the terminal equipment in the remote display process, the page in the terminal equipment is re-rendered through the information of the control tree, and the operation in the current process is executed based on the page remote control terminal equipment, and therefore, development efficiency of a developer of financial and securities application programs is improved, the developer can be guaranteed to safely and safely debug the page, the user can be tested and conveniently, and the user can experience is guaranteed.

Based on the foregoing remote operation method of the terminal device provided in the embodiments corresponding to fig. 10 to fig. 12, based on the same concept, the embodiment of the present disclosure further provides a remote operation device of the terminal device, as shown in fig. 15.

The remote operation device of the terminal equipment comprises: an instruction receiving module 1501, a page element grabbing module 1502 and an information sending module 1503, wherein:

the instruction receiving module 1501 receives a page element grabbing instruction of a target page sent by a platform front-end server when the platform front-end server detects that the target page currently displayed by a terminal device cannot be normally displayed remotely;

a page element grabbing module 1502, configured to grab information of a control tree constructed by a page control grabbed in a target page of the terminal device based on the page element grabbing instruction;

the information sending module 1503 sends the information of the control tree to the platform front end server, where the information of the control tree is used to trigger the platform front end server to render the control tree in a page that cannot be normally and remotely displayed at present based on the information of the control tree, to obtain a remote rendering page corresponding to the target page, and based on an operation behavior of a user in the remote rendering page, remotely control, by using the device, the terminal device to execute an operation corresponding to the operation behavior in the target page.

In an embodiment of the present disclosure, the apparatus further includes:

the screening module is used for screening the control tree based on a preset data screening rule and the information of the control tree to obtain the information of the screened control tree, wherein the data screening rule comprises a rule for screening out data except the preset attribute information of the operable components in the page control in the control tree and the text data used for displaying in the inoperable components;

the information sending module 1503 sends the information of the screened control tree to the platform front end server, where the information of the screened control tree is used to trigger the platform front end server to render the control tree in a page that cannot be normally and remotely displayed at present based on the information of the screened control tree, to obtain a remote rendering page corresponding to the target page, and based on an operation behavior of a user in the remote rendering page, remotely control, by the upper computer, the terminal device to execute an operation corresponding to the operation behavior in the target page.

The embodiment of the specification provides a remote operation device of terminal equipment, a target page which cannot be normally and remotely displayed at present is detected through a platform front-end server, a page element grabbing instruction of the target page sent by the platform front-end server is received, then, based on the page element grabbing instruction, the information of a control tree constructed by page controls grabbed in the target page of the terminal equipment is finally sent to the platform front-end server, the information of the control tree is used for triggering the platform front-end server to render the control tree in the page which cannot be normally and remotely displayed at present based on the information of the control tree, a remote rendering page corresponding to the target page is obtained, and based on the operation behavior of a user in the remote rendering page, the operation corresponding to the operation behavior is executed in the target page through an upper computer, so that the platform front-end server obtains the information of the control tree of the page which is normally displayed in the terminal equipment but is abnormal in the remote display process, and the page of the control tree is finally, the control tree is re-displayed in the terminal equipment at the server end, and the current page is executed based on the remote control terminal equipment, so that the operation class of the control tree is improved, the application program is more convenient and safe to the user experience is ensured, the application program is guaranteed, the application program is developed, and the safety and the application program is more safe and the user is ensured.

The above is a remote operation device of a terminal device provided in the embodiment of the present disclosure, and based on the same concept, the embodiment of the present disclosure further provides a remote operation device of a terminal device, as shown in fig. 16.

The remote operation device of the terminal device may provide a server, a platform front-end server, an upper computer, etc. for the above embodiments.

The remote operation device of the terminal device may have a relatively large difference due to different configurations or capabilities, and may include one or more processors 1601 and a memory 1602, where the memory 1602 may store one or more storage applications or data. Wherein the memory 1602 may be a transient storage or a persistent storage. The application programs stored in the memory 1602 may include one or more modules (not shown), each of which may include a series of computer executable instructions in a remote operation device for a terminal device. Still further, the processor 1601 may be configured to communicate with a memory 1602, and execute a series of computer executable instructions in the memory 1602 on a remote operating device of the terminal device. The remote operation device of the terminal device may also include one or more power supplies 1603, one or more wired or wireless network interfaces 1604, one or more input/output interfaces 1605, and one or more keyboards 1606.

In particular, in this embodiment, the remote operation device of the terminal device includes a memory, and one or more programs, where the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions in the remote operation device of the terminal device, and executing the one or more programs by the one or more processors includes computer executable instructions for:

when detecting that a target page which is not normally displayed at present by the remote display terminal equipment is detected, sending a page element grabbing instruction of the target page to the terminal equipment;

receiving control tree information constructed by the page controls grabbed in the target page by the terminal equipment based on the page element grabbing instruction;

rendering the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page;

based on the operation behaviors of the user in the remote rendering page, remotely controlling the terminal equipment to execute operations corresponding to the operation behaviors in the target page.

Further, in this embodiment, the remote operation device of the terminal device includes a memory, and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions in the remote operation device of the terminal device, and the execution of the one or more programs by the one or more processors includes computer executable instructions for:

when detecting that a target page which cannot be displayed at present by the normal remote display terminal equipment is detected, sending a page element grabbing instruction of the target page to an upper computer;

receiving control tree information constructed by the page controls grabbed in the target page of the terminal equipment by the upper computer based on the page element grabbing instruction;

based on the operation behaviors of the user in the remote rendering page, the terminal equipment is remotely controlled by the upper computer to execute the operation corresponding to the operation behaviors in the target page.

when a platform front-end server detects a target page which cannot be normally displayed at present by a remote display terminal device, receiving a page element grabbing instruction of the target page sent by the platform front-end server;

information of a control tree constructed by page controls grabbed in a target page of the terminal equipment based on the page element grabbing instruction;

and the information of the control tree is used for triggering the platform front-end server to render the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page, and based on the operation behavior of a user in the remote rendering page, the terminal equipment is remotely controlled by the upper computer to execute the operation corresponding to the operation behavior in the target page.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the remote operation device embodiment of the terminal device, since it is substantially similar to the method embodiment, the description is relatively simple, and the relevant points are referred to in the description of the method embodiment.

According to the remote operation equipment of the terminal equipment, when the target page which cannot be normally displayed by the terminal equipment at present is detected, a page element grabbing instruction of the target page is sent to the terminal equipment, then, information of a control tree constructed by a page control which is grabbed in the target page by the terminal equipment based on the page element grabbing instruction is received, then, the control tree is rendered in the page which cannot be normally displayed at present based on the information of the control tree, a remote rendering page corresponding to the target page is obtained, finally, the terminal equipment can be remotely controlled to execute an operation corresponding to the operation behavior in the remote rendering page based on the operation behavior of a user, and therefore, information of the control tree of the page which is normally displayed in the terminal equipment but is abnormal in remote display is obtained through an automation tool, the page in the terminal equipment is re-rendered at a server side based on the information of the control tree, and the operation in the current page is executed by the terminal equipment is remotely controlled based on the page, so that development efficiency of developers of financial and securities application programs is improved, and meanwhile, the developers can conduct and test work more conveniently and guarantee stability and user experience are provided.

Further, based on the methods shown in fig. 2 to 7 and fig. 9 to 12, one or more embodiments of the present disclosure further provide a storage medium for storing computer executable instruction information, where in a specific embodiment, the storage medium may be a U-disc, an optical disc, a hard disk, etc., and the computer executable instruction information stored in the storage medium can implement the following flow when executed by a processor:

In addition, in another specific embodiment, the storage medium may be a usb disk, an optical disc, a hard disk, or the like, where the computer executable instruction information stored in the storage medium, when executed by the processor, can implement the following flow:

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for one of the above-described storage medium embodiments, since it is substantially similar to the method embodiment, the description is relatively simple, and reference is made to the description of the method embodiment for relevant points.

The embodiment of the specification provides a storage medium, when a target page which cannot be normally and remotely displayed by terminal equipment is detected, a page element grabbing instruction of the target page is sent to the terminal equipment, then, the information of a control tree constructed by a page control which is grabbed in the target page by the terminal equipment based on the page element grabbing instruction is received, then, the control tree is rendered in the page which cannot be normally and remotely displayed at present based on the information of the control tree, a remote rendering page corresponding to the target page is obtained, finally, the terminal equipment can be remotely controlled to execute an operation corresponding to the operation behavior in the target page based on the operation behavior of a user in the remote rendering page, so that the information of the control tree of the page which is normally and abnormally displayed in the terminal equipment is obtained through an automatic tool, the page in the terminal equipment is re-rendered at a server end based on the information of the control tree, and the operation in the current page is executed by the terminal equipment is remotely controlled based on the page, so that the development efficiency of a developer of financial application program and securities application program is improved, the developer can conveniently conduct debugging and testing work while the information security is ensured, and the stability of the application program and the user are effectively ensured.

Further, based on the methods shown in fig. 2 to 7 and fig. 9 to 12, one or more embodiments of the present disclosure further provide a computer program product, including a computer program, where the computer program in the computer program product can implement the following flow when executed by a processor:

Furthermore, in another specific embodiment, the computer program product comprises a computer program, which computer program in the computer program product is capable of realizing the following flow when executed by a processor:

According to the embodiment of the specification, when the target page which cannot be normally and remotely displayed by the terminal equipment is detected, a page element grabbing instruction of the target page is sent to the terminal equipment, then, the information of a control tree constructed by a page control which is grabbed in the target page by the terminal equipment based on the page element grabbing instruction is received, then, the control tree is rendered in the page which cannot be normally and remotely displayed at present based on the information of the control tree, a remote rendering page corresponding to the target page is obtained, finally, the terminal equipment can be remotely controlled to execute an operation corresponding to the operation behavior in the remote rendering page based on the operation behavior of a user, so that the information of the control tree of the page which is normally and abnormally displayed in the terminal equipment is obtained through an automation tool, the page in the terminal equipment is re-rendered at a server end based on the information of the control tree, and the operation in the current page is executed by the terminal equipment is remotely controlled based on the page, so that the development efficiency of developers of financial and securities application programs is improved, the developers can conveniently conduct experience and test work while the safety of the information is ensured, and the stability of the application programs is ensured, and the user is effectively debugged.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable Gate Array, FPGA)) is an integrated circuit whose logic function is determined by the programming of the device by a user. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented by using "logic compiler" software, which is similar to the software compiler used in program development and writing, and the original code before the compiling is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but not just one of the hdds, but a plurality of kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), lava, lola, myHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing one or more embodiments of the present description.

It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

Embodiments of the present description are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable fraud case serial-to-parallel device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable fraud case serial-to-parallel device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

One or more embodiments of the present specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. One or more embodiments of the present description may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing is merely exemplary of the present disclosure and is not intended to limit the present disclosure. Various modifications and alterations to this specification will become apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of the present description, are intended to be included within the scope of the claims of the present description.

Claims

1. A method for remotely operating a terminal device, the method comprising:

2. The method of claim 1, wherein the page element crawling instruction includes a uiacometer dump command, and the receiving information of a control tree constructed by the terminal device based on the page control crawled by the page element crawling instruction in the target page includes:

and receiving XML information of a control tree constructed by the terminal equipment based on the UIAutometer dump command in the page element grabbing instruction and grabbing the page control in the target page.

3. The method according to claim 1 or 2, wherein the rendering the control tree in the page that cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page includes:

and converting the information of the control tree into control tree data in a preset data exchange format, and rendering the control tree in a page which cannot be normally and remotely displayed at present based on the converted control tree data to obtain a remote rendering page corresponding to the target page, wherein the preset data exchange format comprises one or more of a JSON format, a protocol buffer Protobuf format, a BSON format and a message packet message pack format.

4. The method of claim 3, wherein the rendering the control tree in the page that cannot be normally and remotely displayed at present based on the information of the control tree to obtain the remote rendering page corresponding to the target page includes:

screening the control tree based on a preset data screening rule and the information of the control tree to obtain the information of the screened control tree, wherein the data screening rule comprises a rule for screening out data except preset attribute information of an operable component in a page control in the control tree and text data used for displaying in an inoperable component;

And based on the information of the screened control tree, rendering the control tree in the page which cannot be normally and remotely displayed at present, and obtaining a remote rendering page corresponding to the target page.

5. The method of claim 4, wherein the performing, based on the filtered information of the control tree, rendering processing of the control tree on a page that cannot be normally and remotely displayed at present to obtain a remote rendered page corresponding to the target page includes:

and highlighting the operable component in a page which cannot be normally and remotely displayed at present based on the information of the screened control tree, and displaying text data used for displaying in the inoperable component so as to perform rendering processing of the control tree and obtain a remote rendering page corresponding to the target page.

6. The method of claim 5, wherein the target page comprises a page for which a screenshot operation or a screen recording operation is not allowed, and the target page for which the remote display terminal device cannot normally display currently comprises a target page for which the remote display terminal device displays currently being a black screen page.

7. The method of claim 6, wherein the remotely controlling the terminal device to perform an operation corresponding to the operation behavior in the target page based on the operation behavior of the user in the remote rendering page comprises:

Acquiring data of operation behaviors of a user in the remote rendering page;

and sending the data of the operation behaviors to the terminal equipment, wherein the data of the operation behaviors are used for triggering the terminal equipment to execute the operation corresponding to the operation behaviors in the target page.

8. A remote operation system of a terminal device, characterized in that the system comprises a server and a terminal device, wherein:

the server is configured to send a page element grabbing instruction of a target page to the terminal equipment when detecting that the target page currently displayed by the terminal equipment cannot be normally and remotely displayed;

the terminal equipment is configured to send the control tree information to the server based on the control tree information constructed by the page controls grabbed in the target page by the page element grabbing instruction;

the server is configured to render the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page, and remotely control the terminal equipment to execute operations corresponding to the operation behaviors in the target page based on the operation behaviors of a user in the remote rendering page.

9. The system of claim 8, wherein the page element crawling instruction includes a uiacomputer dump command, and the terminal device is configured to send XML information of a control tree constructed based on a page control crawled in the target page by the uiacomputer dump command in the page element crawling instruction to the server.

10. The system of claim 8, wherein the server is configured to obtain data of the user's operational behavior in the remote rendered page; transmitting the data of the operation behavior to the terminal equipment;

the terminal device is configured to execute an operation corresponding to the operation behavior in the target page based on the data of the operation behavior.

11. A remote operation method of a terminal device, applied to a platform front-end server, the method comprising:

12. The method according to claim 11, wherein when detecting that the target page currently displayed by the remote display terminal device cannot be normally displayed, sending a page element grabbing instruction of the target page to an upper computer includes:

when detecting that the target page displayed by the terminal equipment cannot be normally displayed remotely, sending a page element grabbing instruction of the target page to an upper computer through a preset communication protocol, wherein the communication protocol comprises a WebSocket protocol.

13. The method of claim 11, wherein the page element grabbing instruction includes a uiacometer dump command, and the receiving information of a control tree constructed by a page control grabbed by the upper computer in a target page of the terminal device based on the page element grabbing instruction includes:

And receiving XML information of a control tree constructed by the page control grabbed in the target page of the terminal equipment by the upper computer based on the UIAcutonomor dump command in the page element grabbing instruction.

14. The remote operation method of the terminal equipment is applied to an upper computer and is characterized by comprising the following steps of:

15. The method of claim 14, wherein the method further comprises:

the sending the control tree information to the platform front-end server comprises the following steps:

and sending the information of the screened control tree to the platform front-end server, wherein the information of the screened control tree is used for triggering the platform front-end server to render the control tree in a page which cannot be normally and remotely displayed at present based on the information of the screened control tree, so as to obtain a remote rendering page corresponding to the target page, and remotely controlling the terminal equipment to execute operation corresponding to the operation behavior in the target page through the upper computer based on the operation behavior of a user in the remote rendering page.

16. The remote operation system of the terminal equipment is characterized by comprising a platform front-end server, an upper computer and the terminal equipment, wherein:

The platform front-end server is configured to send a page element grabbing instruction of a target page to an upper computer when detecting that the target page currently displayed by the terminal equipment cannot be normally displayed remotely;

the upper computer is configured to send an instruction for grabbing a control tree constructed by page controls in the target page to the terminal equipment based on the page element grabbing instruction;

the terminal equipment is configured to send the information of the control tree constructed by the page control in the grabbed target page to the upper computer;

the upper computer is configured to send the control tree information to the platform front-end server;

the platform front-end server is configured to render the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree, obtain a remote rendering page corresponding to the target page, and remotely control the terminal equipment to execute operation corresponding to the operation behavior in the target page through the upper computer based on the operation behavior of a user in the remote rendering page.

17. The system of claim 16, wherein the upper computer is configured to convert the information of the control tree into control tree data in a preset data exchange format, and send the converted control tree data to the platform front-end server, and the preset data exchange format may include one or more of JSON format, protocol buffer Protobuf format, BSON format, and packet message pack format;

The platform front-end server is configured to render the control tree in a page which cannot be normally and remotely displayed at present based on the converted control tree data, so as to obtain a remote rendering page corresponding to the target page.

18. The system of claim 16, wherein the platform front-end server is configured to obtain data of operation behaviors of a user in the remote rendering page, and send the data of operation behaviors to the upper computer;

the upper computer is configured to send the data of the operation behavior to the terminal equipment;

the terminal device is configured to execute an operation corresponding to the operation behavior in a target page based on the data of the operation behavior.

19. A remote operation device of a terminal apparatus, the device comprising:

the instruction sending module is used for sending a page element grabbing instruction of the target page to the terminal equipment when detecting that the target page which is not normally displayed at present by the terminal equipment can not be displayed remotely;

the information receiving module is used for receiving information of a control tree constructed by the terminal equipment based on the page control grabbed in the target page by the page element grabbing instruction;

The rendering module is used for rendering the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page;

and the remote control module is used for remotely controlling the terminal equipment to execute the operation corresponding to the operation behavior in the target page based on the operation behavior of the user in the remote rendering page.

20. A remote operation device of a terminal apparatus, the device comprising:

the system comprises a grabbing instruction sending module, a grabbing instruction processing module and a display module, wherein the grabbing instruction sending module is used for sending a page element grabbing instruction of a target page to an upper computer when detecting that the target page which is not normally displayed at present by a remote display terminal device is detected;

the information receiving module is used for receiving the information of a control tree constructed by the page control grabbed in the target page of the terminal equipment by the upper computer based on the page element grabbing instruction;

the page rendering module is used for rendering the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree to obtain a remote rendering page corresponding to the target page;

and the remote control module is used for remotely controlling the terminal equipment to execute the operation corresponding to the operation behavior in the target page through the upper computer based on the operation behavior of the user in the remote rendering page.

21. A remote operation device of a terminal apparatus, the device comprising:

the instruction receiving module is used for receiving a page element grabbing instruction of a target page sent by the platform front-end server when the platform front-end server detects that the target page which is not normally displayed at present by the remote display terminal equipment;

the page element grabbing module is used for grabbing information of a control tree constructed by page controls grabbed in a target page of the terminal equipment based on the page element grabbing instruction;

the information sending module is used for sending the information of the control tree to the platform front-end server, the information of the control tree is used for triggering the platform front-end server to render the control tree in a page which cannot be normally and remotely displayed at present based on the information of the control tree, a remote rendering page corresponding to the target page is obtained, and based on the operation behavior of a user in the remote rendering page, the terminal equipment is remotely controlled to execute the operation corresponding to the operation behavior in the target page through the device.

22. A remote operation device of a terminal device, characterized in that the remote operation device of the terminal device comprises:

A processor; and

a memory arranged to store computer executable instructions which when executed implement the steps of the method of remote operation of a terminal device as claimed in any one of claims 1 to 7 or any one of claims 11 to 15.