CN114816401B - Interface element positioning method, electronic equipment and storage medium - Google Patents
Interface element positioning method, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114816401B CN114816401B CN202210386808.2A CN202210386808A CN114816401B CN 114816401 B CN114816401 B CN 114816401B CN 202210386808 A CN202210386808 A CN 202210386808A CN 114816401 B CN114816401 B CN 114816401B
- Authority
- CN
- China
- Prior art keywords
- dynamic link
- link library
- code dynamic
- interface element
- software client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/38—Creation or generation of source code for implementing user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
Abstract
The application provides a positioning method of an interface element, electronic equipment and a storage medium, wherein the method is executed by the electronic equipment, the electronic equipment is provided with robot software and a software client terminal using Cef technology, and the method comprises the following steps: the robot software starts a software client and injects a first code dynamic link library into the process of the software client; the first code dynamic link library intercepts an interface element loading notification sent by a software client, modifies the interface element loading notification and then forwards the modified interface element loading notification to a second code dynamic link library of the software client; loading interface elements of the software client by the second code dynamic link library; the first code dynamic link library intercepts a loading completion notice sent by the second code dynamic link library to the software client, calls a programming interface of the second code dynamic link library and executes an interface element positioning script. The scheme enables the Cef software client which does not support the accessibility technology to accurately position the interface elements.
Description
Technical Field
The present disclosure relates to the field of robot process automation technologies, and in particular, to a method for positioning an interface element, an electronic device, and a storage medium.
Background
Interface elements refer to various elements on the software client that interact with the user, such as buttons, input boxes, selection boxes, forms, and the like. The software client refers to general software in an operating system, such as instant messaging software APP, a music player and the like.
Windows access is a programming interface provided by the Microsoft Windows operating system, and can locate most of the software client interface elements. General method using IAccessible technology:
(1) Calling windows api accessibleObjectFromWindow to return an object pointer of IAccessible through a main window of a software client, wherein the object represents a main window control of target software; (2) Calling IAccessible, wherein get _ accChild traverses all child control (interface element) objects of the IAccessible object; (3) Calling IAccessible, wherein get _ accValue obtains the text information of the control; (4) Calling IAccessible:: accLocation to obtain the position information of the control; (5) And acquiring other related information by using other interfaces of the IAccessible.
The limitations of using windows access technology are: if the interface element of the positioned software client is created by using standard windows control technology, only the controls such as a button, an input box and the like provided by the windows can be used, and the access capability technology is automatically supported by the target software.
If the interface elements of the software client are drawn by using drawing technologies such as GDI (graphics device interface), the software client must internally provide support for the access technology, a window message WM _ GETOBJECT needs to be responded, and a proper IAccessfulle object is returned.
The Cef frame is a Google-based chrome open source library, the core of the chrome is a browser technology, so that interface elements of a software client manufactured by using the Cef frame are essentially web page elements and are drawn through a drawing technology, and the chrome needs to support the access technology by itself.
The interface displayed by the software client made by using the Cef frame is actually a webpage, the webpage can be designed very complicated, the interface elements can be thousands of interface elements, and the access availability technology inside the chromium is not perfectly supported, so that the problems that some elements cannot be positioned or the positions of some elements are not accurate can occur.
Disclosure of Invention
The embodiment of the application provides a method for positioning an interface element, which is used for positioning the interface element for a cef software client which does not support the access technology.
The embodiment of the application discloses a method for positioning an interface element, which comprises the following steps:
the method is performed by an electronic device, the electronic device being installed with robot software and a software client using Cef technology, the method comprising:
the robot software starts the software client and injects a first code dynamic link library into the process of the software client;
the first code dynamic link library intercepts an interface element loading notification sent by the software client, modifies the interface element loading notification and forwards the modified interface element loading notification to a second code dynamic link library of the software client;
the second code dynamic link library loads the interface element of the software client;
and the first code dynamic link library intercepts a loading completion notification sent by the second code dynamic link library to the software client, calls a programming interface of the second code dynamic link library and executes an interface element positioning script.
In one embodiment, the executing the interface element positioning script comprises:
the software client communicates with the robot software through a socket network communication technology and receives mouse position information sent by the robot software;
and the software client returns interface element information corresponding to the mouse position information to the robot software.
In one embodiment, the executing the interface element positioning script comprises:
the software client receives interface element information and operation information sent by the robot software;
finding out corresponding interface elements according to the interface element information;
and executing corresponding operation on the found interface element according to the operation information.
In an embodiment, the intercepting, by the first code dynamic link library, an interface element loading notification sent by the software client, modifying the interface element loading notification, and forwarding the modified interface element loading notification to the second code dynamic link library of the software client includes:
the first code dynamic link library intercepts an interface element loading notification sent to the second code dynamic link library by the software client;
the first code dynamic link library modifies the interface element loading notification;
and the first code dynamic link library forwards the modified interface element loading notification to the second code dynamic link library.
In an embodiment, the intercepting, by the first code dynamic link library, an interface element loading notification sent by the software client, modifying the interface element loading notification, and forwarding the modified interface element loading notification to the second code dynamic link library of the software client includes:
skipping to a parameter modification function of the first code dynamic link library by a function hooking technology before the software client calls an object creation function of the second code dynamic link library;
the parameter modification function modifies an address acquisition function transmitted by the software client to be a return function of the first code dynamic link library and transmits the return function to the second code dynamic link library;
and calling an object creating function of the second code dynamic link library to create a browser object.
In an embodiment, the intercepting, by the first code dynamic link library, an interface element loading notification sent by the software client, modifying the interface element loading notification, and forwarding the modified interface element loading notification to the second code dynamic link library of the software client includes:
the software client calls a derived function of the first code dynamic link library and jumps to a corresponding function of the second code dynamic link library;
when an object creating function of the first code dynamic link library is called, modifying an address acquisition function transmitted by the software client into a return function of the first code dynamic link library, and transmitting the return function into the second code dynamic link library;
and calling an object creating function of the second code dynamic link library to create a browser object.
In one embodiment, the second code dynamically linked library loads the interface element of the software client, including:
and drawing the interface elements of the software client through the browser object.
In an embodiment, the first code dynamic link library intercepts a load completion notification sent by the second code dynamic link library to the software client, calls a programming interface of the second code dynamic link library, and executes an interface element positioning script, including:
the second code dynamic link library calls a return function of the first code dynamic link library to obtain an execution function of the first code dynamic link library;
the second code dynamic link library calls the execution function;
the execution function calls a notification function of the software client to notify the software client that the interface loading is completed;
and the execution function calls a specified function of the second code dynamic link library to execute the interface element positioning script.
The embodiment of the application provides a positioning device for interface elements, which comprises:
the code injection module is used for starting a software client through robot software and injecting a first code dynamic link library into the process of the software client;
the loading notification module is used for intercepting an interface element loading notification sent by the software client through the first code dynamic link library, modifying the interface element loading notification and then forwarding the modified interface element loading notification to a second code dynamic link library of the software client;
the element loading module is used for loading the interface elements of the software client through the second code dynamic link library;
and the script execution module is used for intercepting a loading completion notification sent by the second code dynamic link library to the software client through the first code dynamic link library, calling a programming interface of the second code dynamic link library and executing an interface element positioning script.
An embodiment of the present application provides an electronic device, which includes:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to execute the positioning method of the interface element.
The embodiment of the application provides a computer-readable storage medium, and the storage medium stores a computer program, and the computer program can be executed by a processor to complete the positioning method of the interface element.
According to the technical scheme provided by the embodiment of the application, the first code dynamic link library is injected into the process of the software client, the interface element loading notification sent by the software client is intercepted, the modified interface element loading notification is forwarded to the second code dynamic link library of the software client, the second code dynamic link library is intercepted and sent to the loading completion notification of the software client, the programming interface of the second code dynamic link library is called, and the interface element positioning script is executed, so that the interface element can be accurately positioned even if the software client does not support the accessibility technology.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be briefly described below.
Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a method for positioning an interface element according to an embodiment of the present application;
FIG. 3 is a schematic diagram illustrating a method for positioning an interface element according to an embodiment of the present application;
FIG. 4 is a detailed flowchart of step S220 in the corresponding embodiment of FIG. 2;
FIG. 5 is a flow chart illustrating interface element loading by a normal software client;
FIG. 6 is a schematic flow chart diagram illustrating loading of an interface element according to an embodiment of the present application;
FIG. 7 is a schematic flow chart diagram of an interface element positioning script provided by an embodiment of the present application;
FIG. 8 is a flowchart illustrating details of step S220 in the corresponding embodiment of FIG. 2 according to another embodiment of the present application;
FIG. 9 is a schematic flow chart diagram illustrating loading of interface elements according to another embodiment of the present application;
fig. 10 is a block diagram of a positioning apparatus for an interface element according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
Like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not construed as indicating or implying relative importance.
Fig. 1 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may be configured to execute the method for positioning an interface element provided in the embodiment of the present application. As shown in fig. 1, the electronic device 100 includes: one or more processors 102, and one or more memories 104 storing processor-executable instructions. Wherein the processor 102 is configured to execute a positioning method of an interface element provided in the following embodiments of the present application.
The processor 102 may be a gateway, or may be a smart terminal, or may be a device including a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or other forms of processing unit having data processing capabilities and/or instruction execution capabilities, and may process data of other components in the electronic device 100, and may control other components in the electronic device 100 to perform desired functions.
The memory 104 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium and executed by processor 102 to implement the method for positioning interface elements described below. Various applications and various data, such as various data used and/or generated by the applications, may also be stored in the computer-readable storage medium.
In one embodiment, the electronic device 100 shown in FIG. 1 may further include an input device 106, an output device 108, and a data acquisition device 110, which are interconnected via a bus system 112 and/or other form of connection mechanism (not shown). It should be noted that the components and structure of the electronic device 100 shown in fig. 1 are merely exemplary and not limiting, and the electronic device 100 may have other components and structures as desired.
The input device 106 may be a device used by a user to input instructions and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like. The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, and the like. The data acquisition device 110 may acquire an image of a subject and store the acquired image in the memory 104 for use by other components. Illustratively, the data acquisition device 110 may be a camera.
In an embodiment, the components of the example electronic device 100 for implementing the method for positioning the interface element according to the embodiment of the present application may be integrally disposed or disposed separately, such as the processor 102, the memory 104, the input device 106, and the output device 108 being integrally disposed together, and the data acquisition device 110 being disposed separately.
In an embodiment, the example electronic device 100 for implementing the positioning method of the interface element of the embodiment of the present application may be implemented as an intelligent terminal, such as a smart phone, a tablet computer, a desktop computer, an in-vehicle device, and the like.
Fig. 2 is a schematic flowchart of a method for positioning an interface element, according to an embodiment of the present application, where the method is performed by an electronic device, and the electronic device is installed with robot software and a software client using Cef technology, and as shown in fig. 2, the method includes: step S210-step S240.
Step S210: and the robot software starts the software client and injects the first code dynamic link library into the process of the software client.
The software client refers to software for developing an interactive interface by using the Cef technology, such as instant messaging software, music playing software and the like by using the Cef technology. Cef (chemically Embedded Framework) is a Framework developed based on a google chemically open library, and people can display a software client-side interactive interface in a webpage mode by using the Cef.
It should be noted that the general steps of writing software using the Cef framework are to call the API CefBrowserHost provided by Cef, create a cefbrower object, and then use the cefbrower to load the remote-addressed web page or local web page. Calling CefBrowserHost:: createBrowser needs to transfer a CefWindowInfo object as a parameter, the CefWindowInfo object is used for setting the parameter of a window, if calling API CefWindowInfo:: setAsPop is a mode with a window, if calling API CefWindowInfo:: setAsWindowless is a mode without a window.
The windowed model means that a window can be created inside the Cef frame, a developer of a software client does not need to provide an additional window, and the developed software can support the windows access availability technology, because google is already implemented inside, but the problem that the positioning interface elements are inaccurate can be caused.
The windowless mode means that a software client developer needs to create a window by himself, the Cef frame cannot create the window internally, so that window messages cannot be received, the windows access technology cannot be supported, and the software client developer and the Cef frame do not provide other methods for positioning interface elements, so that the technology cannot position the interface elements.
As can be seen, the Cef software client in the window phantom mode can support windows access availability technology, but there is a problem that the interface elements are not accurately positioned, while the Cef software client in the windowless phantom mode has no technology to position the interface elements. The method provided by the embodiment of the application can accurately position the interface elements of the Cef software client, including a window mode and a window-free mode.
Robotic software is software used to locate interface elements of a software client and perform automated processes.
The first code dynamic link library is distinguished from a second code dynamic link library which is described below, the first code dynamic link library (mycef.dll) is made by a robot software developer and is used for being injected into the process of a software client, so that the communication between the software client and the second code dynamic link library (libref.dll) is intercepted, and the software client executes a JavaScript (namely, an interface element positioning script which is described below) which is prepared in advance by the developer when the interface element is loaded. Specifically, a DLL (Dynamic Link Library) injection technology may be used, or a code Dynamic Link Library libref.dll in a software client directory may be renamed to libcef2.dll (i.e., a second code Dynamic Link Library), and then the code Dynamic Link Library mycef.dll written by the developer itself is copied to the directory and named to libref.dll (i.e., a first Dynamic Link Library).
Dll is the code dynamic link library of Cef, and software client developers develop software client interfaces through the programming interface provided by this dynamic link library.
Step S220: and the first code dynamic link library intercepts an interface element loading notification sent by the software client, modifies the interface element loading notification and forwards the modified interface element loading notification to a second code dynamic link library of the software client.
As shown in FIG. 3, the first code dynamic link library intercepts communication between the software client and the second code dynamic link library. After the interface element loading notification is sent to the first code dynamic link library, the first code dynamic link library modifies the interface element loading notification and then forwards the interface element loading notification to the second code dynamic link library. Specifically, the address acquisition function transmitted by the software client can be modified into a return function of the first code dynamic link library, so that an autonomously developed interface element positioning script can be inserted.
In an embodiment, as shown in fig. 4, the step S220 specifically includes: step S221 to step S223.
It should be noted that, since the description of step S220 involves a plurality of functions, we describe these functions by using codes, which are not actual function names, and the following descriptions are given to these codes:
LA: the second code dynamically links the export function of the library, which is used for creating a browser object and is named as an object creating function.
FA: and the address acquisition function of the software client is used for enabling a second code dynamic link library (libref. Dll) to acquire the address of the function FB.
FB: a function in the software client, which functions to allow the second code dynamic link library (libref. Dll) to notify the software client that the loading of the interface is complete.
MA: and the first code dynamically links the parameter modification function of the library, and the function of the function is to modify the FA.
MB: the first code dynamically links the return function of the library, which functions to return the MC.
MC: the first code dynamically links the execution function of the library, and the function is to call the function for executing JavaScript code (interface element positioning script) and call FB to inform the software that the client interface is loaded completely.
Step S221: and skipping to a parameter modification function of the first code dynamic link library by a function hooking technology before the software client calls an object creation function of the second code dynamic link library.
Fig. 5 is a schematic flowchart of loading an interface element by a conventional software client. It should be noted that, as shown in fig. 5, when the software client uses the Cef technology process interface for presentation, in the first step, the software client calls the derived function LA of the second code dynamic link library (libref. Dll) to create a browser object (browser object), and when the function LA is called, the function FA in the software client is introduced as a parameter. And secondly, calling a browser object loading page by the software client. And thirdly, calling a function FA by the second code dynamic link library to obtain a function FB of the software client. And fourthly, calling a function FB by the second code dynamic link library to inform the software of the completion of the loading of the client interface.
In the embodiment of the present application, in order to insert an interface element positioning script that is developed autonomously, a developer may use C + + to make a dynamic link library, which is named as mycef.dll (i.e., a first code dynamic link library), and derive a callable function, where the function name may be called InitCef, and the function functions: using function hooking technique, the derived function LA used to create the browser object in libref.dll (i.e. the second code dynamic link library) is intercepted, so that it jumps to the parameter modification function (code number function MA) in the first code dynamic link library (mycef.dll) first when executing.
Step S222: and the parameter modifying function modifies the address acquisition function transmitted by the software client to be a return function of the first code dynamic link library and transmits the return function to the second code dynamic link library.
When the software client calls the object creation function LA of the second code dynamic link library, because the called object creation function LA is hooked by a developer, the object creation function LA jumps to a parameter modification function MA of the mycef.dll at the moment, the parameter modification function MA is called, an address acquisition function FA transmitted by the software client is replaced by a return function MB in the mycef.dll, and the MB is transmitted into libref.dll.
Step S223: and calling an object creating function of the second code dynamic link library to create a browser object.
As shown in fig. 6, in step (1), the software client calls an object creation function LA to create a browser object (browser object), and transmits an address acquisition function FA; before calling an object creation function LA, jumping to an execution function MA (replacing FA as MB, calling LA to create a browser object); and (3) calling a browser object loading page by the software client.
Step S230: and the second code dynamic link library loads the interface element of the software client.
As shown in fig. 6, the interface element of the software client is drawn through a browser object (browser object).
Step S240: intercepting a loading completion notification sent by the second code dynamic link library to the software client through the first code dynamic link library, calling a programming interface of the second code dynamic link library, and executing an interface element positioning script.
After the interface element is loaded, as shown in fig. 6, in step (4), the second code dynamic link library calls a return function MB of the first code dynamic link library to obtain an execution function MC of the first code dynamic link library. And (5) the second code dynamic link library calls the execution function MC to inform the interface of finishing loading. Step (6), the execution function MC calls a notification function FB of the software client to notify the software client that the interface loading is completed; and (7) the execution function MC calls a specified function (namely a programming interface) of the second code dynamic link library to execute the interface element positioning script (JavaScript code).
Positioning the interface element refers to obtaining various information of the interface element, such as position, size, content, and the like, and may reposition the interface element in reverse according to the information.
In one embodiment, as shown in fig. 7, the specific content of the interface element positioning script includes:
step S701: the software client communicates with the robot software through a socket network communication technology, and receives mouse position information sent by the robot software.
Socket is a network communication technology, and is used for communication among a plurality of programs of the same computer. When a user moves on the software client and performs interface element selection operation, the robot software communicates with a JavaScript code executed in the software client through a socket and sends current mouse position information, and the mouse position information can be represented by coordinates (x, y) of a mouse.
Step S702: and the software client returns interface element information corresponding to the mouse position information to the robot software.
The JavaScript code executed in the software client returns interface element information corresponding to the mouse position to the robot software, and the interface element information can comprise information such as element size, coordinates, title and the like. For example, when the user moves to a button, the coordinates, size, title, etc. of the button may be returned.
Step S703: and receiving interface element information and operation information sent by the robot software.
When the automatic flow is executed, the JavaScript code executed in the software client receives the interface element information and the operation information sent by the robot software, wherein the operation information can be clicking, inputting and the like. Different interface elements can have different operations.
Step S704: and finding out the corresponding interface element according to the interface element information.
Specifically, the JavaScript code executed in the software client may find, according to the coordinates included in the interface element information, the interface element corresponding to the coordinates, for example, a button, an input box, or the like.
Step S705: and executing corresponding operation on the found interface element according to the operation information.
For example, the operation information may be a click, and the interface element may be a button, and the button may be automatically clicked after the button is found.
According to the technical scheme provided by the embodiment of the application, the first code dynamic link library is injected into the process of the software client, the interface element loading notification sent by the software client is intercepted, the modified interface element loading notification is forwarded to the second code dynamic link library of the page loading frame, the second code dynamic link library is intercepted and sent to the loading completion notification of the software client, the programming interface of the second code dynamic link library is called, and the interface element positioning script is executed, so that the interface element can be accurately positioned even if the software client does not support the accessibility technology.
In another embodiment, as shown in fig. 8, the step S220 specifically includes: step S801 to step S803.
Step S801: and the software client calls the export function of the first code dynamic link library and jumps to the corresponding function of the second code dynamic link library.
libref.dll is a code library provided in the Cef framework, and there are about 200 derived functions for the developer to use, so if the original libref.dll is to be replaced, the same derived functions must be provided, and then jump to the original libref.dll in the derived functions.
In one embodiment, as shown in fig. 9, the libref.dll (i.e. the second code dynamic link library) in the software client directory can be renamed to libref 2.dll, and then the mycef.dll (i.e. the first code dynamic link library) written by the developer is copied to the past to implement code injection, at which time the mycef.dll can be named libref.dll. So that the software client loads libref.dll (first code dynamic link library) written by the developer, and libref.dll reloads libref 2.dll (i.e. second code dynamic link library) and acquires all its derived function addresses for jumping.
When the software client is displayed by using a Cef technology process interface, various derived functions of libref.dll are called and directly jump to corresponding functions in libref 2.dll except an object creating function LA for creating a browser object.
Step S802: when the object creating function of the first code dynamic link library is called, the address obtaining function transmitted by the software client side is modified to be the return function of the first code dynamic link library, and the return function is transmitted to the second code dynamic link library.
As shown in fig. 9, in step (1), the software client calls an object creation function LA, and an address acquisition function FA in the software client is introduced as a parameter. The function of the address acquisition function FA is that libcef2.Dll is used for acquiring the address of the notification function FB in the target software client, and the function of the notification function FB is that libcef2.Dll is used for notifying the target software interface of the completion of loading.
As shown in fig. 9, the function LA in step (2) libref.dll (first code dynamic link library) modifies the incoming address acquisition function FA, replaces the address acquisition function FA with the return function MB in libref.dll, and sends libref 2.dll, and calls the function LA of libref 2.dll to create the browser object. Wherein the return function MB functions as an execution function MC in the return function libref. The execution function MC functions to execute JavaScript code and then call the notification function FB.
Step S803: and calling an object creating function LA of the second code dynamic link library to create a browser object. Specifically, in step (3), the interface element of the software client may be drawn through a browser object (browser object).
Then, referring to the above embodiment, when the interface loading of the software client is completed, libcef2.Dll tries to call the address obtaining function FA to obtain the notification function FB, but since FA is replaced by the return function MB, the second code dynamic link library in step (4) calls the return function MB of the first code dynamic link library to obtain the execution function MC of the first code dynamic link library. And (5) the second code dynamic link library calls the execution function MC to inform the interface of finishing loading. Step (6), the execution function MC calls a notification function FB of the software client to notify the software client that the interface loading is completed; and (7) calling a specific function in libcef2.Dll by the execution function MC to execute the developed JavaScript code (namely the interface element positioning script), and establishing socket communication connection with the robot software. The robot software communicates with the executed JavaScript code in the software client through the socket, sends a mouse position message, positions the interface element by the executed JavaScript code through the mouse position, and returns the interface element information to the robot software. The robot software executes an automated process such as mouse clicking by using interface element information (such as a position on a screen).
The following is an embodiment of the apparatus of the present application, which can be used to execute embodiments of the method for positioning the interface element described above in the present application. For details not disclosed in the embodiments of the apparatus of the present application, please refer to the embodiments of the positioning method of the interface element of the present application.
Fig. 10 is a block diagram of a positioning apparatus for an interface element according to an embodiment of the present application. As shown in fig. 10, the apparatus includes: a code injection module 1010, a load notification module 1020, an element load module 1030, and an element execution module 1040.
The code injection module 1010 is used for starting a software client through robot software and injecting a first code dynamic link library into the process of the software client;
a loading notification module 1020, configured to intercept, through the first code dynamic link library, an interface element loading notification sent by the software client, modify the interface element loading notification, and forward the modified interface element loading notification to a second code dynamic link library of the software client;
an element loading module 1030, configured to load an interface element of the software client through the second code dynamic link library;
the script execution module 1040 is configured to intercept, through the first code dynamic link library, a loading completion notification sent by the second code dynamic link library to the software client, and call a programming interface of the second code dynamic link library to execute an interface element positioning script.
The implementation process of the function and the action of each module in the device is specifically described in the implementation process of the corresponding step in the positioning method of the interface element, and is not described again here.
In the embodiments provided in the present application, the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes.
Claims (8)
1. A method for positioning an interface element, the method being performed by an electronic device having robotic software and a software client using Cef technology installed thereon, the method comprising:
the robot software starts the software client and injects a first code dynamic link library into the process of the software client;
the first code dynamic link library intercepts an interface element loading notification sent by the software client, modifies the interface element loading notification and forwards the modified interface element loading notification to a second code dynamic link library of the software client, and the method comprises the following steps: skipping to a parameter modification function of the first code dynamic link library by a function hooking technology before the software client calls an object creation function of the second code dynamic link library; the parameter modification function modifies an address acquisition function transmitted by the software client to be a return function of the first code dynamic link library and transmits the return function to the second code dynamic link library; calling an object creating function of the second code dynamic link library to create a browser object; or the software client calls a derivation function of the first code dynamic link library and jumps to a corresponding function of the second code dynamic link library; when an object creating function of the first code dynamic link library is called, modifying an address acquisition function transmitted by the software client into a return function of the first code dynamic link library, and transmitting the return function into the second code dynamic link library; calling an object creating function of the second code dynamic link library to create a browser object;
the second code dynamic link library loads the interface elements of the software client;
and the first code dynamic link library intercepts a loading completion notification sent by the second code dynamic link library to the software client, calls a programming interface of the second code dynamic link library and executes an interface element positioning script.
2. The method of claim 1, wherein executing the interface element positioning script comprises:
the software client communicates with the robot software through a socket network communication technology and receives mouse position information sent by the robot software;
and the software client returns interface element information corresponding to the mouse position information to the robot software.
3. The method of claim 2, wherein executing the interface element positioning script comprises:
the software client receives interface element information and operation information sent by the robot software;
finding out corresponding interface elements according to the interface element information;
and executing corresponding operation on the found interface elements according to the operation information.
4. The method of claim 1, wherein the first code dynamic link library intercepts an interface element loading notification sent by the software client, modifies the interface element loading notification and forwards the modified interface element loading notification to a second code dynamic link library of the software client, and the method comprises the following steps:
the first code dynamic link library intercepts an interface element loading notification sent to the second code dynamic link library by the software client;
the first code dynamic link library modifies the interface element loading notification;
and the first code dynamic link library forwards the modified interface element loading notification to the second code dynamic link library.
5. The method of claim 1, wherein the second code dynamically linked library loads the interface element of the software client, and comprises:
and drawing the interface elements of the software client through the browser object.
6. The method of claim 1, wherein the first code dynamic link library intercepts a load completion notification sent by the second code dynamic link library to the software client, calls a programming interface of the second code dynamic link library, and executes an interface element positioning script, comprising:
the second code dynamic link library calls a return function of the first code dynamic link library to obtain an execution function of the first code dynamic link library;
the second code dynamic link library calls the execution function;
the execution function calls a notification function of the software client to notify the software client that the interface is loaded;
and the execution function calls a specified function of the second code dynamic link library to execute the interface element positioning script.
7. An electronic device, characterized in that the electronic device comprises:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to perform the method of positioning an interface element of any one of claims 1-6.
8. A computer-readable storage medium, characterized in that the storage medium stores a computer program executable by a processor to perform the method of positioning an interface element according to any one of claims 1-6.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210386808.2A CN114816401B (en) | 2022-04-13 | 2022-04-13 | Interface element positioning method, electronic equipment and storage medium |
| PCT/CN2022/139985 WO2023197656A1 (en) | 2022-04-13 | 2022-12-19 | Interface element positioning method, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210386808.2A CN114816401B (en) | 2022-04-13 | 2022-04-13 | Interface element positioning method, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114816401A CN114816401A (en) | 2022-07-29 |
| CN114816401B true CN114816401B (en) | 2022-12-23 |
Family
ID=82536955
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210386808.2A Active CN114816401B (en) | 2022-04-13 | 2022-04-13 | Interface element positioning method, electronic equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114816401B (en) |
| WO (1) | WO2023197656A1 (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114816401B (en) * | 2022-04-13 | 2022-12-23 | 上海弘玑信息技术有限公司 | Interface element positioning method, electronic equipment and storage medium |
| CN115495807A (en) * | 2022-09-26 | 2022-12-20 | 上海弘玑信息技术有限公司 | Automatic drawing method and electronic equipment |
| CN116302621A (en) * | 2023-05-19 | 2023-06-23 | 杭州实在智能科技有限公司 | Method and system for picking up information-created environment elements based on QT interface library |
| CN117130717B (en) * | 2023-10-27 | 2024-02-13 | 杭州实在智能科技有限公司 | Element positioning method and system of HTMLayout application program in RPA scene |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102023805A (en) * | 2010-12-16 | 2011-04-20 | 浙江大学 | Method for assisting in browsing text messages on software interface |
| CN106325703A (en) * | 2015-07-03 | 2017-01-11 | 宁波弘讯科技股份有限公司 | Webpage man-machine interface control device and running method thereof |
| CN107357586A (en) * | 2017-07-14 | 2017-11-17 | 腾讯科技（深圳）有限公司 | Control method, device and the equipment of application program |
| CN110162249A (en) * | 2019-04-23 | 2019-08-23 | 北京小米移动软件有限公司 | A kind of control application program image target method and device, equipment and storage medium |
| US10558824B1 (en) * | 2019-02-04 | 2020-02-11 | S2 Systems Corporation | Application remoting using network vector rendering |
| CN111459589A (en) * | 2020-03-30 | 2020-07-28 | 深信服科技股份有限公司 | Redirection processing method, virtual desktop infrastructure and readable storage medium |
| CN111857710A (en) * | 2020-06-18 | 2020-10-30 | 远光软件股份有限公司 | Method and device for developing Eclipse plug-in based on embedded chromium |
| CN113796023A (en) * | 2019-05-03 | 2021-12-14 | 三星电子株式会社 | Method and UE for reporting accessibility measurement in wireless communication system |
| CN114327481A (en) * | 2021-12-30 | 2022-04-12 | 北京百度网讯科技有限公司 | Code processing method, device, equipment and storage medium |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6611878B2 (en) * | 1996-11-08 | 2003-08-26 | International Business Machines Corporation | Method and apparatus for software technology injection for operating systems which assign separate process address spaces |
| US9678747B2 (en) * | 2011-02-08 | 2017-06-13 | Openspan, Inc. | Code injection and code interception in an operating system with multiple subsystem environments |
| CN106203069B (en) * | 2016-06-27 | 2019-10-15 | 珠海豹趣科技有限公司 | A kind of hold-up interception method of dynamic link library file, device and terminal device |
| RU2634168C1 (en) * | 2016-07-29 | 2017-10-24 | Акционерное общество "Лаборатория Касперского" | System and method for blocking access to protected applications |
| US11144338B2 (en) * | 2019-08-20 | 2021-10-12 | Hyland Software, Inc. | Computing system for macro generation, modification, verification, and execution |
| CN111078339B (en) * | 2019-11-29 | 2022-11-11 | 苏宁云计算有限公司 | Interface element positioning method and device, computer equipment and storage medium |
| CN114816401B (en) * | 2022-04-13 | 2022-12-23 | 上海弘玑信息技术有限公司 | Interface element positioning method, electronic equipment and storage medium |
| CN115061685B (en) * | 2022-06-30 | 2023-07-21 | 上海弘玑信息技术有限公司 | Interface element positioning method of software client, electronic equipment and storage medium |
-
2022
- 2022-04-13 CN CN202210386808.2A patent/CN114816401B/en active Active
- 2022-12-19 WO PCT/CN2022/139985 patent/WO2023197656A1/en unknown
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102023805A (en) * | 2010-12-16 | 2011-04-20 | 浙江大学 | Method for assisting in browsing text messages on software interface |
| CN106325703A (en) * | 2015-07-03 | 2017-01-11 | 宁波弘讯科技股份有限公司 | Webpage man-machine interface control device and running method thereof |
| CN107357586A (en) * | 2017-07-14 | 2017-11-17 | 腾讯科技（深圳）有限公司 | Control method, device and the equipment of application program |
| US10558824B1 (en) * | 2019-02-04 | 2020-02-11 | S2 Systems Corporation | Application remoting using network vector rendering |
| CN110162249A (en) * | 2019-04-23 | 2019-08-23 | 北京小米移动软件有限公司 | A kind of control application program image target method and device, equipment and storage medium |
| CN113796023A (en) * | 2019-05-03 | 2021-12-14 | 三星电子株式会社 | Method and UE for reporting accessibility measurement in wireless communication system |
| CN111459589A (en) * | 2020-03-30 | 2020-07-28 | 深信服科技股份有限公司 | Redirection processing method, virtual desktop infrastructure and readable storage medium |
| CN111857710A (en) * | 2020-06-18 | 2020-10-30 | 远光软件股份有限公司 | Method and device for developing Eclipse plug-in based on embedded chromium |
| CN114327481A (en) * | 2021-12-30 | 2022-04-12 | 北京百度网讯科技有限公司 | Code processing method, device, equipment and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| Linux | c语言静态链接库和动态链接库以及动态加载库;frozen122;《https://blog.csdn.net/frozen122/article/details/116277992》;20210510;1-8 * |
| Simulation of triaxial compression tests with polyhedral discrete elements;Seung Jae Le等;《Computers and Geotechnics》;20120630;第43卷;92-100 * |
| 一个组件安全自动化测试平台的设计与实现;陈锦富等;《计算机科学》;20081231;第35卷(第12期);229-233 * |
| 面向iOS系统移动终端保密通信技术研究;钱泽虹;《中国优秀硕士学位论文全文数据库 信息科技辑》;20180415;I138-1889 * |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023197656A1 (en) | 2023-10-19 |
| CN114816401A (en) | 2022-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114816401B (en) | Interface element positioning method, electronic equipment and storage medium | |
| CN111026396B (en) | Page rendering method and device, electronic equipment and storage medium | |
| EP3822828A2 (en) | Method, apparatus, device and storage medium for determining page theme and computer program product | |
| US10419568B2 (en) | Manipulation of browser DOM on server | |
| US20140108914A1 (en) | Webpages with onboard hardware functions | |
| CN115061685B (en) | Interface element positioning method of software client, electronic equipment and storage medium | |
| CN111294395A (en) | Terminal page transmission method, device, medium and electronic equipment | |
| US20220407907A1 (en) | Web application component migration to a cloud computing system | |
| CN113656005B (en) | Application component library construction method, application component configuration method and related devices | |
| US11604662B2 (en) | System and method for accelerating modernization of user interfaces in a computing environment | |
| CN111782999B (en) | Page display method, device, equipment and system | |
| CN111934985A (en) | Media content sharing method, device and equipment and computer readable storage medium | |
| US9361131B1 (en) | Network resource access via a mobile shell | |
| US11558321B2 (en) | Integration of an email client with hosted applications | |
| CN114168875A (en) | Page program generation method and device, computer equipment and storage medium | |
| CN111931111A (en) | Webpage development method, webpage display method, device and equipment | |
| Johnson | Using MVVM Light with your Xamarin Apps | |
| CN111984278A (en) | Method and device for acquiring animation image | |
| CN112015507A (en) | Application guide method and device | |
| US20210109749A1 (en) | Providing local variable copies of global state for explicit computer executable instructions depending whether the global state is modified | |
| CN111026466A (en) | File processing method and device, computer readable storage medium and electronic equipment | |
| US20230214310A1 (en) | Infrastructure to integrate an integrated development environment (ide) with game engines | |
| CN112306324B (en) | Information processing method, apparatus, device and medium | |
| US11704014B1 (en) | Supporting micro swipes for generating custom user interface responses | |
| CN116055463A (en) | Method and device for realizing open interface |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |