CN113176921B - Desktop automatic customizing system - Google Patents
Desktop automatic customizing system Download PDFInfo
- Publication number
- CN113176921B CN113176921B CN202110478696.9A CN202110478696A CN113176921B CN 113176921 B CN113176921 B CN 113176921B CN 202110478696 A CN202110478696 A CN 202110478696A CN 113176921 B CN113176921 B CN 113176921B
- Authority
- CN
- China
- Prior art keywords
- control
- elements
- desktop
- name
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The application discloses a desktop automation customization system, which comprises a UI element detection module, wherein the UI element detection module automatically switches to traversal search logic when a native UIA cannot return a correct control element according to screen coordinates, and achieves timely stopping by using a greedy algorithm and a pruning algorithm; and the flow information generation module records key information of the target control element to provide basis for positioning the control during playback, wherein the key information comprises a control type and a control name. The desktop automatic customizing system can repair the defects that a plurality of elements exist in the top layer mask, so that the mask can only be detected when the elements of the inner control are required to be positioned; the search application time can be greatly shortened, and the user experience is improved; dynamic or even trap attribute access can be bypassed, and the recording process is normally completed; the automatic customizing system of the desktop can avoid the Bayesian error.
Description
Technical Field
The application relates to the technical field of desktop automation, in particular to a desktop automation customizing system.
Background
RPA (Robotic Process Automation), i.e. robot flow automation, the function of which is to automatically complete a series of computer software operations, aimed at eliminating repetitive, redundant human labor with definite steps and programmable; the process comprises two stages: recording UI element information and action information of manual operation, namely recording; the robot reproduces the manual operation using the aforementioned information, called playback.
Due to the diversity of implementation manners and technical frameworks of computer software, RPA needs to have various technical means corresponding to the implementation manners and technical frameworks to achieve better coverage, including but not limited to the following: desktop automation, WEB automation, mobile phone automation and the like; the present application will focus on desktop automation.
The most common and sophisticated technology for desktop Automation is UI Automation, UIA for short. It provides a relevant interface for obtaining and setting the attribute values of interface elements based on some searching and executing algorithm. However, software developers may inadvertently break through this algorithm, causing the UIA to obtain unreasonable results. For example, when the target software has a top mask panel, it may not be possible to locate useful controls on the inner layer; when a software developer rewrites the acquisition logic of the control properties but there is a bug, accessing the relevant properties may fall into a dead loop or crash the target application.
Therefore, we propose a desktop automation customization system to solve the above-mentioned problems.
Disclosure of Invention
The application aims to provide a desktop automation customizing system which is used for solving the problem that most desktop automation in the background technology cannot position useful controls on the inner layer when target software has a top layer mask panel; when a software developer rewrites the acquisition logic of the control properties but there is a bug, accessing the relevant properties may be a matter of dead-cycling or crashing the target application.
In order to achieve the above purpose, the present application provides the following technical solutions: a desktop automation customization system, comprising:
the UI element detection module is used for automatically switching to traversal search logic when the original UIA cannot return the correct control element according to the screen coordinates, and stopping timely by using a greedy algorithm and a pruning algorithm;
the flow information generation module records key information of target control elements to provide basis for positioning the control during playback, wherein the key information comprises a control type and a control name;
the rule description configuration file contains four types of key information, namely special application identification rules, features of elements to be ignored, suspension conditions of element detection and element attributes required for generating a positioning sub (basis of a positioning control during playback);
and the playback execution module encapsulates interfaces of various common actions and gives a default implementation, and customization of the application is completed through different implementations of the default implementation.
Preferably, the UI element detection module, when completing the automatic switching to the traversal search logic and implementing timely stopping of the process using a greedy algorithm and a pruning algorithm, depends on one of pruning and suspension conditions, and is different in each application.
Preferably, the input content of the UI element detection module is: screen coordinates, configuration files; the output content of the UI element detection module is as follows: the innermost interface element on the coordinates.
Preferably, the flow information generating module not only needs to obtain the path information of the control in a bottom-up mode, but also simulates a one-time playback process so as to ensure that the generated information can uniquely correspond to the target control; wherein the key information used is flexibly provided by the configuration file.
Preferably, the input content of the flow information generating module is: target control elements, configuration files; the output content of the flow information generation module is as follows: a target key information sequence.
Preferably, the application specific identification rule describes a process flow requiring customization to be enabled.
Preferably, the feature of the element to be ignored describes the element that needs to be pruned during UI detection.
Preferably, the suspension condition of the element detection describes that the search is suspended when an element with specific characteristics is encountered during the UI detection; default to the first leaf element located at a given coordinate.
Preferably, the element attribute required for generating the positioning sub (the basis of the positioning control during playback) defines positioning the target element by the attribute of the element; not only the non-dynamic attribute is needed to be selected, but also the attribute with access error is avoided.
Preferably, the available attributes in the element attributes include:
process Name: a process name;
class Name: class name;
automation Id: the unique identification of the control element defines the introduced control attribute in the UI Automation technology, and the initial aim is to locate for software Automation test;
name: a control name;
roller: a control type;
bounding Rectangle: control position and size information;
is Enabled: whether or not it is available;
is Offscreen: whether the control position is outside the screen;
access Key: and the control shortcut key is used for rapidly operating the control mode.
Compared with the prior art, the application has the beneficial effects that:
1. the top layer mask exists in many elements, so that the mask can only be detected when the inner control element needs to be positioned;
2. the element detection module needs to highlight the target control in real time, and the complete search application control tree is quite time-consuming, so that the time can be greatly shortened and the user experience can be improved through a greedy and pruning algorithm under given conditions;
3. the standard control provided by Microsoft provides an interface for acquiring interface element information by default, but a third party control or a custom control can rewrite related methods, so that errors or dead loops are trapped when certain control attributes are accessed; the specific application is configured with specific characteristic attributes for generating a locator, so that dynamic or even trap attribute access can be bypassed, and the recording process is normally completed;
4. in some applications, invoking a generic interface to perform a specified action does not actually achieve the goal, and needs to be accomplished in other ways; the execution module capable of being updated by heat can quickly repair the problems, and prevent the automatic customizing system of the desktop from being in a Bayesian error.
Drawings
Fig. 1 is a schematic diagram of recording and revisiting procedures according to the present application.
Detailed Description
The technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, but not all embodiments, and all other embodiments obtained by those skilled in the art without making creative efforts based on the embodiments of the present application are included in the protection scope of the present application.
Referring to fig. 1, the present application provides a technical solution: a desktop automation customization system, comprising:
the UI element detection module is used for automatically switching to traversal search logic when the original UIA cannot return the correct control element according to the screen coordinates, and stopping timely by using a greedy algorithm and a pruning algorithm;
the flow information generation module records key information of target control elements to provide basis for positioning the control during playback, wherein the key information comprises a control type and a control name;
the rule description configuration file contains four types of key information, namely special application identification rules, features of elements to be ignored, suspension conditions of element detection and element attributes required for generating a positioning sub (basis of a positioning control during playback);
the playback execution module encapsulates interfaces of various common actions, gives out a default implementation, and completes customization of the application through different implementations of the default implementation; here, a hot update may be supported, storing the dedicated link library in a designated directory, and automatically loading it to perform the required actions.
In the application, the UI element detection module is further dependent on one of pruning and suspension conditions when completing automatic switching to traversal search logic and realizing timely stopping of the process by using a greedy algorithm and a pruning algorithm, and is different in each application, such as filtering out nonsensical panels, uneditable blank text boxes and the like; by distributing the corresponding profile information, the feedback of the specific application can be well customized.
In the application, the input content of the UI element detection module is as follows: screen coordinates, configuration files; the output content of the UI element detection module is as follows: the innermost interface element on the coordinates.
In the application, the flow information generating module not only needs to obtain the path information of the control in a bottom-up mode, but also simulates a one-time playback process so as to ensure that the generated information can uniquely correspond to the target control; wherein the key information used is flexibly provided by the configuration file.
In the application, the input content of the flow information generating module is as follows: target control elements, configuration files; the output content of the flow information generation module is as follows: a target key information sequence.
In the application, the special application identification rule describes the processing flow needing to enable customization; for example, the process name of the target application is XXX and the window class name is XXX.
In the application, the feature of the element to be ignored describes the element to be pruned during UI detection; such as invisible elements or nonsensical container-like controls to avoid creating a strange highlighting effect at recording, reducing the user experience.
The application further provides that the suspension condition of the element detection describes that the search is suspended when the element with specific characteristics is encountered during UI detection; default to the first leaf element located at a given coordinate.
The application further provides the method, wherein the element attribute required by the generation of the positioning sub (the basis of the positioning control during playback) limits the positioning of the target element through the element attribute; not only the non-dynamic attribute is needed to be selected, but also the attribute with access error is avoided.
The application further provides that the available attributes in the element attributes include:
process Name: a process name;
class Name: class name;
automation Id: the unique identification of the control element defines the introduced control attribute in the UI Automation technology, and the initial aim is to locate for software Automation test;
name: a control name;
roller: a control type; such as List, check Box, button, etc.;
bounding Rectangle: control position and size information;
is Enabled: whether or not it is available;
is Offscreen: whether the control position is outside the screen;
access Key: and the control shortcut key is used for rapidly operating the control mode.
Examples of configuration files are as follows:
in the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, of the flowcharts and block diagrams in the figures that illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
It should be noted that in the description of the present specification, descriptions of terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics may be combined in any suitable manner in any one or more embodiments or examples, which are described as being relatively simple as being substantially similar to the method embodiments, as relevant in part to the description of the method embodiments. The system embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements illustrated as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present application without undue burden.
In addition, functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes. It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above description is only of the preferred embodiments of the present application and is not intended to limit the present application, but various modifications and variations can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application. It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (3)
1. A desktop automation customization system, comprising:
the UI element detection module is used for automatically switching to traversal search logic when the original UIA cannot return the correct control element according to the screen coordinates, and stopping timely by using a greedy algorithm and a pruning algorithm;
the flow information generation module records key information of target control elements to provide basis for positioning the control during playback, wherein the key information comprises a control type and a control name;
a rule description configuration file containing special application identification rules, features possessed by elements to be ignored, suspension conditions for element detection, and element attributes required for generating a locator;
the playback execution module encapsulates the interface of the action and gives a default implementation, and customization of the application is completed through different implementations of the default implementation;
the UI element detection module depends on one of pruning and suspension conditions when finishing automatic switching to traversal search logic and realizing timely stopping by using a greedy algorithm and a pruning algorithm;
the input content of the UI element detection module is as follows: screen coordinates, configuration files; the output content of the UI element detection module is as follows: an innermost interface element on the coordinates;
the flow information generation module not only needs to obtain path information of the control in a bottom-up mode, but also simulates a one-time playback process so as to ensure that the generated information can uniquely correspond to the target control; wherein the key information used is provided by a configuration file;
the input content of the flow information generation module is as follows: target control elements, configuration files; the output content of the flow information generation module is as follows: a target key information sequence;
the special application identification rule describes a processing flow requiring customization to be started;
the features of the elements to be ignored describe the elements which need to be pruned during UI detection;
the suspension condition of the element detection describes that the search is suspended when the element with specific characteristics is encountered during UI detection; default to the first leaf element located at a given coordinate.
2. A desktop automation customization system according to claim 1, wherein: the element attributes required for generating the locator define locating the target element through the attributes of the element; not only the non-dynamic attribute is needed to be selected, but also the attribute with access error is avoided.
3. A desktop automation customization system according to claim 2, wherein: the attributes available in the element attributes include:
ProcessName: a process name;
ClassName: class name;
AutomationId: a unique identification of the control element;
name: a control name;
roller: a control type;
BoundingRectangle: control position and size information;
IsEnabled: whether or not it is available;
IsOffscreen: whether the control position is outside the screen;
access Key: and the control shortcut key is used for rapidly operating the control mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110478696.9A CN113176921B (en) | 2021-04-30 | 2021-04-30 | Desktop automatic customizing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110478696.9A CN113176921B (en) | 2021-04-30 | 2021-04-30 | Desktop automatic customizing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113176921A CN113176921A (en) | 2021-07-27 |
| CN113176921B true CN113176921B (en) | 2023-09-19 |
Family
ID=76925593
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110478696.9A Active CN113176921B (en) | 2021-04-30 | 2021-04-30 | Desktop automatic customizing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113176921B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114090155B (en) * | 2021-11-17 | 2024-08-20 | 中国建设银行股份有限公司 | Robot flow automatic interface element positioning method, device and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106776319A (en) * | 2016-12-15 | 2017-05-31 | 广州酷狗计算机科技有限公司 | Automatic test approach and device |
| CN106886418A (en) * | 2017-03-27 | 2017-06-23 | 湛江市霞山区新软佳科技有限公司 | Html interface control is pasted |
| US9870114B1 (en) * | 2012-09-12 | 2018-01-16 | Google Llc | Media element navigation using a virtual jog wheel |
| CN107622008A (en) * | 2016-07-14 | 2018-01-23 | 阿里巴巴集团控股有限公司 | The traversal method and device of a kind of five application page |
| CN108196536A (en) * | 2017-12-21 | 2018-06-22 | 同济大学 | A kind of improved unmanned vehicle rapidly-exploring random tree paths planning method |
| US10078417B1 (en) * | 2016-07-29 | 2018-09-18 | Freedom Scientific, Inc. | View port array of graphic user interface components |
| CN108572913A (en) * | 2018-02-26 | 2018-09-25 | 挖财网络技术有限公司 | Method, apparatus and the application of App homepages are positioned in Android client UI automation processes |
| CN112231032A (en) * | 2019-12-10 | 2021-01-15 | 北京来也网络科技有限公司 | Software interface element access method and device combining RPA and AI |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8073869B2 (en) * | 2008-07-03 | 2011-12-06 | The Regents Of The University Of California | Method for efficiently supporting interactive, fuzzy search on structured data |
| US20110191676A1 (en) * | 2010-01-29 | 2011-08-04 | Microsoft Corporation | Cross-Browser Interactivity Recording, Playback, and Editing |
| US10817063B2 (en) * | 2019-02-11 | 2020-10-27 | Volvo Car Corporation | Facilitating interaction with a vehicle touchscreen using haptic feedback |
| US10970097B2 (en) * | 2019-06-19 | 2021-04-06 | Sap Se | Adaptive web-based robotic process automation |
| US11849196B2 (en) * | 2019-09-11 | 2023-12-19 | Educational Vision Technologies, Inc. | Automatic data extraction and conversion of video/images/sound information from a slide presentation into an editable notetaking resource with optional overlay of the presenter |
-
2021
- 2021-04-30 CN CN202110478696.9A patent/CN113176921B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9870114B1 (en) * | 2012-09-12 | 2018-01-16 | Google Llc | Media element navigation using a virtual jog wheel |
| CN107622008A (en) * | 2016-07-14 | 2018-01-23 | 阿里巴巴集团控股有限公司 | The traversal method and device of a kind of five application page |
| US10078417B1 (en) * | 2016-07-29 | 2018-09-18 | Freedom Scientific, Inc. | View port array of graphic user interface components |
| CN106776319A (en) * | 2016-12-15 | 2017-05-31 | 广州酷狗计算机科技有限公司 | Automatic test approach and device |
| CN106886418A (en) * | 2017-03-27 | 2017-06-23 | 湛江市霞山区新软佳科技有限公司 | Html interface control is pasted |
| CN108196536A (en) * | 2017-12-21 | 2018-06-22 | 同济大学 | A kind of improved unmanned vehicle rapidly-exploring random tree paths planning method |
| CN108572913A (en) * | 2018-02-26 | 2018-09-25 | 挖财网络技术有限公司 | Method, apparatus and the application of App homepages are positioned in Android client UI automation processes |
| CN112231032A (en) * | 2019-12-10 | 2021-01-15 | 北京来也网络科技有限公司 | Software interface element access method and device combining RPA and AI |
Non-Patent Citations (2)
| Title |
|---|
| 一种基于QoS剪枝的Top-k自动服务组合方法;杨汝涛;张绍谦;窦万春;;电子学报(第07期);第1489-1491页 * |
| 基于操作记录与回放技术的远程同步教学工具;陈德健;孙艳春;黄罡;张颖;;计算机工程与应用(第06期);第65-71页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113176921A (en) | 2021-07-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7778968B2 (en) | Systems and methods for compiling applications on a test server | |
| US20090265689A1 (en) | Generic validation test famework for graphical user interfaces | |
| US9223647B2 (en) | Automatic classification adjustment of recorded actions for automation script | |
| US20080015911A1 (en) | Methods and apparatuses for developing business solutions | |
| CN113176922A (en) | Universal automatic detection system for interface elements | |
| CN113176921B (en) | Desktop automatic customizing system | |
| MX2008003417A (en) | Declaratively defined control actions. | |
| CN113918198A (en) | Model-code synchronization method | |
| KR20150097181A (en) | A system for providing an application developing environment | |
| KR20150098962A (en) | A recording media recognized by computer for providing an application developing environment program | |
| Hillier | Programming SharePoint Services | |
| KR20150099176A (en) | A recording media for providing an application developing environment program | |
| KR20150098958A (en) | A recording media recognized by computer for providing an application developing environment program | |
| KR20150099352A (en) | A recording media specialized for an application developing environment program recording recognized by computer | |
| KR20150098989A (en) | A recording media recognized by computer for providing an application developing environment program | |
| KR20150099321A (en) | A recording media specialized for an application developing environment program recording recognized by computer | |
| KR20150099348A (en) | A recording media recognized by computer for providing an application developing environment program | |
| KR20150099274A (en) | A system for providing an application developing environment | |
| KR20150099258A (en) | A recording media recognized by computer for providing an application developing environment program | |
| KR20150098764A (en) | A recording media recognized by computer for providing an application developing environment program | |
| KR20150099001A (en) | A recording media specialized for an application developing environment program recording recognized by computer | |
| KR20150099177A (en) | A recording media recognized by computer for providing an application developing environment program | |
| KR20150099267A (en) | A recording media specialized for an application developing environment program recording recognized by computer | |
| KR20150099194A (en) | A recording media for providing an application developing environment program | |
| KR20150098810A (en) | A recording media recognized by computer for providing an application developing environment program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |