CN114090155A - Robot process automation interface element positioning method and device and storage medium - Google Patents

Robot process automation interface element positioning method and device and storage medium Download PDF

Info

Publication number
CN114090155A
CN114090155A CN202111364188.4A CN202111364188A CN114090155A CN 114090155 A CN114090155 A CN 114090155A CN 202111364188 A CN202111364188 A CN 202111364188A CN 114090155 A CN114090155 A CN 114090155A
Authority
CN
China
Prior art keywords
target
searched
control
list
target element
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111364188.4A
Other languages
Chinese (zh)
Other versions
CN114090155B (en
Inventor
徐立宇
孙歌睿
林震宇
林晨
陈佳雯
廖婉蓉
陈艺辉
王金哲
胡雪惠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Bank Corp
Original Assignee
China Construction Bank Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp filed Critical China Construction Bank Corp
Priority to CN202111364188.4A priority Critical patent/CN114090155B/en
Publication of CN114090155A publication Critical patent/CN114090155A/en
Application granted granted Critical
Publication of CN114090155B publication Critical patent/CN114090155B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The embodiment of the invention relates to the technical field of artificial intelligence, and discloses a robot process automation interface element positioning method, a device and a storage medium. The description character string of the target element is obtained by describing the target element according to the general set description mode for each interface interaction technology, so that when the target element is positioned, the interface interaction technology corresponding to the control tree to which the target element belongs can be determined according to the label of the interface interaction technology corresponding to the control tree to which the target element belongs, and when the target element is positioned, the target element can be quickly positioned according to the attribute value of the element attribute in the description path.

Description

Robot process automation interface element positioning method and device and storage medium
Technical Field
The embodiment of the invention relates to the technical field of artificial intelligence, in particular to a method and a device for positioning an interface element in robot process automation and a storage medium.
Background
Robot Process Automation (RPA), also known as digital labor, can use and understand existing applications of an enterprise through a user interface, automate regular operations based on rules, and replace a human executing office processes with high regularity and repeatability in front of a computer.
The RPA completes the automation of the office process by simulating the operation of natural people on a user interface, and when the natural people operate software, the natural people generally identify elements on the interface through vision and operate equipment such as a mouse, a keyboard and the like to interact with the elements; however, when the RPA accesses the user interface, if the interface element is recognized by computer vision, a recognition error is easily caused due to a change in screen resolution, window size, and the like, and therefore the element is often searched and located by attributes such as a name and a type of the interface element.
However, because there are a plurality of mainstream user interface drawing technologies, the attribute definitions of the user interface elements are different, and the access modes are also incompatible, so that the RPA developer is required to select a corresponding element grabbing tool according to different user interface drawing technologies when developing the robot, thereby increasing the development difficulty of the robot and influencing the element positioning speed.
Disclosure of Invention
The embodiment of the invention provides a robot process automation interface element positioning method, a device and a storage medium, which are used for automatically identifying an interface element at the current mouse position in an interface described by at least two control trees and quickly positioning a target element according to a description character string in a set description mode.
In a first aspect, an embodiment of the present invention provides a method for positioning an interface element in robot process automation, including:
determining target elements according to the at least two control trees, wherein the target elements are elements with the smallest control area in the elements containing the current mouse coordinates in each control tree;
describing the target element according to a set description mode to obtain a description character string of the target element, wherein the description character string comprises a label of an interface interaction technology corresponding to a control tree to which the target element belongs and a description path from a tree root of the control tree to which the target element belongs to the target element;
and positioning the target element according to the description character string of the target element.
Optionally, determining the target element according to at least two control trees includes:
acquiring a current mouse coordinate corresponding to the current mouse position;
for each control tree, searching each layer of the control tree layer by layer from the control tree root, and determining the bottommost layer element containing the current mouse coordinate in the control tree;
and determining the element with the smallest control area in the bottommost layer elements containing the current mouse coordinates in each control tree as a target element.
Optionally, for each control tree, searching each layer of the control tree layer by layer from the control tree root, and determining the bottommost layer element containing the current mouse coordinate in the control tree, including:
for each control tree, adding a first layer of the control tree into a first list to be searched, and recording that an element to be selected is empty;
traversing the first list to be searched, acquiring a control containing the current mouse coordinate, and setting the control containing the current mouse coordinate as an element to be selected after the control containing the current mouse coordinate is acquired;
clearing a first list to be searched;
when the element to be selected contains the sub-element, adding the sub-element of the element to be selected into a first list to be searched, skipping to traverse the first list to be searched, acquiring a control containing the current mouse coordinate, and setting the control containing the current mouse coordinate as the element to be selected after acquiring the control containing the current mouse coordinate;
and when the element to be selected does not contain the sub-element, determining the element to be selected as the bottommost element containing the current mouse coordinate and finishing searching.
Optionally, the robot process automation interface element positioning method further includes: and after traversing the first to-be-searched list, determining the element to be selected as the bottommost element containing the current mouse coordinate when the control containing the current mouse coordinate does not exist in the first to-be-searched list, and finishing searching.
Optionally, determining, as a target element, an element with the smallest control area in the bottommost element containing the current mouse coordinate in each control tree, including:
acquiring the size of a control corresponding to the bottommost element containing the current mouse coordinate in each control tree;
determining the area of the control corresponding to the bottommost layer element containing the current mouse coordinate according to the size information of the control corresponding to the bottommost layer element containing the current mouse coordinate in the control tree;
and taking the element with the smallest control area in the bottommost layer elements containing the current mouse coordinate in each control tree as a target element.
Optionally, describing the target element according to a set description mode to obtain a description character string of the target element, including:
recording a control tree root of a control tree to which a target element belongs and the target element in a form of at least attribute name-attribute value pairs to obtain a description path of the target element, wherein adjacent elements in the description path of the target element are divided by adopting a first divider;
and dividing the description path of the label and the target element by the second divider to obtain the description character string of the target element.
Optionally, at least recording a control tree root of a control tree to which the target element belongs and the target element in the form of an attribute name-attribute value pair to obtain a description path of the target element, where the description path includes:
and respectively recording the control tree root of the control tree to which the target element belongs to each layer of element of the target element in the form of attribute name-attribute value pairs to obtain the description path of the target element.
Optionally, the labels of different interface interaction technologies are different.
Optionally, when the interface interaction technologies corresponding to the control tree to which the target element belongs are different, the value ranges of the attribute names in the attribute name-attribute value pairs are not completely the same.
Optionally, locating the target element according to the description character string of the target element includes:
segmenting the description character string according to the second segmentation character, and determining a second list to be searched according to the obtained label corresponding to the interface interaction technology;
dividing the description path of the target element according to the first divider to obtain an element set consisting of all elements in the description path;
and determining the first element to the last element in the element set as search elements one by one, comparing the search elements with the elements in the second list to be searched one by one as comparison elements when each element in the element set is determined as a search element, and updating the second list to be searched according to the sub-elements of the comparison elements after comparison to locate the target element.
Optionally, the segmenting the description character string according to the second segmenter, and determining the second list to be searched according to the obtained label corresponding to the interface interaction technology, includes:
dividing the description character string according to the second divider to obtain a label corresponding to the interface interaction technology, and adding a first-layer element of a control tree corresponding to the label into a second list to be searched;
determining a first element to a last element in the element set as search elements one by one, comparing the search elements with corresponding elements in a second list to be searched when each element in the element set is determined as a search element, and updating the second list to be searched according to sub-elements of the comparison elements after comparison to locate a target element, wherein the method comprises the following steps:
when i is not more than n, for the ith element, taking out the first element of the second list to be searched as a comparison element when the second list to be searched is not empty; wherein i is more than or equal to 1 and less than or equal to n, and n represents the total number of elements in the element set; after the first element is taken out from the second list to be searched, the second list to be searched does not comprise the element any more;
comparing each attribute name-attribute value pair in the ith element with a comparison element;
when the attribute name-attribute value pair corresponds to the attribute value of the comparison element, emptying a second to-be-searched list, adding all sub-elements of the comparison element into the second to-be-searched list, and setting the comparison element as a search result of the current round;
adding 1 to i and jumping to the situation that when i is not more than n, for the ith element, when the second list to be searched is not empty, taking out the first element of the second list to be searched as a comparison element;
and positioning the search result corresponding to the nth element as a target element.
Optionally, when all attribute name-attribute value pairs in the ith element do not correspond to the attribute values of the comparison elements and the comparison elements include sub-elements, adding all the sub-elements of the comparison elements to the second list to be searched, and jumping to a step of taking the first element of the second list to be searched as the comparison element when i is not more than n for the ith element and the second list to be searched is not empty.
Optionally, when all attribute name-attribute value pairs in the ith element do not correspond to the attribute values of the comparison element and the comparison element does not include a sub-element, jumping to a step of taking the first element of the second list to be searched as the comparison element when i is not greater than n for the ith element when the second list to be searched is not empty.
In a second aspect, an embodiment of the present invention further provides a robot process automation interface element positioning apparatus, including:
the target element determining module is used for determining target elements according to at least two boundary control trees, wherein the target elements are elements with the smallest control area in the elements containing the current mouse coordinates in each control tree;
the description module is used for describing the target element according to a set description mode to obtain a description character string of the target element, wherein the description character string comprises a label of an interface interaction technology corresponding to a control tree to which the target element belongs and a description path from a tree root of the control tree to which the target element belongs to the target element;
and the positioning module is used for positioning the target element according to the description character string of the target element.
In a third aspect, an embodiment of the present invention further provides an apparatus, including:
one or more processors;
storage means for storing one or more programs;
when executed by one or more processors, cause the one or more processors to implement the above-described robotic process automation interface element positioning method.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the above-mentioned method for positioning an interface element in robot process automation.
In a fifth aspect, embodiments of the present invention provide a computer program product comprising computer-executable instructions for implementing the method of the first aspect when executed.
The embodiment of the invention provides a robot process automation interface element positioning method, a device and a storage medium, wherein a target element is determined according to at least two control trees, and the target element is determined to be the element with the smallest control area in the elements containing current mouse coordinates in each control tree, so that no matter what kind of known interface drawing technology is adopted for drawing an interface, the method can automatically identify the element containing the current mouse coordinates from the control trees corresponding to the interface interaction technologies corresponding to the known interface drawing technologies, and the situation that the target element cannot be grasped can be avoided. The description character string of the target element is obtained by describing the target element according to the general set description mode for each interface interaction technology, so that when the target element is positioned according to the description character string of the target element, the interface interaction technology corresponding to the control tree to which the target element belongs can be determined according to the label of the interface interaction technology corresponding to the control tree to which the target element belongs, the attribute of the interface element is described by the interface interaction technology, and when the target element is positioned, the target element can be quickly positioned according to the attribute value of the element attribute in the description path.
Drawings
Fig. 1 is a flowchart of a method for positioning an interface element in robot process automation according to an embodiment of the present invention;
fig. 2 is a flowchart of another method for positioning elements in a robot process automation interface according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating an example of a user interface provided by an embodiment of the invention;
FIG. 4 is a flow chart of another method for robot process automation interface element positioning provided by an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a robot process automation interface element positioning apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an apparatus according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Fig. 1 is a flowchart of a method for positioning a robot process automation interface element according to an embodiment of the present invention, where the embodiment is applicable to a situation where positioning is performed based on different robot process automation interface elements, and the method may be executed by a robot process automation interface element positioning apparatus, and the apparatus may be implemented in a software and/or hardware manner, and the apparatus may be configured in an electronic device, such as a server or a terminal device, where a typical terminal device includes a mobile terminal, and specifically includes a mobile phone, a computer, or a tablet computer. As shown in fig. 1, the method may specifically include:
step 110, determining a target element according to at least two control trees;
and the target element is the element with the smallest control area in the elements containing the current mouse coordinate in each control tree.
Wherein the control tree corresponds to the interface interaction technology. An interface interaction technology may correspond to one control tree or at least two control trees, and this embodiment is not specifically limited herein. That is, at least two control trees in this step may correspond to one interface interaction technology, or may correspond to at least two interface interaction technologies.
Specifically, taking the currently mainstream operating system Windows as an example, the mainstream user interface interaction technology includes: UIAutomation accessing common Windows forms, JavaAccess bridge accessing Java program forms, mshtml accessing IE web pages and the like. Because the drawing logics of the user interfaces are similar, the mainstream interface interaction technology describes the user interfaces in a tree structure-control tree mode. For example, in a user form a, there is a table T in the form, where the table T has two rows of contents, respectively R1 and R2, and each row includes a button B, the control tree is as follows:
Figure BDA0003360296510000081
Figure BDA0003360296510000091
if a plurality of forms drawn by different technologies exist on a screen at the same time, a plurality of control trees exist at the same time, each element only appears on a control tree corresponding to an interface interaction technology, in the prior art, when an interface element is positioned, the interface element is generally searched and positioned on the corresponding control tree through the own attribute of the element, the attributes describing the interface element of different interface interaction technologies are also different, for example, UIAutomation has attributes such as Automation ID, Name, ClassName, ControlType and the like, and the attribute of JavaAccessBridge is Role, Name, States and the like. Since the elements only appear on the corresponding control tree, if the elements do not correspond to the access technology, the situation that the elements cannot be found occurs in any method.
Unlike the prior art, in step 110, the target element is determined according to at least two control trees, optionally, the target element is determined according to the control trees corresponding to the known interface interaction technologies, specifically, the target element is determined according to the control area of the element containing the current mouse coordinate in the control tree corresponding to each known interface interaction technology. The target elements are elements with the smallest control area in the control tree corresponding to each interface interaction technology and the elements containing the current mouse coordinates. That is, in this step 110, an element with the smallest control area among elements including current mouse coordinates in each control tree is determined as a target element, so that no matter what known interface drawing technology is used for drawing the interface, the method of this embodiment can automatically identify the element including the current mouse coordinates from the control tree corresponding to the interface interaction technology corresponding to each known interface drawing technology, and a situation that the target element cannot be grasped cannot occur.
Step 120, describing the target element according to a set description mode to obtain a description character string of the target element; the description character string comprises a label of an interface interaction technology corresponding to the control tree to which the target element belongs and a description path from the tree root of the control tree to the target element to which the target element belongs.
The setting description mode may be a description mode common to each interface interaction technology. The description character string of the target element obtained by describing the target element in the set description mode comprises a label of the interface interaction technology corresponding to the control tree to which the target element belongs, wherein the label of the interface interaction technology corresponding to the control tree to which the target element belongs is used for identifying the interface interaction technology corresponding to the control tree to which the target element belongs. The description character string of the target element obtained by describing the target element in the set description mode also comprises a description path from the tree root of the control tree to which the target element belongs to the target element, the description path is used for describing the position of the target element in the control tree, and the target element can be found from the control tree through the description path.
And step 130, positioning the target element according to the description character string of the target element.
Specifically, the description character string of the target element includes a label of the interface interaction technology corresponding to the control tree to which the target element belongs, and in this step, the interface interaction technology corresponding to the control tree to which the target element belongs may be determined according to the label. The description character string of the target element also comprises a description path from the root of the control tree to which the target element belongs to the target element, so that after the interface interaction technology corresponding to the control tree to which the target element belongs is determined, the attribute of the interface element described by the interface interaction technology can also be determined, and when the target element is positioned, the target element can be quickly positioned according to the attribute value of the element attribute in the description path, and a user does not need to deeply know the interface interaction technologies in the whole process, so that the development and use threshold of the robot is reduced, and the development efficiency is improved.
According to the robot flow automation interface element positioning method provided by the embodiment, the target element is determined according to at least two control trees, and the target element is determined as the element with the smallest control area in the elements containing the current mouse coordinates in each control tree, so that no matter what known interface drawing technology is adopted for drawing the interface, the method of the embodiment can automatically identify the element containing the current mouse coordinates from the control trees corresponding to the interface interaction technologies corresponding to the known interface drawing technologies, and the situation that the target element cannot be grasped can be avoided. The description character string of the target element is obtained by describing the target element according to the general set description mode for each interface interaction technology, so that when the target element is positioned according to the description character string of the target element, the interface interaction technology corresponding to the control tree to which the target element belongs can be determined according to the label of the interface interaction technology corresponding to the control tree to which the target element belongs, the attribute of the interface element is described by the interface interaction technology, and when the target element is positioned, the target element can be quickly positioned according to the attribute value of the element attribute in the description path.
In the following embodiments, an interface interaction technology corresponding to a control tree is taken as an example for explanation.
Fig. 2 is a flowchart of another robot process automation interface element positioning method provided in an embodiment of the present invention, and with reference to fig. 2, optionally, the robot process automation interface element positioning method includes:
and step 210, obtaining a current mouse coordinate corresponding to the current mouse position.
Specifically, the current mouse coordinate may be a two-dimensional coordinate value of the current mouse position on the screen, optionally, a certain fixed point of the screen is selected as an origin of coordinates, exemplarily, an upper left corner of the screen is selected as the origin of coordinates, and after the origin of coordinates is determined, the current mouse coordinate corresponding to the current mouse position may be determined.
And step 220, for each control tree, searching each layer of the control tree layer by layer from the control tree root, and determining the bottommost layer element containing the current mouse coordinate in the control tree.
In particular, for a control tree, the control area of the lower-level elements is generally smaller. In this step, by searching each layer of the control tree layer by layer from the control tree root to the control tree, determining the bottommost layer element containing the current mouse coordinate in the control tree, and further determining the element containing the minimum control area of the current mouse coordinate in the control tree.
Optionally, in step 220, the control tree may include the following steps:
(1) and adding the first layer of the control tree into the first list to be searched, and recording that the element to be selected is empty.
(2) Traversing the first list to be searched, acquiring a control containing the current mouse coordinate, and setting the control containing the current mouse coordinate as an element to be selected after the control containing the current mouse coordinate is acquired;
(3) clearing a first list to be searched;
(4) when the element to be selected contains the sub-element, adding the sub-element of the element to be selected into a first list to be searched, skipping to traverse the first list to be searched, acquiring a control containing the current mouse coordinate, and setting the control containing the current mouse coordinate as the element to be selected after acquiring the control containing the current mouse coordinate;
(5) and when the element to be selected does not contain the sub-element, determining the element to be selected as the bottommost element containing the current mouse coordinate and finishing searching.
The step (2) in the step 220 further includes: and after traversing the first to-be-searched list, determining the element to be selected as the bottommost element containing the current mouse coordinate when the control containing the current mouse coordinate does not exist in the first to-be-searched list, and finishing searching.
Fig. 3 is a schematic diagram of an example of a user interface provided by an embodiment of the present invention. Referring to fig. 3, the on-screen user interface in this example includes Windows W1, Java frame J1, web page frame I1, with the current mouse position S1 in Windows frame W1 and Java frame J1.
For the interface interaction technology UIAutomation corresponding to the Windows form W1, if the first layer of the control tree is W1, the step 220 includes the steps of:
adding W1 into the first list to be searched, and recording the element to be selected as null (corresponding to the step (1) of the step 220);
acquiring that the control containing the current mouse coordinate is W1, and setting W1 as an element to be selected (corresponding to the step (2) of the step 220);
emptying the first list to be searched (corresponding to the step (3) of the step 220);
no child element is contained in W1, and W1 is determined as the bottom-most element containing the current mouse coordinates as the end lookup (corresponding to step (5) of step 220).
For the interface interaction technology JavaAccessBridge corresponding to the Java form J1, if the first layer of the control tree is J1, the step 220 includes the steps of:
adding J1 into the first list to be searched, and recording the element to be selected as null (corresponding to the step (1) of the step 220);
traversing the first list to be searched, acquiring a control J1 containing the current mouse coordinate, and setting a control J1 as an element to be selected (corresponding to the step (2) of the step 220);
emptying the first list to be searched (corresponding to step (3) of step 220);
j1 includes two sub-elements, namely a first sub-element B1 and a second sub-element B2, and adds the first sub-element B1 and the second sub-element B2 to the first list to be searched (corresponding to step (4) of step 220);
jump to step (2) of step 220:
traversing the first list to be searched, acquiring a control containing the current mouse coordinate, namely a second sub-element B2, and setting a second sub-element B2 as an element to be selected (corresponding to the step (2) of the step 220);
emptying the first list to be searched (corresponding to step (3) of step 220);
b2 includes a sub-element, i.e., a third sub-element X, which is added to the first list to be searched (corresponding to the (4) step of step 220);
jump to step (2) of step 220:
traversing the first list to be searched to obtain the control containing the current mouse coordinate, determining the element B2 to be selected as the bottommost element containing the current mouse coordinate because the element containing the current mouse coordinate no longer exists in the first list to be searched, and ending the search (corresponding to the step (2) of the step 220).
For interface interaction technology mshtml corresponding to web page form I1, if the first level of the control tree is I1, step 220 includes the steps of:
adding I1 into the first list to be searched, and recording the element to be selected as null (corresponding to the step (1) of the step 220);
and traversing the first to-be-searched list, acquiring a control containing the current mouse coordinate, determining the element to be selected (which is empty) as the bottommost element containing the current mouse coordinate and finishing searching (corresponding to the step (2) of the step 220) because the element containing the current mouse coordinate does not exist in the first to-be-searched list.
It should be noted that, the above example is only given for traversing the control trees of the interface interaction technologies corresponding to different forms, and for the known interface interaction technologies, when no corresponding form exists on the screen (the corresponding control tree is empty), step 220 is also executed, and when no form corresponding to some interface interaction technology exists on the screen, the acquired bottommost layer element including the current mouse coordinate in the control tree corresponding to the interface interaction technology is empty.
And step 230, determining the element with the smallest control area in the bottommost layer elements containing the current mouse coordinates in each control tree as a target element.
Specifically, in step 220, after determining the bottommost layer element containing the current mouse coordinate in each control tree, comparing the control areas of the bottommost layer element containing the current mouse coordinate in each control tree, and determining the element of the control with the smallest area as the target element.
Optionally, step 230 includes the following steps:
(1) and acquiring the size information of the control corresponding to the bottommost element containing the current mouse coordinate in each control tree.
Specifically, the size information of the control is related to the shape of the control, optionally, when the shape of the control is a rectangle, the size information of the control includes length information and width information of the control, the length information and the width information of the control can be obtained by obtaining vertex coordinates of the control of the bottommost element, for example, four vertex coordinates of the rectangle can be obtained, and then the length information and the width information of the space can be obtained according to the four vertex coordinates. Optionally, when the shape of the control is a circle, the size information of the control includes radius information or diameter information of the control.
(2) And determining the area of the control corresponding to the bottommost layer element containing the current mouse coordinate according to the size information of the control corresponding to the bottommost layer element containing the current mouse coordinate in the control tree.
After the length and size information of the control is determined, the area of the control can be obtained according to the size information, for example, when the control is a rectangle, the area of the control is the length multiplied by the width.
(3) And taking the element with the smallest control area in the bottommost layer elements containing the current mouse coordinate in each control tree as a target element.
The element with the smallest control area in the bottommost layer element containing the current mouse coordinate in each control tree is used as a target element, so that the element of the interface where the current mouse coordinate is located can be accurately positioned, and the condition that the target element cannot be grasped in the prior art is avoided.
Step 240, describing the target element according to a set description mode to obtain a description character string of the target element, wherein the description character string comprises a label of an interface interaction technology corresponding to a control tree to which the target element belongs and a description path from a tree root of the control tree to which the target element belongs to the target element; this step is the same as step 120 in the above embodiments, and is not described herein again.
Step 250, positioning the target element according to the description character string of the target element; this step is the same as step 130 in the above embodiment, and is not described herein again.
The method for positioning the robot process automation interface element of the embodiment obtains the current mouse coordinate corresponding to the current mouse position, for each control tree, searching each layer of the control tree layer by layer from the control tree root, determining the bottommost layer element containing the current mouse coordinate in the control tree, and adding the current mouse coordinate to the control tree, the element with the smallest control area in the bottommost element containing the current mouse coordinate is determined as the target element, the bottommost element containing the current mouse coordinates can be found from each control tree and the control areas can be compared, therefore, no matter what known interface drawing technology is adopted for drawing the interface, the method of the embodiment can automatically identify the element containing the current mouse coordinate from the control tree corresponding to the interface interaction technology corresponding to each known interface drawing technology, and the situation that the target element cannot be grasped can not occur.
Fig. 4 is a flowchart of another robot process automation interface element positioning method according to an embodiment of the present invention, and referring to fig. 4, the robot process automation interface element positioning method includes:
step 310, determining a target element according to at least two control trees;
the target elements are elements with the smallest control area in the elements containing the current mouse coordinates in each control tree; this step is the same as step 110 in the above embodiment, and is not described again here.
Step 320, recording a control tree root of a control tree to which the target element belongs and the target element at least in the form of an attribute name-attribute value pair to obtain a description path of the target element;
wherein adjacent elements in the description path of the target element are partitioned by a first partition character.
The attribute name-attribute value pair is a key-value pair, wherein key is the attribute name and value is the attribute value.
Optionally, the step 320 includes: and respectively recording the control tree root of the control tree to which the target element belongs to each layer of element of the target element in the form of attribute name-attribute value pairs to obtain the description path of the target element.
Take the user interface shown in fig. 3 as an example. After the target element (B2) is determined, recording the control tree root of the control tree to which the target element belongs to each layer of elements of the target element in the form of attribute name-attribute value pairs respectively, wherein the control tree root (namely the first layer of the control tree) of the control tree to which the target element belongs is J1, and J1 has two attributes of name and id, which can be described as { "name": J1"," id ": J1" }; the second layer of the control tree to which the target element belongs is B1 and B2, and the target element is in the B2 element of the second layer, so that B2 is described in the description path, and B2 only has one attribute of name, and can be described as { "name": B2 "}. The description path of the target element can be represented as { "name": J1"," id ": J1" } - > { "name": B2"}, assuming that the first delimiter can be described as" - >.
It should be noted that the above only takes the user interface shown in fig. 3 as an example to describe the path by the target element. To more clearly illustrate the description manner of the description path of the target element, an example of the description path of the target element (X) is given below by taking the current mouse position within the X element of B2 as an example.
Wherein J1 has two attributes of name and id, which can be described as { "name": J1"," id ": J1" };
b2 has only one attribute of name, and can be described as { "name": B2 "};
x has name and value attributes and can be described as { "name": X "," value ": text value" };
then, the description path of the target element (X) obtained by recording the description path of the target element from the control tree root of the control tree to which the target element belongs to each layer element of the target element in the form of the attribute name-attribute value pair may be represented as:
{ "name": J1"," id ": J1" } - > { "name": "B2" } - > { "name": X "," value ": text value" }.
The description path of the target element obtained by obtaining the description path of the target element from the control tree root of the control tree to which the target element belongs to each layer of element of the target element is recorded in the form of attribute name-attribute value pairs, so that the search range is smaller when the target element is subsequently positioned according to the description path of the target element, and the positioning speed is further improved.
In another alternative embodiment of the present invention, the description of the control tree layer between the control tree root and the target element may be omitted, for example, when the target element is X, B2 may be omitted, and the description path of the target element (X) may be represented as:
{ "name": J1"," id ": J1" } - > { "name": X "," value ": text value" }.
The description of the control tree layer between the control tree root and the target element is omitted, and wrong guidance caused by low element attribute specificity in the control tree layer between the control tree root and the target element can be eliminated.
According to the description path of the target element, in the description path of the target element, the path from the control tree root to the target element is recorded in a mode of 'parent element- > descendant element of descendant element'.
When the interface interaction technologies corresponding to the control tree to which the target element belongs are different, the value ranges of the attribute names in the attribute name-attribute value pairs are not completely the same. Because the element attributes of different interface interaction technologies are different, the value ranges of the attribute names are different.
And step 330, dividing the description path of the label and the target element by using the second divider to obtain the description character string of the target element.
Optionally, the labels of different interface interaction technologies are different. A unique tag can be set for each interface access technology, tags are not repeated among different access technologies, for example, the tag of the UIAutomation interface interaction technology is Winauto; the label of the JavaAccessbridge interface interaction technology is JAB; the tag of the mshtml interface interaction technology is mHtml.
Illustratively, the second delimiter is "═ then the description string of the target element (B2) corresponding to the user interface shown in fig. 3 is JAB {" name ": J1", "id": J1"} - > {" name ": B2".
And 340, segmenting the description character string according to the second segmentation character, and determining a second list to be searched according to the obtained label corresponding to the interface interaction technology.
Specifically, in this step, the interface interaction technology corresponding to the label can be determined according to the label of the interface interaction technology. And determining a second list to be searched according to the obtained label corresponding to the interface interaction technology, optionally, segmenting the description character string according to the second segmentation symbol to obtain a label corresponding to the interface interaction technology, and adding the first-layer element of the control tree corresponding to the label to the second list to be searched. Still taking the user interface shown in fig. 3 as an example, the description string of the target element is obtained in step 330, i.e., JAB { "name": "J1", "id": "J1" } - > { "name": "B2" }. And according to the second segmentation character string, obtaining a label corresponding to the interface interaction technology as JAB, and adding a first-layer element J1 of the space tree corresponding to the label into the second list to be searched.
And 350, dividing the description path of the target element according to the first divider to obtain an element set consisting of the elements in the description path.
Still taking the user interface shown in fig. 3 as an example, the description string of the target element is obtained in step 330, that is, JAB { "name": i.e., "J1", "id": J1"} - > {" name ":" B2"}, and the description path of the target element is {" name ":" J1"," id ":" J1"} - > {" name ": B2" }. And segmenting the description path of the target element according to the first segmenter to obtain an element set formed by elements in the description path as ({ "name": J1"," id ": J1" }, { "name": B2 "}).
And step 360, determining the first element to the last element in the element set as search elements one by one, comparing the search elements with the elements in the second list to be searched one by one as comparison elements when each element in the element set is determined as a search element, and updating the second list to be searched according to the sub-elements of the comparison elements after comparison to locate the target element.
Wherein, in two adjacent elements in the element set, the latter element is the descendant element of the former element. Still taking the user interface shown in fig. 3 as an example, segmenting the description path of the target element according to the first segmenter results in that the element set composed of the elements in the description path is ({ "name": J1"," id ": J1" }, { "name": B2"}), i.e., the first element in the element set is {" name ": J1", "id": J1"}, and the second element is {" name ": B2" }. Determining a first element to a last element in the element set as search elements one by one, comparing the search elements with elements in a second list to be searched one by one as comparison elements when each element in the element set is determined as a search element, and updating the second list to be searched according to sub-elements of the comparison elements after comparison, wherein when the first element in the element set is determined as a search element, the second list to be searched is a first-layer element of a control tree, and when the second element in the element set is determined as a search element, the second list to be searched is a second-layer element of the control tree.
Optionally, the step 360 includes:
(a) when i is not more than n, for the ith element, taking out the first element of the second list to be searched as a comparison element when the second list to be searched is not empty; wherein i is more than or equal to 1 and less than or equal to n, and n represents the total number of elements in the element set; after the first element is taken out from the second list to be searched, the second list to be searched does not comprise the element any more.
(b) Comparing each attribute name-attribute value pair in the ith element with a comparison element;
(c) when the attribute name-attribute value pair corresponds to the attribute value of the comparison element, emptying a second to-be-searched list, adding all sub-elements of the comparison element into the second to-be-searched list, and setting the comparison element as a search result of the current round;
(d) adding 1 to i and jumping to the situation that when i is not more than n, for the ith element, when the second list to be searched is not empty, taking out the first element of the second list to be searched as a comparison element;
(e) and positioning the search result corresponding to the nth element as a target element.
Optionally, step (c) of step 360 further includes:
when all attribute name-attribute value pairs in the ith element do not correspond to the attribute values of the comparison elements and the comparison elements comprise sub-elements, adding all the sub-elements of the comparison elements into a second list to be searched, and jumping to a step of taking out a first element of the second list to be searched as the comparison element when i is not more than n for the ith element and the second list to be searched is not empty;
and when all attribute name-attribute value pairs in the ith element do not correspond to the attribute values of the comparison element and the comparison element does not comprise sub-elements, jumping to a step of taking the first element of the second list to be searched as the comparison element when i is not more than n and the second list to be searched is not empty for the ith element.
Still taking the user interface shown in fig. 3 as an example, the description path of the target element is segmented according to the first segmenter to obtain a set of elements consisting of elements in the description path as ({ "name": J1"," id ": J1" }, { "name": B2"}), that is, the first element in the set of elements is {" name ": J1", "id": J1"}, the second element is {" name ": B2" }, and the total number n of elements in the set of elements is 2. Then according to the steps included in step 360 above, the process of locating the target element is as follows:
for the 1 st element { "name": J1"," id ": J1" }, in the second to-be-searched list (according to step 340, the second to-be-searched list is J1), the second to-be-searched list is not empty, and the first element J1 of the second to-be-searched list is taken out as an alignment element; after the first element is taken out from the second list to be searched, the second list to be searched does not include the element (corresponding to the step (a) of the step 360);
aligning each attribute name-attribute value pair of the 1 st element { "name": J1"," id ": J1" } with the alignment element J1 (corresponding to step (b) of step 360);
attribute name-attribute value pairs { "name": J1"," id ": J1" } correspond to the attribute value of the alignment element J1, empty the second to-be-searched list, add all the sub-elements (the first sub-element B1 and the second sub-element B2) of the alignment element to the second to-be-searched list, and set the alignment element J1 as the search result of the current round (corresponding to the (c) step of step 360);
adding 1 to 1, jumping to a step of taking out the first element of the second list to be searched as a comparison element when the second list to be searched is not empty for the ith element when i is not more than n (corresponding to the step (d) of the step 360);
then continue to execute
For the 2 nd element { "name": B2"}, in the second to-be-searched list (according to step 350, the second to-be-searched list is B1 and B2), the second to-be-searched list is not empty, and the first element B1 of the second to-be-searched list is taken as the alignment element; after the first element is taken out from the second list to be searched, the second list to be searched no longer includes the element, and at this time, the second list to be searched only includes B2 (corresponding to step (a) of step 360);
aligning each attribute name-attribute value pair of the 2 nd element { "name": "B2" } with the alignment element B1 (corresponding to step (B) of step 360);
if the attribute name-attribute value pair { "name": B2"} does not correspond to the attribute value of the alignment element B1 and the alignment element B1 does not include a sub-element, jumping to a step of, for the ith element, taking the first element of the second list to be searched as the alignment element when i is not greater than n and the second list to be searched is not empty (step (c) corresponding to step 360);
for the 2 nd element { "name": "B2" }, the second list to be searched (only including B2) is not empty, and the first element B2 of the second list to be searched is taken out as an alignment element; after the first element is taken out from the second list to be searched, the second list to be searched does not include the element (corresponding to (a) of step 360);
aligning each attribute name-attribute value pair of the 2 nd element { "name": "B2" } with the alignment element B2 (corresponding to step (B) of step 360);
the attribute name-attribute value pair { "name": B2"} alignment element B2 corresponds to the attribute value, the second to-be-searched list is emptied, all sub-elements (empty) of the alignment element are added to the second to-be-searched list, and the alignment element B2 is set as the search result of the current round (corresponding to step (c) of step 360);
adding 1 to 2, jumping to the situation that when i is not more than n, for the ith element, when the second list to be searched is not empty, taking out the first element of the second list to be searched as a comparison element; since 2+1 is 3 and is greater than the total number of elements included in the element set by 2, the condition that i is less than or equal to n is not satisfied, and therefore, the step of taking out the first element of the second list to be searched as the comparison element when the second list to be searched is not empty is no longer performed for the ith element (step (d) corresponding to step 360).
The search result B2 corresponding to the 2 nd element is located as the target element (corresponding to the (e) step of step 360).
According to the robot process automation interface element positioning method, the description path of the target element is obtained by recording the control tree root of the control tree to which the target element belongs and the target element at least in the form of attribute name-attribute value pairs, adjacent elements in the description path of the target element are divided by the first divider, and the description character string of the target element is obtained by dividing the label and the description path of the target element by the second divider, so that elements in any interface can be described in a uniform and general manner. And then segmenting the description character string according to a second segmentation symbol, determining a second list to be searched according to a label corresponding to the obtained interface interaction technology, segmenting a description path of a target element according to the first segmentation symbol to obtain an element set consisting of elements in the description path, determining a first element to a last element in the element set as search elements one by one, comparing the search elements with elements in the second list to be searched one by one as comparison elements when each element in the element set is determined as a search element, updating the second list to be searched according to sub-elements of the comparison elements after comparison, and rapidly positioning the target element according to a uniform description mode of the target element to improve positioning speed.
An embodiment of the present invention further provides a robot flow automation interface element positioning apparatus, and fig. 5 is a schematic structural diagram of the robot flow automation interface element positioning apparatus provided in the embodiment of the present invention. The robot flow automation interface element positioning device provided by the embodiment of the invention can execute the robot flow automation interface element positioning method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
The device specifically comprises a target element determining module 410, configured to determine a target element according to at least two control trees, where the target element is an element with a smallest control area among elements including current mouse coordinates in each control tree;
the description module 420 is configured to describe the target element in a set description manner to obtain a description character string of the target element, where the description character string includes a label of an interface interaction technology corresponding to the control tree to which the target element belongs and a description path from a tree root of the control tree to which the target element belongs to the target element;
and the positioning module 430 is configured to position the target element according to the description character string of the target element.
The robot process automation interface element positioning device provided by the embodiment of the invention can execute the robot process automation interface element positioning method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Fig. 6 is a schematic structural diagram of an apparatus according to an embodiment of the present invention. FIG. 6 illustrates a block diagram of an exemplary device suitable for use to implement embodiments of the present invention. The device 41 shown in fig. 6 is only an example and should not bring any limitation to the function and the range of use of the embodiment of the present invention.
As shown in fig. 6, the device 41 is represented in the form of a general-purpose device. The components of device 41 may include, but are not limited to: one or more processors 416, a storage device 428, and a bus 418 that couples the various system components including the storage device 428 and the processors 416.
Bus 418 represents one or more of any of several types of bus structures, including a memory device bus or memory device controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
Device 41 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by device 41 and includes both volatile and nonvolatile media, removable and non-removable media.
Storage 428 may include computer system readable media in the form of volatile Memory, such as Random Access Memory (RAM) 431 and/or cache Memory 432. The device 41 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 434 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 6, commonly referred to as a "hard drive"). Although not shown in FIG. 6, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk such as a Compact disk Read-Only Memory (CD-ROM), Digital Video disk Read-Only Memory (DVD-ROM), or other optical media may be provided. In these cases, each drive may be connected to bus 418 by one or more data media interfaces. Storage 428 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 441 having a set (at least one) of program modules 442 may be stored, for instance, in storage 428, such program modules 442 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may include an implementation of a network environment. The program modules 442 generally perform the functions and/or methodologies of the described embodiments of the invention.
The device 41 may also communicate with one or more external devices 414 (e.g., keyboard, pointing terminal, display 424, etc.), with one or more terminals that enable a user to interact with the device 41, and/or with any terminals (e.g., network card, modem, etc.) that enable the device 41 to communicate with one or more other computing terminals. Such communication may occur via input/output (I/O) interfaces 422. Also, the device 41 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public Network such as the internet) through the Network adapter 421. As shown in FIG. 6, network adapter 421 communicates with the other modules of device 41 via bus 418. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with device 41, including but not limited to: microcode, end drives, Redundant processors, external disk drive Arrays, RAID (Redundant Arrays of Independent Disks) systems, tape drives, and data backup storage systems, among others.
The processor 416 executes various functional applications and data processing by running programs stored in the storage device 428, for example, implementing a robot process automation interface element positioning method provided by an embodiment of the present invention, the method including:
determining target elements according to the at least two control trees, wherein the target elements are elements with the smallest control area in the elements containing the current mouse coordinates in each control tree;
describing the target element according to a set description mode to obtain a description character string of the target element, wherein the description character string comprises a label of an interface interaction technology corresponding to a control tree to which the target element belongs and a description path from a tree root of the control tree to which the target element belongs to the target element;
and positioning the target element according to the description character string of the target element.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a method for positioning an interface element in robot process automation, where the method includes:
determining target elements according to the at least two control trees, wherein the target elements are the elements with the smallest control area in the elements containing the current mouse coordinate in each control tree;
describing the target element according to a set description mode to obtain a description character string of the target element, wherein the description character string comprises a label of an interface interaction technology corresponding to a control tree to which the target element belongs and a description path from a tree root of the control tree to which the target element belongs to the target element;
and positioning the target element according to the description character string of the target element.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or terminal. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
Embodiments of the present invention also provide a computer program product comprising computer executable instructions for performing the method of robot process automation interface element positioning in the above method embodiments when executed by a computer processor.
Of course, the computer program product provided in the embodiments of the present application has computer-executable instructions that are not limited to the method operations described above, and may also perform related operations in the method provided in any embodiments of the present application.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the particular embodiments described herein, and that various obvious changes, rearrangements and substitutions will now be apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (17)

1. A robot process automation interface element positioning method is characterized by comprising the following steps:
determining target elements according to at least two control trees, wherein the target elements are elements with the smallest control area in the elements containing the current mouse coordinate in each control tree;
describing the target element according to a set description mode to obtain a description character string of the target element, wherein the description character string comprises a label of an interface interaction technology corresponding to a control tree to which the target element belongs and a description path from a tree root of the control tree to which the target element belongs to the target element;
and positioning the target element according to the description character string of the target element.
2. The method of claim 1, wherein determining the target element from at least two control trees comprises:
acquiring a current mouse coordinate corresponding to a current mouse position;
for each control tree, searching each layer of the control tree layer by layer from the control tree root, and determining the bottommost layer element containing the current mouse coordinate in the control tree;
and determining the element with the smallest control area in the bottommost layer elements containing the current mouse coordinate in each control tree as the target element.
3. The method of claim 2, wherein for each of the control trees, searching each layer of the control tree layer by layer starting from the control tree root, and determining the lowest layer element of the control tree containing the current mouse coordinate comprises:
for each control tree, adding a first layer of the control tree into a first list to be searched, and recording that an element to be selected is empty;
traversing the first list to be searched, acquiring a control containing the current mouse coordinate, and setting the control containing the current mouse coordinate as the element to be selected after acquiring the control containing the current mouse coordinate;
clearing the first list to be searched;
when the element to be selected contains a sub-element, adding the sub-element of the element to be selected into the first list to be searched, skipping to traverse the first list to be searched, acquiring a control containing the current mouse coordinate, and setting the control containing the current mouse coordinate as the element to be selected after acquiring the control containing the current mouse coordinate;
and when the element to be selected does not contain the sub-element, determining the element to be selected as the bottommost element containing the current mouse coordinate and finishing searching.
4. The method of claim 3, further comprising: and after traversing the first to-be-searched list, when the control containing the current mouse coordinate does not exist in the first to-be-searched list, determining the element to be selected as the bottommost element containing the current mouse coordinate and finishing searching.
5. The method according to claim 2, wherein the determining, as the target element, an element with a smallest control area in the bottommost element containing the current mouse coordinate in each control tree comprises:
acquiring the size of a control corresponding to the bottommost element containing the current mouse coordinate in each control tree;
determining the area of the control corresponding to the bottommost layer element containing the current mouse coordinate according to the size information of the control corresponding to the bottommost layer element containing the current mouse coordinate in the control tree;
and taking the smallest control area in the bottommost layer element containing the current mouse coordinate in each control tree as the target element.
6. The method according to claim 1, wherein the describing the target element according to the set description mode to obtain a description character string of the target element comprises:
recording a control tree root of a control tree to which the target element belongs and the target element in a form of at least attribute name-attribute value pairs to obtain a description path of the target element, wherein adjacent elements in the description path of the target element are divided by adopting a first divider;
and dividing the description path of the label and the target element by using a second divider to obtain a description character string of the target element.
7. The method according to claim 6, wherein the recording the control tree root of the control tree to which the target element belongs and the description path of the target element obtained by the target element at least in the form of attribute name-attribute value pairs comprises:
and respectively recording the control tree root of the control tree to which the target element belongs to each layer of element of the target element in the form of attribute name-attribute value pairs to obtain the description path of the target element.
8. The method of claim 1 or 6, wherein the labels of different interface interaction technologies are different.
9. The method according to claim 6, wherein value ranges of the attribute names in the attribute name-attribute value pairs are not completely the same when interface interaction techniques corresponding to the control trees to which the target elements belong are different.
10. The method of claim 6, wherein the locating the target element according to the description string of the target element comprises:
dividing the description character string according to the second divider, and determining a second list to be searched according to the obtained label corresponding to the interface interaction technology;
dividing the description path of the target element according to a first divider to obtain an element set formed by each element in the description path;
and determining the first element to the last element in the element set as search elements one by one, comparing the search elements with elements in the second to-be-searched list as comparison elements one by one when each element in the element set is determined as the search element, and updating the second to-be-searched list according to sub-elements of the comparison elements after comparison to locate the target element.
11. The method according to claim 10, wherein the segmenting the description character string according to the second segmentation character, and determining a second list to be searched according to the obtained label corresponding to the interface interaction technology comprises:
dividing the description character string according to the second divider to obtain the label corresponding to the interface interaction technology, and adding the first layer of elements of the control tree corresponding to the label into the second list to be searched;
determining a first element to a last element in the element set one by one as a search element, comparing the search element with a corresponding element in the second to-be-searched list when each element in the element set is determined as the search element, and updating the second to-be-searched list according to sub-elements of the comparison element after comparison to locate the target element, including:
when i is not more than n, for the ith element, taking out the first element of the second list to be searched as a comparison element when the second list to be searched is not empty; wherein i is more than or equal to 1 and less than or equal to n, and n represents the total number of elements in the element set; after the first element is taken out of the second list to be searched, the second list to be searched does not comprise the element any more;
comparing each attribute name-attribute value pair in the ith element with the comparison element;
when the attribute name-attribute value pair corresponds to the attribute value of the comparison element, emptying the second to-be-searched list, adding all sub-elements of the comparison element into the second to-be-searched list, and setting the comparison element as a search result of the current round;
adding 1 to i and jumping to the step of taking the first element of the second list to be searched as a comparison element when the second list to be searched is not empty for the ith element when i is not more than n;
and positioning the search result corresponding to the nth element as the target element.
12. The method according to claim 11, wherein when all the attribute name-attribute value pairs in the ith element do not correspond to the attribute values of the comparison element and the comparison element includes a sub-element, all the sub-elements of the comparison element are added to the second list to be searched, and when i is not greater than n, for the ith element, when the second list to be searched is not empty, the first element of the second list to be searched is taken out as the comparison element.
13. The method according to claim 11, wherein when all the attribute name-attribute value pairs in the ith element do not correspond to the attribute values of the comparison element and the comparison element does not include a sub-element, the method jumps to a step of taking out the first element of the second list to be searched as the comparison element when i is not greater than n for the ith element and the second list to be searched is not empty.
14. A robot process automation interface element positioning device, comprising:
the target element determining module is used for determining target elements according to at least two control trees, wherein the target elements are elements with the smallest control area in the elements containing the current mouse coordinates in each control tree;
the description module is used for describing the target element according to a set description mode to obtain a description character string of the target element, wherein the description character string comprises a label of an interface interaction technology corresponding to a control tree to which the target element belongs and a description path from a tree root of the control tree to which the target element belongs to the target element;
and the positioning module is used for positioning the target element according to the description character string of the target element.
15. An apparatus, characterized in that the apparatus comprises:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the robotic process automation interface element positioning method of any of claims 1-13.
16. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method for robot process automation interface element positioning according to any of claims 1-13.
17. A computer program product comprising computer-executable instructions for implementing the method of any one of claims 1-13 when executed.
CN202111364188.4A 2021-11-17 2021-11-17 Robot flow automatic interface element positioning method, device and storage medium Active CN114090155B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111364188.4A CN114090155B (en) 2021-11-17 2021-11-17 Robot flow automatic interface element positioning method, device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111364188.4A CN114090155B (en) 2021-11-17 2021-11-17 Robot flow automatic interface element positioning method, device and storage medium

Publications (2)

Publication Number Publication Date
CN114090155A true CN114090155A (en) 2022-02-25
CN114090155B CN114090155B (en) 2024-08-20

Family

ID=80301645

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111364188.4A Active CN114090155B (en) 2021-11-17 2021-11-17 Robot flow automatic interface element positioning method, device and storage medium

Country Status (1)

Country Link
CN (1) CN114090155B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115062206A (en) * 2022-05-30 2022-09-16 上海弘玑信息技术有限公司 Webpage element searching method and electronic equipment
CN116185411A (en) * 2023-03-23 2023-05-30 苏州峰之鼎信息科技有限公司 RPA interface determination method, RPA interface determination device, computer equipment and storage medium
CN117130717A (en) * 2023-10-27 2023-11-28 杭州实在智能科技有限公司 Element positioning method and system of HTMLayout application program in RPA scene

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070043701A1 (en) * 2005-08-17 2007-02-22 Microsoft Corporation Query-based identification of user interface elements
CN104699608A (en) * 2015-03-10 2015-06-10 于秀山 Automatic test method and device for graphic user interface software
CN107957948A (en) * 2017-12-07 2018-04-24 郑州云海信息技术有限公司 A kind of user interface automatic test device and method
CN111045941A (en) * 2019-12-09 2020-04-21 广州品唯软件有限公司 Positioning method and device of user interface control and storage medium
CN112182468A (en) * 2020-10-14 2021-01-05 北京新纽科技有限公司 Positioning and analyzing method compatible with client interface element and webpage element
CN113176921A (en) * 2021-04-30 2021-07-27 上海云扩信息科技有限公司 Automatic customization system of desktop
CN113176922A (en) * 2021-04-30 2021-07-27 上海云扩信息科技有限公司 Universal automatic detection system for interface elements

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070043701A1 (en) * 2005-08-17 2007-02-22 Microsoft Corporation Query-based identification of user interface elements
CN104699608A (en) * 2015-03-10 2015-06-10 于秀山 Automatic test method and device for graphic user interface software
CN107957948A (en) * 2017-12-07 2018-04-24 郑州云海信息技术有限公司 A kind of user interface automatic test device and method
CN111045941A (en) * 2019-12-09 2020-04-21 广州品唯软件有限公司 Positioning method and device of user interface control and storage medium
CN112182468A (en) * 2020-10-14 2021-01-05 北京新纽科技有限公司 Positioning and analyzing method compatible with client interface element and webpage element
CN113176921A (en) * 2021-04-30 2021-07-27 上海云扩信息科技有限公司 Automatic customization system of desktop
CN113176922A (en) * 2021-04-30 2021-07-27 上海云扩信息科技有限公司 Universal automatic detection system for interface elements

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115062206A (en) * 2022-05-30 2022-09-16 上海弘玑信息技术有限公司 Webpage element searching method and electronic equipment
CN115062206B (en) * 2022-05-30 2023-04-07 上海弘玑信息技术有限公司 Webpage element searching method and electronic equipment
CN116185411A (en) * 2023-03-23 2023-05-30 苏州峰之鼎信息科技有限公司 RPA interface determination method, RPA interface determination device, computer equipment and storage medium
CN116185411B (en) * 2023-03-23 2024-04-30 苏州峰之鼎信息科技有限公司 RPA interface determination method, RPA interface determination device, computer equipment and storage medium
CN117130717A (en) * 2023-10-27 2023-11-28 杭州实在智能科技有限公司 Element positioning method and system of HTMLayout application program in RPA scene
CN117130717B (en) * 2023-10-27 2024-02-13 杭州实在智能科技有限公司 Element positioning method and system of HTMLayout application program in RPA scene

Also Published As

Publication number Publication date
CN114090155B (en) 2024-08-20

Similar Documents

Publication Publication Date Title
US10970097B2 (en) Adaptive web-based robotic process automation
US11836662B2 (en) Systems and methods for identifying processes for robotic automation and building models therefor
CN114090155B (en) Robot flow automatic interface element positioning method, device and storage medium
US11256912B2 (en) Electronic form identification using spatial information
US10169053B2 (en) Loading a web page
US10210193B2 (en) Ingestion plan based on table uniqueness
Sun et al. Ui components recognition system based on image understanding
CN114868109B (en) Automatic anchor point determination and target graphic element identification in user interface automation
CN115759100A (en) Data processing method, device, equipment and medium
EP4394618A1 (en) Log data query method and apparatus, and device and medium
CN113392014B (en) Test case generation method, device, electronic equipment and medium
CN115421831A (en) Method, device, equipment and storage medium for generating calling relation of activity component
CN113687880A (en) Method, device, equipment and medium for calling component
CN111428724B (en) Examination paper handwriting statistics method, device and storage medium
CN113221035A (en) Method, apparatus, device, medium, and program product for determining an abnormal web page
CN110321435B (en) Data source dividing method, device, equipment and storage medium
US10620946B1 (en) Dynamic modeling for opaque code during static analysis
US8775936B2 (en) Displaying dynamic and shareable help data for images a distance from a pointed-to location
CN113139542A (en) Target detection method, device, equipment and computer readable storage medium
WO2024031983A1 (en) Code management method and related device
CN112954025B (en) Information pushing method, device, equipment and medium based on hierarchical knowledge graph
WO2024143655A1 (en) Electronic device for processing training data in neural network model by using uncertainty value, and operation method therefor
US10489272B2 (en) Automatic instrumentation of code
CN115658939A (en) Modeling method and device for pedestrian protection head type point location database for deep learning
CN116107587A (en) Buried point method, device, equipment and medium applied to page

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant