CN113360151A - UI data set automatic generation method and system for RPA system - Google Patents

UI data set automatic generation method and system for RPA system Download PDF

Info

Publication number
CN113360151A
CN113360151A CN202110612219.7A CN202110612219A CN113360151A CN 113360151 A CN113360151 A CN 113360151A CN 202110612219 A CN202110612219 A CN 202110612219A CN 113360151 A CN113360151 A CN 113360151A
Authority
CN
China
Prior art keywords
partition
control
resolution size
user interface
value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110612219.7A
Other languages
Chinese (zh)
Other versions
CN113360151B (en
Inventor
杨圣洪
李肯立
林灏昶
蔡宇辉
秦云川
吴帆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hunan University
Zhongdian Jinxin Software Co Ltd
Original Assignee
Hunan University
Zhongdian Jinxin Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hunan University, Zhongdian Jinxin Software Co Ltd filed Critical Hunan University
Priority to CN202110612219.7A priority Critical patent/CN113360151B/en
Publication of CN113360151A publication Critical patent/CN113360151A/en
Application granted granted Critical
Publication of CN113360151B publication Critical patent/CN113360151B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/30—Creation or generation of source code
    • G06F8/38—Creation or generation of source code for implementing user interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/40—Transformation of program code
    • G06F8/41—Compilation

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • User Interface Of Digital Computer (AREA)
  • Digital Computer Display Output (AREA)

Abstract

The invention discloses a UI data set automatic generation method for an RPA system, which comprises the following steps: generating a GUI program source code, automatically compiling the GUI program source code to generate a GUI program executable file, automatically operating the GUI program executable file to generate a user interface window, automatically performing screenshot processing on the user interface window to generate an image file of a user interface data set, automatically analyzing the GUI program source code to generate a label information file of the user interface data set, and packaging the image file of the user interface data set and the label information file of the user interface data set to generate a user interface data set aiming at the RPA system target detection model. The method can solve the technical problems of complicated steps, high cost and long time consumption of the whole establishing process in the existing establishing method of the target detection data set for the RPA system and the technical problem that the precision of the data set is influenced by artificial uncertain factors caused by the need of artificially marking the target interval.

Description

UI data set automatic generation method and system for RPA system
Technical Field
The invention belongs to the technical field of User Interface (UI) and computer image recognition, and particularly relates to a method and a system for automatically generating a UI data set for an RPA system.
Background
Robot Process Automation (RPA) is a Process task automatically executed according to rules by simulating human operations on a computer through specific robot software.
At present, the method for establishing a target detection data set for an RPA system specifically includes: firstly, labeling a target area of an image by using labeling software, artificially delimiting a target interval and giving a labeling label value; then, putting the target image picture file and the image annotation information file into corresponding folders; finally, the picture file and the markup file are associated with each other.
However, the above method has some non-negligible drawbacks: firstly, the whole establishing process has complicated steps, high cost and long consumed time; secondly, the method needs to artificially mark the target interval, but the artificial uncertain factors can have certain influence on the accuracy of the data set.
Disclosure of Invention
Aiming at the defects or improvement requirements in the prior art, the invention provides a method and a system for automatically generating a UI (user interface) data set for an RPA (resilient packet access) system, and aims to solve the technical problems of complicated steps, high cost and long time consumption of the whole establishing process in the existing method for establishing a target detection data set for the RPA system and the technical problem that the precision of the data set is influenced by artificial uncertain factors due to the fact that target intervals need to be artificially marked.
To achieve the above object, according to one aspect of the present invention, there is provided an automatic UI data set generation method for an RPA system, comprising the steps of:
(1) generating a GUI program source code;
(2) and (3) automatically compiling the source code of the GUI program obtained in the step (1) to generate a GUI program executable file.
(3) Automatically operating the GUI program executable file obtained in the step (2) to generate a user interface window;
(4) and (4) automatically carrying out screenshot processing on the user interface window obtained in the step (3) to generate an image file of the user interface data set.
(5) And (3) automatically analyzing the GUI program source code obtained in the step (1) to generate a labeling information file of the user interface data set.
(6) And (5) packing the image file of the user interface data set obtained in the step (4) and the label information file of the user interface data set obtained in the step (5), thereby generating a user interface data set aiming at the RPA system target detection model.
Preferably, step (1) comprises the sub-steps of:
(1-1) acquiring a screen size through an interface of an operating system, and further acquiring a range of a user interface window.
(1-2) setting a start coordinate of the user interface window (which comprises a start abscissa and a start ordinate of the user interface window), randomly selecting a resolution size (which comprises a height value and a width value of the resolution size of the user interface window) and a partition number of the user interface window within the range of the user interface window obtained in the step (1-1), and determining a start coordinate (which comprises the start abscissa and the start ordinate of the partition) and a resolution size (which comprises the width value and the height value of the resolution size of the partition) of each partition according to the resolution size and the partition number of the user interface window. And determines the number of controls in each partition.
(1-3) according to the number of the partitions, the starting coordinate and the resolution size of each partition and the number of the controls in each partition obtained in the step (1-2), obtaining the starting coordinate of each control frame in the partition (which comprises the starting abscissa and the starting ordinate of the control frame) and the resolution size of the control frame (which comprises the width value and the height value of the resolution size of the control frame), randomly obtaining a control (which specifically comprises a button, an input frame, a text frame and a file list), determining the starting coordinate of the control (which comprises the starting abscissa and the starting ordinate of the control) according to the resolution size of the control frame, determining the resolution size of the control (which comprises the width value and the height value of the resolution size of the control) according to the control type of the control, and determining the control type of the control, And combining the starting coordinates of the control and the resolution size of the control into a control information list.
And (1-4) generating a GUI program source code according to the control information list obtained in the step (1-3).
Preferably, the start coordinate of the user interface window includes a start abscissa of the user interface window and a start ordinate of the user interface window, both of which are 0;
the height value of the resolution size of the user interface window is randomly acquired between one third of the screen resolution height value and the screen resolution height value;
the width value of the resolution size of the user interface window is R of the width value of the resolution size of the user interface windoww Multiple, Rw ∈[1,2.5]；
And if the width value of the resolution size of the user interface window exceeds the width value of the resolution size of the screen, re-acquiring the width value of the resolution size of the user interface window.
Preferably, the number of partitions, K, is a randomly derived integer and has K ∈ [1,4 ].
If the partition number K is 1, the user interface window is a first partition, the initial coordinate of the first partition is equal to the initial coordinate of the user interface window, and the resolution size of the first partition is equal to the resolution size of the user interface window;
if the number of the partitions K is 2, the user interface window is divided into a first partition and a second partition, wherein the width value of the resolution size of each partition is equal to the width value of the resolution size of the user interface window divided by the number of the partitions, and the height value of the resolution size of each partition is equal to the height value of the resolution size of the user interface window; the starting abscissa of the second partition is the starting abscissa of the first partition plus the width value of the resolution size of the first partition, and the starting ordinate of the second partition is equal to the starting ordinate of the first partition.
If the number of the partitions K is 3, the user interface window is divided into a first partition, a second partition and a third partition. Wherein the height value of the resolution size of the first partition is the height value R of the window resolution size of the user interfacewh31Multiple, Rwh31∈[0.1，0.3](ii) a The width value of the resolution size of the first partition is equal to the width value of the resolution size of the user interface window; the width value of the resolution size of the partition number two is R of the width of the resolution size of the user interface windowww32Multiple, Rwh32∈[0.2，0.4]The height value of the resolution size of the partition number two is equal to the height value of the resolution size of the user interface window minus the height value of the resolution size of the partition number one. The starting ordinate of the second partition is the starting ordinate of the first partition plus the height value of the resolution size of the first partition. The starting abscissa of the second partition is equal to the starting abscissa of the second partition; the remainder of the user interface window may be divided into three sections.
If the number of the partitions K is 4, the user interface window is divided into a first partition, a second partition, a third partition and a fourth partition. Wherein the height value of the resolution size of the first partition is the height value R of the resolution size of the user interface windowwh41Multiple, Rwh41∈[0.1，0.3](ii) a The width value of the resolution size of the first partition is equal to the width value of the resolution size of the user interface window; the width value of the resolution size of the second partition is the width value R of the resolution size of the user interface windowwh42Multiple, Rww42∈[0.2，0.4]The height value of the resolution size of the second partition is the height value R of the resolution size of the user interface windowwh42Height value of resolution size of multiple minus one division, Rwh42∈[0.6，0.85]. The initial ordinate of the partition number two is equal to the height value of the initial ordinate of the partition number one plus the resolution size of the partition number one, andthe starting abscissa of the number zone is equal to the starting abscissa of the number one zone; the width value of the resolution size of the third partition is the width value of the resolution size of the user interface window minus the width value of the resolution size of the second partition, and the height value of the resolution size of the third partition is equal to the height value of the resolution size of the second partition. The initial ordinate of the third partition is equal to the initial ordinate of the second partition, and the initial abscissa of the third partition is equal to the initial abscissa of the second partition plus the width value of the resolution size of the second partition; the remainder of the user interface window may be divided into four sections.
Preferably, if the partition number K is 1, the number of controls C in the first partition1Is a random integer and has C1∈[6，12]。
If the partition number K is 2, the number of controls C in the first partition21Is a random integer and has C21∈[4，8]Number of controls C in partition number two22Is a random integer and has C22∈[4，8]
If the partition number K is 3, the number of controls C in the first partition31Is a random integer and has C31∈[0，2]Number of controls C in partition number two32Is a random integer and has C32∈[0，2]Number of controls C in partition number three33Is a random integer and has C33∈[6，10]。
If the partition number K is 4, the number of controls C in the first partition41Is a random integer and has C41∈[0，2]Number of controls C in partition number two42Is a random integer and has C42∈[0，2]Number of controls C in partition number three43Is a random integer and has C43∈[6，10]Number of controls C in partition number four44Is a random integer and has C44∈[0，2]。
Preferably, the start coordinates of the control frame
Figure BDA0003096323300000055
And ycanvas_ijAnd resolution size W of control framecanvasAnd HcanvasIs obtained by the following function:
Figure BDA0003096323300000051
Figure BDA0003096323300000052
xcavas_ij＝i*Wcanvas，i∈[0，Krow-1]
ycanvas_ij＝j*Hcanvas，j∈[0，Kcolumn-1]
wherein:
Krow＝Rand(S)，S∈[1，2]
Figure BDA0003096323300000053
wherein KrowFor the number of controls in each row within each partition, which is obtained by a random function Rand (), KcolumnFor the number of column controls, the number of controls in each column in each partition is indicated, WareaWidth value of resolution size for each partition, HareaHeight value, W, for resolution size of each partitioncanvasWidth value, H, representing resolution size of control framecanvasA height value representing the resolution size of the control box.
Figure BDA0003096323300000054
And ycanvas_ijAnd respectively representing the starting abscissa and the starting ordinate of the control box in the ith row and the jth column in each partition.
Preferably, the height value of the resolution size of the control is T times of the height value of the resolution size of the control frame, and T belongs to [0.3, 0.7 ];
the width value of the resolution size of the control is X times of the height value of the resolution size of the control, and X is a natural number; wherein:
if the control type is a button, X belongs to [1, 2 ];
if the control type is an input box, X belongs to [2, 5 ];
if the control type is a text box, X belongs to [2, 5 ];
if the control type is a file list, then X belongs to [1, 3 ].
Preferably, the starting coordinate of the control is determined by the position of the control;
if the position of the control in the control frame is deviated to the left, the starting abscissa of the control is P of the width value of the resolution size of the control framewX is wherein PwIs a random number and has Pw∈[0.05，0.15]The initial ordinate of the control is the height value P of the resolution size of the control framehMultiple, PhIs a random number and has Ph∈[0.05，0.15](ii) a If the width value of the starting abscissa of the control plus the resolution size of the control exceeds the width value of the resolution size of the control picture frame, randomly acquiring the width value of the resolution size of the control again; and if the height value of the initial ordinate of the control plus the resolution size of the control exceeds the height value of the resolution size of the control picture frame, randomly acquiring the height value of the resolution size of the control again.
If the position of the control is located in the middle of the control frame, acquiring a central value of the control frame (the abscissa of the central value is equal to one half of the width value of the resolution size of the control frame, and the ordinate of the central value is equal to one half of the height value of the resolution size of the control frame) and a central value of the control (the abscissa of the central value is equal to one half of the width value of the resolution size of the control frame, and the ordinate of the central value is equal to one half of the height value of the resolution size of the control). Subtracting the abscissa of the central value of the control frame from the abscissa of the central value of the control to obtain the initial abscissa of the control, and subtracting the ordinate of the central value of the control frame from the ordinate of the central value of the control to obtain the initial ordinate of the control.
If the position of the control is positioned on the right side of the control frame, the control termination width value is P of the width value of the resolution size of the control framewThe number of times of the total number of the parts,Pwis a random number and has Pw∈[0.85，0.95]. P with the control termination height value being the height value of the resolution size of the control framehMultiple, PhIs a random number and has Ph∈[0.85，0.95](ii) a The initial abscissa of the control is the width value obtained by subtracting the resolution size of the control from the control termination width value, and the initial ordinate of the control is the height value obtained by subtracting the resolution size of the control from the control termination height value; if the width value of the starting abscissa of the control plus the resolution size of the control exceeds the width value of the resolution size of the control picture frame, randomly acquiring the width value of the resolution size of the control again; and if the height value of the initial ordinate of the control plus the resolution size of the control exceeds the height value of the resolution size of the control picture frame, randomly acquiring the height value of the resolution size of the control again.
Preferably, step (5) is specifically: and (4) analyzing the file content in the GUI program source code to extract the control type, the initial coordinate of the control and the resolution size of the control in the file content, and writing the file into the annotation information file with the same name as the image file of the user interface data set obtained in the step (4), so as to generate the annotation information file of the user interface data set.
According to another aspect of the present invention, there is provided an automatic UI data set generation system for an RPA system, comprising:
the first module is used for generating GUI program source codes;
and the second module is used for automatically compiling the source code of the GUI program obtained by the second module to generate the GUI program executable file.
The third module is used for automatically operating the GUI program executable file obtained by the second module so as to generate a user interface window;
and the fourth module is used for automatically carrying out screenshot processing on the user interface window obtained by the third module so as to generate an image file of the user interface data set.
And the fifth module is used for automatically analyzing the GUI program source code obtained by the first module so as to generate a labeling information file of the user interface data set.
And the sixth module is used for packaging the image file of the user interface data set obtained by the fourth module and the label information file of the user interface data set obtained by the fifth module, so as to generate a user interface data set aiming at the RPA system target detection model.
In general, compared with the prior art, the above technical solution contemplated by the present invention can achieve the following beneficial effects:
(1) the invention adopts the steps (1) to (6) to automatically generate the image file of the user interface data set and automatically generate the label information file of the user interface data set, thereby solving the technical problems of complicated steps, high cost and long time consumption of the existing target detection data set establishment method for the RPA system;
(2) the invention adopts the step (5) to automatically analyze the source code of the GUI program and obtain the control type of the control in the source code, the initial coordinate of the control and the resolution size of the control, thereby solving the technical problem that the prior target detection data set establishment method for the RPA system needs to artificially mark the target interval, and the artificial uncertain factors can have certain influence on the precision of the data set
Drawings
FIG. 1 is a flow chart of a UI dataset auto-generation method for an RPA system of the present invention;
FIG. 2 is a detailed flow chart of step (1) of the method of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
The method mainly comprises the steps of randomly generating GUI program source codes, automatically compiling, operating and analyzing, generating image files of user interface data sets and label files of corresponding user interface data sets, and packaging to generate user interface data sets aiming at the RPA system target detection model. On one hand, the GUI program source code is automatically compiled, operated and analyzed, and an image file of a user interface data set and a corresponding label file of the user interface data set are generated; and on the other hand, packing the image file of the user interface data set and the label file of the corresponding user interface data set to generate the user interface data set aiming at the RPA system target detection model.
As shown in fig. 1, the present invention provides a method for automatically generating UI data set for RPA system, comprising the following steps:
(1) generating a GUI program source code;
as shown in fig. 2, step (1) includes the following sub-steps:
(1-1) acquiring a screen size through an interface of an operating system, and further acquiring a range of a user interface window.
(1-2) setting a start coordinate of the user interface window (which comprises a start abscissa and a start ordinate of the user interface window), randomly selecting a resolution size (which comprises a height value and a width value of the resolution size of the user interface window) and a partition number of the user interface window within the range of the user interface window obtained in the step (1-1), and determining a start coordinate (which comprises the start abscissa and the start ordinate of the partition) and a resolution size (which comprises the width value and the height value of the resolution size of the partition) of each partition according to the resolution size and the partition number of the user interface window. And determines the number of controls in each partition.
Take the height of the screen resolution size as 1980 and the width as 1080 as an example. The start coordinate of the user interface window (which includes the start abscissa of the user interface window and the start ordinate of the user interface window) is set to 0. The height value of the resolution size of the user interface window is randomly acquired between 360 (which includes one third of the screen resolution height value) and 1080 (which includes the screen resolution height value), and the width value of the resolution size of the user interface window is R of the width value of the resolution size of the user interface windowwlMultiple, Rwl∈[1，2.5]. Wherein if the user isAnd if the width value of the resolution size of the interface window exceeds the width value of the resolution size of the screen, re-acquiring the width value of the resolution size of the user interface window.
The step (1-2) has the advantages that the user interface window is ensured to be wholly positioned in the screen, and the adverse condition that part of the user interface window exceeds the display range of the screen is avoided, so that the generation effect of the data set picture file is influenced.
The number of partitions, K, is a randomly derived integer and has K ∈ [1,4 ].
Specifically, if the number of the partitions K is 1, the user interface window is a first partition, the initial coordinate of the first partition is equal to the initial coordinate of the user interface window, and the resolution size of the first partition is equal to the resolution size of the user interface window;
if the number of the partitions K is 2, the user interface window is divided into a first partition and a second partition, wherein the width value of the resolution size of each partition is equal to the width value of the resolution size of the user interface window divided by the number of the partitions, and the height value of the resolution size of each partition is equal to the height value of the resolution size of the user interface window;
the starting abscissa of the second partition is the starting abscissa of the first partition plus the width value of the resolution size of the first partition, and the starting ordinate of the second partition is equal to the starting ordinate of the first partition.
If the number of the partitions K is 3, the user interface window is divided into a first partition, a second partition and a third partition. Wherein the height value of the resolution size of the first partition is the height value R of the window resolution size of the user interfacewh31Multiple, Rwh31∈[0.1，0.3](ii) a The width value of the resolution size of the first partition is equal to the width value of the resolution size of the user interface window.
The width value of the resolution size of the partition number two is R of the width of the resolution size of the user interface windowww32Multiple, Rwh32∈[0.2，0.4]The height value of the resolution size of the partition number two is equal to the height value of the resolution size of the user interface window minus the height value of the resolution size of the partition number one. Second branchThe start ordinate of the zone is the start ordinate of the zone number one plus the height value of the resolution size of the zone number one. The starting abscissa of the second partition is equal to the starting abscissa of the second partition.
The remainder of the user interface window may be divided into three sections.
If the number of the partitions K is 4, the user interface window is divided into a first partition, a second partition, a third partition and a fourth partition. Wherein the height value of the resolution size of the first partition is the height value R of the resolution size of the user interface windowwh41Multiple, Rwh41∈[0.1，0.3](ii) a The width value of the resolution size of the first partition is equal to the width value of the resolution size of the user interface window.
The width value of the resolution size of the second partition is the width value R of the resolution size of the user interface windowwh42Multiple, Rww42∈[0.2，0.4]The height value of the resolution size of the second partition is the height value R of the resolution size of the user interface windowwh42Height value of resolution size of multiple minus one division, Rwh42∈[0.6，0.85]. The starting ordinate of the partition No. two is equal to the starting ordinate of the partition No. one plus the height value of the resolution size of the partition No. one, and the starting abscissa of the partition No. two is equal to the starting abscissa of the partition No. one.
The width value of the resolution size of the third partition is the width value of the resolution size of the user interface window minus the width value of the resolution size of the second partition, and the height value of the resolution size of the third partition is equal to the height value of the resolution size of the second partition. The initial ordinate of the third partition is equal to the initial ordinate of the second partition, and the initial abscissa of the third partition is equal to the initial abscissa of the second partition plus the width value of the resolution size of the second partition.
The remainder of the user interface window may be divided into four sections.
If the partition number K is 1, the number of controls C in the first partition1Is a random integer and has C1∈[6，12]。
If the partition number K is 2, the number of controls C in the first partition21Is a random integer and has C21∈[4，8]Number of controls C in partition number two22Is a random integer and has C22∈[4，8]
If the partition number K is 3, the number of controls C in the first partition31Is a random integer and has C31∈[0，2]Number of controls C in partition number two32Is a random integer and has C32∈[0，2]Number of controls C in partition number three33Is a random integer and has C33∈[6，10]。
If the partition number K is 4, the number of controls C in the first partition41Is a random integer and has C41∈[0，2]Number of controls C in partition number two42Is a random integer and has C42∈[0，2]Number of controls C in partition number three43Is a random integer and has C43∈[6，10]Number of controls C in partition number four44Is a random integer and has C44∈[0，2]。
The advantage of the above step (1-2) is that the concept of partitioning is also applied because of the design of user interface windows in reality. According to different partitions, different operation effects and operation logics exist, and correspondingly, different control numbers exist. Therefore, the steps ensure that the generated user interface window has practical significance to a certain extent.
(1-3) according to the number of the partitions, the starting coordinate and the resolution size of each partition and the number of the controls in each partition obtained in the step (1-2), obtaining the starting coordinate of each control frame in the partition (which comprises the starting abscissa and the starting ordinate of the control frame) and the resolution size of the control frame (which comprises the width value and the height value of the resolution size of the control frame), randomly obtaining a control (which specifically comprises a button, an input frame, a text frame and a file list), determining the starting coordinate of the control (which comprises the starting abscissa and the starting ordinate of the control) according to the resolution size of the control frame, determining the resolution size of the control (which comprises the width value and the height value of the resolution size of the control) according to the control type of the control, and determining the control type of the control, And combining the starting coordinates of the control and the resolution size of the control into a control information list.
In particular, the start coordinates of the control frame
Figure BDA0003096323300000125
And ycanvas_ijAnd resolution size W of control framecanvasAnd HcanvasIs obtained by the following function:
Figure BDA0003096323300000121
Figure BDA0003096323300000122
xcavas_ij＝i*Wcanvas，i∈[0，Krow-1]
ycanvas_ij＝j*Hcanvas，j∈[0，Kcolumn-1]
wherein:
Krow＝Rand(S)，S∈[1，2]
Figure BDA0003096323300000123
wherein KrowFor the number of controls in each row within each partition, which is obtained by a random function Rand (), KcolumnFor the number of column controls, the number of controls in each column in each partition is indicated, WareaWidth value of resolution size for each partition, HareaHeight value, W, for resolution size of each partitioncanvasWidth value, H, representing resolution size of control framecanvasA height value representing the resolution size of the control box.
Figure BDA0003096323300000124
And ycanvas_ijRespectively representing the control box in the ith row and the jth column in each partitionThe starting abscissa and the starting ordinate.
The height value of the resolution size of the control is T times of the height value of the resolution size of the control frame, and T belongs to [0.3, 0.7], the width value of the resolution size of the control is X times of the height value of the resolution size of the control, X is a natural number, wherein if the control type is a button, X belongs to [1, 2 ]; if the control type is an input box, X belongs to [2, 5 ]; if the control type is a text box, X belongs to [2, 5 ]; if the control type is a file list, then X belongs to [1, 3 ].
The step (1-3) has the advantages that the resolution size proportion of the control is randomized, the diversity of the resolution sizes of the control is ensured, and the generated resolution size proportion of the control is in a certain range, so that the control has certain practical visual significance.
The starting coordinates of the control are determined by where the control is located.
If the position of the control in the control frame is deviated to the left, the starting abscissa of the control is P of the width value of the resolution size of the control framewX is wherein PwIs a random number and has Pw∈[0.05，0.15]The initial ordinate of the control is the height value P of the resolution size of the control framehMultiple, PhIs a random number and has Ph∈[0.05，0.15]。
If the width value of the initial abscissa of the control plus the resolution size of the control exceeds the width value of the resolution size of the control picture frame, randomly acquiring the width value of the resolution size of the control again; and if the height value of the initial ordinate of the control plus the resolution size of the control exceeds the height value of the resolution size of the control picture frame, randomly acquiring the height value of the resolution size of the control again.
If the position of the control is located in the middle of the control frame, acquiring a central value of the control frame (the abscissa of the central value is equal to one half of the width value of the resolution size of the control frame, and the ordinate of the central value is equal to one half of the height value of the resolution size of the control frame) and a central value of the control (the abscissa of the central value is equal to one half of the width value of the resolution size of the control frame, and the ordinate of the central value is equal to one half of the height value of the resolution size of the control). Subtracting the abscissa of the central value of the control frame from the abscissa of the central value of the control to obtain the initial abscissa of the control, and subtracting the ordinate of the central value of the control frame from the ordinate of the central value of the control to obtain the initial ordinate of the control.
If the position of the control is positioned on the right side of the control frame, the control termination width value is P of the width value of the resolution size of the control framewMultiple, PwIs a random number and has Pw∈[0.85，0.95]. P with the control termination height value being the height value of the resolution size of the control framehMultiple, PhIs a random number and has Ph∈[0.85，0.95]. The starting abscissa of the control is the width value of the ending width value of the control minus the resolution size of the control, and the starting ordinate of the control is the height value of the ending height value of the control minus the resolution size of the control.
If the width value of the initial abscissa of the control plus the resolution size of the control exceeds the width value of the resolution size of the control picture frame, randomly acquiring the width value of the resolution size of the control again; and if the height value of the initial ordinate of the control plus the resolution size of the control exceeds the height value of the resolution size of the control picture frame, randomly acquiring the height value of the resolution size of the control again.
The step (1-3) has the advantages that the position of each control is ensured to be located in the control frame, that is, the controls are not overlapped, and the size of the resolution of the generated control is significant.
And (1-4) generating a GUI program source code according to the control information list obtained in the step (1-3).
(2) And (3) automatically compiling the source code of the GUI program obtained in the step (1) to generate a GUI program executable file.
The method comprises the following specific steps: and (3) automatically processing the GUI program source code obtained in the step (1) by using a mode of simulating command line input so as to compile the source code into a GUI program executable file.
(3) Automatically operating the GUI program executable file obtained in the step (2) to generate a user interface window;
the method comprises the following steps: and (3) automatically running the GUI program executable file obtained in the step (2) by using a mode of simulating command line input so as to generate a user interface window.
(4) And (4) automatically carrying out screenshot processing on the user interface window obtained in the step (3) to generate an image file of the user interface data set.
The method comprises the following steps: and (4) automatically performing screenshot processing on the user interface window obtained in the step (3) by using a mode of simulating command line input, saving the result as an image file, closing the user interface window after one second, and taking the image file as the image file of the user interface data set.
The step (3) and the step (4) have the advantages that the user interface window is closed regularly, and the integrity of the running logic of the GUI program is ensured. And when the operation life of the GUI program is finished, the GUI program is terminated in time, the memory is recycled, the memory overflow is prevented, and the operation robustness of the GUI program is enhanced.
(5) And (3) automatically analyzing the GUI program source code obtained in the step (1) to generate a labeling information file of the user interface data set.
The method comprises the following steps: analyzing the file content in the GUI program source code to extract the control type, the initial coordinate of the control and the resolution size of the control in the file content, and writing the file into the annotation information file with the same name as the image file of the user interface data set obtained in the step (4), so as to generate the annotation information file of the user interface data set;
(6) packing the image file of the user interface data set obtained in the step (4) and the label information file of the user interface data set obtained in the step (5) so as to generate a user interface data set aiming at an RPA system target detection model;
it can be understood that the invention aims at the problem of overlarge data integration cost of manually constructing a target detection model user interface of an RPA system, and realizes an automatic generation method capable of automatically and quickly generating a UI data set for the RPA system. The automatic generation of the user interface data set aiming at the RPA system target detection model is successfully established, and the consumption of the enterprise material and financial cost and the time cost is reduced.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims (10)

1. A UI data set automatic generation method for an RPA system is characterized by comprising the following steps:
(1) generating a GUI program source code;
(2) and (3) automatically compiling the source code of the GUI program obtained in the step (1) to generate a GUI program executable file.
(3) Automatically operating the GUI program executable file obtained in the step (2) to generate a user interface window;
(4) and (4) automatically carrying out screenshot processing on the user interface window obtained in the step (3) to generate an image file of the user interface data set.
(5) And (3) automatically analyzing the GUI program source code obtained in the step (1) to generate a labeling information file of the user interface data set.
(6) And (5) packing the image file of the user interface data set obtained in the step (4) and the label information file of the user interface data set obtained in the step (5), thereby generating a user interface data set aiming at the RPA system target detection model.
2. The automatic UI data set generation method for RPA system according to claim 1, wherein the step (1) comprises the sub-steps of:
(1-1) acquiring a screen size through an interface of an operating system, and further acquiring a range of a user interface window.
(1-2) setting a start coordinate of the user interface window (which comprises a start abscissa and a start ordinate of the user interface window), randomly selecting a resolution size (which comprises a height value and a width value of the resolution size of the user interface window) and a partition number of the user interface window within the range of the user interface window obtained in the step (1-1), and determining a start coordinate (which comprises the start abscissa and the start ordinate of the partition) and a resolution size (which comprises the width value and the height value of the resolution size of the partition) of each partition according to the resolution size and the partition number of the user interface window. And determines the number of controls in each partition.
(1-3) according to the number of the partitions, the starting coordinate and the resolution size of each partition and the number of the controls in each partition obtained in the step (1-2), obtaining the starting coordinate of each control frame in the partition (which comprises the starting abscissa and the starting ordinate of the control frame) and the resolution size of the control frame (which comprises the width value and the height value of the resolution size of the control frame), randomly obtaining a control (which specifically comprises a button, an input frame, a text frame and a file list), determining the starting coordinate of the control (which comprises the starting abscissa and the starting ordinate of the control) according to the resolution size of the control frame, determining the resolution size of the control (which comprises the width value and the height value of the resolution size of the control) according to the control type of the control, and determining the control type of the control, And combining the starting coordinates of the control and the resolution size of the control into a control information list.
And (1-4) generating a GUI program source code according to the control information list obtained in the step (1-3).
3. The automatic generation method of UI data sets for RPA system according to claim 1 or 2, characterized in that,
the starting coordinate of the user interface window comprises a starting abscissa of the user interface window and a starting ordinate of the user interface window, and both the starting abscissa and the starting ordinate are 0;
the height value of the resolution size of the user interface window is randomly acquired between one third of the screen resolution height value and the screen resolution height value;
the width value of the resolution size of the user interface window is R of the width value of the resolution size of the user interface windowwlMultiple, Rwl∈[1,2.5]；
And if the width value of the resolution size of the user interface window exceeds the width value of the resolution size of the screen, re-acquiring the width value of the resolution size of the user interface window.
4. The automatic generation method of UI data sets for RPA system according to any of claims 1-3, characterized in that,
the number of partitions, K, is a randomly derived integer and has K ∈ [1,4 ].
If the partition number K is 1, the user interface window is a first partition, the initial coordinate of the first partition is equal to the initial coordinate of the user interface window, and the resolution size of the first partition is equal to the resolution size of the user interface window;
if the number of the partitions K is 2, the user interface window is divided into a first partition and a second partition, wherein the width value of the resolution size of each partition is equal to the width value of the resolution size of the user interface window divided by the number of the partitions, and the height value of the resolution size of each partition is equal to the height value of the resolution size of the user interface window; the starting abscissa of the second partition is the starting abscissa of the first partition plus the width value of the resolution size of the first partition, and the starting ordinate of the second partition is equal to the starting ordinate of the first partition.
If the number of the partitions K is 3, the user interface window is divided into a first partition, a second partition and a third partition. Wherein the height value of the resolution size of the first partition is the height value R of the window resolution size of the user interfacewh31Multiple, Rwh31∈[0.1,0.3](ii) a The width value of the resolution size of the first partition is equal to the width value of the resolution size of the user interface window; the width value of the resolution size of the partition number two is R of the width of the resolution size of the user interface windowww32Multiple, Rwh32∈[0.2,0.4]The height value of the resolution size of the partition number two is equal to the height value of the resolution size of the user interface window minus the height value of the resolution size of the partition number one. The starting ordinate of the second partition is the starting ordinate of the first partition plus the height value of the resolution size of the first partition. The starting abscissa of the second partition is equal to that of the second partitionThe starting abscissa of (a); the remainder of the user interface window may be divided into three sections.
If the number of the partitions K is 4, the user interface window is divided into a first partition, a second partition, a third partition and a fourth partition. Wherein the height value of the resolution size of the first partition is the height value R of the resolution size of the user interface windowwh41Multiple, Rwh41∈[0.1,0.3](ii) a The width value of the resolution size of the first partition is equal to the width value of the resolution size of the user interface window; the width value of the resolution size of the second partition is the width value R of the resolution size of the user interface windowwh42Multiple, Rww42∈[0.2,0.4]The height value of the resolution size of the second partition is the height value R of the resolution size of the user interface windowwh42Height value of resolution size of multiple minus one division, Rwhh2∈[0.6,0.85]. The initial ordinate of the second partition is equal to the initial ordinate of the first partition plus the height value of the resolution size of the first partition, and the initial abscissa of the second partition is equal to the initial abscissa of the first partition; the width value of the resolution size of the third partition is the width value of the resolution size of the user interface window minus the width value of the resolution size of the second partition, and the height value of the resolution size of the third partition is equal to the height value of the resolution size of the second partition. The initial ordinate of the third partition is equal to the initial ordinate of the second partition, and the initial abscissa of the third partition is equal to the initial abscissa of the second partition plus the width value of the resolution size of the second partition; the remainder of the user interface window may be divided into four sections.
5. The automatic generation method of UI data sets for RPA system of claim 4, wherein,
if the partition number K is 1, the number of controls C in the first partition1Is a random integer and has C1∈[6，12]。
If the partition number K is 2, the number of controls C in the first partition21Is a random integer and has C21∈[4，8]Number of controls C in partition number two22Is a random integerAnd has C22∈[4，8]
If the partition number K is 3, the number of controls C in the first partition31Is a random integer and has C31∈[0，2]Number of controls C in partition number two32Is a random integer and has C32∈[0，2]Number of controls C in partition number three33Is a random integer and has C33∈[6，10]。
If the partition number K is 4, the number of controls C in the first partition41Is a random integer and has C41∈[0，2]Number of controls C in partition number two42Is a random integer and has C42∈[0，2]Number of controls C in partition number three43Is a random integer and has C43∈[6，10]Number of controls C in partition number four44Is a random integer and has C44∈[0，2]。
6. The method of claim 2, wherein the start coordinates of the control box are the same as the start coordinates of the control box
Figure FDA0003096323290000043
And ycanvas_ijAnd resolution size W of control framecanvasAnd HcanvasIs obtained by the following function:
Figure FDA0003096323290000041
Figure FDA0003096323290000042
xcavas_ij＝i*Wcanvas，i∈[0，Krow-1]
ycanvas_ij＝j*Hcanvas，j∈[0，Kcolumn-1]
wherein:
Krow＝Rand(S)，S∈[1，2]
Figure FDA0003096323290000051
wherein KrowFor the number of controls in each row within each partition, which is obtained by a random function Rand (), KcolumnFor the number of column controls, the number of controls in each column in each partition is indicated, WareaWidth value of resolution size for each partition, HareaHeight value, W, for resolution size of each partitioncanvasWidth value, H, representing resolution size of control framecanvasA height value representing the resolution size of the control box.
Figure FDA0003096323290000052
And ycanvas_ijAnd respectively representing the starting abscissa and the starting ordinate of the control box in the ith row and the jth column in each partition.
7. The automatic generation method of UI data sets for RPA system of claim 6, wherein,
the height value of the resolution size of the control is T times of the height value of the resolution size of the control frame, and T belongs to [0.3, 0.7 ];
the width value of the resolution size of the control is X times of the height value of the resolution size of the control, and X is a natural number; wherein:
if the control type is a button, X belongs to [1, 2 ];
if the control type is an input box, X belongs to [2, 5 ];
if the control type is a text box, X belongs to [2, 5 ];
if the control type is a file list, then X belongs to [1, 3 ].
8. The automatic generation method of UI data sets for RPA system of claim 7, wherein,
the initial coordinate of the control is determined by the position of the control;
if the position of the control in the control frame is deviated to the left, the starting abscissa of the control is P of the width value of the resolution size of the control framewX is wherein PwIs a random number and has Pw∈[0.05，0.15]The initial ordinate of the control is the height value P of the resolution size of the control framehMultiple, PhIs a random number and has Ph∈[0.05,0.15](ii) a If the width value of the starting abscissa of the control plus the resolution size of the control exceeds the width value of the resolution size of the control picture frame, randomly acquiring the width value of the resolution size of the control again; and if the height value of the initial ordinate of the control plus the resolution size of the control exceeds the height value of the resolution size of the control picture frame, randomly acquiring the height value of the resolution size of the control again.
If the position of the control is located in the middle of the control frame, acquiring a central value of the control frame (the abscissa of the central value is equal to one half of the width value of the resolution size of the control frame, and the ordinate of the central value is equal to one half of the height value of the resolution size of the control frame) and a central value of the control (the abscissa of the central value is equal to one half of the width value of the resolution size of the control frame, and the ordinate of the central value is equal to one half of the height value of the resolution size of the control). Subtracting the abscissa of the central value of the control frame from the abscissa of the central value of the control to obtain the initial abscissa of the control, and subtracting the ordinate of the central value of the control frame from the ordinate of the central value of the control to obtain the initial ordinate of the control.
If the position of the control is positioned on the right side of the control frame, the control termination width value is P of the width value of the resolution size of the control framewMultiple, PwIs a random number and has Pw∈[0.85,0.95]. P with the control termination height value being the height value of the resolution size of the control framehMultiple, PhIs a random number and has Ph∈[0.85,0.95](ii) a The initial abscissa of the control is the width value obtained by subtracting the resolution size of the control from the control termination width value, and the initial ordinate of the control is the height value obtained by subtracting the resolution size of the control from the control termination height value; wherein if the controlIf the width value of the resolution size of the control plus the initial abscissa exceeds the width value of the resolution size of the control frame, the width value of the resolution size of the control is randomly obtained again; and if the height value of the initial ordinate of the control plus the resolution size of the control exceeds the height value of the resolution size of the control picture frame, randomly acquiring the height value of the resolution size of the control again.
9. The method for automatically generating a UI data set for an RPA system according to claim 1, wherein the step (5) is specifically: and (4) analyzing the file content in the GUI program source code to extract the control type, the initial coordinate of the control and the resolution size of the control in the file content, and writing the file into the annotation information file with the same name as the image file of the user interface data set obtained in the step (4), so as to generate the annotation information file of the user interface data set.
10. An automatic UI dataset generation system for an RPA system, comprising:
the first module is used for generating GUI program source codes;
and the second module is used for automatically compiling the source code of the GUI program obtained by the second module to generate the GUI program executable file.
The third module is used for automatically operating the GUI program executable file obtained by the second module so as to generate a user interface window;
and the fourth module is used for automatically carrying out screenshot processing on the user interface window obtained by the third module so as to generate an image file of the user interface data set.
And the fifth module is used for automatically analyzing the GUI program source code obtained by the first module so as to generate a labeling information file of the user interface data set.
And the sixth module is used for packaging the image file of the user interface data set obtained by the fourth module and the label information file of the user interface data set obtained by the fifth module, so as to generate a user interface data set aiming at the RPA system target detection model.
CN202110612219.7A 2021-06-02 2021-06-02 UI data set automatic generation method and system for RPA system Active CN113360151B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110612219.7A CN113360151B (en) 2021-06-02 2021-06-02 UI data set automatic generation method and system for RPA system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110612219.7A CN113360151B (en) 2021-06-02 2021-06-02 UI data set automatic generation method and system for RPA system

Publications (2)

Publication Number Publication Date
CN113360151A true CN113360151A (en) 2021-09-07
CN113360151B CN113360151B (en) 2024-03-08

Family

ID=77531077

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110612219.7A Active CN113360151B (en) 2021-06-02 2021-06-02 UI data set automatic generation method and system for RPA system

Country Status (1)

Country Link
CN (1) CN113360151B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114020267A (en) * 2021-11-04 2022-02-08 四川启睿克科技有限公司 Method for detecting RPA window control

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050248545A1 (en) * 2004-05-07 2005-11-10 Takanori Nishimura Method, apparatus, and software program for processing information
WO2016170288A1 (en) * 2015-04-23 2016-10-27 Real Digit System for creating and reproducing a composite document
US20180203571A1 (en) * 2017-01-18 2018-07-19 Oracle International Corporation Generating a graphical user interface model from an image
US20190250891A1 (en) * 2018-02-12 2019-08-15 Oracle International Corporation Automated code generation

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050248545A1 (en) * 2004-05-07 2005-11-10 Takanori Nishimura Method, apparatus, and software program for processing information
WO2016170288A1 (en) * 2015-04-23 2016-10-27 Real Digit System for creating and reproducing a composite document
US20180203571A1 (en) * 2017-01-18 2018-07-19 Oracle International Corporation Generating a graphical user interface model from an image
US20190250891A1 (en) * 2018-02-12 2019-08-15 Oracle International Corporation Automated code generation

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
娄宇;王琼;: "基于XML的嵌入式图形界面自动生成技术", 计算机应用, no. 1 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114020267A (en) * 2021-11-04 2022-02-08 四川启睿克科技有限公司 Method for detecting RPA window control

Also Published As

Publication number Publication date
CN113360151B (en) 2024-03-08

Similar Documents

Publication Publication Date Title
CN1517904B (en) Ink marking device and associated application programmed interface
US20220350955A1 (en) System and method for collaborative ink management
US11579856B2 (en) Multi-chip compatible compiling method and device
Ma et al. Ladv: Deep learning assisted authoring of dashboard visualizations from images and sketches
CN106951298A (en) A kind of method and apparatus for lifting application toggle speed
WO2023138437A1 (en) Application development platform, micro-program generation method, and device and storage medium
CN114020256A (en) Front-end page generation method, device and equipment and readable storage medium
US20140380205A1 (en) Interface Development and Operation
CN115311130A (en) Method, system and terminal for migrating styles of Chinese, calligraphy and digital images in multiple lattices
CN111310155A (en) System architecture for automatic identification of slider verification code and implementation method
CN112416788A (en) Hierarchical standard Web application UI automatic test method
CN113360151A (en) UI data set automatic generation method and system for RPA system
Plimmer et al. Sketchnode: Intelligent sketching support and formal diagramming
KR100807061B1 (en) Automatic design method with MS Excel
Stapleton et al. Combining sketching and traditional diagram editing tools
US20200364034A1 (en) System and Method for Automated Code Development and Construction
Chugh Prodirect manipulation: bidirectional programming for the masses
JP2830354B2 (en) Equivalent data item detection method
CN112084103B (en) Interface test method, device, equipment and medium
US6683624B1 (en) System and method for managing programming object visual representations participating in alternative execution paths
Plimmer et al. Rata: codeless generation of gesture recognizers
CN112686014A (en) Cable number plate compiling system and method
CN107908429B (en) Human-computer interaction and programming system applied to robot software engineer
US7698659B1 (en) Methods and apparatus for formatting portion of content
EP1248189A1 (en) Program reproducing method and device, and medium on which a program for program reproduction recording

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant