CN113360151A - UI data set automatic generation method and system for RPA system - Google Patents

UI data set automatic generation method and system for RPA system Download PDF

Info

Publication number
CN113360151A
CN113360151A CN202110612219.7A CN202110612219A CN113360151A CN 113360151 A CN113360151 A CN 113360151A CN 202110612219 A CN202110612219 A CN 202110612219A CN 113360151 A CN113360151 A CN 113360151A
Authority
CN
China
Prior art keywords
control
partition
resolution size
user interface
starting
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110612219.7A
Other languages
Chinese (zh)
Other versions
CN113360151B (en
Inventor
杨圣洪
李肯立
林灏昶
蔡宇辉
秦云川
吴帆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hunan University
Zhongdian Jinxin Software Co Ltd
Original Assignee
Hunan University
Zhongdian Jinxin Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hunan University, Zhongdian Jinxin Software Co Ltd filed Critical Hunan University
Priority to CN202110612219.7A priority Critical patent/CN113360151B/en
Publication of CN113360151A publication Critical patent/CN113360151A/en
Application granted granted Critical
Publication of CN113360151B publication Critical patent/CN113360151B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/30—Creation or generation of source code
    • G06F8/38—Creation or generation of source code for implementing user interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/40—Transformation of program code
    • G06F8/41—Compilation

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Digital Computer Display Output (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

本发明公开了一种用于RPA系统的UI数据集自动生成方法，包括：生成GUI程序源码，对GUI程序源码进行自动编译源码处理，生成GUI程序可执行文件，将GUI程序可执行文件自动运行，以生成用户界面窗口，将用户界面窗口自动进行截图处理，以生成用户界面数据集的图像文件，将GUI程序源码进行自动解析，以生成用户界面数据集的标注信息文件，将用户界面数据集的图像文件和用户界面数据集的标注信息文件进行打包，从而生成针对RPA系统目标检测模型用户界面数据集。本发明能够解决现有用于RPA系统的目标检测数据集的建立方法中存在的整个建立过程步骤繁琐，成本高，耗时长的技术问题，以及由于需要人为标注目标区间，导致人为的不确定因素影响数据集精度的技术问题。

Figure 202110612219

The invention discloses a method for automatically generating a UI data set for an RPA system. , to generate a user interface window, automatically perform screenshot processing of the user interface window to generate an image file of the user interface data set, automatically parse the GUI program source code to generate the annotation information file of the user interface data set, and convert the user interface data set. The image file and the annotation information file of the user interface dataset are packaged to generate a user interface dataset for the RPA system target detection model. The invention can solve the technical problems of cumbersome steps, high cost and long time in the entire establishment process in the existing method for establishing the target detection data set for the RPA system, as well as the need to manually mark the target interval, resulting in the influence of artificial uncertain factors Technical issues with dataset accuracy.

Figure 202110612219

Description

UI data set automatic generation method and system for RPA system
Technical Field
The invention belongs to the technical field of User Interface (UI) and computer image recognition, and particularly relates to a method and a system for automatically generating a UI data set for an RPA system.
Background
Robot Process Automation (RPA) is a Process task automatically executed according to rules by simulating human operations on a computer through specific robot software.
At present, the method for establishing a target detection data set for an RPA system specifically includes: firstly, labeling a target area of an image by using labeling software, artificially delimiting a target interval and giving a labeling label value; then, putting the target image picture file and the image annotation information file into corresponding folders; finally, the picture file and the markup file are associated with each other.
However, the above method has some non-negligible drawbacks: firstly, the whole establishing process has complicated steps, high cost and long consumed time; secondly, the method needs to artificially mark the target interval, but the artificial uncertain factors can have certain influence on the accuracy of the data set.
Disclosure of Invention
Aiming at the defects or improvement requirements in the prior art, the invention provides a method and a system for automatically generating a UI (user interface) data set for an RPA (resilient packet access) system, and aims to solve the technical problems of complicated steps, high cost and long time consumption of the whole establishing process in the existing method for establishing a target detection data set for the RPA system and the technical problem that the precision of the data set is influenced by artificial uncertain factors due to the fact that target intervals need to be artificially marked.
To achieve the above object, according to one aspect of the present invention, there is provided an automatic UI data set generation method for an RPA system, comprising the steps of:
(1) generating a GUI program source code;
(2) and (3) automatically compiling the source code of the GUI program obtained in the step (1) to generate a GUI program executable file.
(3) Automatically operating the GUI program executable file obtained in the step (2) to generate a user interface window;
(4) and (4) automatically carrying out screenshot processing on the user interface window obtained in the step (3) to generate an image file of the user interface data set.
(5) And (3) automatically analyzing the GUI program source code obtained in the step (1) to generate a labeling information file of the user interface data set.
(6) And (5) packing the image file of the user interface data set obtained in the step (4) and the label information file of the user interface data set obtained in the step (5), thereby generating a user interface data set aiming at the RPA system target detection model.
Preferably, step (1) comprises the sub-steps of:
(1-1) acquiring a screen size through an interface of an operating system, and further acquiring a range of a user interface window.
(1-2) setting a start coordinate of the user interface window (which comprises a start abscissa and a start ordinate of the user interface window), randomly selecting a resolution size (which comprises a height value and a width value of the resolution size of the user interface window) and a partition number of the user interface window within the range of the user interface window obtained in the step (1-1), and determining a start coordinate (which comprises the start abscissa and the start ordinate of the partition) and a resolution size (which comprises the width value and the height value of the resolution size of the partition) of each partition according to the resolution size and the partition number of the user interface window. And determines the number of controls in each partition.
(1-3) according to the number of the partitions, the starting coordinate and the resolution size of each partition and the number of the controls in each partition obtained in the step (1-2), obtaining the starting coordinate of each control frame in the partition (which comprises the starting abscissa and the starting ordinate of the control frame) and the resolution size of the control frame (which comprises the width value and the height value of the resolution size of the control frame), randomly obtaining a control (which specifically comprises a button, an input frame, a text frame and a file list), determining the starting coordinate of the control (which comprises the starting abscissa and the starting ordinate of the control) according to the resolution size of the control frame, determining the resolution size of the control (which comprises the width value and the height value of the resolution size of the control) according to the control type of the control, and determining the control type of the control, And combining the starting coordinates of the control and the resolution size of the control into a control information list.
And (1-4) generating a GUI program source code according to the control information list obtained in the step (1-3).
Preferably, the start coordinate of the user interface window includes a start abscissa of the user interface window and a start ordinate of the user interface window, both of which are 0;
the height value of the resolution size of the user interface window is randomly acquired between one third of the screen resolution height value and the screen resolution height value;
the width value of the resolution size of the user interface window is R of the width value of the resolution size of the user interface windoww Multiple, Rw ∈[1,2.5]；
And if the width value of the resolution size of the user interface window exceeds the width value of the resolution size of the screen, re-acquiring the width value of the resolution size of the user interface window.
Preferably, the number of partitions, K, is a randomly derived integer and has K ∈ [1,4 ].
If the partition number K is 1, the user interface window is a first partition, the initial coordinate of the first partition is equal to the initial coordinate of the user interface window, and the resolution size of the first partition is equal to the resolution size of the user interface window;
if the number of the partitions K is 2, the user interface window is divided into a first partition and a second partition, wherein the width value of the resolution size of each partition is equal to the width value of the resolution size of the user interface window divided by the number of the partitions, and the height value of the resolution size of each partition is equal to the height value of the resolution size of the user interface window; the starting abscissa of the second partition is the starting abscissa of the first partition plus the width value of the resolution size of the first partition, and the starting ordinate of the second partition is equal to the starting ordinate of the first partition.
If the number of the partitions K is 3, the user interface window is divided into a first partition, a second partition and a third partition. Wherein the height value of the resolution size of the first partition is the height value R of the window resolution size of the user interfacewh31Multiple, Rwh31∈[0.1，0.3](ii) a The width value of the resolution size of the first partition is equal to the width value of the resolution size of the user interface window; the width value of the resolution size of the partition number two is R of the width of the resolution size of the user interface windowww32Multiple, Rwh32∈[0.2，0.4]The height value of the resolution size of the partition number two is equal to the height value of the resolution size of the user interface window minus the height value of the resolution size of the partition number one. The starting ordinate of the second partition is the starting ordinate of the first partition plus the height value of the resolution size of the first partition. The starting abscissa of the second partition is equal to the starting abscissa of the second partition; the remainder of the user interface window may be divided into three sections.
If the number of the partitions K is 4, the user interface window is divided into a first partition, a second partition, a third partition and a fourth partition. Wherein the height value of the resolution size of the first partition is the height value R of the resolution size of the user interface windowwh41Multiple, Rwh41∈[0.1，0.3](ii) a The width value of the resolution size of the first partition is equal to the width value of the resolution size of the user interface window; the width value of the resolution size of the second partition is the width value R of the resolution size of the user interface windowwh42Multiple, Rww42∈[0.2，0.4]The height value of the resolution size of the second partition is the height value R of the resolution size of the user interface windowwh42Height value of resolution size of multiple minus one division, Rwh42∈[0.6，0.85]. The initial ordinate of the partition number two is equal to the height value of the initial ordinate of the partition number one plus the resolution size of the partition number one, andthe starting abscissa of the number zone is equal to the starting abscissa of the number one zone; the width value of the resolution size of the third partition is the width value of the resolution size of the user interface window minus the width value of the resolution size of the second partition, and the height value of the resolution size of the third partition is equal to the height value of the resolution size of the second partition. The initial ordinate of the third partition is equal to the initial ordinate of the second partition, and the initial abscissa of the third partition is equal to the initial abscissa of the second partition plus the width value of the resolution size of the second partition; the remainder of the user interface window may be divided into four sections.
Preferably, if the partition number K is 1, the number of controls C in the first partition1Is a random integer and has C1∈[6，12]。
If the partition number K is 2, the number of controls C in the first partition21Is a random integer and has C21∈[4，8]Number of controls C in partition number two22Is a random integer and has C22∈[4，8]
If the partition number K is 3, the number of controls C in the first partition31Is a random integer and has C31∈[0，2]Number of controls C in partition number two32Is a random integer and has C32∈[0，2]Number of controls C in partition number three33Is a random integer and has C33∈[6，10]。
If the partition number K is 4, the number of controls C in the first partition41Is a random integer and has C41∈[0，2]Number of controls C in partition number two42Is a random integer and has C42∈[0，2]Number of controls C in partition number three43Is a random integer and has C43∈[6，10]Number of controls C in partition number four44Is a random integer and has C44∈[0，2]。
Preferably, the start coordinates of the control frame
Figure BDA0003096323300000055
And ycanvas_ijAnd resolution size W of control framecanvasAnd HcanvasIs obtained by the following function:
Figure BDA0003096323300000051
Figure BDA0003096323300000052
xcavas_ij＝i*Wcanvas，i∈[0，Krow-1]
ycanvas_ij＝j*Hcanvas，j∈[0，Kcolumn-1]
wherein:
Krow＝Rand(S)，S∈[1，2]
Figure BDA0003096323300000053
wherein KrowFor the number of controls in each row within each partition, which is obtained by a random function Rand (), KcolumnFor the number of column controls, the number of controls in each column in each partition is indicated, WareaWidth value of resolution size for each partition, HareaHeight value, W, for resolution size of each partitioncanvasWidth value, H, representing resolution size of control framecanvasA height value representing the resolution size of the control box.
Figure BDA0003096323300000054
And ycanvas_ijAnd respectively representing the starting abscissa and the starting ordinate of the control box in the ith row and the jth column in each partition.
Preferably, the height value of the resolution size of the control is T times of the height value of the resolution size of the control frame, and T belongs to [0.3, 0.7 ];
the width value of the resolution size of the control is X times of the height value of the resolution size of the control, and X is a natural number; wherein:
if the control type is a button, X belongs to [1, 2 ];
if the control type is an input box, X belongs to [2, 5 ];
if the control type is a text box, X belongs to [2, 5 ];
if the control type is a file list, then X belongs to [1, 3 ].
Preferably, the starting coordinate of the control is determined by the position of the control;
if the position of the control in the control frame is deviated to the left, the starting abscissa of the control is P of the width value of the resolution size of the control framewX is wherein PwIs a random number and has Pw∈[0.05，0.15]The initial ordinate of the control is the height value P of the resolution size of the control framehMultiple, PhIs a random number and has Ph∈[0.05，0.15](ii) a If the width value of the starting abscissa of the control plus the resolution size of the control exceeds the width value of the resolution size of the control picture frame, randomly acquiring the width value of the resolution size of the control again; and if the height value of the initial ordinate of the control plus the resolution size of the control exceeds the height value of the resolution size of the control picture frame, randomly acquiring the height value of the resolution size of the control again.
If the position of the control is located in the middle of the control frame, acquiring a central value of the control frame (the abscissa of the central value is equal to one half of the width value of the resolution size of the control frame, and the ordinate of the central value is equal to one half of the height value of the resolution size of the control frame) and a central value of the control (the abscissa of the central value is equal to one half of the width value of the resolution size of the control frame, and the ordinate of the central value is equal to one half of the height value of the resolution size of the control). Subtracting the abscissa of the central value of the control frame from the abscissa of the central value of the control to obtain the initial abscissa of the control, and subtracting the ordinate of the central value of the control frame from the ordinate of the central value of the control to obtain the initial ordinate of the control.
If the position of the control is positioned on the right side of the control frame, the control termination width value is P of the width value of the resolution size of the control framewThe number of times of the total number of the parts,Pwis a random number and has Pw∈[0.85，0.95]. P with the control termination height value being the height value of the resolution size of the control framehMultiple, PhIs a random number and has Ph∈[0.85，0.95](ii) a The initial abscissa of the control is the width value obtained by subtracting the resolution size of the control from the control termination width value, and the initial ordinate of the control is the height value obtained by subtracting the resolution size of the control from the control termination height value; if the width value of the starting abscissa of the control plus the resolution size of the control exceeds the width value of the resolution size of the control picture frame, randomly acquiring the width value of the resolution size of the control again; and if the height value of the initial ordinate of the control plus the resolution size of the control exceeds the height value of the resolution size of the control picture frame, randomly acquiring the height value of the resolution size of the control again.
Preferably, step (5) is specifically: and (4) analyzing the file content in the GUI program source code to extract the control type, the initial coordinate of the control and the resolution size of the control in the file content, and writing the file into the annotation information file with the same name as the image file of the user interface data set obtained in the step (4), so as to generate the annotation information file of the user interface data set.
According to another aspect of the present invention, there is provided an automatic UI data set generation system for an RPA system, comprising:
the first module is used for generating GUI program source codes;
and the second module is used for automatically compiling the source code of the GUI program obtained by the second module to generate the GUI program executable file.
The third module is used for automatically operating the GUI program executable file obtained by the second module so as to generate a user interface window;
and the fourth module is used for automatically carrying out screenshot processing on the user interface window obtained by the third module so as to generate an image file of the user interface data set.
And the fifth module is used for automatically analyzing the GUI program source code obtained by the first module so as to generate a labeling information file of the user interface data set.
And the sixth module is used for packaging the image file of the user interface data set obtained by the fourth module and the label information file of the user interface data set obtained by the fifth module, so as to generate a user interface data set aiming at the RPA system target detection model.
In general, compared with the prior art, the above technical solution contemplated by the present invention can achieve the following beneficial effects:
(1) the invention adopts the steps (1) to (6) to automatically generate the image file of the user interface data set and automatically generate the label information file of the user interface data set, thereby solving the technical problems of complicated steps, high cost and long time consumption of the existing target detection data set establishment method for the RPA system;
(2) the invention adopts the step (5) to automatically analyze the source code of the GUI program and obtain the control type of the control in the source code, the initial coordinate of the control and the resolution size of the control, thereby solving the technical problem that the prior target detection data set establishment method for the RPA system needs to artificially mark the target interval, and the artificial uncertain factors can have certain influence on the precision of the data set
Drawings
FIG. 1 is a flow chart of a UI dataset auto-generation method for an RPA system of the present invention;
FIG. 2 is a detailed flow chart of step (1) of the method of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
The method mainly comprises the steps of randomly generating GUI program source codes, automatically compiling, operating and analyzing, generating image files of user interface data sets and label files of corresponding user interface data sets, and packaging to generate user interface data sets aiming at the RPA system target detection model. On one hand, the GUI program source code is automatically compiled, operated and analyzed, and an image file of a user interface data set and a corresponding label file of the user interface data set are generated; and on the other hand, packing the image file of the user interface data set and the label file of the corresponding user interface data set to generate the user interface data set aiming at the RPA system target detection model.
As shown in fig. 1, the present invention provides a method for automatically generating UI data set for RPA system, comprising the following steps:
(1) generating a GUI program source code;
as shown in fig. 2, step (1) includes the following sub-steps:
(1-1) acquiring a screen size through an interface of an operating system, and further acquiring a range of a user interface window.
(1-2) setting a start coordinate of the user interface window (which comprises a start abscissa and a start ordinate of the user interface window), randomly selecting a resolution size (which comprises a height value and a width value of the resolution size of the user interface window) and a partition number of the user interface window within the range of the user interface window obtained in the step (1-1), and determining a start coordinate (which comprises the start abscissa and the start ordinate of the partition) and a resolution size (which comprises the width value and the height value of the resolution size of the partition) of each partition according to the resolution size and the partition number of the user interface window. And determines the number of controls in each partition.
Take the height of the screen resolution size as 1980 and the width as 1080 as an example. The start coordinate of the user interface window (which includes the start abscissa of the user interface window and the start ordinate of the user interface window) is set to 0. The height value of the resolution size of the user interface window is randomly acquired between 360 (which includes one third of the screen resolution height value) and 1080 (which includes the screen resolution height value), and the width value of the resolution size of the user interface window is R of the width value of the resolution size of the user interface windowwlMultiple, Rwl∈[1，2.5]. Wherein if the user isAnd if the width value of the resolution size of the interface window exceeds the width value of the resolution size of the screen, re-acquiring the width value of the resolution size of the user interface window.
The step (1-2) has the advantages that the user interface window is ensured to be wholly positioned in the screen, and the adverse condition that part of the user interface window exceeds the display range of the screen is avoided, so that the generation effect of the data set picture file is influenced.
The number of partitions, K, is a randomly derived integer and has K ∈ [1,4 ].
Specifically, if the number of the partitions K is 1, the user interface window is a first partition, the initial coordinate of the first partition is equal to the initial coordinate of the user interface window, and the resolution size of the first partition is equal to the resolution size of the user interface window;
if the number of the partitions K is 2, the user interface window is divided into a first partition and a second partition, wherein the width value of the resolution size of each partition is equal to the width value of the resolution size of the user interface window divided by the number of the partitions, and the height value of the resolution size of each partition is equal to the height value of the resolution size of the user interface window;
the starting abscissa of the second partition is the starting abscissa of the first partition plus the width value of the resolution size of the first partition, and the starting ordinate of the second partition is equal to the starting ordinate of the first partition.
If the number of the partitions K is 3, the user interface window is divided into a first partition, a second partition and a third partition. Wherein the height value of the resolution size of the first partition is the height value R of the window resolution size of the user interfacewh31Multiple, Rwh31∈[0.1，0.3](ii) a The width value of the resolution size of the first partition is equal to the width value of the resolution size of the user interface window.
The width value of the resolution size of the partition number two is R of the width of the resolution size of the user interface windowww32Multiple, Rwh32∈[0.2，0.4]The height value of the resolution size of the partition number two is equal to the height value of the resolution size of the user interface window minus the height value of the resolution size of the partition number one. Second branchThe start ordinate of the zone is the start ordinate of the zone number one plus the height value of the resolution size of the zone number one. The starting abscissa of the second partition is equal to the starting abscissa of the second partition.
The remainder of the user interface window may be divided into three sections.
If the number of the partitions K is 4, the user interface window is divided into a first partition, a second partition, a third partition and a fourth partition. Wherein the height value of the resolution size of the first partition is the height value R of the resolution size of the user interface windowwh41Multiple, Rwh41∈[0.1，0.3](ii) a The width value of the resolution size of the first partition is equal to the width value of the resolution size of the user interface window.
The width value of the resolution size of the second partition is the width value R of the resolution size of the user interface windowwh42Multiple, Rww42∈[0.2，0.4]The height value of the resolution size of the second partition is the height value R of the resolution size of the user interface windowwh42Height value of resolution size of multiple minus one division, Rwh42∈[0.6，0.85]. The starting ordinate of the partition No. two is equal to the starting ordinate of the partition No. one plus the height value of the resolution size of the partition No. one, and the starting abscissa of the partition No. two is equal to the starting abscissa of the partition No. one.
The width value of the resolution size of the third partition is the width value of the resolution size of the user interface window minus the width value of the resolution size of the second partition, and the height value of the resolution size of the third partition is equal to the height value of the resolution size of the second partition. The initial ordinate of the third partition is equal to the initial ordinate of the second partition, and the initial abscissa of the third partition is equal to the initial abscissa of the second partition plus the width value of the resolution size of the second partition.
The remainder of the user interface window may be divided into four sections.
If the partition number K is 1, the number of controls C in the first partition1Is a random integer and has C1∈[6，12]。
If the partition number K is 2, the number of controls C in the first partition21Is a random integer and has C21∈[4，8]Number of controls C in partition number two22Is a random integer and has C22∈[4，8]
If the partition number K is 3, the number of controls C in the first partition31Is a random integer and has C31∈[0，2]Number of controls C in partition number two32Is a random integer and has C32∈[0，2]Number of controls C in partition number three33Is a random integer and has C33∈[6，10]。
If the partition number K is 4, the number of controls C in the first partition41Is a random integer and has C41∈[0，2]Number of controls C in partition number two42Is a random integer and has C42∈[0，2]Number of controls C in partition number three43Is a random integer and has C43∈[6，10]Number of controls C in partition number four44Is a random integer and has C44∈[0，2]。
The advantage of the above step (1-2) is that the concept of partitioning is also applied because of the design of user interface windows in reality. According to different partitions, different operation effects and operation logics exist, and correspondingly, different control numbers exist. Therefore, the steps ensure that the generated user interface window has practical significance to a certain extent.
(1-3) according to the number of the partitions, the starting coordinate and the resolution size of each partition and the number of the controls in each partition obtained in the step (1-2), obtaining the starting coordinate of each control frame in the partition (which comprises the starting abscissa and the starting ordinate of the control frame) and the resolution size of the control frame (which comprises the width value and the height value of the resolution size of the control frame), randomly obtaining a control (which specifically comprises a button, an input frame, a text frame and a file list), determining the starting coordinate of the control (which comprises the starting abscissa and the starting ordinate of the control) according to the resolution size of the control frame, determining the resolution size of the control (which comprises the width value and the height value of the resolution size of the control) according to the control type of the control, and determining the control type of the control, And combining the starting coordinates of the control and the resolution size of the control into a control information list.
In particular, the start coordinates of the control frame
Figure BDA0003096323300000125
And ycanvas_ijAnd resolution size W of control framecanvasAnd HcanvasIs obtained by the following function:
Figure BDA0003096323300000121
Figure BDA0003096323300000122
xcavas_ij＝i*Wcanvas，i∈[0，Krow-1]
ycanvas_ij＝j*Hcanvas，j∈[0，Kcolumn-1]
wherein:
Krow＝Rand(S)，S∈[1，2]
Figure BDA0003096323300000123
wherein KrowFor the number of controls in each row within each partition, which is obtained by a random function Rand (), KcolumnFor the number of column controls, the number of controls in each column in each partition is indicated, WareaWidth value of resolution size for each partition, HareaHeight value, W, for resolution size of each partitioncanvasWidth value, H, representing resolution size of control framecanvasA height value representing the resolution size of the control box.
Figure BDA0003096323300000124
And ycanvas_ijRespectively representing the control box in the ith row and the jth column in each partitionThe starting abscissa and the starting ordinate.
The height value of the resolution size of the control is T times of the height value of the resolution size of the control frame, and T belongs to [0.3, 0.7], the width value of the resolution size of the control is X times of the height value of the resolution size of the control, X is a natural number, wherein if the control type is a button, X belongs to [1, 2 ]; if the control type is an input box, X belongs to [2, 5 ]; if the control type is a text box, X belongs to [2, 5 ]; if the control type is a file list, then X belongs to [1, 3 ].
The step (1-3) has the advantages that the resolution size proportion of the control is randomized, the diversity of the resolution sizes of the control is ensured, and the generated resolution size proportion of the control is in a certain range, so that the control has certain practical visual significance.
The starting coordinates of the control are determined by where the control is located.
If the position of the control in the control frame is deviated to the left, the starting abscissa of the control is P of the width value of the resolution size of the control framewX is wherein PwIs a random number and has Pw∈[0.05，0.15]The initial ordinate of the control is the height value P of the resolution size of the control framehMultiple, PhIs a random number and has Ph∈[0.05，0.15]。
If the width value of the initial abscissa of the control plus the resolution size of the control exceeds the width value of the resolution size of the control picture frame, randomly acquiring the width value of the resolution size of the control again; and if the height value of the initial ordinate of the control plus the resolution size of the control exceeds the height value of the resolution size of the control picture frame, randomly acquiring the height value of the resolution size of the control again.
If the position of the control is located in the middle of the control frame, acquiring a central value of the control frame (the abscissa of the central value is equal to one half of the width value of the resolution size of the control frame, and the ordinate of the central value is equal to one half of the height value of the resolution size of the control frame) and a central value of the control (the abscissa of the central value is equal to one half of the width value of the resolution size of the control frame, and the ordinate of the central value is equal to one half of the height value of the resolution size of the control). Subtracting the abscissa of the central value of the control frame from the abscissa of the central value of the control to obtain the initial abscissa of the control, and subtracting the ordinate of the central value of the control frame from the ordinate of the central value of the control to obtain the initial ordinate of the control.
If the position of the control is positioned on the right side of the control frame, the control termination width value is P of the width value of the resolution size of the control framewMultiple, PwIs a random number and has Pw∈[0.85，0.95]. P with the control termination height value being the height value of the resolution size of the control framehMultiple, PhIs a random number and has Ph∈[0.85，0.95]. The starting abscissa of the control is the width value of the ending width value of the control minus the resolution size of the control, and the starting ordinate of the control is the height value of the ending height value of the control minus the resolution size of the control.
If the width value of the initial abscissa of the control plus the resolution size of the control exceeds the width value of the resolution size of the control picture frame, randomly acquiring the width value of the resolution size of the control again; and if the height value of the initial ordinate of the control plus the resolution size of the control exceeds the height value of the resolution size of the control picture frame, randomly acquiring the height value of the resolution size of the control again.
The step (1-3) has the advantages that the position of each control is ensured to be located in the control frame, that is, the controls are not overlapped, and the size of the resolution of the generated control is significant.
And (1-4) generating a GUI program source code according to the control information list obtained in the step (1-3).
(2) And (3) automatically compiling the source code of the GUI program obtained in the step (1) to generate a GUI program executable file.
The method comprises the following specific steps: and (3) automatically processing the GUI program source code obtained in the step (1) by using a mode of simulating command line input so as to compile the source code into a GUI program executable file.
(3) Automatically operating the GUI program executable file obtained in the step (2) to generate a user interface window;
the method comprises the following steps: and (3) automatically running the GUI program executable file obtained in the step (2) by using a mode of simulating command line input so as to generate a user interface window.
(4) And (4) automatically carrying out screenshot processing on the user interface window obtained in the step (3) to generate an image file of the user interface data set.
The method comprises the following steps: and (4) automatically performing screenshot processing on the user interface window obtained in the step (3) by using a mode of simulating command line input, saving the result as an image file, closing the user interface window after one second, and taking the image file as the image file of the user interface data set.
The step (3) and the step (4) have the advantages that the user interface window is closed regularly, and the integrity of the running logic of the GUI program is ensured. And when the operation life of the GUI program is finished, the GUI program is terminated in time, the memory is recycled, the memory overflow is prevented, and the operation robustness of the GUI program is enhanced.
(5) And (3) automatically analyzing the GUI program source code obtained in the step (1) to generate a labeling information file of the user interface data set.
The method comprises the following steps: analyzing the file content in the GUI program source code to extract the control type, the initial coordinate of the control and the resolution size of the control in the file content, and writing the file into the annotation information file with the same name as the image file of the user interface data set obtained in the step (4), so as to generate the annotation information file of the user interface data set;
(6) packing the image file of the user interface data set obtained in the step (4) and the label information file of the user interface data set obtained in the step (5) so as to generate a user interface data set aiming at an RPA system target detection model;
it can be understood that the invention aims at the problem of overlarge data integration cost of manually constructing a target detection model user interface of an RPA system, and realizes an automatic generation method capable of automatically and quickly generating a UI data set for the RPA system. The automatic generation of the user interface data set aiming at the RPA system target detection model is successfully established, and the consumption of the enterprise material and financial cost and the time cost is reduced.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims (10)

1.一种用于RPA系统的UI数据集自动生成方法，其特征在于，包括如下步骤：1. a UI data set automatic generation method for RPA system, is characterized in that, comprises the steps: (1)生成GUI程序源码；(1) Generate GUI program source code; (2)对步骤(1)中得到的GUI程序源码进行自动编译源码处理，生成GUI程序可执行文件。(2) Perform automatic compilation source code processing on the GUI program source code obtained in step (1) to generate a GUI program executable file. (3)将步骤(2)中得到的GUI程序可执行文件自动运行，以生成用户界面窗口；(3) the GUI program executable file obtained in the step (2) is automatically run to generate a user interface window; (4)将步骤(3)中得到的用户界面窗口自动进行截图处理，以生成用户界面数据集的图像文件。(4) Automatically perform screenshot processing on the user interface window obtained in step (3) to generate an image file of the user interface data set. (5)将步骤(1)得到的GUI程序源码进行自动解析，以生成用户界面数据集的标注信息文件。(5) Automatically parse the GUI program source code obtained in step (1) to generate an annotation information file of the user interface data set. (6)将步骤(4)得到的用户界面数据集的图像文件和步骤(5)得到的用户界面数据集的标注信息文件进行打包，从而生成针对RPA系统目标检测模型用户界面数据集。(6) Package the image file of the user interface data set obtained in step (4) and the annotation information file of the user interface data set obtained in step (5), thereby generating a user interface data set for the RPA system target detection model. 2.根据权利要求1所述的用于RPA系统的UI数据集自动生成方法，其特征在于，步骤(1)包括以下子步骤：2. the UI data set automatic generation method for RPA system according to claim 1, is characterized in that, step (1) comprises following substep: (1-1)通过操作系统的接口获取屏幕尺寸大小，并进而获得用户界面窗口的范围。(1-1) Obtain the screen size through the interface of the operating system, and then obtain the scope of the user interface window. (1-2)设置用户界面窗口的起始坐标(其包括用户界面窗口的起始横坐标与起始纵坐标)，在步骤(1-1)得到的用户界面窗口的范围内随机选择用户界面窗口的分辨率尺寸(其包括用户界面窗口的分辨率尺寸的高度值与宽度值)和分区数量，根据用户界面窗口的分辨率尺寸和分区数量确定每个分区的起始坐标(其包括分区的起始横坐标与起始纵坐标)与分辨率尺寸(其包括该分区的分辨率尺寸的宽度值与高度值)。并确定每个分区中的控件数量。(1-2) Set the initial coordinates of the user interface window (which includes the initial abscissa and the initial ordinate of the user interface window), and randomly select a user interface within the range of the user interface window obtained in step (1-1) The resolution size of the window (which includes the height value and width value of the resolution size of the user interface window) and the number of partitions, the starting coordinates of each partition (which includes the resolution size of the user interface window and the number of partitions) The starting abscissa and starting ordinate) and the resolution size (which includes the width and height values of the resolution size of the partition). And determine the number of controls in each partition. (1-3)根据步骤(1-2)得到的分区数量、每个分区的起始坐标与分辨率尺寸、以及每个分区中的控件数量，获取该分区内每个控件画框的起始坐标(其包括控件画框的起始横坐标和起始纵坐标)、以及该控件画框的分辨率尺寸(其包括控件画框的分辨率尺寸的宽度值与高度值)，随机获取控件(其具体包括按钮、输入框、文本框、以及文件列表)，根据该控件画框的分辨率尺寸确定该控件的起始坐标(其包括控件的起始横坐标与起始纵坐标)，根据该控件的控件类型确定该控件的分辨率尺寸(其包括控件的分辨率尺寸的宽度值和高度值)，并将该控件的控件类型、控件的起始坐标、控件的分辨率尺寸合并成控件信息列表。(1-3) According to the number of partitions obtained in step (1-2), the starting coordinates and resolution size of each partition, and the number of controls in each partition, obtain the start of each control frame in the partition Coordinates (which include the starting abscissa and starting ordinate of the control picture frame), and the resolution size of the control picture frame (which includes the width value and height value of the resolution size of the control picture frame), randomly obtain the control ( It specifically includes a button, an input box, a text box, and a file list), according to the resolution size of the control picture frame to determine the starting coordinate of the control (which includes the starting abscissa and starting ordinate of the control), according to the The control type of the control determines the resolution size of the control (which includes the width value and height value of the resolution size of the control), and combines the control type of the control, the starting coordinates of the control, and the resolution size of the control into control information list. (1-4)根据步骤(1-3)得到的控件信息列表生成GUI程序源码。(1-4) Generate GUI program source code according to the control information list obtained in step (1-3). 3.根据权利要求1或2所述的用于RPA系统的UI数据集自动生成方法，其特征在于，3. the UI data set automatic generation method for RPA system according to claim 1 and 2, is characterized in that, 用户界面窗口的起始坐标包括用户界面窗口的起始横坐标与用户界面窗口的起始纵坐标，二者均为0；The initial coordinates of the user interface window include the initial horizontal coordinate of the user interface window and the initial vertical coordinate of the user interface window, both of which are 0; 用户界面窗口的分辨率尺寸的高度值是在屏幕分辨率高度值的三分之一与屏幕分辨率高度值之间随机获取；The height value of the resolution size of the user interface window is randomly obtained between one-third of the screen resolution height value and the screen resolution height value; 用户界面窗口的分辨率尺寸的宽度值为用户界面窗口的分辨率尺寸的宽度值的Rwl倍，Rwl∈[1,2.5]；The width value of the resolution size of the user interface window is R wl times the width value of the resolution size of the user interface window, R wl ∈ [1, 2.5]; 若用户界面窗口的分辨率尺寸的宽度值超过屏幕分辨率尺寸的宽度值，则重新获取用户界面窗口的分辨率尺寸的宽度值。If the width value of the resolution size of the user interface window exceeds the width value of the screen resolution size, the width value of the resolution size of the user interface window is reacquired. 4.根据权利要求1至3中任意一项所述的用于RPA系统的UI数据集自动生成方法，其特征在于，4. the UI data set automatic generation method for RPA system according to any one of claims 1 to 3, is characterized in that, 分区数量K是随机获取的整数，且有K∈[1,4]。The number of partitions K is a randomly obtained integer and has K∈[1,4]. 若分区数量K为1，则用户界面窗口为一号分区，一号分区的起始坐标等于用户界面窗口的起始坐标，一号分区的分辨率尺寸就等于用户界面窗口的分辨率尺寸；If the number of partitions K is 1, the user interface window is the first partition, the starting coordinates of the first partition are equal to the starting coordinates of the user interface window, and the resolution size of the first partition is equal to the resolution size of the user interface window; 若分区数量K为2，则用户界面窗口会被划分为一号分区与二号分区，其中每个分区的分辨率尺寸的宽度值就等于该用户界面窗口的分辨率尺寸的宽度值除以分区数量，每个分区的分辨率尺寸的高度值等于用户界面窗口的分辨率尺寸的高度值；第二分区的起始横坐标为一号分区的起始横坐标加上一号分区的分辨率尺寸的宽度值，第二分区的起始纵坐标等于一号分区的起始纵坐标。If the number of partitions K is 2, the user interface window will be divided into the first partition and the second partition, and the width value of the resolution size of each partition is equal to the width value of the resolution size of the user interface window divided by the partition. Quantity, the height value of the resolution size of each partition is equal to the height value of the resolution size of the user interface window; the starting abscissa of the second partition is the starting abscissa of the first partition plus the resolution size of the first partition The width value of , the starting ordinate of the second partition is equal to the starting ordinate of the first partition. 若分区数量K为3，用户界面窗口会被划分为一号分区、二号分区与三号分区。其中一号分区的分辨率尺寸的高度值为用户界面的窗口分辨率尺寸的高度值Rwh31倍，Rwh31∈[0.1,0.3]；一号分区的分辨率尺寸的宽度值与用户界面窗口的分辨率尺寸的宽度值相等；二号分区的分辨率尺寸的宽度值为户界面窗口的的分辨率尺寸的宽度的Rww32倍，Rwh32∈[0.2,0.4]，二号分区的分辨率尺寸的高度值等于用户界面窗口的分辨率尺寸的高度值减去一号分区的分辨率尺寸的高度值。第二分区的起始纵坐标为一号分区的起始纵坐标加上一号分区的分辨率尺寸的高度值。第二分区的起始横坐标等于第二分区的起始横坐标；用户界面窗口其余部分会被划分为三号区间。If the number of partitions K is 3, the user interface window will be divided into the first partition, the second partition and the third partition. The height value of the resolution size of partition No. 1 is R wh31 times the height value of the window resolution size of the user interface, R wh31 ∈ [0.1, 0.3]; the width value of the resolution size of partition No. 1 is the same as that of the user interface window The width value of the resolution size is equal; the width value of the resolution size of the second partition is R ww32 times the width of the resolution size of the user interface window, R wh32 ∈ [0.2, 0.4], the resolution size of the second partition The height value is equal to the height value of the resolution size of the UI window minus the height value of the resolution size of the first partition. The starting ordinate of the second partition is the starting ordinate of the first partition plus the height value of the resolution size of the first partition. The starting abscissa of the second partition is equal to the starting abscissa of the second partition; the rest of the user interface window will be divided into the third interval. 若分区数量K为4，用户界面窗口会被划分为一号分区、二号分区、三号分区与四号区间。其中一号分区的分辨率尺寸的高度值为用户界面窗口的分辨率尺寸的高度值Rwh41倍，Rwh41∈[0.1,0.3]；一号分区的分辨率尺寸的宽度值与用户界面窗口的分辨率尺寸的宽度值相等；二号分区的分辨率尺寸的宽度值为用户界面窗口的分辨率尺寸的宽度值Rwh42倍，Rww42∈[0.2,0.4]，二号分区的分辨率尺寸的高度值为用户界面窗口的分辨率尺寸的高度值Rwh42倍减去一号分区的分辨率尺寸的高度值，Rwhh2∈[0.6,0.85]。二号分区的起始纵坐标等于一号分区的起始纵坐标加上一号分区的分辨率尺寸的高度值，二号分区的起始横坐标等于一号分区的起始横坐标；三号分区的分辨率尺寸的宽度值为用户界面窗口的分辨率尺寸的宽度值减去二号分区的分辨率尺寸的宽度值，三号分区的分辨率尺寸的高度值等于二号分区的分辨率尺寸的高度值。三号分区的起始纵坐标等于二号分区的起始纵坐标，三号分区的起始横坐标等于二号分区的起始横坐标加上二号分区的分辨率尺寸的宽度值；用户界面窗口其余部分会被划分为四号区间。If the number of partitions K is 4, the user interface window will be divided into the first partition, the second partition, the third partition and the fourth partition. The height value of the resolution size of partition No. 1 is R wh41 times the height value of the resolution size of the user interface window, R wh41 ∈ [0.1, 0.3]; the width value of the resolution size of partition No. 1 is the same as that of the user interface window. The width value of the resolution size is equal; the width value of the resolution size of the second partition is R wh42 times the width value of the resolution size of the user interface window, R ww42 ∈ [0.2, 0.4], the resolution size of the second partition is The height value is the height value of the resolution size of the user interface window, R wh42 times minus the height value of the resolution size of the first partition, R whh2 ∈ [0.6, 0.85]. The starting ordinate of the No. 2 partition is equal to the starting ordinate of the No. 1 partition plus the height value of the resolution size of the No. 1 partition. The starting abscissa of the No. 2 partition is equal to the starting abscissa of the No. 1 partition; The width value of the resolution size of the partition is the width value of the resolution size of the user interface window minus the width value of the resolution size of the second partition, and the height value of the resolution size of the third partition is equal to the resolution size of the second partition height value. The starting ordinate of the No. 3 partition is equal to the starting ordinate of the No. 2 partition, and the starting abscissa of the No. 3 partition is equal to the starting abscissa of the No. 2 partition plus the width value of the resolution size of the No. 2 partition; user interface The rest of the window will be divided into four intervals. 5.根据权利要求4所述的用于RPA系统的UI数据集自动生成方法，其特征在于，5. the UI data set automatic generation method for RPA system according to claim 4 is characterized in that, 若分区数量K为1，一号分区中的控件数量C1为随机的整数，且有C1∈[6，12]。If the number of partitions K is 1, the number of controls C 1 in the first partition is a random integer, and there is C 1 ∈ [6, 12]. 若分区数量K为2，一号分区中的控件数量C21为随机的整数，且有C21∈[4，8]，二号分区中的控件数量C22为随机的整数，且有C22∈[4，8]If the number of partitions K is 2, the number of controls C 21 in the first partition is a random integer, and there is C 21 ∈ [4, 8], and the number of controls in the second partition C 22 is a random integer, and there is C 22 ∈ [4, 8] 若分区数量K为3，一号分区中的控件数量C31为随机的整数，且有C31∈[0，2]，二号分区中的控件数量C32为随机的整数，且有C32∈[0，2]，三号分区中的控件数量C33为随机的整数，且有C33∈[6，10]。If the number of partitions K is 3, the number of controls C 31 in the first partition is a random integer, and there is C 31 ∈ [0, 2], the number of controls in the second partition C 32 is a random integer, and there is C 32 ∈ [0, 2], the number of controls C 33 in the third partition is a random integer, and there is C 33 ∈ [6, 10]. 若分区数量K为4，则一号分区中的控件数量C41为随机的整数，且有C41∈[0，2]，二号分区中的控件数量C42为随机的整数，且有C42∈[0，2]，三号分区中的控件数量C43为随机的整数，且有C43∈[6，10]，四号分区中的控件数量C44为随机的整数，且有C44∈[0，2]。If the number of partitions K is 4, the number of controls C 41 in the first partition is a random integer, and there is C 41 ∈ [0, 2], the number of controls in the second partition C 42 is a random integer, and there is C 42 ∈ [0, 2], the number of controls C 43 in the third partition is a random integer, and there is C 43 ∈ [6, 10], the number of controls in the fourth partition C 44 is a random integer, and there is C 44 ∈ [0, 2]. 6.根据权利要求2所述的用于RPA系统的UI数据集自动生成方法，其特征在于，控件画框的起始坐标
Figure FDA0003096323290000043
和ycanvas_ij和控件画框的分辨率尺寸Wcanvas和Hcanvas是通过以下函数获得：6. the UI data set automatic generation method that is used for RPA system according to claim 2 is characterized in that, the starting coordinate of control picture frame
Figure FDA0003096323290000043
and y canvas_ij and the resolution size of the control frame W canvas and H canvas are obtained by the following functions:
Figure FDA0003096323290000041
Figure FDA0003096323290000041
Figure FDA0003096323290000042
Figure FDA0003096323290000042
 xcavas_ij＝i*Wcanvas，i∈[0，Krow-1]x cavas_ij = i*W canvas , i∈[0, K row -1] ycanvas_ij＝j*Hcanvas，j∈[0，Kcolumn-1]y canvas_ij = j*H canvas , j∈[0, K column -1] 其中：in: Krow＝Rand(S)，S∈[1，2]K row = Rand(S), S∈[1, 2]
Figure FDA0003096323290000051
Figure FDA0003096323290000051
 其中Krow为每个分区内每一行中的控件数目，其是通过随机函数Rand()获取，Kcolumn为列控件数，表示每个分区内每一列中的控件数，Warea为每个分区的分辨率尺寸的宽度值，Harea为每个分区的分辨率尺寸的高度值，Wcanvas表示控件画框的分辨率尺寸的宽度值，Hcanvas表示控件画框的分辨率尺寸的高度值。
Figure FDA0003096323290000052
和ycanvas_ij分别表示每个分区中第i行第j列的控件画框的起始横坐标与起始纵坐标。Among them, K row is the number of controls in each row in each partition, which is obtained by the random function Rand(), K column is the number of column controls, indicating the number of controls in each column in each partition, and W area is each partition. The width value of the resolution size, H area is the height value of the resolution size of each partition, W canvas represents the width value of the resolution size of the control picture frame, and H canvas represents the height value of the resolution size of the control picture frame.
Figure FDA0003096323290000052
and y canvas_ij respectively represent the starting abscissa and starting ordinate of the control frame in the i-th row and the j-th column in each partition. 7.根据权利要求6所述的用于RPA系统的UI数据集自动生成方法，其特征在于，7. the UI data set automatic generation method for RPA system according to claim 6, is characterized in that, 控件的分辨率尺寸的高度值为控件画框的分辨率尺寸的高度值的T倍，且有T∈[0.3，0.7]；The height value of the resolution size of the control is T times the height value of the resolution size of the control picture frame, and has T∈[0.3, 0.7]; 控件的分辨率尺寸的宽度值为控件的分辨率尺寸的高度值的X倍，X为自然数；其中：The width value of the resolution size of the control is X times the height value of the resolution size of the control, where X is a natural number; where: 若控件类型为按钮，则X∈[1，2]；If the control type is a button, then X∈[1, 2]; 若控件类型为输入框，则X∈[2，5]；If the control type is an input box, then X∈[2,5]; 若控件类型为文本框，则X∈[2，5]；If the control type is a text box, then X∈[2,5]; 若控件类型为文件列表，则X∈[1，3]。If the control type is a file list, X∈[1, 3]. 8.根据权利要求7所述的用于RPA系统的UI数据集自动生成方法，其特征在于，8. the UI data set automatic generation method for RPA system according to claim 7, is characterized in that, 控件的起始坐标是由该控件所在位置决定的；The starting coordinates of the control are determined by the location of the control; 若控件的位置在控件画框中的位置为偏左，则控件的起始横坐标为控件画框的分辨率尺寸的宽度值的Pw倍，其中Pw为随机数，且有Pw∈[0.05，0.15]，控件的起始纵坐标为控件画框的分辨率尺寸的高度值Ph倍，Ph为随机数，且有Ph∈[0.05,0.15]；其中若控件的起始横坐标加控件的分辨率尺寸的宽度值超过控件画框的分辨率尺寸的宽度值，则再次随机获取控件的分辨率尺寸的宽度值；若控件的起始纵坐标加控件的分辨率尺寸的高度值超过控件画框的分辨率尺寸的高度值，则再次随机获取控件的分辨率尺寸的高度值。If the position of the control in the control frame is to the left, the starting abscissa of the control is P w times the width value of the resolution size of the control frame, where P w is a random number, and there is P w ∈ [0.05, 0.15], the starting ordinate of the control is P h times the height value of the resolution size of the control picture frame, P h is a random number, and has P h ∈ [0.05, 0.15]; If the width value of the abscissa plus the resolution size of the control exceeds the width value of the resolution size of the control picture frame, the width value of the resolution size of the control is randomly obtained again; if the starting ordinate of the control plus the resolution size of the control If the height value exceeds the height value of the resolution size of the control picture frame, the height value of the resolution size of the control is randomly obtained again. 若控件的位置位于控件画框的中间，则获取控件画框的中心值(其横坐标等于控件画框的分辨率尺寸的宽度值的二分之一，其纵坐标等于控件画框的分辨率尺寸的高度值的二分之一)与控件的中心值(其横坐标等于控件的分辨率尺寸的宽度值的二分之一，其纵坐标等于控件的分辨率尺寸的高度值的二分之一)。将控件画框的中心值的横坐标与控件的中心值的横坐标相减，从而得到控件的起始横坐标，将控件画框的中心值的纵坐标与控件的中心值的纵坐标相减，从而得到控件的起始纵坐标。If the position of the control is located in the middle of the control picture frame, get the center value of the control picture frame (its abscissa is equal to half of the width value of the resolution size of the control picture frame, and its ordinate is equal to the resolution of the control picture frame 1/2 of the height value of the size) and the center value of the control (whose abscissa is equal to 1/2 of the width value of the resolution size of the control, and its ordinate is equal to 1/2 of the height value of the resolution size of the control one). Subtract the abscissa of the center value of the control picture frame from the abscissa of the center value of the control to obtain the starting abscissa of the control, and subtract the ordinate of the center value of the control picture frame from the ordinate of the center value of the control , so as to get the starting ordinate of the control. 若控件的位置位于控件画框的右侧，则控件终止宽度值为控件画框的分辨率尺寸的宽度值的Pw倍，Pw为随机数，且有Pw∈[0.85,0.95]。控件终止高度值为控件画框的分辨率尺寸的高度值的Ph倍，Ph为随机数，且有Ph∈[0.85,0.95]；控件的起始横坐标为控件终止宽度值减去控件的分辨率尺寸的宽度值，控件的起始纵坐标为控件终止高度值减去控件的分辨率尺寸的高度值；其中若控件的起始横坐标加控件的分辨率尺寸的宽度值超过控件画框的分辨率尺寸的宽度值，则再次随机获取控件的分辨率尺寸的宽度值；若控件的起始纵坐标加控件的分辨率尺寸的高度值超过控件画框的分辨率尺寸的高度值，则再次随机获取控件的分辨率尺寸的高度值。If the position of the control is on the right side of the control frame, the end width of the control is Pw times the width of the resolution size of the control frame, and Pw is a random number with Pw∈ [ 0.85,0.95 ]. The control termination height value is P h times the height value of the resolution size of the control picture frame, P h is a random number, and has P h ∈ [0.85, 0.95]; the starting abscissa of the control is the control termination width minus the value of the control termination width. The width value of the resolution size of the control, the starting vertical coordinate of the control is the ending height value of the control minus the height value of the resolution size of the control; if the starting abscissa of the control plus the width value of the resolution size of the control exceeds the control The width value of the resolution size of the picture frame, the width value of the resolution size of the control is randomly obtained again; if the starting ordinate of the control plus the height value of the resolution size of the control exceeds the height value of the resolution size of the control picture frame , the height value of the resolution size of the control is randomly obtained again. 9.根据权利要求1所述的用于RPA系统的UI数据集自动生成方法，其特征在于，步骤(5)具体为：解析GUI程序源码中的文件内容，以提取该文件内容中控件的控件类型、控件的起始坐标、以及控件的分辨率尺寸，将其写入与步骤(4)得到的用户界面数据集的图像文件具有相同名字的标注信息文件内，从而生成用户界面数据集的标注信息文件。9. the UI data set automatic generation method that is used for RPA system according to claim 1 is characterized in that, step (5) is specially: analyze the file content in the GUI program source code, to extract the control of the control in this file content type, the starting coordinates of the control, and the resolution size of the control, and write them into the annotation information file with the same name as the image file of the user interface dataset obtained in step (4), thereby generating the annotation of the user interface dataset. information file. 10.一种用于RPA系统的UI数据集自动生成系统，其特征在于，包括：10. A UI data set automatic generation system for RPA system is characterized in that, comprising: 第一模块，用于生成GUI程序源码；The first module is used to generate GUI program source code; 第二模块，用于对第二模块得到的GUI程序源码进行自动编译源码处理，生成GUI程序可执行文件。The second module is used for automatically compiling the source code of the GUI program obtained by the second module, and generating the executable file of the GUI program. 第三模块，用于将第二模块得到的GUI程序可执行文件自动运行，以生成用户界面窗口；The third module is used to automatically run the GUI program executable file obtained by the second module to generate a user interface window; 第四模块，用于将第三模块得到的用户界面窗口自动进行截图处理，以生成用户界面数据集的图像文件。The fourth module is used for automatically taking screenshots of the user interface window obtained by the third module to generate an image file of the user interface data set. 第五模块，用于将第一模块得到的GUI程序源码进行自动解析，以生成用户界面数据集的标注信息文件。The fifth module is used to automatically parse the GUI program source code obtained by the first module to generate an annotation information file of the user interface data set. 第六模块，用于将第四模块得到的用户界面数据集的图像文件和第五模块得到的用户界面数据集的标注信息文件进行打包，从而生成针对RPA系统目标检测模型用户界面数据集。The sixth module is used to package the image file of the user interface data set obtained by the fourth module and the annotation information file of the user interface data set obtained by the fifth module, thereby generating a user interface data set for the target detection model of the RPA system.
CN202110612219.7A 2021-06-02 2021-06-02 UI data set automatic generation method and system for RPA system Active CN113360151B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110612219.7A CN113360151B (en) 2021-06-02 2021-06-02 UI data set automatic generation method and system for RPA system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110612219.7A CN113360151B (en) 2021-06-02 2021-06-02 UI data set automatic generation method and system for RPA system

Publications (2)

Publication Number Publication Date
CN113360151A true CN113360151A (en) 2021-09-07
CN113360151B CN113360151B (en) 2024-03-08

Family

ID=77531077

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110612219.7A Active CN113360151B (en) 2021-06-02 2021-06-02 UI data set automatic generation method and system for RPA system

Country Status (1)

Country Link
CN (1) CN113360151B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114020267A (en) * 2021-11-04 2022-02-08 四川启睿克科技有限公司 Method for detecting RPA window control

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050248545A1 (en) * 2004-05-07 2005-11-10 Takanori Nishimura Method, apparatus, and software program for processing information
WO2016170288A1 (en) * 2015-04-23 2016-10-27 Real Digit System for creating and reproducing a composite document
US20180203571A1 (en) * 2017-01-18 2018-07-19 Oracle International Corporation Generating a graphical user interface model from an image
US20190250891A1 (en) * 2018-02-12 2019-08-15 Oracle International Corporation Automated code generation

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050248545A1 (en) * 2004-05-07 2005-11-10 Takanori Nishimura Method, apparatus, and software program for processing information
WO2016170288A1 (en) * 2015-04-23 2016-10-27 Real Digit System for creating and reproducing a composite document
US20180203571A1 (en) * 2017-01-18 2018-07-19 Oracle International Corporation Generating a graphical user interface model from an image
US20190250891A1 (en) * 2018-02-12 2019-08-15 Oracle International Corporation Automated code generation

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
娄宇;王琼;: "基于XML的嵌入式图形界面自动生成技术", 计算机应用, no. 1 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114020267A (en) * 2021-11-04 2022-02-08 四川启睿克科技有限公司 Method for detecting RPA window control

Also Published As

Publication number Publication date
CN113360151B (en) 2024-03-08

Similar Documents

Publication Publication Date Title
CN111796831B (en) Compiling method and device for multi-chip compatibility
CN1517904B (en) Ink marking device and associated application programmed interface
US20190303107A1 (en) Automated software programming guidance
US7523440B2 (en) Dynamic generation of formatted user interfaces in software environments
CN109753636A (en) Machine processing and text error correction method and device calculate equipment and storage medium
CN110968294B (en) Business domain model establishing system and method
WO2018196345A1 (en) Automatic compiling method for use in graphical programming
US9910591B2 (en) Visualizations of memory layouts in software programs
US11593540B1 (en) Systems and methods for performing concurrency analysis in simulation environments
US20140380205A1 (en) Interface Development and Operation
CN117313802A (en) Neural network model conversion method, device, equipment and storage medium
WO2023138437A1 (en) Application development platform, micro-program generation method, and device and storage medium
CN113360151A (en) UI data set automatic generation method and system for RPA system
CN112084103B (en) Interface test method, device, equipment and medium
US20110191760A1 (en) Method and apparatus for enhancing comprehension of code time complexity and flow
US10275154B2 (en) Building memory layouts in software programs
US6683624B1 (en) System and method for managing programming object visual representations participating in alternative execution paths
Chugh Prodirect manipulation: bidirectional programming for the masses
CN118034706A (en) A programming code loop fragment completion method and device based on data flow analysis
US9552192B2 (en) Context-based generation of memory layouts in software programs
US10467120B2 (en) Software optimization for multicore systems
CN111708572B (en) A method for automatic generation of control flow diagram based on Clang program structure
US10353793B2 (en) Identifying improvements to memory usage of software programs
CA2359079A1 (en) Program reproducing method and device, and medium on which a program for program reproduction recording
CN112148271A (en) Method for automatically generating and injecting assembly process codes

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant