CN113641569B - Robot flow automation method - Google Patents
Robot flow automation method Download PDFInfo
- Publication number
- CN113641569B CN113641569B CN202110704432.0A CN202110704432A CN113641569B CN 113641569 B CN113641569 B CN 113641569B CN 202110704432 A CN202110704432 A CN 202110704432A CN 113641569 B CN113641569 B CN 113641569B
- Authority
- CN
- China
- Prior art keywords
- image
- information system
- judgment
- page
- rpa
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000012360 testing method Methods 0.000 claims abstract description 23
- 238000004088 simulation Methods 0.000 claims abstract description 11
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 10
- 238000013515 script Methods 0.000 claims abstract description 9
- 238000012549 training Methods 0.000 claims abstract description 7
- 238000004801 process automation Methods 0.000 claims abstract description 6
- 238000005516 engineering process Methods 0.000 claims abstract description 4
- 238000007689 inspection Methods 0.000 claims abstract description 4
- 238000010801 machine learning Methods 0.000 claims abstract description 4
- 238000013528 artificial neural network Methods 0.000 claims description 9
- 230000009471 action Effects 0.000 claims description 3
- 238000012937 correction Methods 0.000 claims description 3
- 238000001035 drying Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 3
- 238000009826 distribution Methods 0.000 claims description 2
- 230000010354 integration Effects 0.000 claims description 2
- 230000001360 synchronised effect Effects 0.000 claims description 2
- 238000012546 transfer Methods 0.000 claims description 2
- 238000002372 labelling Methods 0.000 claims 2
- 239000000284 extract Substances 0.000 abstract 1
- 238000012423 maintenance Methods 0.000 description 6
- 238000012790 confirmation Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- BUGBHKTXTAQXES-UHFFFAOYSA-N Selenium Chemical compound [Se] BUGBHKTXTAQXES-UHFFFAOYSA-N 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 229910052711 selenium Inorganic materials 0.000 description 1
- 239000011669 selenium Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/103—Workflow collaboration or project management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Quality & Reliability (AREA)
- Strategic Management (AREA)
- Human Resources & Organizations (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Entrepreneurship & Innovation (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Molecular Biology (AREA)
- Computer Hardware Design (AREA)
- Health & Medical Sciences (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Economics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the field of office automation, in particular to a robot process automation method, which uses RPA technology to record the basic operation of a user and generate test cases, runs the test cases to play back in an information system and converts the test cases into automation scripts; acquiring webpage data through an automatic script simulation information system; the simulation operation is inspected in a manual judgment mode, whether the information system operates normally is judged according to inspection results, and image features in a browser page and image features in a database are compared and judged according to an image similarity comparison and difference marking method; simulating a manual judgment mode through a machine learning algorithm to obtain configuration recommendation of a judgment result; and finally, through repeated training and testing, an accurate operation rule base and a judgment rule base are established. The method sets RPA operation aiming at daily affairs, optimizes the basic flow of enterprises, collects and extracts information to generate forms required by users, and automatically enters a business system.
Description
Technical Field
The invention relates to the field of office automation, in particular to a robot flow automation method based on image similarity comparison and difference marking.
Background
The information system automation operation and maintenance comprises a development environment, a test environment, a pre-release environment, a production environment and the like, can effectively deploy deployment packages to different environments, and has the functions of log management, automation monitoring, alarm response and the like. At present, the automatic operation and maintenance of an information system mainly has the following problems:
the existing information system needs to use a browser to display and handle the service, the terminal displays the service correctly only by combining the data, css style files or js script files with the browser, and then the server is checked to return partial content in the data to judge whether the service is normal or not. The method has low accuracy and cannot check specific service nodes;
the system has very large implementation workload in compatibility test before software change and full coverage test after change, and consumes time and labor by means of manual mode;
the test work before and after the release of the information system is mainly borne by first-line customer service personnel, the automatic test work in the industry is usually a tool customized on the basis of selenium, test specific codes are required to be written for each service, and the writing of the codes requires support of programmers, so that the popularization and application of the automatic test in the first-line customer service are limited;
a large amount of repeated data input works, such as user account numbers and authorized maintenance of the 4A platform, can be pushed to an integrated related system and the 4A platform only by operating on a foreground page, and the tedious input works are finished by manpower, are time-consuming and labor-consuming, and severely restrict the development of automatic operation and maintenance works.
If the RPA (robot process automation method) is used to optimize the automation operation and maintenance work, the confirmation of the execution result of the existing RPA application software is based on the text content of the execution feedback, and the problems of difficult and inaccurate confirmation exist for complex web applications, especially pages with multiple layers of iframes.
Disclosure of Invention
Aiming at a plurality of problems existing in the automatic operation and maintenance work of the existing information system, the invention provides a robot flow automation method based on image similarity comparison and difference marking.
The robot process automation method is characterized by comprising the following steps of:
1) RPA simulates manual operation: recording basic operation of a user by using an RPA technology, generating a test case, running the test case for playback in an information system, and converting the test case into an automatic script; acquiring webpage data through an automatic script simulation information system;
2) RPA simulation manual judgment: the simulation operation is inspected in a manual judgment mode, whether the information system operates normally is judged according to inspection results, and image features in a browser page and image features in a database are compared and judged according to an image similarity comparison and difference marking method; simulating a manual judgment mode through a machine learning algorithm to obtain configuration recommendation of a judgment result;
3) Establishing an operation rule base and a judgment rule base: through repeated operation training and action correction, accurate operation characteristics are finally formed, a series of trained operation characteristics are arranged together according to a specific sequence, and a specific operation rule base is finally established; and (3) integrating the judgment results in the pictures by using a method of comparing the image similarity and the difference mark and finally establishing an accurate judgment rule base through repeated training and testing.
Specifically, the method for comparing the image similarity and marking the difference in the step 2) adopts a FREAK algorithm to extract image characteristics of the image frames in the page in the information system, and uses a convolution algorithm to extract image edges, and performs smoothing, blurring, drying and sharpening treatment on the image; and comparing the image characteristics of the browser page with the image characteristics in the database by using a left neural network and a right neural network, mapping the image characteristics to a new space by using the two neural networks, evaluating the similarity of the two inputs by calculating the loss degree, and marking the difference of the image characteristics.
Furthermore, the method for comparing the image similarity and marking the difference adopts the distributed integrated scheduling method to realize the cooperation work of the cloud end and the client end, and is used for accelerating the feature extraction and marking the difference of the image, and the distributed integrated scheduling method comprises the following steps: the cloud end and the client end work cooperatively through a scheduler, and the task state executed by the system is updated and synchronized regularly through a timer; the task execution can be switched and transferred between the client and the cloud through the transfer device; and comprehensively evaluating the client and cloud execution efficiency according to the resource use condition and the task execution efficiency to decide task distribution scheduling so as to achieve optimal use of resources. The conventional RPA application software confirms text content based on execution feedback, solves the problems of difficult and inaccurate confirmation of complex web application, especially a page with multiple layers of iframes, marks image differences through image feature extraction and image similarity comparison, comprehensively judges the execution result of RPA service, effectively improves the accuracy of RPA, and further ensures that the application of RPA replaces manual batch service handling to be more efficient and accurate.
Specifically, the judging result in the biological relative relationship integration picture in the step 3) is applied to the multi-layer nested web page, and comprises the steps that the parent page calls the child iframe page, the child iframe page calls the parent page, the brother iframe pages in the page call each other, the two-layer or multi-layer iframe call is crossed, the role of the page is given by the method for allowing the biological relative relationship to locate and search, the element number of the multi-layer nested web page is related through the father-son relationship and the brother relationship, and the element number search is simplified. Aiming at the problem that html page elements are difficult to find mainly according to id, class, tagname labels when the web application RPA is used at present, the elements of a complex page are difficult to locate and find through association finding of parent-child elements and brother elements, so that the application range of RPA business handling is expanded.
According to the robot process automation method, RPA operation is set for daily transactions with high repeatability, standardization, clear rules and large batch in different industries, the enterprise basic process is optimized, information is collected and extracted to generate a form required by a user, and the form is automatically input into a business system, so that repeated manpower work is reduced, and the happiness of staff is improved.
Detailed Description
Example 1: the robot process automation method is characterized by comprising the following steps of:
1) RPA simulates manual operation: recording basic operation of a user by using an RPA technology, generating a test case, running the test case for playback in an information system, and converting the test case into an automatic script; acquiring webpage data through an automatic script simulation information system;
2) RPA simulation manual judgment: the simulation operation is inspected in a manual judgment mode, whether the information system operates normally is judged according to inspection results, and then image characteristics of a picture frame in a page in the information system are extracted by adopting a FREAK algorithm through an image similarity comparison and difference marking method, and an image edge is extracted by adopting a convolution algorithm, and smoothing, blurring, drying and sharpening are carried out on the image; and comparing the image characteristics of the browser page with the image characteristics in the database by using a left neural network and a right neural network, mapping the image characteristics to a new space by using the two neural networks, evaluating the similarity of the two inputs by calculating the loss degree, and marking the difference of the image characteristics. And obtaining configuration recommendation of the judgment result by simulating a manual judgment mode through a machine learning algorithm.
3) Establishing an operation rule base and a judgment rule base: through repeated operation training and action correction, accurate operation characteristics are finally formed, a series of trained operation characteristics are arranged together according to a specific sequence, and a specific operation rule base is finally established. For the multi-layer nested web page, the role of the page is given by a method for allowing the biological relative relationship to locate and search, so that the element number of the multi-layer nested web page is associated through father-son relationship and brother relationship. And finally, an accurate judgment rule base is established through repeated training and testing.
Claims (4)
1. The robot process automation method is characterized by comprising the following steps of:
1) RPA simulates manual operation: recording basic operation of a user by using an RPA technology, generating a test case, running the test case for playback in an information system, and converting the test case into an automatic script; acquiring webpage data through an automatic script simulation information system;
2) RPA simulation manual judgment: the simulation operation is inspected in a manual judgment mode, whether the information system operates normally is judged according to inspection results, and image features in a browser page and image features in a database are compared and judged according to an image similarity comparison and difference marking method; simulating a manual judgment mode through a machine learning algorithm to obtain configuration recommendation of a judgment result;
3) Establishing an operation rule base and a judgment rule base: through repeated operation training and action correction, accurate operation characteristics are finally formed, a series of trained operation characteristics are arranged together according to a specific sequence, and a specific operation rule base is finally established; and (3) integrating the judgment results in the pictures by using a method of comparing the image similarity and the difference mark and finally establishing an accurate judgment rule base through repeated training and testing.
2. The automated robot process method according to claim 1, wherein the image similarity comparison and difference labeling method in step 2) uses a frak algorithm to extract image features of the image frames in the pages in the information system, and uses a convolution algorithm to extract image edges, and performs smoothing, blurring, de-drying, and sharpening processes on the images; and comparing the image characteristics of the browser page with the image characteristics in the database by using a left neural network and a right neural network, mapping the image characteristics to a new space by using the two neural networks, evaluating the similarity of the two inputs by calculating the loss degree, and marking the difference of the image characteristics.
3. The automated method of robot flow according to claim 2, wherein the cooperation between the cloud and the client is implemented by using a distributed integrated scheduling method, for accelerating feature extraction and difference labeling of images, the distributed integrated scheduling method comprising: the cloud end and the client end work cooperatively through a scheduler, and the task state executed by the system is updated and synchronized regularly through a timer; the task execution can be switched and transferred between the client and the cloud through the transfer device; comprehensively evaluating the client and cloud execution efficiency according to the resource use condition and the task execution efficiency to decide task distribution scheduling.
4. The automated robot flow method of claim 1, wherein the judging result in the biological relationship integration picture in step 3) is applied to a multi-layer nested web page, and comprises a parent page calling child iframe pages, child iframe pages calling parent pages, and intra-page sibling iframe pages calling each other, and cross-layer or multi-layer iframe calling, wherein the method of allowing biological relationship positioning searching gives the role of the page, so that the element number of the multi-layer nested web page is related through father-son relationship and sibling relationship, and the element number searching is simplified.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110704432.0A CN113641569B (en) | 2021-06-24 | 2021-06-24 | Robot flow automation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110704432.0A CN113641569B (en) | 2021-06-24 | 2021-06-24 | Robot flow automation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113641569A CN113641569A (en) | 2021-11-12 |
| CN113641569B true CN113641569B (en) | 2023-11-14 |
Family
ID=78416221
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110704432.0A Active CN113641569B (en) | 2021-06-24 | 2021-06-24 | Robot flow automation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113641569B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114253728A (en) * | 2021-12-23 | 2022-03-29 | 上海交通大学 | Heterogeneous multi-node cooperative distributed neural network deployment system based on webpage ecology |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110806934A (en) * | 2019-11-15 | 2020-02-18 | 四川中电启明星信息技术有限公司 | RPA technology-based intelligent all-in-one machine development and multi-service rapid processing method |
| CN112101357A (en) * | 2020-11-03 | 2020-12-18 | 杭州实在智能科技有限公司 | RPA robot intelligent element positioning and picking method and system |
| CN112926954A (en) * | 2021-04-26 | 2021-06-08 | 南京微雀信息技术有限公司 | Cross-network government affair information exchange system and method based on artificial intelligent robot |
| KR102261793B1 (en) * | 2020-12-11 | 2021-06-09 | 주식회사 이노룰스 | System for rpa robot agent clutstering |
-
2021
- 2021-06-24 CN CN202110704432.0A patent/CN113641569B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110806934A (en) * | 2019-11-15 | 2020-02-18 | 四川中电启明星信息技术有限公司 | RPA technology-based intelligent all-in-one machine development and multi-service rapid processing method |
| CN112101357A (en) * | 2020-11-03 | 2020-12-18 | 杭州实在智能科技有限公司 | RPA robot intelligent element positioning and picking method and system |
| KR102261793B1 (en) * | 2020-12-11 | 2021-06-09 | 주식회사 이노룰스 | System for rpa robot agent clutstering |
| CN112926954A (en) * | 2021-04-26 | 2021-06-08 | 南京微雀信息技术有限公司 | Cross-network government affair information exchange system and method based on artificial intelligent robot |
Non-Patent Citations (1)
| Title |
|---|
| 自动化运维工具在电力企业信息系统管理中的作用;杨震乾等;电子技术与软件工程;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113641569A (en) | 2021-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105426394B (en) | Based on cross-platform mobile report form generation method and system | |
| CN107957940B (en) | Test log processing method, system and terminal | |
| US20170109636A1 (en) | Crowd-Based Model for Identifying Executions of a Business Process | |
| US20140365827A1 (en) | Architecture for end-to-end testing of long-running, multi-stage asynchronous data processing services | |
| CN109408763B (en) | Method and system for managing resume of different templates | |
| CN105868956A (en) | Data processing method and device | |
| CN111881105B (en) | Labeling model of business data and model training method thereof | |
| CN111931809A (en) | Data processing method and device, storage medium and electronic equipment | |
| CN113641569B (en) | Robot flow automation method | |
| CN113778894A (en) | Test case construction method, device, equipment and storage medium | |
| CN112506778A (en) | WEB user interface automatic test method, device, equipment and storage medium | |
| CN115657890A (en) | PRA robot customizable method | |
| CN111865673A (en) | Automatic fault management method, device and system | |
| CN104008042A (en) | UI (user interface) automated testing method, system and device | |
| CN107368407A (en) | Information processing method and device | |
| CN113312260A (en) | Interface testing method, device, equipment and storage medium | |
| Figalist et al. | An end-to-end framework for productive use of machine learning in software analytics and business intelligence solutions | |
| Ulrich et al. | Operator timing of task level primitives for use in computation-based human reliability analysis | |
| CN115035044A (en) | Be applied to intelligent AI platform of industry quality inspection | |
| CN115982272A (en) | Data labeling method and device for urban big data management and computer storage medium | |
| CN114880239A (en) | Interface automation testing framework and method based on data driving | |
| CN113420080A (en) | Toxicology experiment data management system | |
| US11507728B2 (en) | Click to document | |
| CN112968941B (en) | Data acquisition and man-machine collaborative annotation method based on edge calculation | |
| CN113836037B (en) | Interface interaction testing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |