CN114971539A - Simulated manual operation method based on image matching - Google Patents
Simulated manual operation method based on image matching Download PDFInfo
- Publication number
- CN114971539A CN114971539A CN202210523708.XA CN202210523708A CN114971539A CN 114971539 A CN114971539 A CN 114971539A CN 202210523708 A CN202210523708 A CN 202210523708A CN 114971539 A CN114971539 A CN 114971539A
- Authority
- CN
- China
- Prior art keywords
- image
- positioning
- task
- manual operation
- positioning image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 18
- 238000013515 script Methods 0.000 claims abstract description 36
- 230000006399 behavior Effects 0.000 claims abstract description 8
- 230000009471 action Effects 0.000 claims abstract description 3
- 238000004422 calculation algorithm Methods 0.000 claims description 3
- 238000000354 decomposition reaction Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Strategic Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Entrepreneurship & Innovation (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Economics (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a simulated manual operation method based on image matching, which comprises the following steps: step 1: selecting a positioning image of an operation interface; the positioning image is a screenshot of the component; the characteristic elements of the positioning image are unique; step 2: providing a sequence of operational events; the operation event sequence comprises operation behaviors on the positioning image; and step 3: repeating the step 1 and the step 2 to form a set of task scripts; and 4, step 4: and running the task script to automatically finish the office action. The invention provides a simulated manual operation method based on image matching. Since the operation area of each step is positioned by image matching, the positional area variation of the operation interface cannot affect the positioning.
Description
Technical Field
The invention relates to the technical field of office automation, in particular to a simulated manual operation method based on image matching.
Background
With the development of modern information technology, office automation technology combining office and computer technology is an emerging comprehensive technology. The system not only can realize the automatic processing of office affairs, but also can greatly improve the efficiency of individual or team office affair processing. Currently, whether governments, enterprises or individual operators, a large amount of document-class work with flow and repeatability may exist, and manual operation is boring and easy to cause errors. The automatic operation software can save people from complicated repetitive work by simulating the behavior of manual operation and replacing the manual operation with a computer to process the work. Meanwhile, the error rate of manual operation can be greatly reduced by machine operation.
At present, a mainstream technical idea of simulating manual operation is to record related events (mouse click events, keyboard input events, etc.) generated by an operating system during manual operation through codes, and then send the recorded events to the operating system in sequence during the process of simulating manual operation. This technique generally records the corresponding operation area when the event occurs, and the change of the operation area may cause the behavior of the machine simulation to fail. Another problem is that for some scenes with time requirements in the previous and subsequent steps, the factors such as time intervals need to be considered carefully when the person performs the demonstration operation, so as to avoid the phenomenon that the object to be operated does not appear yet and the simulation event is generated, which finally causes the task failure.
Disclosure of Invention
In order to solve the problem that the computer simulation manual operation process is affected by operation area change and time difference between the previous step and the next step, the embodiment of the invention provides a simulation manual operation method based on image matching. The technical scheme is as follows:
in one aspect, a method for simulating manual operation based on image matching is provided, which includes:
step 1: selecting a positioning image of an operation interface; the positioning image is a screenshot of the component; the characteristic elements of the positioning image are unique;
step 2: providing a sequence of operational events; the operation event sequence comprises operation behaviors on the positioning image;
and 3, step 3: repeating the step 1 and the step 2 to form a set of task scripts;
and 4, step 4: and running the task script to automatically finish the office action.
Further, in step 1, the specific step of selecting the positioning image of the operation interface includes:
acquiring an operation interface window image;
then calculating whether a plurality of point positions meeting the positioning image characteristics exist in the window image through an image comparison algorithm; if a plurality of point locations exist, the feature elements of the positioning image cannot be uniquely positioned, and the positioning elements need to be reselected.
Further, in step 2, the specific step of providing the operation event sequence includes:
the operation event is selected first and then the input content is provided.
Further, in step 3, the specific steps of forming a set of task scripts include:
and (3) an operation step of decomposing the preset task, sequentially executing the step (1) and the step (2) according to a decomposition sequence, and storing the task script after all processing is finished. .
The technical scheme provided by the embodiment of the invention has the following beneficial effects:
the invention provides a simulated manual operation method based on image matching. Since the operation area of each step is positioned by image matching, the positional area variation of the operation interface cannot affect the positioning. In addition, if the software does not detect the positioning image of a certain step, namely the current step is not started yet, the software enters a waiting mode, waits for the next step to start, does not simulate the behavior to be operated of the object which does not appear yet, and ensures the continuity and stability of the task.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic diagram of a method for simulating manual operation based on image matching according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The embodiment provides a simulated manual operation method based on image matching, which comprises the following steps:
step 1: selecting a positioning image A; the positioning image A is usually a screenshot of a certain button, a text box and other components, and the image characteristics of the A are required to be ensured to be unique and obvious as much as possible.
Specifically, the positioning check needs to acquire an image of the whole operation interface window, which is called S, and a positioning element, which is called a, and then calculates whether a plurality of point locations satisfying the a feature exist in S through an image comparison algorithm. If a plurality of point positions exist, the characteristic element A cannot be uniquely positioned, and the positioning element needs to be reselected.
And 2, step: providing an operation event sequence B; the content included in the event sequence B is an operation behavior for positioning the image a, such as clicking a button, and inputting a text box.
Specifically, an operation event is selected first, and input content is provided; such as a keyboard entry event, may require the provision of text to be entered.
And step 3: forming a set of task scripts by repeating the steps 1 and 2, wherein the content of the task scripts is a series of operations for completing a preset task;
specifically, the operation steps of a certain task are decomposed, the step 1 and the step 2 are sequentially executed according to the decomposition sequence, and the task script file is stored after all the processing is finished.
In this embodiment, taking accessing a web page as an example, accessing a web page requires:
1) clicking a browser;
2) inputting a website in an address bar;
3) the access button is clicked.
The corresponding script is:
1) selecting the browser icon in the desktop as the positioning image A1 and the mouse click event as the content of the event sequence B1, and the step is called T1 as a part of the task script.
2) The address bar image is selected as the location image a2, the text entry event and the web address as the content of the event sequence B2, which is referred to as T2 as part of the task script.
3) An access button image is selected as the navigation image a3 and a mouse click event is selected as the event sequence B3 content, this step being referred to as T3 as part of the task script.
The T1, the T2 and the T3 form a task script T for accessing the webpage in sequence.
And 4, step 4: when the task script T is executed, the desktop restores the initial image P0, and the following operations are executed according to the operation sequence in the task script T:
1) executing the task script T1, taking out the positioning image A1 in the task script T1, searching the positioning image A1 in the image P0 in a matching mode, determining the position coordinates (x1, y1) of an area in an operating system when the image features accord with the features of the positioning image A1, and simulating a mouse moving event to move the mouse to the position (x1, y 1). The sequence of events B1 is entered at this location through an interface provided by the operating system. After all, the execution of the task script T1 is completed.
2) After the task script T1 is executed, the desktop forms a new image P1. And executing the task script T2, taking out the positioning image A2 in the task script T2, searching the positioning image A2 in the image P1 in a matching mode, and inputting an event sequence B2 after finding the positioning image A2. And after all, T2 execution is completed.
3) The task script T2 executes the desktop to form a new image P2. And executing the task script T3, taking out the positioning image A3 in the task script T3, searching the positioning image A3 in the image P2 in a matching manner, and inputting an event sequence B3 after finding the positioning image A3. After all, the task script T3 is completed.
After all the task scripts T1, T2, and T3 are executed, the task of the task script T is completed, that is, the task of accessing a certain web page is completed.
In specific operation, see fig. 1:
1. and (3) opening the software by the user, creating a task file, determining the current office task, and decomposing each step of the task.
2. And turning to a single-step operation interface, selecting a positioning image by using software, and checking the positioning reliability by using the software, so that the software can determine a certain step when the task file is executed. After the checking is passed, temporarily storing the positioning image into an internal memory; the detection is not by prompting the user for a reselection.
3. And prompting the user to select the operation event, and storing the operation event and the content provided by the user in the memory.
4. Prompting the user whether to continue to compile the script operation, if so, turning to the step 2; if not, the script file indicates that the script file is used for completing the writing of all script contents, and the software stores all the contents in the memory into the task file in a persistent mode.
5. And the user runs the task file by using software, and the software automatically completes the office task pointed by the task file according to the positioning image and the event content of each step stored in the task file.
Since the operation area of each step is positioned by image matching, the positional area variation of the operation interface cannot affect the positioning. In addition, if the software does not detect the positioning image of a certain step, namely the current step is not started yet, the software enters a waiting mode, waits for the next step to start, does not simulate the behavior to be operated of the object which does not appear yet, and ensures the continuity and stability of the task.
The present invention is not limited to the above preferred embodiments, and any modifications, equivalent replacements, improvements, etc. within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (4)
1. A simulated manual operation method based on image matching is characterized by comprising the following steps:
step 1: selecting a positioning image of an operation interface; the positioning image is a screenshot of the component; the characteristic elements of the positioning image are unique;
step 2: providing a sequence of operational events; the operation event sequence comprises operation behaviors on the positioning image;
and step 3: repeating the step 1 and the step 2 to form a set of task scripts;
and 4, step 4: and running the task script to automatically finish the office action.
2. The method for simulating manual operation based on image matching as claimed in claim 1, wherein in step 1, the specific step of selecting the positioning image of the operation interface comprises:
acquiring an operation interface window image;
then calculating whether a plurality of point positions meeting the characteristics of the positioning image exist in the window image through an image comparison algorithm; if a plurality of point positions exist, the feature elements of the positioning image cannot be uniquely positioned, and the positioning elements need to be reselected.
3. The method for simulating manual operation based on image matching as claimed in claim 2, wherein in step 2, the specific step of providing the operation event sequence comprises:
the operation event is selected first and then the input content is provided.
4. The method for simulating manual operation based on image matching as claimed in claim 3, wherein in step 3, the specific steps of forming a set of task scripts comprise:
and (3) an operation step of decomposing the preset task, sequentially executing the step (1) and the step (2) according to a decomposition sequence, and storing the task script after all processing is finished.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210523708.XA CN114971539A (en) | 2022-05-13 | 2022-05-13 | Simulated manual operation method based on image matching |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210523708.XA CN114971539A (en) | 2022-05-13 | 2022-05-13 | Simulated manual operation method based on image matching |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114971539A true CN114971539A (en) | 2022-08-30 |
Family
ID=82983304
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210523708.XA Pending CN114971539A (en) | 2022-05-13 | 2022-05-13 | Simulated manual operation method based on image matching |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114971539A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101276270A (en) * | 2007-03-28 | 2008-10-01 | 张登科 | Automatic control software technique based on analog input and image analysis |
| CN104951393A (en) * | 2014-03-25 | 2015-09-30 | 中国电信股份有限公司 | Testing method and device |
| CN109324864A (en) * | 2018-10-24 | 2019-02-12 | 北京赢销通软件技术有限公司 | A kind of acquisition methods and device of man-machine interactive operation information |
-
2022
- 2022-05-13 CN CN202210523708.XA patent/CN114971539A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101276270A (en) * | 2007-03-28 | 2008-10-01 | 张登科 | Automatic control software technique based on analog input and image analysis |
| CN104951393A (en) * | 2014-03-25 | 2015-09-30 | 中国电信股份有限公司 | Testing method and device |
| CN109324864A (en) * | 2018-10-24 | 2019-02-12 | 北京赢销通软件技术有限公司 | A kind of acquisition methods and device of man-machine interactive operation information |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7398068B2 (en) | software testing | |
| CN110928772B (en) | Test method and device | |
| CN102184138B (en) | Method and system for automatically reproducing and positioning software error | |
| US8935673B1 (en) | System and method for debugging computer program based on execution history | |
| US9740506B2 (en) | Automating interactions with software user interfaces | |
| CN102667696B (en) | For the System and method for of the object identity in user interface | |
| US20080294985A1 (en) | Graphical User Interface (Gui) Noise Reduction in a Cognitive Control Framework | |
| CN102804147A (en) | Code inspection executing system for performing a code inspection of ABAP source codes | |
| EP2705441B1 (en) | Automatic classification adjustment of recorded actions for automation script | |
| CN112241370B (en) | API interface class checking method, system and device | |
| CN111679976A (en) | Method and device for searching page object | |
| US7783927B2 (en) | Intelligent processing tools | |
| CN108399125A (en) | Automated testing method, device, computer equipment and storage medium | |
| JP7557431B2 (en) | Apparatus and method for supporting source code modification | |
| CN113032279A (en) | Web application testing and repairing method based on semantic path search | |
| CN110286882B (en) | Foreground system design and verification method based on model detection | |
| CN114971539A (en) | Simulated manual operation method based on image matching | |
| CN115481025A (en) | Script recording method and device for automatic test, computer equipment and medium | |
| CN112181809A (en) | Automatic graphical user interface testing method based on multiple positioning methods | |
| CN112612469A (en) | Interface element processing method and device and electronic equipment | |
| CN112015638A (en) | Automatic testing method and system | |
| CN113220596B (en) | Application testing method, device, equipment, storage medium and program product | |
| CN112148608B (en) | Mobile terminal automated software testing method based on control function labeling | |
| Hellmann | Automated GUI Testing for Agile Development Environments | |
| CN117762786A (en) | Method for generating test case, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |