CN112579431A - Cross-platform script recording and playback method based on image recognition - Google Patents
Cross-platform script recording and playback method based on image recognition Download PDFInfo
- Publication number
- CN112579431A CN112579431A CN201910951890.7A CN201910951890A CN112579431A CN 112579431 A CN112579431 A CN 112579431A CN 201910951890 A CN201910951890 A CN 201910951890A CN 112579431 A CN112579431 A CN 112579431A
- Authority
- CN
- China
- Prior art keywords
- script
- equipment
- recording
- matching
- playback
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013515 script Methods 0.000 title claims abstract description 72
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000012360 testing method Methods 0.000 claims abstract description 62
- 238000005516 engineering process Methods 0.000 claims abstract description 19
- 230000008569 process Effects 0.000 claims abstract description 13
- 238000012015 optical character recognition Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000001514 detection method Methods 0.000 claims description 3
- 238000003709 image segmentation Methods 0.000 claims description 3
- 230000002159 abnormal effect Effects 0.000 claims description 2
- 230000005540 biological transmission Effects 0.000 claims description 2
- 238000006243 chemical reaction Methods 0.000 claims description 2
- 238000003708 edge detection Methods 0.000 claims description 2
- 230000009466 transformation Effects 0.000 claims description 2
- 238000012423 maintenance Methods 0.000 abstract description 4
- 230000018109 developmental process Effects 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000013522 software testing Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000000275 quality assurance Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000010998 test method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
Abstract
The invention provides a cross-platform script recording and playback method based on image recognition, which is used for reducing the writing threshold of an automatic test script and avoiding the repeated writing and maintenance expenses of the script under different equipment and different platforms by the same application in the field of mobile application automatic test. (2) The image recognition and layout matching technology is combined to realize the running playback of the recorded script under different platforms of a plurality of devices, the cross-platform capability of the test script is greatly improved while the automatic script generation process is simplified, and the effectiveness and the universality of the automatic test are improved.
Description
Technical Field
The invention belongs to the technical field of software testing in computer technology, in particular to the technical field of automatic testing in software testing, which is used for recording mobile application testing operation, generating an automatic testing script and realizing single recording, multi-equipment and cross-platform running playback of the testing script by combining with an image recognition technology.
Background
With the comprehensive and rapid development of the mobile internet, the user requirements are not limited to the functional level any more, and people pay more attention to the software quality. As quality assurance of mobile application, software testing is particularly important in the whole development process. The types of tests available in the industry can be divided into unit tests, interface tests, and ui (user interface) tests. Because unit testing, interface testing and the like are relatively stable, the existing testing framework can sufficiently meet the requirements of the two types of testing. Frequent iteration of the mobile application version enables a large amount of test resources to be consumed in the regression test, and the UI test is taken as an important ring in the regression test, so that the efficiency is optimized and improved, and the test cost of the mobile application can be greatly reduced.
Currently, UI testing in the industry is implemented by two ways, namely manual testing and writing of automated testing scripts. Mobile applications inherently have multi-platform, multi-device operational characteristics, which also makes UI testing a lot of repetitive work. In the aspect of manual testing, because the subjectivity of a tester is high, the testing accuracy depends on the tester, and the problems of incomplete testing, unredetailed defect record, difficulty in reproduction and the like exist in large-scale manual testing results for a long time. Modern software development usually adopts an agile development concept, and manual testing with low efficiency greatly influences the software development process. Test developers write an automatic script mode, and the test efficiency of the mobile application is improved to a certain extent. However, due to the diversity of mobile equipment models and platforms and the variability of the UI interface, a large amount of repeated modification and maintenance exist in the UI automation test script in the actual production process. Meanwhile, the test script is written based on a high-level language, and the technical requirements on testers are high. A large number of repeated scripts under different platforms and models need to be modified and maintained, so that the enterprise testing cost is high.
Comparing the difference between the manual test and the existing automatic test, it can be found that the angle of the manual test is based on the identification and operation of the UI interface to complete the test. From the angle, the encoding type script development is simplified into the operation type script recording, namely, remote equipment control is realized based on Web, an application control is clicked, and attributes such as a control picture, a layout position and an XML file are automatically extracted at the rear end to generate a test script. And in the script playback stage, positioning of the control on different equipment interfaces is realized by means of an image recognition technology and combining script information, and the playback of script operation is realized by means of a bottom layer tool after coordinate information is acquired. Therefore, from the angle of simulating manual testing, the problem of repeated writing of multiple models and multiple platforms of the test script can be better solved, one-time script recording and multiple model and multiple platform playback testing are realized, and the complexity of script development and maintenance is greatly reduced. Compared with the traditional automatic test method in which test script codes are compiled in different languages under different platforms, the script clicking recording mode is easier for testers to accept.
Disclosure of Invention
The invention aims to solve the problems that: a method for recording and replaying cross-platform scripts based on image recognition is provided. The technical scheme of the invention is as follows:
1) and entering an equipment remote control interface to initialize one connected equipment, and installing and starting the application to be tested on the equipment.
2) Monitoring a series of test operations such as clicking, sliding and inputting executed by a user on a remote control interface in real time, acquiring and storing attribute information of a control corresponding to the operation, converting the operation into an equipment executable instruction, sending the equipment executable instruction to a remote real machine, synchronizing an equipment instruction interface to a client, judging whether the recording of the script is finished or not, and generating a recorded automatic test script when the recording is finished.
3) Selecting a playback script and target equipment, initializing remote equipment, acquiring a single-step operation file and a current equipment page screenshot from a recorded automatic test script, and trying to position a control by sequentially adopting an image matching positioning technology and a layout matching technology.
4) The image matching technology firstly detects and describes image features and constructs a feature point set, matches the feature point set, eliminates error matching through ratio testing, calculates distortion, and finally obtains matching area coordinates through transmission transformation.
5) The layout matching technology carries out gray level conversion on the equipment page screenshot, completes image segmentation through edge detection, expansion and contour detection, carries out OCR (optical character recognition) portrayal on the layout in the screenshot, and merges the final result to obtain layout coordinates.
6) If the control matching and positioning are successful, returning the actual coordinates of the control in the current equipment, generating an operation executable instruction, sending the operation executable instruction to the equipment, running and replaying, and repeating the process until the script replaying is finished; and if the control fails to be matched and positioned, prompting that the playback of the test script is abnormal, and ending the playback.
The invention has the beneficial effects that: the automatic test script generation process is simplified by adopting a recording-playback mode, and the script generation threshold is reduced. Meanwhile, the single recording of the script, the multi-device and cross-platform operation playback are realized by combining the image recognition technology, the cross-platform capability of the script is enhanced, the repeated writing of the script caused by different models and platforms in the same application under the traditional automatic test framework is avoided, and the script maintenance cost is greatly reduced. .
Drawings
FIG. 1 general architecture diagram
FIG. 2 device management module flow diagram
FIG. 3 flow chart of script recording module
FIG. 4 script playback Module flow diagram
FIG. 5 image matching algorithm flow chart
FIG. 6 layout matching algorithm flow chart
Detailed description of the invention
The method mainly comprises three parts of equipment management, script recording and script playback, and the whole architecture is shown in figure 1. The script recording and the script playback are core parts of the method; the equipment management is mainly responsible for the access, state monitoring, interface acquisition, instruction distribution, execution and the like of the remote mobile equipment; the image recognition and processing technology in script playback is a key point for realizing multi-device and cross-platform playback of test scripts.
The device management module is mainly used for accessing the device, monitoring the device state, transmitting the device data, and querying the device, and fig. 2 describes the operation flow of the whole module.
When the system is started, the equipment monitor and the timer are started, the equipment monitor triggers an access event when new equipment is accessed into the system, then software and hardware information of the newly accessed mobile equipment is acquired, newly added equipment is recorded into an equipment list, and the equipment is in an idle state. And meanwhile, the monitor monitors the state of the equipment in the system, and correspondingly modifies the equipment list maintained in the system when the equipment state is transferred in an idle state and an occupied state due to equipment calling.
The script recording module is the core part of the method. The module is mainly a transfer layer module for interaction between a Web end operation interface and a bottom layer equipment management module, and provides script information recording, updating, recording of an imaging control element attribute file required by script playback construction and the like. The main flow is shown in fig. 3 and roughly includes the processes of remote connection of devices, recording of single step operation, script generation and the like.
The script playback is based on the script file generated in the recording process, and the process of playing back the script file on different platforms and model devices is performed on the script file. The main flow of the module is shown in fig. 4, and mainly includes remote connection of equipment, script analysis, and positioning of a control. The control positioning is realized based on two modes of image matching and layout matching, and the stability of the script is ensured.
In order to realize cross-device and cross-platform playback of the recorded script, the system utilizes the control and page image information extracted in the recording process, and utilizes the image processing technology to position the control on new equipment by utilizing the attributes of the traditional control and the like. The system provides two control positioning strategies of image matching and layout matching, and the control can be positioned on new equipment.
The image feature matching algorithm takes the target image and the image to be matched as input, and takes the coordinate value of the vertex of the region finally matched on the image to be matched as an output result. The algorithm mainly comprises five main processes of preprocessing, feature extraction, feature matching, mismatch elimination and distortion calculation, and the flow of the image matching algorithm is shown in figure 5.
Layout matching is based on a REMAII tool which mainly realizes the function of reversely generating page codes through interface pictures. Wherein the depiction of the layout of the pages is the basis for the layout matching method used and developed herein. The tool respectively carves the image page layout by using an image segmentation technology Canny and an OCR (optical character recognition), and processes the result by combining two modes to generate the page layout. The interface is divided into rows and columns, and layout two-dimensional coordinates are given to each element for positioning the control. The specific processing flow is shown in fig. 6.
Claims (3)
1. A cross-platform script recording and playback method based on image recognition is characterized in that: (1) the remote real machine is put to a Web end in real time for a user to operate, and the state of the equipment is managed and dynamically updated; (2) automatically recording test operation executed by a user at a Web end and generating an automatic test script, and supporting recording clicking, sliding, inputting and other operations; (3) the recorded script is played back on the appointed equipment of the user by adopting an image recognition technology, so that the selection of multiple equipment is supported, and the cross-platform playback is supported.
2. The image recognition-based cross-platform script recording and playback method as claimed in claim 1, wherein remote real machine debugging is provided at a webpage end for a user to operate a remote real machine, manage and dynamically update the equipment state, record test operations such as clicking, sliding, inputting and the like in the user operation process in real time and generate an automatic test script, the user of the recorded script can select to play back on any appointed machine type, and image recognition technology and layout matching technology are adopted to realize matching and positioning of a return visit time control component, so that one-time recording, multi-equipment and cross-platform playback of the automatic test script are realized.
3. The image recognition-based cross-platform script recording and playback method as claimed in claims 1 and 2, which is characterized by comprising the following specific steps:
1) entering an equipment remote control interface to select and initialize one connected equipment, and installing and starting an application to be tested on the equipment;
2) monitoring a series of test operations such as clicking, sliding and inputting executed by a user on a remote control interface in real time, acquiring and storing attribute information of a control corresponding to the operation, converting the operation into an equipment executable instruction, sending the equipment executable instruction to a remote real machine, synchronizing an equipment instruction interface to a client, judging whether the recording of the script is finished or not, and generating a recorded automatic test script when the recording is finished;
3) selecting a playback script and target equipment, initializing remote equipment, acquiring a single-step operation file and a current equipment page screenshot from a recorded automatic test script, and trying to position a control by sequentially adopting an image matching positioning technology and a layout matching technology;
4) the image matching technology comprises the steps of firstly carrying out image feature detection, describing and constructing a feature point set, matching the feature point set, eliminating error matching through ratio testing, calculating distortion, and finally obtaining a matching area coordinate through transmission transformation;
5) the layout matching technology carries out gray level conversion on the equipment page screenshot, completes image segmentation through edge detection, expansion and contour detection, carries out OCR (optical character recognition) portrayal on the layout in the screenshot, and merges the final results to obtain layout coordinates;
6) if the control matching and positioning are successful, returning the actual coordinates of the control in the current equipment, generating an operation executable instruction, sending the operation executable instruction to the equipment, running and replaying, and repeating the process until the script replaying is finished; and if the control fails to be matched and positioned, prompting that the playback of the test script is abnormal, and ending the playback.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910951890.7A CN112579431A (en) | 2019-09-30 | 2019-09-30 | Cross-platform script recording and playback method based on image recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910951890.7A CN112579431A (en) | 2019-09-30 | 2019-09-30 | Cross-platform script recording and playback method based on image recognition |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112579431A true CN112579431A (en) | 2021-03-30 |
Family
ID=75117231
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910951890.7A Pending CN112579431A (en) | 2019-09-30 | 2019-09-30 | Cross-platform script recording and playback method based on image recognition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579431A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113392007A (en) * | 2021-06-17 | 2021-09-14 | 网易（杭州）网络有限公司 | Compatibility testing method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101620564A (en) * | 2008-06-30 | 2010-01-06 | 福建升腾资讯有限公司 | Method for automatically testing recording playback mode of terminal system |
| CN105550114A (en) * | 2015-12-23 | 2016-05-04 | 广东欧珀移动通信有限公司 | Automated testing method and apparatus and mobile terminal |
| CN105955881A (en) * | 2016-04-22 | 2016-09-21 | 百度在线网络技术（北京）有限公司 | Automated test step recording and playback method and apparatus |
| CN107357724A (en) * | 2017-06-27 | 2017-11-17 | 深圳市泰衡诺科技有限公司上海分公司 | Automatic software test method and device in Android system |
| CN110275834A (en) * | 2019-06-25 | 2019-09-24 | 中国工商银行股份有限公司 | User interface automatization test system and method |
-
2019
- 2019-09-30 CN CN201910951890.7A patent/CN112579431A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101620564A (en) * | 2008-06-30 | 2010-01-06 | 福建升腾资讯有限公司 | Method for automatically testing recording playback mode of terminal system |
| CN105550114A (en) * | 2015-12-23 | 2016-05-04 | 广东欧珀移动通信有限公司 | Automated testing method and apparatus and mobile terminal |
| CN105955881A (en) * | 2016-04-22 | 2016-09-21 | 百度在线网络技术（北京）有限公司 | Automated test step recording and playback method and apparatus |
| CN107357724A (en) * | 2017-06-27 | 2017-11-17 | 深圳市泰衡诺科技有限公司上海分公司 | Automatic software test method and device in Android system |
| CN110275834A (en) * | 2019-06-25 | 2019-09-24 | 中国工商银行股份有限公司 | User interface automatization test system and method |
Non-Patent Citations (1)
| Title |
|---|
| 赵文远: "基于图像识别的跨平台测试脚本录制与回放系统的设计与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》, no. 7, pages 138 - 436 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113392007A (en) * | 2021-06-17 | 2021-09-14 | 网易（杭州）网络有限公司 | Compatibility testing method and device |
| CN113392007B (en) * | 2021-06-17 | 2023-09-08 | 网易（杭州）网络有限公司 | Compatibility testing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110309071B (en) | Test code generation method and module, and test method and system | |
| Baker et al. | Model-driven engineering in a large industrial context—Motorola case study | |
| CN107665168A (en) | A kind of embedded software GUI automated test devices | |
| CN105760296A (en) | Automation testing control method, device and terminal | |
| CN103180834A (en) | An automated operating system test framework | |
| CN107526679A (en) | Automated test frame, the automated testing method based on thereon, storage medium and computer equipment | |
| CN107622017B (en) | Analysis method for universal automation software test | |
| Tuovenen et al. | MAuto: Automatic mobile game testing tool using image-matching based approach | |
| CN111241111B (en) | Data query method and device, data comparison method and device, medium and equipment | |
| CN113505082A (en) | Application program testing method and device | |
| CN109101414B (en) | Massive UI test generation method and device based on buried point data | |
| CN112579431A (en) | Cross-platform script recording and playback method based on image recognition | |
| CN116630990B (en) | RPA flow element path intelligent restoration method and system | |
| CN113448845A (en) | UI automation test method and system | |
| CN110286882B (en) | Foreground system design and verification method based on model detection | |
| Leotta et al. | Assessor: a po-based webdriver test suites generator from selenium ide recordings | |
| CN111008140B (en) | Cross-platform UI (user interface) automatic testing method and device | |
| CN114090459A (en) | Test system and test method based on microkernel | |
| CN112559377A (en) | Method and device for generating first test case | |
| CN109669868A (en) | The method and system of software test | |
| CN114692382B (en) | Management method and device for nuclear power simulation model development data and computer equipment | |
| CN103377118A (en) | Method and device for testing Flash engine | |
| Tiwari et al. | Combining Model-Based Testing and Automated Analysis of Behavioural Models using GraphWalker and UPPAAL | |
| US11907111B2 (en) | Database troubleshooting with automated functionality | |
| Mansurov | Automatic synthesis of SDL from MSC and its applications in forward and reverse engineering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210330 |
|
| WD01 | Invention patent application deemed withdrawn after publication |