CN108197030B - Software interface automatic test cloud platform device based on deep learning and test method - Google Patents
Software interface automatic test cloud platform device based on deep learning and test method Download PDFInfo
- Publication number
- CN108197030B CN108197030B CN201810034716.1A CN201810034716A CN108197030B CN 108197030 B CN108197030 B CN 108197030B CN 201810034716 A CN201810034716 A CN 201810034716A CN 108197030 B CN108197030 B CN 108197030B
- Authority
- CN
- China
- Prior art keywords
- module
- neural network
- image
- database
- web
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a software interface automatic test cloud platform device based on deep learning, which comprises a plurality of user terminals and a cloud server, wherein each user terminal comprises a Web browser and an automatic screen capture module, each cloud server comprises a Web server, a database, a deep neural network model, a screen capture receiving module and a Web crawler module, the Web browser is in communication connection with the Web server, the automatic screen capture module is connected with the screen capture receiving module, the screen capture receiving module and the Web crawler module are respectively connected to the database, and the database is respectively connected with the Web server and the deep neural network model. The test method comprises the following steps: (1) automatically capturing a picture, and marking the captured picture with aesthetic degree; (2) training a deep neural network model; (3) and evaluating the newly input software interface image to evaluate the aesthetic degree of the newly input software interface image. The invention can realize automatic classification of different software interfaces according to the aesthetic degree, saves time and labor and has high efficiency.
Description
Technical Field
The invention relates to a cloud platform device, in particular to a software interface automatic test cloud platform device and a test method based on deep learning.
Background
Interface test (UI test for short) is used for testing whether the layout of functional modules of a user interface is reasonable, whether the overall style is consistent, whether the placement positions of all controls accord with the use habits of customers, and also for testing the convenience of interface operation, simplicity and comprehensiveness of navigation, usability of page elements, whether characters in the interface are correct, whether names are unified, whether pages are attractive, whether combinations of the characters and pictures are perfect, and the like.
The traditional software interface test mainly takes manual work as a main part, wastes time and labor, and has the problems of repeated test and the like.
Disclosure of Invention
The invention aims to provide a deep learning-based automatic testing cloud platform device and a deep learning-based automatic testing method for a software interface, so as to solve the problems in the background technology.
In order to achieve the purpose, the invention provides the following technical scheme:
the utility model provides a software interface automatic testing cloud platform device based on deep learning, includes a plurality of user terminal and cloud ware, every user terminal includes Web browser and automatic screenshot module, the cloud ware includes Web server, database, deep neural network model, screenshot receiving module, Web crawler module, Web browser and Web server communication are connected, automatic screenshot module is connected with screenshot receiving module, screenshot receiving module and Web crawler module are connected to the database respectively, the database is connected with Web server and deep neural network model respectively, Web crawler module is connected with the image search engine.
As a further scheme of the invention: the automatic screen capture module is deployed at a user terminal, captures different interfaces of a user when the user uses software, and uploads the different interfaces to the cloud server.
As a still further scheme of the invention: the screenshot receiving communication module is deployed in the cloud server, collects images uploaded by the user terminal automatic screenshot module, stores the images in a file system, and stores image information in a database.
As a still further scheme of the invention: the Web server is deployed in the cloud server and used for displaying the interface image to the user and receiving the evaluation of the user on the interface.
As a still further scheme of the invention: the network crawler module is deployed in a cloud server, downloads software interface images by using an image search engine on the Internet, stores the crawled images into a file system, and stores image information into a database.
As a still further scheme of the invention: the database is used for storing the information of the software interface screenshot image, including the serial number, the storage path and the corresponding software name.
As a still further scheme of the invention: the deep neural network module completes the function of automatically classifying the interface aesthetic degree based on the image.
As a still further scheme of the invention: the testing method for the automatic testing cloud platform device based on the deep learning software interface comprises the following steps:
(1) automatically capturing a picture when a user uses software, and marking the captured picture with aesthetic degree;
(2) training a deep neural network model by using the interface image with the mark;
(3) and evaluating the newly input software interface image by using the deep neural network model obtained by training, and evaluating the aesthetic degree of the newly input software interface image.
Compared with the prior art, the invention has the beneficial effects that: the invention can realize automatic classification of different software interfaces according to the aesthetic degree. The invention is based on the deep learning technology, carries out interface test on software, saves time and labor and has high efficiency.
Drawings
Fig. 1 is a schematic structural diagram of a deep learning-based software interface automatic test cloud platform device.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, in an embodiment of the present invention, an automatic software interface testing cloud platform device based on deep learning includes a plurality of user terminals and a cloud server, each of the user terminals includes a Web browser and an automatic screen capture module, the cloud server includes a Web server, a database, a Deep Neural Network (DNN) model, a screen capture receiving module, and a Web crawler module, the Web browser is in communication connection with the Web server, the Web server is disposed in the cloud server and is configured to display an interface image to a user and receive an evaluation of the interface by the user, the automatic screen capture module is connected with the screen capture receiving module, the automatic screen capture module is disposed in the user terminal and is configured to capture different interfaces of the user when the user uses software and upload the different interfaces to the cloud server, the screen capture receiving communication module is disposed in the cloud server and receives an image uploaded by the automatic screen capture module of the user terminal, storing the image into a file system and storing the image information into a database; the screenshot receiving module and the web crawler module are respectively connected to a database, the web crawler module is deployed on a cloud server, an image search engine on the Internet is used for downloading a software interface image, the crawled image is stored in a file system, and image information is stored in the database; the database is used for storing the information of the software interface screenshot image, including the serial number, the storage path and the corresponding software name; the database is respectively connected with a Web server and a Deep Neural Network (DNN) model, the Deep Neural Network (DNN) module completes the function of automatically classifying (evaluating) the aesthetic degree of an interface based on an image, and the Web crawler module is connected with an image search engine.
Several of the user terminals include the following functions: automatically screen-capturing, uploading, browsing and marking software interface screenshot images by a software interface; the cloud server includes the following functions: image data reception, storage, Deep Neural Network (DNN) model training and storage, and deployment of a Web server.
One, Deep Neural Network (DNN) model
1. The DNN model that Tensorflow implements automatic classification of images was selected. It is characterized in that:
(1) it has stable APIs with high compatibility (e.g., Keras and SkFlow, etc.), and its combination with Numpy is convenient to use.
(2) Which has significant platform flexibility. TensorFlow may be applicable in a variety of machines from HPC to embedded systems.
(3) With the support of Google and its community, more and more technologies and models are being added to the TensorFlow.
The invention utilizes TensorFlow to realize two Convolution Neural Network (CNN) algorithms. CNN is characterized by its good classification accuracy for all image classification datasets.
2. The CNN network model is selected, AlexNet and ResNet are selected, and the CNN network model is characterized by a deeper structure and good classification performance in image classification application.
(1) Alexnet, one of the CNN models we want to implement in this project, is characterized by: alexnet differs from the traditional LeNet CNN in that it is more accurate and efficient than LeNet due to its algorithm and network structure. Alexnet may be trained on multiple GPUs.
(2) With the continuous deepening of the neural network hierarchy, the classification result of the flat network has no obvious change. ResNet (residual neural network) aims to handle deeper network problems and to influence classification results. The residual neural network avoids the vanishing gradient problem by introducing short paths, which can deliver gradients over a very deep network.
Web technology
Ajax (asynchronous JavaScript and XML)
In the embodiment of the invention, the category labels of the pictures need to be browsed synchronously in the webpage and uploaded to the database of the server side. In addition to using the basic languages including HTML5, CSS, and JavaScript, and the PHP document for connection on the server side, embodiments of the present invention use Ajax to update portions of a web page, which is characterized by the fact that the entire page does not need to be reloaded.
With the present invention, when screenshots of the tested applications are uploaded to the cloud server, the Web page should display them immediately without the need to refresh the page to reload the content. Therefore, Ajax (asynchronous JavaScript and XML) can be used as a suitable technology to meet the requirement of the server to exchange a small amount of data with the server. It is more intuitive for testers and customers to browse through pictures and labels.
The Web technology used by the invention has the following characteristics:
1. there is an Ajax based MVC (model-view-controller) optimization that provides more visual data representation and better user experience.
2. The working process is as follows: first, Ajax sends a request from the browser. The server then accepts the request and returns a JSON data to the browser. Finally there will be an interface rendering process using front-end tools.
3. The optimized MVC is reduced to front-end and back-end partitioning patterns using REST techniques. REST and Ajax have relative rigidity, and a connection framework between the front end and the back end of the Web can be constructed optionally.
Third, automatic screenshot and uploading technology
In the invention, the application module automatically identifies the software user interface when finishing and capturing the image and uploads the software user interface to the cloud server. The embodiment of the invention takes C # as a development language, and is characterized by having a higher-level syntax system than Java, and the running platform required by C # is built in internal Windows, and ensures the compatibility of the software. In order to achieve the required screenshot function, the invention uses a method for calling Windows API, which is characterized in that the handle of the active window can be more easily obtained, the size and the position of the window can be obtained, and then the screenshot function can be realized.
After screenshot data are intercepted, the image data are uploaded to a cloud server, so that the method provides an uploading function in an automatic screenshot module and connects a cloud MySQL database server with the local. It requires remote access rights and can be accessed using any remote terminal.
By configuring the server binding address and adding the corresponding portal, the embodiment of the invention can connect the server at the client through the ADO. After the picture data is uploaded to the database, the DNN model can be used for training, and the picture data can also be displayed by a Web server.
In the embodiment of the invention, the test method for automatically testing the cloud platform device based on the deep learning software interface comprises the following steps:
(1) and automatically capturing the picture when the user uses the software, and marking the aesthetic degree (whether the picture is attractive or not) of the captured picture.
(2) And (4) training the deep neural network model by using the marked interface image.
(3) And evaluating the newly input software interface image by using the deep neural network model obtained by training, and evaluating the aesthetic degree of the newly input software interface image.
The construction of the model is recommended to be carried out by using a mature deep learning platform such as TensorFlow, Caffe and the like.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Furthermore, it should be understood that although the present description refers to embodiments, not every embodiment may contain only a single embodiment, and such description is for clarity only, and those skilled in the art should integrate the description, and the embodiments may be combined as appropriate to form other embodiments understood by those skilled in the art.
Claims (2)
1. A software interface automatic test cloud platform device based on deep learning comprises a plurality of user terminals and a cloud server, and is characterized in that each user terminal comprises a Web browser and an automatic screen capture module, the cloud server comprises a Web server, a database, a deep neural network model, a screen capture receiving module and a Web crawler module, the Web browser is in communication connection with the Web server, the automatic screen capture module is connected with the screen capture receiving module, the screen capture receiving module and the Web crawler module are respectively connected to the database, the database is respectively connected with the Web server and the deep neural network model, and the Web crawler module is connected with an image search engine; the automatic screen capturing module is deployed at a user terminal, captures different interfaces of a user when the user uses software, and uploads the different interfaces to the cloud server; the screenshot receiving module is deployed in the cloud server, receives the images uploaded by the automatic screenshot module of the user terminal, stores the images in the file system and stores the image information in the database; the Web server is deployed in the cloud server and used for displaying an interface image to a user and receiving the evaluation of the user on the interface; the network crawler module is deployed in a cloud server, downloads software interface images by using an image search engine on the Internet, stores the crawled images in a file system, and stores image information in a database; the database is used for storing the information of the software interface screenshot image, including the serial number, the storage path and the corresponding software name; the deep neural network module completes the function of automatically classifying the interface aesthetic degree based on the image;
selecting TensorFlow to realize a deep neural network model of automatic image classification, utilizing TensorFlow to realize two convolution neural network algorithms, specifically selecting the convolution neural network model, selecting AlexNet and ResNet, training the AlexNet on a plurality of GPUs, and leading in a short path by the ResNet to avoid the problem of vanishing gradient and transmitting the gradient in the deep network range.
2. The method for automatically testing the cloud platform device based on the deep learning software interface as claimed in claim 1, comprising the following steps:
(1) automatically capturing a picture when a user uses software, and marking the captured picture with aesthetic degree;
(2) training a deep neural network model by using the interface image with the mark;
(3) and evaluating the newly input software interface image by using the deep neural network model obtained by training, and evaluating the aesthetic degree of the newly input software interface image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810034716.1A CN108197030B (en) | 2018-01-15 | 2018-01-15 | Software interface automatic test cloud platform device based on deep learning and test method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810034716.1A CN108197030B (en) | 2018-01-15 | 2018-01-15 | Software interface automatic test cloud platform device based on deep learning and test method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108197030A CN108197030A (en) | 2018-06-22 |
| CN108197030B true CN108197030B (en) | 2021-03-30 |
Family
ID=62589393
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810034716.1A Active CN108197030B (en) | 2018-01-15 | 2018-01-15 | Software interface automatic test cloud platform device based on deep learning and test method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108197030B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109086201A (en) * | 2018-07-16 | 2018-12-25 | 曙光信息产业(北京)有限公司 | Automatic software test method and system |
| CN109324844A (en) * | 2018-09-30 | 2019-02-12 | 武汉斗鱼网络科技有限公司 | A kind of method, apparatus and computer equipment at detection window interface |
| CN110008993A (en) * | 2019-03-01 | 2019-07-12 | 华东师范大学 | A kind of end-to-end image-recognizing method based on deep neural network |
| CN112306846B (en) * | 2019-07-31 | 2022-02-11 | 北京大学 | Mobile application black box testing method based on deep learning |
| CN110647940A (en) * | 2019-09-25 | 2020-01-03 | 捻果科技(深圳)有限公司 | Airport apron foreign person monitoring method based on video analysis and deep learning |
| CN113837627B (en) * | 2021-09-28 | 2024-03-15 | 卡斯柯信号有限公司 | Title review platform and method based on text processing |
| CN114661612A (en) * | 2022-04-08 | 2022-06-24 | 郑州大学第一附属医院 | GUI testing method and system based on deep learning and storage device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101295371A (en) * | 2008-04-18 | 2008-10-29 | 浙江大学 | Computer estimation method of Chinese character writing morphology beauty |
| CN103544101A (en) * | 2012-07-16 | 2014-01-29 | 深圳市世纪光速信息技术有限公司 | Wireless platform development automatic test system and method |
| CN105894025A (en) * | 2016-03-30 | 2016-08-24 | 中国科学院自动化研究所 | Natural image aesthetic feeling quality assessment method based on multitask deep learning |
| CN107301425A (en) * | 2017-06-09 | 2017-10-27 | 浙江工业大学 | Children doodling scoring method based on deep learning |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9665802B2 (en) * | 2014-11-13 | 2017-05-30 | Nec Corporation | Object-centric fine-grained image classification |
-
2018
- 2018-01-15 CN CN201810034716.1A patent/CN108197030B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101295371A (en) * | 2008-04-18 | 2008-10-29 | 浙江大学 | Computer estimation method of Chinese character writing morphology beauty |
| CN103544101A (en) * | 2012-07-16 | 2014-01-29 | 深圳市世纪光速信息技术有限公司 | Wireless platform development automatic test system and method |
| CN105894025A (en) * | 2016-03-30 | 2016-08-24 | 中国科学院自动化研究所 | Natural image aesthetic feeling quality assessment method based on multitask deep learning |
| CN107301425A (en) * | 2017-06-09 | 2017-10-27 | 浙江工业大学 | Children doodling scoring method based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108197030A (en) | 2018-06-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108197030B (en) | Software interface automatic test cloud platform device based on deep learning and test method | |
| CN106294816B (en) | Page information processing method and device | |
| CN107885848B (en) | Webpage screen capturing method based on web technology | |
| CN109101425B (en) | Index point burying method and device for dynamic page AB test | |
| CN105808417B (en) | Automated testing method and proxy server | |
| CN106484383A (en) | page rendering method, device and equipment | |
| JP2009528555A5 (en) | ||
| CN103246678A (en) | Method and device for previewing web page contents | |
| CN108230113A (en) | User's portrait generation method, device, equipment and readable storage medium storing program for executing | |
| CN109375914A (en) | Information remote exchange method and system | |
| CN103761303A (en) | Method and device for picture arrangement display | |
| CN102279884A (en) | Data acquiring and displaying method, system and Web server based on Web pages | |
| CN108023918A (en) | A kind of information on services acquisition methods, apparatus and system | |
| CN104765746B (en) | Data processing method and device for mobile communication terminal browser | |
| CN104484361B (en) | Information shows method and device | |
| CN104504096B (en) | A kind of information transferring method and web page browsing device of inter-network page | |
| CN106844705A (en) | Method and apparatus for showing content of multimedia | |
| CN103544272A (en) | Method and device for displaying pictures in browser | |
| CN106941538A (en) | Information display system, method and apparatus | |
| Tehrani et al. | Heuristic evaluation for Virtual Museum on smartphone | |
| CN103971412B (en) | Electronic map shows method and device | |
| CN106570750A (en) | Browser plug-in-based automatic tax declaration method, system and browser plug-in | |
| CN103646115A (en) | Method and system for client side to access to webpage | |
| CN103685498A (en) | Wallpaper loading method and browser device | |
| CN110866712B (en) | Component intelligent logistics positioning method and system based on BIM model plug-in |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |