CN112650796A - Automatic application data collection and storage management system - Google Patents
Automatic application data collection and storage management system Download PDFInfo
- Publication number
- CN112650796A CN112650796A CN201910912434.1A CN201910912434A CN112650796A CN 112650796 A CN112650796 A CN 112650796A CN 201910912434 A CN201910912434 A CN 201910912434A CN 112650796 A CN112650796 A CN 112650796A
- Authority
- CN
- China
- Prior art keywords
- data
- data collection
- module
- storage
- control
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an automatic application data collection and storage management system, which comprises: the system comprises a data collection module, a data preprocessing module and a data storage module; the data collection module is used for collecting application data in a page automatic traversal and user behavior tracking mode to obtain control information and a data screenshot: the data preprocessing module is used for extracting information in the control according to a predefined template, performing character recognition on the intercepted picture in a character definition and recognition mode, and performing data cleaning and duplicate removal; and the data storage module is used for storing the collected data after the cleaning and the de-duplication according to a preset storage rule. Compared with the traditional manual method, the invention can save more than 20 times of time for the application data collection efficiency of the IOS, android and other mobile devices, and can save more than 10 times of time for the data collection efficiency of the application on the computer terminal.
Description
Technical Field
The invention relates to the technical field of application data receiving and processing, in particular to an automatic application data collecting and storing management system.
Background
At present, various types of applications are diversified in the market, and how to quickly collect terminal data is a great problem encountered in enterprises. Most enterprises spend a lot of money and time to construct business systems and office automation systems of online transaction OLTP for recording various relevant data of transactions. How enterprises convert data into information and knowledge through various technical means has become a major bottleneck for improving the core competitiveness of the enterprises. ETL is one of the main technical means. To make ETL, data collection is needed first, and data in various types of applications is collected quickly and efficiently, so that various transformations and transformations can be performed on the data subsequently, and data easy to analyze and understand can be constructed. In the case that data storage cannot be directly read, for example, when a database cannot be accessed, a common acquisition method needs to collect data, including a crawler and the like, but the crawler can only be directed to web services. Compared with the web data source which is relatively clear in structure and easy to acquire data, the method has the advantage that various kinds of software and applications which are mainly faced by the method are relatively more difficult to acquire data. For various application programs, common methods such as manual extraction are time-consuming and labor-consuming. The method of embedding points in application is only suitable for the condition that the program source code is owned and the modification authority is obtained, and has great limitation. The inventor of the application finds that the method in the prior art at least has the following technical problems that the existing program data collection method is large in limitation and cannot adapt to different application scenes. At the same time, an automated data collection method and storage and management of collected data are lacking.
Disclosure of Invention
The invention aims to provide an automatic application data collection and storage management system aiming at the technical defects in the prior art, which can support applications in mobile devices such as IOS (input operation system), android and the like and applications at a computer end.
The technical scheme adopted for realizing the purpose of the invention is as follows:
an automated application data collection, storage management system, comprising:
the system comprises a data collection module, a data preprocessing module and a data storage module;
the data collection module is used for collecting application data in a page automatic traversal and user behavior tracking mode to obtain control information and a data screenshot;
the data preprocessing module is used for extracting information in the control according to a predefined template, performing character recognition on the intercepted picture in a character definition and recognition mode, and performing data cleaning and duplicate removal;
and the data storage module is used for storing the collected data after the cleaning and the de-duplication according to a preset storage rule.
The automatic application data collection and storage management system further comprises a data management module, wherein the data management module is used for providing management for the data collection module, the data preprocessing module, the state monitoring and service calling of the data storage module, monitoring the real-time service condition and data collection speed of the data collection module in real time, monitoring the data storage condition and capacity of the data storage module in real time, and starting, suspending and stopping the data collection service.
When the data preprocessing module is used for character recognition, firstly, preprocessing is carried out on an image, the area where the character is located is judged through image processing, and then, a neural network model is used for character recognition.
The data storage module adopts three-level data storage from bottom to top: the first-stage data storage comprises that each collection terminal firstly stores original data locally and carries out preprocessing, the second-stage data storage comprises that each type terminal collects data stored in the corresponding first stage respectively and carries out unified storage, and the third-stage data storage comprises that all data stored in the second-stage storage are collected and carried out unified storage.
The automatic page traversal of the data collection module is realized by matching a test frame operation terminal with a depth-first traversal algorithm to perform page traversal, acquiring handle and control information and performing data screenshot in the traversal process, and generating a corresponding function process diagram; in the page traversal process, each behavior triggers the recording of information in the control and the screenshot, and data collection is completed in this way; and establishing indexes for collected data for optimization.
Furthermore, in the automatic traversal of the page, the packaged data interface can be operated by combining with the test framework, and the related operation script can be directly compiled.
The user behavior tracking of the data collection module comprises monitoring specific operations of a user on each control or a specific screen coordinate position and outputting a process file of detailed behaviors of the user; then, analyzing which controls and positions are operated by the user and what operations are respectively performed, extracting the sequential relation of the control and the operations of each position during simulation, and outputting a script for simulation operation; and finally, simulating a behavior mode used by a user for collecting data automatically through simulation, completely operating each page of the user interface in sequence, regularly acquiring the operation interface of the software user, finishing screenshot of the data page and simultaneously storing information in the control.
Wherein, the specific operation comprises clicking, inputting and dragging.
The monitoring user operation acquires the control currently operated when detecting the occurrence of the control event, the control information comprises the control type, the control type name, the number, the position coordinate of the control and the control name, and the handle information of the operation window is collected.
According to the invention, aiming at the automatic data collection processing of various types of application programs, the time can be saved by more than 20 times aiming at the application data collection efficiency of mobile equipment such as IOS, android and the like compared with the traditional manual method, and the time can be saved by more than 10 times aiming at the data collection efficiency of computer-side application.
Drawings
FIG. 1 is a schematic diagram of an automated application data collection, storage management system of the present invention;
FIG. 2 is a flow diagram of a data collection process for an automated application data collection, storage management system;
FIG. 3 is an architecture diagram of an automated application data collection, storage management system;
FIG. 4 is a flow diagram illustrating a process for automated traversal of a page during data collection;
FIG. 5 is a flow chart illustrating the operation of tracking a particular coordinate location in a screen.
Fig. 6 is a flow chart showing operations of listening users.
Detailed Description
The invention is described in further detail below with reference to the figures and specific examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1-6, the automated application data collection and storage management system of the present invention comprises: the system comprises a data collection module, a data preprocessing module, a data storage module and a data management module.
When the data collection module collects data, the data collection module collects applied data in a mode of page automatic traversal and user behavior tracking to obtain control information and a data screenshot:
in the invention, the automatic page traversal is to operate a terminal by means of a test frame (such as airtest) for mobile application, perform page traversal by matching with a depth-first traversal algorithm, acquire handle and control information in the traversal process, perform data screenshot, and simultaneously generate a corresponding functional process diagram. In the page traversal process, each behavior triggers the recording of information in the control and the screenshot, and data collection is completed in this way. The data collection results may be indexed using a hash table or B-tree based on file name to optimize retrieval efficiency.
The specific process is that elements and controls are extracted through an xpath page, the controls are screened according to set relevant rules, finally, path record and redirection, popup (installation process, authority, app prompt) and the like are obtained through depth-first traversal, then handle and control information is obtained, data screenshot is carried out, and meanwhile a corresponding functional process diagram is generated.
In addition, the data interface of the package can be operated in combination with a test framework (such as airtest), and related operation script writing can be directly carried out.
It should be noted that, the automatic page traversal is for common mobile terminals such as IOS, android, or computer-side applications.
In the invention, the user behavior tracking comprises the following specific steps:
firstly, monitoring specific operations of a user on each control or a specific coordinate position in a screen, and outputting a process file of detailed behaviors of the user, wherein the common specific operations comprise clicking, inputting, dragging and the like.
And then analyzing which controls and positions are operated by the user and what operations are respectively performed, extracting the sequential relation of the controls and the positions in the simulation, and outputting a script for the simulation operation. The specific process is that after the collected process record is read in, the script generation judgment is carried out. Reading all records into a multi-dimensional tuple, and further performing judgment function running sequence and cycle judgment of the script by using a simple directed graph model to be generated. For data read according to rows, the calling relationship among functions can be acquired easily, and therefore a corresponding control operation directed graph is generated.
And finally, automatically simulating a behavior mode used by a user for collecting data by using the simulation module, completely operating each page of the user interface in sequence, regularly acquiring the operation interface of the software user, finishing screenshot of the data page and simultaneously storing the information in the control.
It should be noted that, in the present invention, when the monitoring user operation detects the occurrence of a control event, the monitoring user operation may obtain a currently operated control through tools such as observation, UI Automation, airtest, and the like, where the control information includes five properties such as a control type, a control type name, a number, a position coordinate where the control is located, and a control name, and collects handle information of an operation window.
In the invention, the data preprocessing module collects the information data obtained from the control according to a predefined template. And extracting characters in the intercepted picture in a character definition and identification mode.
The character recognition function firstly needs to carry out preprocessing operation on the image, can judge the area where the character is located through image processing, and then carries out character recognition by using a designed neural network model.
In the present invention, the data preprocessing module further includes data cleaning and data integration functions, wherein the data cleaning includes algorithms such as dirty data removal and duplicate data deduplication.
In the invention, the data storage module comprises three stages of data storage devices from bottom to top, the first stage of data storage comprises that each collection terminal firstly stores original data locally and carries out preprocessing, the second stage of data storage comprises that each type terminal respectively collects and stores data stored in the corresponding first stage in a unified way, and the third stage of data storage comprises that all data in the second stage of storage are collected and stored in a unified way. The writing efficiency during data collection is effectively improved through the three-level storage mechanism, organization and management are facilitated, and meanwhile, a disaster recovery mechanism is provided through the main library and the standby library.
In the invention, the data management module provides state monitoring and service calling for the data collection module, the data preprocessing module and the data storage module, can monitor the real-time service condition, the data collection speed and the like of the data collection module in real time, can monitor the data storage condition and the capacity of the data storage module in real time, and can carry out start-pause-stop management and the like on the data collection service.
The invention provides an automatic data collection system for various types of application programs, and also provides a scalable multi-level data collection, storage and management service, compared with the traditional manual method, the automatic data collection system for the IOS, android and other mobile devices can save more than 20 times of time, and the automatic data collection system for the computer-side application can save more than 10 times of time.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (9)
1. An automated application data collection, storage management system, comprising:
the system comprises a data collection module, a data preprocessing module and a data storage module;
the data collection module is used for collecting application data in a page automatic traversal and user behavior tracking mode to obtain control information and a data screenshot;
the data preprocessing module is used for extracting information in the control according to a predefined template, performing character recognition on the intercepted picture in a character definition and recognition mode, and performing data cleaning and duplicate removal;
and the data storage module is used for storing the collected data after the cleaning and the de-duplication according to a preset storage rule.
2. The automated application data collection and storage management system of claim 1, further comprising a data management module configured to provide a start-up, pause, and stop management for the data collection service by monitoring the status of the data collection module, the data preprocessing module, and the data storage module and invoking the service, monitoring the real-time service status of the data collection module, the data collection speed, and monitoring the data storage status and capacity of the data storage module.
3. The automated application data collection and storage management system of claim 1, wherein the data preprocessing module first preprocesses the image during character recognition, determines the region where the character is located through image processing, and then performs character recognition using the neural network model.
4. The automated application data collection, storage management system of claim 1, wherein the data storage module employs a bottom-up three-level data storage: the first-stage data storage comprises that each collection terminal firstly stores original data locally and carries out preprocessing, the second-stage data storage comprises that each type terminal collects data stored in the corresponding first stage respectively and carries out unified storage, and the third-stage data storage comprises that all data stored in the second-stage storage are collected and carried out unified storage.
5. The automated application data collection and storage management system of claim 1, wherein the page automatic traversal of the data collection module is performed by means of a test framework operation terminal in cooperation with a depth-first traversal algorithm, wherein in the traversal process, handle and control information are acquired and data screenshots are performed, and meanwhile, a corresponding functional process diagram is generated; in the page traversal process, each behavior triggers the recording of information in the control and the screenshot, and data collection is completed in this way; and establishing indexes for collected data for optimization.
6. The automated application data collection and storage management system of claim 5, wherein in the page automation traversal, the related operational script writing can be directly performed in combination with the data interface of the test framework operation package.
7. The automated application data collection and storage management system of claim 1, wherein the user behavior tracking of the data collection module comprises monitoring specific operations performed by a user on each control or specific screen coordinate locations, and outputting a process file of detailed behaviors of the user; then, analyzing which controls and positions are operated by the user and what operations are respectively performed, extracting the sequential relation of the control and the operations of each position during simulation, and outputting a script for simulation operation; and finally, simulating a behavior mode used by a user for collecting data automatically through simulation, completely operating each page of the user interface in sequence, regularly acquiring the operation interface of the software user, finishing screenshot of the data page and simultaneously storing information in the control.
8. The automated application data collection, storage management system of claim 7, wherein the specific operations comprise clicking, inputting, dragging.
9. The automated application data collection and storage management system according to claim 7, wherein the monitoring user operation obtains a currently operated control while detecting the occurrence of a control event, the control information includes a control type, a control type name, a number, a position coordinate of the control, and a control name, and collects handle information of an operation window.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910912434.1A CN112650796A (en) | 2019-09-25 | 2019-09-25 | Automatic application data collection and storage management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910912434.1A CN112650796A (en) | 2019-09-25 | 2019-09-25 | Automatic application data collection and storage management system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112650796A true CN112650796A (en) | 2021-04-13 |
Family
ID=75342311
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910912434.1A Withdrawn CN112650796A (en) | 2019-09-25 | 2019-09-25 | Automatic application data collection and storage management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112650796A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342787A (en) * | 2021-04-28 | 2021-09-03 | 杭州千岛湖点通信息技术有限公司 | Big data research and development center data analysis system |
-
2019
- 2019-09-25 CN CN201910912434.1A patent/CN112650796A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342787A (en) * | 2021-04-28 | 2021-09-03 | 杭州千岛湖点通信息技术有限公司 | Big data research and development center data analysis system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107886238B (en) | Business process management system and method based on mass data analysis | |
| US11269822B2 (en) | Generation of automated data migration model | |
| CN109062780B (en) | Development method of automatic test case and terminal equipment | |
| CN109669933B (en) | Transaction data intelligent processing method and device and computer readable storage medium | |
| US10311111B2 (en) | Search method and apparatus and storage medium | |
| CN113688288B (en) | Data association analysis method, device, computer equipment and storage medium | |
| CN106227654A (en) | A kind of test platform | |
| CN111865673A (en) | Automatic fault management method, device and system | |
| CN115657890A (en) | PRA robot customizable method | |
| CN114419631A (en) | Network management virtual system based on RPA | |
| US8850407B2 (en) | Test script generation | |
| CN112650796A (en) | Automatic application data collection and storage management system | |
| CN112148602B (en) | Source code security analysis method based on history optimization feature intelligent learning | |
| Faiz et al. | Achieving better requirements to code traceability: Which refactoring should be done first? | |
| CN115794545A (en) | Automatic processing method of operation and maintenance data and related equipment thereof | |
| CN116069628A (en) | Intelligent-treatment software automatic regression testing method, system and equipment | |
| CN109033196A (en) | A kind of distributed data scheduling system and method | |
| CN114297961A (en) | Chip test case processing method and related device | |
| CN116991659A (en) | Virtual training system-oriented data collection system and method | |
| CN114328159A (en) | Abnormal statement determination method, device, equipment and computer readable storage medium | |
| JP5444071B2 (en) | Fault information collection system, method and program | |
| CN109992475A (en) | A kind of processing method of log, server and storage medium | |
| US11507728B2 (en) | Click to document | |
| CN114329125A (en) | Artificial intelligence data acquisition method and device for smart power grid | |
| CN116860754A (en) | Report data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210413 |