CN115277202A - Automatic data acquisition system and method for android APP - Google Patents

Automatic data acquisition system and method for android APP Download PDF

Info

Publication number
CN115277202A
CN115277202A CN202210896984.0A CN202210896984A CN115277202A CN 115277202 A CN115277202 A CN 115277202A CN 202210896984 A CN202210896984 A CN 202210896984A CN 115277202 A CN115277202 A CN 115277202A
Authority
CN
China
Prior art keywords
android
data
data acquisition
cloud server
app
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210896984.0A
Other languages
Chinese (zh)
Inventor
唐弋淞
徐桢虎
孙唐虎
付楷峰
占钦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sichuan Cover Media Technology Co ltd
Original Assignee
Sichuan Cover Media Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sichuan Cover Media Technology Co ltd filed Critical Sichuan Cover Media Technology Co ltd
Priority to CN202210896984.0A priority Critical patent/CN115277202A/en
Publication of CN115277202A publication Critical patent/CN115277202A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/30Network architectures or network communication protocols for network security for supporting lawful interception, monitoring or retaining of communications or communication related information
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/08Network architectures or network communication protocols for network security for authentication of entities
    • H04L63/0823Network architectures or network communication protocols for network security for authentication of entities using certificates
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/30Network architectures or network communication protocols for network security for supporting lawful interception, monitoring or retaining of communications or communication related information
    • H04L63/308Network architectures or network communication protocols for network security for supporting lawful interception, monitoring or retaining of communications or communication related information retaining data, e.g. retaining successful, unsuccessful communication attempts, internet access, or e-mail, internet telephony, intercept related information or call content
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/22Parsing or analysis of headers

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Hardware Design (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Technology Law (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention relates to the technical field of data acquisition, discloses an automatic data acquisition system and method for android APP (application), aims to solve the problems of time consumption, labor consumption and high difficulty of the conventional data acquisition mode of the android APP, and mainly comprises the following steps: connecting the cloud server with a plurality of android devices through an ADB program; the cloud server acquires identity information of the currently connected android device, acquires a data acquisition task according to the identity information, and executes a corresponding automation script according to the data acquisition task; the proxy server starts the proxy module when the cloud server executes the automation script, the proxy module monitors and acquires requests initiated by all android devices in real time, corresponding response data are acquired according to the requests, required data are acquired from the response data, and data landing is carried out on the required data. The method and the device reduce the consumption of manpower and material resources for APP data acquisition, improve the data acquisition efficiency, and are particularly suitable for news data acquisition.

Description

Automatic data acquisition system and method for android APP
Technical Field
The invention relates to the technical field of data acquisition, in particular to an automatic data acquisition system and method for android APP.
Background
With the development of the mobile internet, netizens rely on mobile phone APPs more and more, and clients are pushed out from various large websites to attract users. At present, mobile phone APP is various in form and complex in content, and is a major difficulty in network public opinion monitoring of a mobile terminal, and data acquisition of a news APP is an important link for public opinion monitoring, and a breakthrough needs to be made. However, as data security and network security are more and more emphasized, the security level of the client of each major website is also higher and higher, automatic acquisition of APP data is difficult to achieve by the conventional packet capturing and reverse APP data acquisition-based scheme, and corresponding programs need to be customized for different APPs, so that time and labor are consumed very much in development, and the application scene requirements of large-scale and large-data APP data acquisition cannot be gradually met.
Disclosure of Invention
The invention aims to solve the problems of time and labor consumption and high difficulty of the existing data acquisition mode of android APP, and provides an automatic data acquisition system and method for the android APP.
The technical scheme adopted by the invention for solving the technical problems is as follows:
in one aspect, an automated data acquisition system for android APP is provided, comprising: the android system comprises a cloud server, a proxy server and a plurality of android devices, wherein the cloud server is connected with each android device through an ADB program;
the cloud server is used for determining the currently connected android device, acquiring the identity information of the currently connected android device, acquiring a data acquisition task according to the identity information, and executing a corresponding automation script according to the data acquisition task, wherein the automation script is used for controlling the android device to execute corresponding operation;
the proxy server is used for starting the proxy module when the cloud server executes the automation script, the proxy module monitors and acquires requests initiated by all android devices in real time, acquires corresponding response data according to the requests, acquires required data from the response data and performs data landing on the data;
the android device is provided with at least one android APP and is used for receiving control of the cloud server and executing corresponding operation on the android APP when the cloud server executes the automation script, receiving monitoring of the proxy server when the operation is executed and sending an initiated request to the proxy server.
Further, the cloud server is specifically configured to:
after the identity information of the currently connected android device is acquired, establishing a process for each android device and simultaneously running all the processes; and in each process, acquiring a data acquisition task according to the identity information of the corresponding android device, and executing an automatic script for controlling the corresponding android device according to the data acquisition task.
Further, after the proxy server starts the proxy module, the method further includes:
the proxy server exposes a monitoring port to the outside, sets a proxy address and a port of the android device connected at present as a server address and a port of the proxy server, and installs a proxy certificate on the android device connected at present.
Further, the cloud server acquires a data acquisition task from a database, a time difference between the last acquisition time corresponding to the data acquisition task and the current time is greater than the acquisition interval time, and the data acquisition task is the data acquisition task with the highest priority or the previous acquisition time corresponding to the same priority.
Further, acquiring the required data from the response data specifically includes:
and performing data analysis on the response data to obtain the required data, wherein the data analysis mode is XPath, CSS or regular expression.
Further, the data is in the form of a mysql database, a mongo database, a redis database, or a kafka database.
Further, the automation script is specifically configured to:
and controlling the android device to install, open and uninstall the android APP, and clicking page elements in the android APP.
On the other hand, an automatic data acquisition method for android APP is provided, which is applied to the automatic data acquisition system for android APP and comprises the following steps:
step 1, connecting a cloud server with a plurality of android devices through an ADB program;
step 2, the cloud server determines the currently connected android device, acquires the identity information of the currently connected android device, acquires a data acquisition task according to the identity information, and executes a corresponding automatic script according to the data acquisition task, wherein the automatic script is used for controlling the android device to execute corresponding operation;
and 3, the proxy server starts a proxy module when the cloud server executes the automation script, the proxy module monitors and acquires requests initiated by all android devices in real time, corresponding response data are acquired according to the requests, required data are acquired from the response data, and data landing is carried out on the required data.
Further, the method specifically comprises:
after the identity information of the currently connected android device is acquired, establishing a process for each android device and simultaneously running all the processes; and in each process, acquiring a data acquisition task according to the identity information of the corresponding android device, and executing an automatic script for controlling the corresponding android device according to the data acquisition task.
Further, the cloud server acquires a data acquisition task from a database, a time difference between the last acquisition time corresponding to the data acquisition task and the current time is greater than the acquisition interval time, and the data acquisition task is the data acquisition task with the highest priority or the previous acquisition time corresponding to the same priority.
The beneficial effects of the invention are: according to the automatic data acquisition system and method for the android APP, the cloud server runs the automatic scripts to control the android devices to execute corresponding operations, and meanwhile, the android APP is subjected to data acquisition when the android devices execute the operations.
Drawings
Fig. 1 is a schematic structural diagram of an automated data acquisition system for android APP according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of an automated data acquisition method for android APP according to an embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
The invention provides an automatic data acquisition system for android APP, which comprises: the android system comprises a cloud server, a proxy server and a plurality of android devices, wherein the cloud server is connected with each android device through an ADB program; the cloud server is used for determining the currently connected android device, acquiring the identity information of the currently connected android device, acquiring a data acquisition task according to the identity information, and executing a corresponding automation script according to the data acquisition task, wherein the automation script is used for controlling the android device to execute corresponding operation; the proxy server is used for starting the proxy module when the cloud server executes the automation script, the proxy module monitors and acquires requests initiated by all android devices in real time, acquires corresponding response data according to the requests, acquires required data from the response data and performs data landing on the data; the android device is provided with at least one android APP and is used for receiving control of the cloud server and executing corresponding operation on the android APP when the cloud server executes the automation script, receiving monitoring of the proxy server when the operation is executed and sending an initiated request to the proxy server.
The android device and the cloud server are connected, the main program automatically acquires all android devices, an optimal task is acquired from a database according to identity information of the android devices, the automatic script stored in a corresponding position of the cloud server is selected to run according to task parameters, the cloud server can control the android devices to open corresponding operations such as APP opening and page element clicking according to script corresponding rules when executing the automatic script, the android devices are in the operation process, data interaction can be generated between a client side of the APP and a server side, requests and response data are generated, the proxy server can monitor requests of the android devices in real time and intercept useful requests of the APPs, response data are acquired, data analysis is performed on the requests by using a certain technical means, then the analyzed data fall to the ground, and subsequent data application is facilitated.
Examples
The automatic data acquisition system for android APP, as shown in fig. 1, includes: the system comprises a cloud server, a proxy server and a plurality of android devices; the cloud server and the proxy server may be the same server.
Based on the above system, the automatic data acquisition method for android APP, described in this embodiment, as shown in fig. 2, includes the following steps:
step 1, connecting a cloud server with a plurality of android devices through an ADB program;
in this embodiment, if the cloud server and the proxy server are the same server, the server is directly connected with each android device through an ADB program; if the cloud server and the proxy server are not the same server, the proxy server needs to be connected with the cloud server and each android device respectively.
The ADB (Android Debug Bridge) is an Android Debug Bridge, and the Android program can be conveniently debugged in Eclipse through the DDMS through the ADB program.
Step 2, the cloud server determines the currently connected android device, acquires the identity information of the currently connected android device, acquires a data acquisition task according to the identity information, and executes a corresponding automatic script according to the data acquisition task, wherein the automatic script is used for controlling the android device to execute corresponding operation;
specifically, in the embodiment, after the identity information of the currently connected android device is acquired, one process is established for each android device and all the processes are run simultaneously; in each process, a data acquisition task is obtained according to the identity information (such as the equipment number) of the corresponding android equipment, and an automation script for controlling the corresponding android equipment is executed according to the data acquisition task.
In this embodiment, the cloud server obtains a data acquisition task from a database, a time difference between a last acquisition time corresponding to the data acquisition task and a current time is greater than an acquisition interval time, and the data acquisition task is a data acquisition task with a highest priority or a previous acquisition time corresponding to the same priority.
For each process, the currently optimal data acquisition task, that is, the data acquisition task with the acquisition time before the acquisition time when the time difference between the last acquisition time and the current time is outside the acquisition interval time and the priority is highest or the same, is read from the database.
The acquisition interval time can be set according to actual conditions, and the embodiment does not limit the acquisition interval time.
After the data acquisition task is acquired, the automatic script stored in the corresponding position of the cloud server is selected to run according to the task parameters, and the cloud server can control the android device to perform corresponding operations such as installation, unloading, opening of the APP and page element clicking according to the script corresponding rule when executing the automatic script, for example, the android device is controlled to be installed and a news client is opened, and the corresponding page element is clicked in a news client interface, so that the news data of the news client can be acquired.
And 3, the proxy server starts a proxy module when the cloud server executes the automation script, the proxy module monitors and acquires requests initiated by all android devices in real time, acquires corresponding response data according to the requests, acquires required data from the response data and performs data landing on the data.
Specifically, the proxy server exposes the monitoring port to the outside after the proxy module is opened, sets the proxy address and the port of the android device connected at present as the server address and the port of the proxy server, and installs the proxy certificate on the android device connected at present. After the configuration is completed, the agent module can monitor all requests sent by the android device in real time and perform corresponding data processing operation according to different requests and different requirements. The module provides the operations of acquiring a request data packet, acquiring a request header and response data, and extracts required data from the response data according to the encapsulation rule.
And after the agent module obtains the response data, performing data analysis on the response data to obtain the required data, and performing data landing on the required data. In this embodiment, the data parsing manner may be XPath (XML Path Language), CSS (Cascading Style Sheets), or regular expression, and the data ground form may be mysql database, mongo database, redis database, or kafka database.
In summary, according to the automatic data acquisition system and method for android APPs described in this embodiment, after an android device is connected to a cloud server, a main program automatically acquires all the android devices, and establishes a process for each android device, so as to implement multi-process parallelism, each process acquires an optimal task from a database, an automatic script stored in a corresponding position of the cloud server is selected to run according to task parameters, the cloud server can control the android device to perform corresponding operations such as opening the APP and clicking page elements according to script correspondence rules, during an operation process of the android device, a client of the APP can perform data interaction with a server to generate request and response data, the proxy server can monitor requests of the android devices in real time and intercept useful requests of the APP, acquire response data corresponding to the requests, perform data analysis on the response data by using a certain technical means, and then place the analyzed data to facilitate data application. The APP data acquisition flow is simplified by the aid of the method, the automation script is easy to develop, manpower and material resources for APP data acquisition are reduced, and meanwhile, data acquisition efficiency is improved.

Claims (10)

1. An automated data acquisition system for android APP, comprising: the android system comprises a cloud server, a proxy server and a plurality of android devices, wherein the cloud server is connected with each android device through an ADB program;
the cloud server is used for determining the currently connected android device, acquiring the identity information of the currently connected android device, acquiring a data acquisition task according to the identity information, and executing a corresponding automation script according to the data acquisition task, wherein the automation script is used for controlling the android device to execute corresponding operation;
the proxy server is used for starting the proxy module when the cloud server executes the automation script, the proxy module monitors and acquires requests initiated by all android devices in real time, acquires corresponding response data according to the requests, acquires required data from the response data and performs data landing on the data;
the android device is provided with at least one android APP and is used for receiving control of the cloud server and executing corresponding operation on the android APP when the cloud server executes the automation script, receiving monitoring of the proxy server when the operation is executed and sending an initiated request to the proxy server.
2. The automated data acquisition system for android APP of claim 1, the cloud server specifically configured to:
after the identity information of the currently connected android device is acquired, establishing a process for each android device and simultaneously running all the processes; and in each process, acquiring a data acquisition task according to the identity information of the corresponding android device, and executing an automatic script for controlling the corresponding android device according to the data acquisition task.
3. The automated data acquisition system for android APP of claim 1 further comprising, after the proxy server starts a proxy module:
the proxy server exposes a monitoring port to the outside, sets a proxy address and a port of the android device connected at present as a server address and a port of the proxy server, and installs a proxy certificate on the android device connected at present.
4. The automated data acquisition system for android APP of claim 1, wherein the cloud server obtains a data acquisition task from a database, a time difference between a last acquisition time corresponding to the data acquisition task and a current time is greater than an acquisition interval time, and the data acquisition task is a data acquisition task with a highest priority or a previous acquisition time corresponding to the same priority.
5. The automated data acquisition system for android APPs of claim 1, in which obtaining desired data from the response data specifically comprises:
and performing data analysis on the response data to obtain the required data, wherein the data analysis mode is XPath, CSS or regular expression.
6. The automated data collection system for android APPs of claim 1, in which the data floor is in the form of a mysql database, a mongo database, a redis database, or a kafka database.
7. The automated data collection system for android APPs of claim 1, the automation script specifically to:
and controlling the android device to install, open and uninstall the android APP, and clicking page elements in the android APP.
8. An automated data acquisition method for android APP, applied to the automated data acquisition system for android APP as claimed in any one of claims 1 to 7, comprising the steps of:
step 1, connecting a cloud server with a plurality of android devices through an ADB program;
step 2, the cloud server determines the currently connected android device, acquires the identity information of the currently connected android device, acquires a data acquisition task according to the identity information, and executes a corresponding automatic script according to the data acquisition task, wherein the automatic script is used for controlling the android device to execute corresponding operation;
and 3, the proxy server starts a proxy module when the cloud server executes the automation script, the proxy module monitors and acquires requests initiated by all android devices in real time, acquires corresponding response data according to the requests, acquires required data from the response data and performs data landing on the data.
9. The automated data collection method for android APP of claim 8, the method specifically comprising:
after the identity information of the currently connected android device is acquired, establishing a process for each android device and simultaneously running all the processes; and in each process, acquiring a data acquisition task according to the identity information of the corresponding android device, and executing an automatic script for controlling the corresponding android device according to the data acquisition task.
10. The automated data collection method for android APPs of claim 8, wherein the cloud server obtains a data collection task from a database, a time difference between a last collection time corresponding to the data collection task and a current time is greater than a collection interval time, and the data collection task is a data collection task with a highest priority or a previous collection time corresponding to the same priority.
CN202210896984.0A 2022-07-28 2022-07-28 Automatic data acquisition system and method for android APP Pending CN115277202A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210896984.0A CN115277202A (en) 2022-07-28 2022-07-28 Automatic data acquisition system and method for android APP

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210896984.0A CN115277202A (en) 2022-07-28 2022-07-28 Automatic data acquisition system and method for android APP

Publications (1)

Publication Number Publication Date
CN115277202A true CN115277202A (en) 2022-11-01

Family

ID=83771057

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210896984.0A Pending CN115277202A (en) 2022-07-28 2022-07-28 Automatic data acquisition system and method for android APP

Country Status (1)

Country Link
CN (1) CN115277202A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116233301A (en) * 2023-02-20 2023-06-06 深圳感臻智能股份有限公司 Method, system and device for configuring WiFisensing by utilizing android system

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106357670A (en) * 2016-10-17 2017-01-25 成都知道创宇信息技术有限公司 Simulator-based android application server side Web vulnerability detection method
CN107733735A (en) * 2017-06-01 2018-02-23 云智慧(北京)科技有限公司 Script recording system and method
US20180262533A1 (en) * 2017-03-13 2018-09-13 Comcast Cable Communications, Llc Monitoring Device Data and Gateway Data
CN109710140A (en) * 2018-12-25 2019-05-03 清创网御(合肥)科技有限公司 A kind of smart phone social application automatic data collection method
CN110825985A (en) * 2018-08-06 2020-02-21 阿里巴巴集团控股有限公司 Data acquisition system, method and device, control equipment and proxy equipment
CN112199568A (en) * 2020-09-28 2021-01-08 时趣互动(北京)科技有限公司 Mobile APP data capturing method, device and system
CN112882953A (en) * 2021-03-24 2021-06-01 京东方科技集团股份有限公司 Automatic demonstration method and system for application program based on android debugging bridge
US11086869B1 (en) * 2016-09-26 2021-08-10 Splunk Inc. Data intake and query system gateway

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11086869B1 (en) * 2016-09-26 2021-08-10 Splunk Inc. Data intake and query system gateway
CN106357670A (en) * 2016-10-17 2017-01-25 成都知道创宇信息技术有限公司 Simulator-based android application server side Web vulnerability detection method
US20180262533A1 (en) * 2017-03-13 2018-09-13 Comcast Cable Communications, Llc Monitoring Device Data and Gateway Data
CN107733735A (en) * 2017-06-01 2018-02-23 云智慧(北京)科技有限公司 Script recording system and method
CN110825985A (en) * 2018-08-06 2020-02-21 阿里巴巴集团控股有限公司 Data acquisition system, method and device, control equipment and proxy equipment
CN109710140A (en) * 2018-12-25 2019-05-03 清创网御(合肥)科技有限公司 A kind of smart phone social application automatic data collection method
CN112199568A (en) * 2020-09-28 2021-01-08 时趣互动(北京)科技有限公司 Mobile APP data capturing method, device and system
CN112882953A (en) * 2021-03-24 2021-06-01 京东方科技集团股份有限公司 Automatic demonstration method and system for application program based on android debugging bridge

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116233301A (en) * 2023-02-20 2023-06-06 深圳感臻智能股份有限公司 Method, system and device for configuring WiFisensing by utilizing android system

Similar Documents

Publication Publication Date Title
CN111083225A (en) Data processing method and device in Internet of things platform and Internet of things platform
CN103248651B (en) A kind of method and system of performance monitoring and client and server
CN110224896B (en) Network performance data acquisition method and device and storage medium
CN109560996B (en) Automatic testing system and method for terminal of Internet of things
CN111158891A (en) Analysis task processing method and device based on Flink technology and storage medium
CN108234246B (en) Method and system for multi-directionally testing network performance of server
CN107229566B (en) Method and system for maintaining functional consistency of SOA service conversion language
CN104932974A (en) Script conversion method and script conversion device
CN115277202A (en) Automatic data acquisition system and method for android APP
CN111211934A (en) Cluster remote communication test method and system
CN109032705B (en) Application program execution method and device, electronic equipment and readable storage medium
CN111523074A (en) Acquisition system for dynamic page sensitive data of front-end rendering website
CN112953983A (en) SFTP transmission method and device
CN114666407A (en) RPA cross-terminal communication method and equipment
CN115329170A (en) Webpage crawling method, device, equipment and storage medium
CN111769979A (en) Cloud resource arranging method, system, terminal and storage medium
CN105095070B (en) QQ group's data capture method and system based on browser testing component
CN112966262B (en) Method and device for generating operation log, electronic equipment and storage medium
CN112256557B (en) Program regression testing method, device, system, computer equipment and storage medium
CN111245917B (en) Katalon-based work order entry device and implementation method thereof
CN104104969B (en) Video interception method and device
CN108874659B (en) Test method, test system and server of digital currency computing equipment
CN115904388A (en) Application program protocol analysis method, device, equipment and storage medium
CN109634856A (en) IOS intelligent terminal remote real machine debugging system and method based on cloud service
CN113821406A (en) Method, device and equipment for monitoring webpage and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination