CN111309550A - Data acquisition method, system, equipment and storage medium of application program - Google Patents
Data acquisition method, system, equipment and storage medium of application program Download PDFInfo
- Publication number
- CN111309550A CN111309550A CN202010080290.0A CN202010080290A CN111309550A CN 111309550 A CN111309550 A CN 111309550A CN 202010080290 A CN202010080290 A CN 202010080290A CN 111309550 A CN111309550 A CN 111309550A
- Authority
- CN
- China
- Prior art keywords
- data
- buried point
- point
- interactive
- user behavior
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000003860 storage Methods 0.000 title claims abstract description 26
- 230000002452 interceptive effect Effects 0.000 claims abstract description 64
- 230000006399 behavior Effects 0.000 claims description 119
- 230000003993 interaction Effects 0.000 claims description 27
- 238000013480 data collection Methods 0.000 claims description 16
- 238000009933 burial Methods 0.000 claims description 13
- 238000007405 data analysis Methods 0.000 claims description 13
- 230000001960 triggered effect Effects 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 9
- 238000005457 optimization Methods 0.000 description 11
- 238000002474 experimental method Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 5
- 230000008878 coupling Effects 0.000 description 4
- 238000010168 coupling process Methods 0.000 description 4
- 238000005859 coupling reaction Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3086—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves the use of self describing data formats, i.e. metadata, markup languages, human readable formats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Library & Information Science (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the technical field of data processing, and provides a data acquisition method, a system, equipment and a storage medium for an application program. The data acquisition method comprises the following steps: receiving an access request of an application program, and generating interactive data, buried point data and a buried point ID, wherein the buried point data comprises an interactive behavior to be monitored; returning the interactive data and the embedded point ID to the application program to generate an interactive page, and sending the embedded point data to a message queue by taking the embedded point ID as an identifier; collecting user behavior data of an interactive behavior to be monitored of an interactive page, and reporting the user behavior data to a message queue by taking a buried point ID as an identifier; and acquiring buried point data and user behavior data with the same buried point ID from the message queue, and storing the buried point ID serving as an associated keyword into a data warehouse. The invention can reduce the data volume returned to the client, improve the running performance of the application program and reduce the flow consumption of the client; and the buried point data and the user behavior data are associated through the buried point ID, so that the accuracy and the efficiency of data acquisition are improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data acquisition method, a system, equipment and a storage medium of an application program.
Background
In the product optimization process, the application program needs to collect user data, and the product function is improved in a targeted manner according to the user data.
In the prior art, more than two improved schemes are often designed by using an ABTEST (A/B test), the improved schemes are respectively pushed according to user flow, users respectively see different scheme designs, and then product optimization decision is helped according to real data feedback of more than two groups of users.
The ABTEST effect is not separated from the collection and analysis of the data of the buried point, namely, the data of the user is obtained by monitoring the interactive behavior of the user in the using process of the application program through the buried point, and then the data analysis is carried out. In the prior art, after different buried point data are generated by a server according to user traffic distribution of an ABTEST, all the buried point data are returned to a client. When a user generates a preset interactive behavior, the client reports the data of the buried points and the data of the user to the server together, and then large data flow in for ABTEST experiment effect analysis.
An application typically includes a plurality of function blocks, each of which may correspond to a business segment. In the optimization process of the application program, each function block needs to be subjected to point burying, collection and analysis, so that the data of the point buried returned to the client is larger and larger, the performance of the client is reduced, the application program runs in a block state, and the waste of user flow is also caused.
It is to be noted that the information applied in the above background section is only for enhancing the understanding of the background of the present invention, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
In view of this, the present invention provides a data collection method, system, device and storage medium for an application program, which optimizes data embedding points and a collection process, and solves the problems of performance reduction and traffic waste of the application program caused by continuous and overstaffed data embedding points of a client.
One aspect of the present invention provides a data acquisition method for an application program, including the steps of: receiving an access request of an application program, and generating interactive data responding to the access request; generating buried point data according to the access request, wherein the buried point data comprises interaction behaviors to be monitored, and generating a buried point ID for uniquely identifying the buried point data; returning the interactive data and the buried point ID to the application program to generate an interactive page, and sending the buried point data to a message queue by taking the buried point ID as an identifier; collecting user behavior data of the interactive behavior to be monitored of the interactive page, and reporting the user behavior data to the message queue by taking the embedded point ID as an identifier; and acquiring buried point data and user behavior data with the same buried point ID from the message queue, and storing the buried point ID as an associated keyword to a data warehouse.
In some embodiments, the buried point data further includes a data object to be monitored, and the data object to be monitored is all or part of the data object in the interactive data; the user behavior data comprises a data object of the interactive behavior to be monitored in the interactive data and the generated interactive behavior to be monitored.
In some embodiments, each data object in the interaction data has a unique identification code, and the interaction behaviors to be listened to include an exposure behavior and a click behavior.
In some embodiments, the buried point ID is generated from a portion of key data in the buried point data that includes the data object to be listened to.
In some embodiments, the step of obtaining the burial point data and the user behavior data with the same burial point ID from the message queue is triggered when the reported user behavior data is received by the message queue.
In some embodiments, the step of obtaining data of burial points and data of user behavior with the same burial point ID from the message queue comprises: traversing the message queue based on the reported buried point ID carried by the user behavior data, and acquiring the buried point data carrying the buried point ID and the user behavior data from the message queue.
In some embodiments, in the step of storing the buried point ID as the associated keyword into a data warehouse for data analysis, the buried point ID is used as a key, the buried point data and the user behavior data are used as values, and the buried point data and the user behavior data associated with the buried point ID are stored as key-value pairs.
Another aspect of the present invention provides a data acquisition system for an application, including: the interactive data generating module is used for receiving an access request of an application program and generating interactive data responding to the access request; the buried point data generating module is used for generating buried point data according to the access request, wherein the buried point data comprises interaction behaviors to be monitored, and a buried point ID used for uniquely identifying the buried point data is generated; the data feedback module is used for returning the interactive data and the embedded point ID to the application program to generate an interactive page, and sending the embedded point data to a message queue by taking the embedded point ID as an identifier; the data reporting module is used for acquiring user behavior data of the interactive behavior to be monitored of the interactive page and reporting the user behavior data to the message queue by taking the embedded point ID as an identifier; and the data collection module is used for acquiring buried point data and user behavior data with the same buried point ID from the message queue, and storing the buried point ID as an associated keyword to a data warehouse for data analysis.
Yet another aspect of the present invention provides a data acquisition apparatus for an application, including: a processor; a memory having stored therein executable instructions of the processor; wherein the processor is configured to perform the steps of the data acquisition method of the application program according to any of the above embodiments by executing the executable instructions.
Yet another aspect of the present invention provides a computer-readable storage medium storing a program which, when executed, implements the steps of the data acquisition method of an application program described in any of the above embodiments.
Compared with the prior art, the invention has the beneficial effects that:
data exchange between the server and the client of the application program is reduced through the embedded point ID, the server generates globally unique embedded point ID while generating embedded point data, and only returns the embedded point ID to the client where the application program is located, and the specific embedded point data is sent to the message queue, so that the data volume returned to the client is reduced, the running performance of the application program is improved, and the flow consumption of the client is reduced;
the buried point ID is used for associating the buried point data and the user behavior data, so that the accuracy and efficiency of data acquisition are improved, a globally unique association relation is established when the buried point data and the user behavior data flow into a data warehouse, and the subsequent big data analysis is facilitated;
particularly for the product optimization scenes of application programs such as ABTEST and the like, a large amount of buried point data and user behavior data exist, the data collection method can greatly reduce the data volume returned to the client, provides powerful support for the client to use the application programs more smoothly, enables the buried point data and the user behavior data to be clearly and accurately associated, and improves the ABTEST experiment accuracy.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
FIG. 1 is a schematic diagram illustrating steps of a data acquisition method for an application program according to an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating an architecture of a data collection method for an application according to an embodiment of the present invention;
FIG. 3 shows a block diagram of a data acquisition system for an application in an embodiment of the invention;
FIG. 4 is a schematic structural diagram of a data acquisition device of an application program in the embodiment of the invention;
fig. 5 shows a schematic structural diagram of a computer-readable storage medium in an embodiment of the invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and thus their repetitive description will be omitted.
Fig. 1 shows the main steps of a data acquisition method of an application program in an embodiment, where the application program is installed in a client, such as a tablet computer, a smart phone, and the like, and a functional service of the application program is provided by a server, such as a business server including interaction with each functional module of the application program. The data acquisition method of the embodiment is mainly realized by the server. Referring to fig. 1, the data acquisition method of the application program includes: in step S110, receiving an access request of an application program, and generating interactive data responding to the access request; in step S120, generating buried point data according to the access request, where the buried point data includes an interaction behavior to be monitored, and generating a buried point ID for uniquely identifying the buried point data; in step S130, returning the interactive data and the embedded point ID to the application program to generate an interactive page, and sending the embedded point data to a message queue with the embedded point ID as an identifier; in step S140, collecting user behavior data of an interaction behavior to be monitored occurring on the interaction page, and reporting the user behavior data to a message queue with a buried point ID as an identifier; and in step S150, acquiring buried point data and user behavior data having the same buried point ID from the message queue, and storing the data in the data warehouse with the buried point ID as an associated keyword.
The buried point id (identity document) is a globally unique identification code for the buried point data. When an application program sends an access request needing to interact with a server, the server generates interaction data and buried point data according to the access request, the interaction data is used for responding to the access request and generating an interaction page for the application program, and the buried point data is used for monitoring interaction behaviors of a user based on the access request and collecting user behavior data. The server generates buried point data, generates a globally unique buried point ID for the buried point data, returns the buried point ID to the client, and sends the specific buried point data to the message queue, so that the data amount returned to the client is reduced, the running performance of an application program is improved, and the flow consumption of the client is reduced. When a specific interactive behavior, namely the interactive behavior to be monitored, is monitored on the interactive page, the client reports the generated user behavior data to the message queue together with the embedded point ID. Subsequently, the embedded point data and the user behavior data with the same embedded point ID in the message queue are stored in a data warehouse in an associated mode, the user behavior data and the service embedded point data are established in an associated relation through the globally unique embedded point ID, and accuracy and efficiency of data collection and analysis are improved. Particularly for the product optimization scenes of application programs such as ABTEST and the like, different product optimization schemes have different data of embedded points, different functional modules in each product optimization scheme can also design different data of embedded points, and the data of embedded points can be generated each time the application program requests data interaction from a server. Under the condition that a large amount of embedded point data exists, the server generates embedded point IDs for each access request of the application program, only the embedded point IDs are returned to the client, the specific embedded point data are sent to the message queue and then flow into the big data platform, the data amount returned to the client is greatly reduced, the problem of bloated client data is solved, powerful support is provided for the client to use the application program more smoothly, and the collection of the embedded point data and the user behavior data is optimized. Through the global unique identification of the buried point ID, the buried point data and the user behavior data can be clearly and accurately associated, and the ABTEST experiment accuracy is improved.
Furthermore, the buried point data also comprises a data object to be monitored, and the data object to be monitored is all data objects or partial data objects in the interactive data. For example, the access request initiated by the application is "search for a source", and the server responds to the access request, and generates interactive data, i.e., source information, including, for example, 20 pieces of source information. Further, the server generates buried point data according to the access request, which includes data objects to be monitored and interaction behaviors to be monitored for the current interaction data, where the data objects to be monitored are, for example, the 20 items of source information or 10 items of source information, and the interaction behaviors to be monitored are some operations with analysis value, such as browsing, clicking, making a deal, etc., that the user may perform on the 20 items of source information. In the ABTEST experiment scene, the buried point data also comprises information of the affiliated experiment group and the like. The user behavior data comprises data objects of the interactive behaviors to be monitored in the interactive data and the generated interactive behaviors to be monitored. For example, if the user clicks on a piece of source information in the interactive page including the 20 pieces of source information, the client reports the clicked source information and the clicking operation as user behavior data to the message queue.
Further, each data object in the interactive data has a unique identification code, and the interactive behaviors to be monitored comprise an exposure behavior and a click behavior. Taking the above 20 items of source information as an example, each item of source information has a unique identification code, for example, source ID "001-020". When some data objects in the interactive data have an exposure behavior or a click behavior, the client takes the goods source ID for identifying the data objects and the interactive behavior as user behavior data, and reports the user behavior data and the buried point ID to a message queue. And the subsequent server side can analyze and obtain which data objects to be monitored have the interactive behaviors according to the user behavior data and the embedded point ID. Furthermore, the embedded point ID is generated according to the part of key data of the data object to be monitored in the embedded point data, so that the embedded point ID can be associated with the data object to be monitored, and the subsequent data analysis is facilitated.
In some embodiments, the specific information of the buried point data may be configured in advance according to requirements, and when an application program initiates an access request, the buried point data may be generated correspondingly. For example, an application program initiates an access request of "searching for a goods source", that is, buried point data for exposing, clicking and monitoring the transaction behavior of all interactive data is generated. For another example, the application program initiates an access request of "view order", that is, buried point data for performing exposure and click behavior monitoring on all interactive data is generated. In some embodiments, the interaction behavior to be monitored is configured when the functional module of the application program is designed, the embedded point ID is generated according to a part of core data including the data object to be monitored in the embedded point data, and when the interaction page of the application program generates a specific interaction behavior, the client uploads the collected user behavior data and the embedded point ID so that the server analyzes which data objects to be monitored have the specific interaction behavior.
In some embodiments, the step of obtaining buried point data and user behavior data having the same buried point ID from the message queue is triggered when the message queue receives the reported user behavior data. That is, each time user behavior data reported by the client is received in the message queue, a real-time consumption step of the message is triggered. Alternatively, in some embodiments, the step of obtaining the burial point data and the user behavior data with the same burial point ID from the message queue may be performed periodically, for example, every 1 minute, every 5 minutes, etc., obtaining the burial point data and the user behavior data with the same burial point ID from the message queue, and storing the obtained burial point data and user behavior data in the data warehouse in an associated manner. The step of obtaining buried point data and user behavior data having the same buried point ID includes, for example: and traversing the message queue based on the buried point ID carried by the reported user behavior data, and acquiring the buried point data carrying the buried point ID and the user behavior data from the message queue. When the association is stored in the data warehouse, the buried point ID is used as a Key, the buried point data and the user behavior data are used as values, and the buried point data and the user behavior data associated with the buried point ID are stored as a Key-Value pair.
Fig. 2 shows a system architecture of the data acquisition method of the application program in the embodiment, and referring to fig. 2, the data acquisition method of the application program includes: in step S210, an application program (App)210 initiates an access request to the service system 220, and after receiving the access request, the service system 220 generates interactive data responding to the access request according to the service logic module 2201. Before or after the service system 220 calls the service logic module 2201 to generate the interactive data, the embedded point logic module 2202 is called in step S220 to generate the embedded point data, and a globally unique embedded point ID of the embedded point data is generated. The buried point logic module 2202 is a general software package for generating buried point data and buried point ID, which can be called by different service systems. In the ABTEST scenario for application 210, the buried point logic 2202 is an ABTEST SDK for service system calls using ABTEST. Next, in step S230, the buried point logic module 2202 returns the buried point ID to the application 210 through the business logic module 2201, and sends the buried point data to the message queue 230. The message queue 230 is, for example, a KAFKA cluster. In step S240, after the user clicks and exposes the interactive page of the application 210, the data reporting service 240 of the application 210 reports the user behavior data and the embedded point ID to the message queue 230. Then, the data collection module 250 executes step S250, and the data of the embedded points and the user behavior data with the same embedded point ID in the consumption message queue 230 are stored in the data warehouse 260, and are associated with each other by the embedded point ID for effect analysis of the big data. The data collection module 250 is a FLINK JOB, and is based on a real-time task developed by the streaming computing engine FLINK, and is used for consuming the user behavior data and the buried point data in the message queue 230 in real time. The data warehouse is a HIVE data warehouse.
In the system architecture described above, the buried node logic module 2202 is developed using JAVA 8, the production of buried node data is performed using KAFKA JAVA API, and the buried node ID generation is performed using distributed ID generation service Leaf. The data collection module 250 uses JAVA 8 for development, FLINK JAVA API for JOB development, HADOOP JAVA API for HADOOP file generation, and JDBC link HIVERSERVER for downloading the newly generated HIVE data file into the HIVE table.
The data acquisition method of the application program reduces data exchange between the server and the client of the application program through the embedded point ID; the embedded point ID is returned to the application program, so that the embedded point data quantity of the client is reduced, the running performance of the application program is improved, and the flow consumption of the client is reduced; sending the buried point data to a message queue, and decoupling the coupling of the buried point data and an application program client; the user behavior data and the buried point data are associated through the buried point ID, and the association relation between the user behavior data and the service buried point data is established, so that the data acquisition accuracy and efficiency are improved, and the subsequent data analysis is facilitated; particularly for the optimization scenes of application programs such as ABTEST and the like, the data volume returned to the client can be greatly reduced, powerful support is provided for the client to use the application programs more smoothly, the data of the embedded points and the user behavior data are clearly and accurately associated, and the ABTEST experiment accuracy is improved.
An embodiment of the present invention further provides a data acquisition system of an application program, and referring to fig. 3, the data acquisition system 3 of the application program includes: the interactive data generating module 310 is configured to receive an access request of an application program, and generate interactive data responding to the access request; the buried point data generating module 320 is configured to generate buried point data according to the access request, where the buried point data includes an interaction behavior to be monitored, and generates a buried point ID for uniquely identifying the buried point data; the data feedback module 330 is configured to return the interactive data and the embedded point ID to the application program to generate an interactive page, and send the embedded point data to a message queue with the embedded point ID as an identifier; the data reporting module 340 is configured to collect user behavior data of an interaction behavior to be monitored occurring on an interaction page, and report the user behavior data to a message queue with a buried point ID as an identifier; and a data collection module 350, configured to obtain buried point data and user behavior data with the same buried point ID from the message queue, and store the buried point ID as an associated keyword in a data warehouse for data analysis. The data acquisition system 3 of the application program of the present embodiment corresponds to the data acquisition method of the application program of the above-described embodiment, and therefore, description thereof will not be repeated.
The data acquisition system of the application program of the embodiment can reduce data exchange between the server and the application program client through the embedded point ID; the embedded point ID is returned to the application program, so that the embedded point data quantity of the client is reduced, the running performance of the application program is improved, and the flow consumption of the client is reduced; sending the buried point data to a message queue, and decoupling the coupling of the buried point data and an application program client; the user behavior data and the buried point data are associated through the buried point ID, and the association relation between the user behavior data and the service buried point data is established, so that the data acquisition accuracy and efficiency are improved, and the subsequent data analysis is facilitated; particularly for the optimization scenes of application programs such as ABTEST and the like, the data volume returned to the client can be greatly reduced, powerful support is provided for the client to use the application programs more smoothly, the data of the embedded points and the user behavior data are clearly and accurately associated, and the ABTEST experiment accuracy is improved.
The embodiment of the present invention further provides a data acquisition device for an application program, which includes a processor and a memory, where the memory stores executable instructions, and the processor is configured to execute the steps of the data acquisition method for an application program in the foregoing embodiments by executing the executable instructions.
As described above, the data acquisition device of the application program of the present invention can reduce data exchange between the server and the application client by using the buried point ID; the embedded point ID is returned to the application program, so that the embedded point data quantity of the client is reduced, the running performance of the application program is improved, and the flow consumption of the client is reduced; sending the buried point data to a message queue, and decoupling the coupling of the buried point data and an application program client; the user behavior data and the buried point data are associated through the buried point ID, and the association relation between the user behavior data and the service buried point data is established, so that the data acquisition accuracy and efficiency are improved, and the subsequent data analysis is facilitated; particularly for the optimization scenes of application programs such as ABTEST and the like, the data volume returned to the client can be greatly reduced, powerful support is provided for the client to use the application programs more smoothly, the data of the embedded points and the user behavior data are clearly and accurately associated, and the ABTEST experiment accuracy is improved.
Fig. 4 is a schematic structural diagram of a data acquisition device of an application program in an embodiment of the present invention, and it should be understood that fig. 4 only schematically illustrates various modules, which may be virtual software modules or actual hardware modules, and the combination, the splitting, and the addition of the remaining modules of these modules are within the scope of the present invention.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" platform.
A data acquisition device (hereinafter, simply referred to as an electronic device) 400 of an application program of the present invention is described below with reference to fig. 4. The electronic device 400 shown in fig. 4 is only an example and should not bring any limitation to the function and the scope of use of the embodiments of the present invention.
As shown in fig. 4, electronic device 400 is embodied in the form of a general purpose computing device. The components of electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one memory unit 420, a bus 430 connecting different platform components (including memory unit 420 and processing unit 410), a display unit 440, and the like.
The storage unit stores a program code, and the program code can be executed by the processing unit 410, so that the processing unit 410 executes the steps of the data acquisition method of the application program described in the above embodiments. For example, the processing unit 410 may perform the steps as shown in fig. 1 or fig. 2.
The storage unit 420 may include readable media in the form of volatile storage units, such as a random access memory unit (RAM)4201 and/or a cache memory unit 4202, and may further include a read only memory unit (ROM) 4203.
The storage unit 420 may also include a program/utility 4204 having a set (at least one) of program modules 4205, such program modules 4205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 400 may also communicate with one or more external devices 500 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 400, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 400 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 450. Also, the electronic device 400 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 460. The network adapter 460 may communicate with other modules of the electronic device 400 via the bus 430. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, to name a few.
The embodiment of the present invention further provides a computer-readable storage medium, which is used for storing a program, and when the program is executed, the steps of the data acquisition method of the application program described in the above embodiment are implemented. In some possible embodiments, the various aspects of the present invention may also be implemented in the form of a program product, which includes program code for causing a terminal device to perform the steps of the data acquisition method of the application program described in the above embodiments, when the program product is run on the terminal device.
As described above, the computer-readable storage medium of the present invention can reduce data exchange of a server and an application client by a buried point ID; the embedded point ID is returned to the application program, so that the embedded point data quantity of the client is reduced, the running performance of the application program is improved, and the flow consumption of the client is reduced; sending the buried point data to a message queue, and decoupling the coupling of the buried point data and an application program client; the user behavior data and the buried point data are associated through the buried point ID, and the association relation between the user behavior data and the service buried point data is established, so that the data acquisition accuracy and efficiency are improved, and the subsequent data analysis is facilitated; particularly for the optimization scenes of application programs such as ABTEST and the like, the data volume returned to the client can be greatly reduced, powerful support is provided for the client to use the application programs more smoothly, the data of the embedded points and the user behavior data are clearly and accurately associated, and the ABTEST experiment accuracy is improved.
Fig. 5 is a schematic structural diagram of a computer-readable storage medium of the present invention. Referring to fig. 5, a program product 600 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (10)
1. A data acquisition method of an application program is characterized by comprising the following steps:
receiving an access request of an application program, and generating interactive data responding to the access request;
generating buried point data according to the access request, wherein the buried point data comprises interaction behaviors to be monitored, and generating a buried point ID for uniquely identifying the buried point data;
returning the interactive data and the buried point ID to the application program to generate an interactive page, and sending the buried point data to a message queue by taking the buried point ID as an identifier;
collecting user behavior data of the interactive behavior to be monitored of the interactive page, and reporting the user behavior data to the message queue by taking the embedded point ID as an identifier; and
and acquiring buried point data and user behavior data with the same buried point ID from the message queue, and storing the buried point ID as an associated keyword to a data warehouse.
2. The data collection method of claim 1, wherein the buried data further comprises data objects to be intercepted, the data objects to be intercepted being all or part of the data objects in the interactive data;
the user behavior data comprises a data object of the interactive behavior to be monitored in the interactive data and the generated interactive behavior to be monitored.
3. The data collection method of claim 2, wherein each data object in the interaction data has a unique identification code, and the interaction behaviors to be listened to include an exposure behavior and a click behavior.
4. The data collection method of claim 2, wherein the buried point ID is generated from a portion of the key data in the buried point data that includes the data object to be monitored.
5. The data collection method of claim 1, wherein the step of obtaining burial point data and user behavior data having the same burial point ID from the message queue is triggered when the message queue receives the reported user behavior data.
6. The data collection method of claim 5, wherein the step of obtaining burial point data and user behavior data with the same burial point ID from the message queue comprises:
traversing the message queue based on the reported buried point ID carried by the user behavior data, and acquiring the buried point data carrying the buried point ID and the user behavior data from the message queue.
7. The data collection method of claim 6, wherein the step of storing the buried point ID as the associated keyword into a data warehouse for data analysis stores the buried point data and the user behavior data associated with the buried point ID as key-value pairs with the buried point ID as a key and the buried point data and the user behavior data as values.
8. A data acquisition system for an application, comprising:
the interactive data generating module is used for receiving an access request of an application program and generating interactive data responding to the access request;
the buried point data generating module is used for generating buried point data according to the access request, wherein the buried point data comprises interaction behaviors to be monitored, and a buried point ID used for uniquely identifying the buried point data is generated;
the data feedback module is used for returning the interactive data and the embedded point ID to the application program to generate an interactive page, and sending the embedded point data to a message queue by taking the embedded point ID as an identifier;
the data reporting module is used for acquiring user behavior data of the interactive behavior to be monitored of the interactive page and reporting the user behavior data to the message queue by taking the embedded point ID as an identifier; and
and the data collection module is used for acquiring buried point data and user behavior data with the same buried point ID from the message queue, and storing the buried point ID as an associated keyword to a data warehouse for data analysis.
9. An application data acquisition device, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the data acquisition method of the application program of any one of claims 1 to 7 via execution of the executable instructions.
10. A computer-readable storage medium storing a program, wherein the program when executed implements the steps of the data acquisition method of an application program of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010080290.0A CN111309550A (en) | 2020-02-05 | 2020-02-05 | Data acquisition method, system, equipment and storage medium of application program |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010080290.0A CN111309550A (en) | 2020-02-05 | 2020-02-05 | Data acquisition method, system, equipment and storage medium of application program |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111309550A true CN111309550A (en) | 2020-06-19 |
Family
ID=71148852
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010080290.0A Withdrawn CN111309550A (en) | 2020-02-05 | 2020-02-05 | Data acquisition method, system, equipment and storage medium of application program |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111309550A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112148574A (en) * | 2020-09-21 | 2020-12-29 | 北京基调网络股份有限公司 | Performance data acquisition method, computer equipment and storage medium |
| CN112181770A (en) * | 2020-09-28 | 2021-01-05 | 北京达佳互联信息技术有限公司 | Method, device and system for setting buried object |
| CN112260965A (en) * | 2020-10-21 | 2021-01-22 | 阳光保险集团股份有限公司 | Message processing method, device, equipment and storage medium |
| CN112306501A (en) * | 2020-11-30 | 2021-02-02 | 杭州网易云音乐科技有限公司 | Business data acquisition method and device, storage medium and computing equipment |
| CN112422445A (en) * | 2020-10-10 | 2021-02-26 | 四川新网银行股份有限公司 | Kafka-based real-time acquisition, calculation and storage method for buried point data |
| CN113032823A (en) * | 2021-02-26 | 2021-06-25 | 加和（北京）信息科技有限公司 | Method and device for generating equipment ID |
| CN113094248A (en) * | 2021-04-28 | 2021-07-09 | 深圳市网旭科技有限公司 | User behavior data analysis method and device, electronic equipment and medium |
| CN113312404A (en) * | 2021-08-02 | 2021-08-27 | 北京华品博睿网络技术有限公司 | Method and system for collecting characteristic samples in real time |
| CN113515427A (en) * | 2021-05-19 | 2021-10-19 | 中国工商银行股份有限公司 | Point burying method and device, electronic equipment and storage medium |
| CN113781088A (en) * | 2021-02-04 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | User tag processing method, device and system |
| CN113901372A (en) * | 2021-10-15 | 2022-01-07 | 四川新网银行股份有限公司 | User behavior data warehouse construction system and method |
| CN115134352A (en) * | 2022-06-27 | 2022-09-30 | 重庆长安汽车股份有限公司 | Method, device, equipment and medium for uploading data of buried points |
-
2020
- 2020-02-05 CN CN202010080290.0A patent/CN111309550A/en not_active Withdrawn
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112148574A (en) * | 2020-09-21 | 2020-12-29 | 北京基调网络股份有限公司 | Performance data acquisition method, computer equipment and storage medium |
| CN112181770A (en) * | 2020-09-28 | 2021-01-05 | 北京达佳互联信息技术有限公司 | Method, device and system for setting buried object |
| CN112422445A (en) * | 2020-10-10 | 2021-02-26 | 四川新网银行股份有限公司 | Kafka-based real-time acquisition, calculation and storage method for buried point data |
| CN112260965A (en) * | 2020-10-21 | 2021-01-22 | 阳光保险集团股份有限公司 | Message processing method, device, equipment and storage medium |
| CN112306501A (en) * | 2020-11-30 | 2021-02-02 | 杭州网易云音乐科技有限公司 | Business data acquisition method and device, storage medium and computing equipment |
| CN112306501B (en) * | 2020-11-30 | 2023-03-28 | 杭州网易云音乐科技有限公司 | Business data acquisition method and device, storage medium and computing equipment |
| CN113781088A (en) * | 2021-02-04 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | User tag processing method, device and system |
| CN113032823A (en) * | 2021-02-26 | 2021-06-25 | 加和（北京）信息科技有限公司 | Method and device for generating equipment ID |
| CN113032823B (en) * | 2021-02-26 | 2023-08-01 | 加和（北京）信息科技有限公司 | Method and device for generating device ID |
| CN113094248A (en) * | 2021-04-28 | 2021-07-09 | 深圳市网旭科技有限公司 | User behavior data analysis method and device, electronic equipment and medium |
| CN113515427A (en) * | 2021-05-19 | 2021-10-19 | 中国工商银行股份有限公司 | Point burying method and device, electronic equipment and storage medium |
| CN113312404B (en) * | 2021-08-02 | 2021-11-02 | 北京华品博睿网络技术有限公司 | Method and system for collecting characteristic samples in real time |
| CN113312404A (en) * | 2021-08-02 | 2021-08-27 | 北京华品博睿网络技术有限公司 | Method and system for collecting characteristic samples in real time |
| CN113901372A (en) * | 2021-10-15 | 2022-01-07 | 四川新网银行股份有限公司 | User behavior data warehouse construction system and method |
| CN113901372B (en) * | 2021-10-15 | 2024-07-02 | 四川新网银行股份有限公司 | User behavior data warehouse construction system and method |
| CN115134352A (en) * | 2022-06-27 | 2022-09-30 | 重庆长安汽车股份有限公司 | Method, device, equipment and medium for uploading data of buried points |
| CN115134352B (en) * | 2022-06-27 | 2023-06-20 | 重庆长安汽车股份有限公司 | Buried point data uploading method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111309550A (en) | Data acquisition method, system, equipment and storage medium of application program | |
| CN109034993B (en) | Account checking method, account checking equipment, account checking system and computer readable storage medium | |
| US10560465B2 (en) | Real time anomaly detection for data streams | |
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| CN107370806B (en) | HTTP status code monitoring method, device, storage medium and electronic equipment | |
| CN110262807B (en) | Cluster creation progress log acquisition system, method and device | |
| CN110928851B (en) | Method, device and equipment for processing log information and storage medium | |
| CN110858172A (en) | Automatic test code generation method and device | |
| US10657099B1 (en) | Systems and methods for transformation and analysis of logfile data | |
| CN110851324A (en) | Log-based routing inspection processing method and device, electronic equipment and storage medium | |
| CN110554951A (en) | Method and device for managing embedded points | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN117271584A (en) | Data processing method and device, computer readable storage medium and electronic equipment | |
| CN112579406A (en) | Log call chain generation method and device | |
| CN115017218B (en) | Processing method and device of distributed call chain, storage medium and electronic equipment | |
| KR101878291B1 (en) | Big data management system and management method thereof | |
| CN112148762A (en) | Statistical method and device for real-time data stream | |
| US10536390B1 (en) | Requesting embedded hypermedia resources in data interchange format documents | |
| US10567469B1 (en) | Embedding hypermedia resources in data interchange format documents | |
| CN115391286A (en) | Link tracking data management method, device, equipment and storage medium | |
| CN112015623B (en) | Report data processing method, device, equipment and readable storage medium | |
| CN113114612B (en) | Determination method and device for distributed system call chain | |
| CN113360558A (en) | Data processing method, data processing device, electronic device, and storage medium | |
| CN113704203A (en) | Log file processing method and device | |
| CN113485897A (en) | Data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200619 |
|
| WW01 | Invention patent application withdrawn after publication |