CN117591381A

CN117591381A - Data reporting method and device

Info

Publication number: CN117591381A
Application number: CN202410070580.5A
Authority: CN
Inventors: 常建龙; 彭彩芳; 朱健; 张辛宇
Original assignee: Shanghai Shouqianba Internet Technology Co ltd; Nanjing Yanli Technology Co ltd
Current assignee: Shanghai Shouqianba Internet Technology Co ltd; Nanjing Yanli Technology Co ltd
Priority date: 2024-01-18
Filing date: 2024-01-18
Publication date: 2024-02-23
Anticipated expiration: 2044-01-18
Also published as: CN117591381B

Abstract

The application provides a data reporting method and device, and relates to the technical field of network communication. The method comprises the following steps: monitoring whether the embedded data in the client is triggered by a development kit; the embedded data comprises event data for recording user behaviors and application states; generating a configuration file based on the triggered buried data through a development kit; and reporting the data to the server based on the configuration file. According to the method and the device, the triggering condition of the buried point data in the client is monitored through the development tool package arranged in the client, so that the request event of the client is monitored, and the corresponding data is recorded when triggered to obtain the configuration file, so that the data can be reported to the server directly based on the configuration file. And a proxy server does not need to be arranged between the client and the server for data reporting, and the log generation and the log transmission are decoupled, so that the integrity and the reliability of data in the configuration file are effectively improved, and the efficiency and the safety of data reporting are improved.

Description

Data reporting method and device

Technical Field

The present application relates to the field of network communications technologies, and in particular, to a data reporting method and apparatus.

Background

At present, the method for reporting the embedded point data of the client is generally that the embedded point reporting is performed by an agent: in a client application, the user's behavior data may be sent to the application or the back-end server of the website via HTTP requests. In the proxy server or middleware, an interceptor or filter is provided for intercepting the HTTP request sent by the client application and extracting the behavior data of the user, such as URL, parameters, header, etc., from the request. In the proxy server or middleware, the extracted behavior data of the user is sent to the data receiving server in a JSON format in an HTTP POST mode, and meanwhile, an original HTTP request is forwarded to a back-end server of an application or a website, so that normal execution of business logic is ensured. In the data receiving server, the behavior data of the user sent by the proxy server or the middleware is received and stored, and the data is analyzed and processed.

The processing mode of proxy embedded point reporting can acquire user interface and interactive data without modifying codes of applications or websites, and can realize real-time acquisition and transmission of data. However, the processing mode has high data acquisition dependency and risk, and is easily affected by a proxy server or middleware, such as performance degradation, fault occurrence, configuration error and the like, so that data transmission failure or abnormality is caused.

Disclosure of Invention

In view of the foregoing, an object of an embodiment of the present application is to provide a data reporting method and apparatus, so as to solve the problem of abnormal data transmission when using middleware such as a proxy server to report data in the prior art.

In order to solve the above problem, in a first aspect, an embodiment of the present application provides a data reporting method, where the method includes:

monitoring whether the embedded data in the client is triggered by a development kit; the embedded data comprises event data for recording user behaviors and application states;

generating, by the development kit, a configuration file based on the triggered buried data;

and reporting data to a server based on the configuration file.

In the implementation process, the triggering condition of the embedded point data in the client is monitored through the development tool package arranged in the client so as to monitor the request event of the client, and the corresponding data is recorded when triggered to obtain the configuration file, so that the data can be directly reported to the server based on the configuration file. And a proxy server does not need to be arranged between the client and the server for data reporting, the integrity and the reliability of data in the configuration file are improved by solidifying the log in the file, and the log generation and the log transmission are decoupled, so that the efficiency and the reliability of data acquisition are improved, and the efficiency and the safety of data reporting are improved.

Optionally, the configuration file includes a first log;

the generating, by the development kit, a configuration file based on the triggered embedded point data includes:

detecting a first request result of a first call request for calling an application interface of the development kit under the condition that the embedded point data is triggered by the development kit;

if the first request result is judged to be call failure, recording a first type of failure request through the development kit, and writing the recorded first type of failure request into the first log based on a preset data format.

In the implementation process, the development kit can record only abnormal request data so as to realize a data reporting mode of combining the main and standby. And detecting a first request result of a first call request for calling an application interface of the client under the condition that the embedded data in the client is triggered by the development kit so as to judge whether the first call request is successfully called, thereby recording a first type of failure request when the first call request fails, and writing the first type of failure request into a corresponding first log through a preset data format. The method and the system can independently collect and process the failed request data, effectively improve the integrity of the first log and reduce the adverse condition of data reporting failure or abnormality caused by network or server abnormality.

Optionally, the reporting the data to the server based on the configuration file includes:

if the first request result is judged to be successful in calling, reporting a first type of successful request to the server side based on an asynchronous communication request through the development kit;

if the first request result is judged to be the call failure, the first log is processed through a log acquisition tool to obtain first abnormal log data; and reporting the first abnormal log data to the server based on a preset transmission protocol through the log acquisition tool.

In the implementation process, in the data reporting mode combining the main part and the standby part, if the first calling request is successfully called, the development kit can directly report the first type of successful request to the server based on the asynchronous communication request, so that the time and the calculation force required for recording are reduced, and the instantaneity of data reporting is improved. If the first call request fails to call, the first log is monitored and processed through the log collection tool, the first abnormal log data obtained through processing is additionally reported to the server for processing based on a preset transmission protocol, and the generation and the sending of the first log are decoupled, so that adverse effects of the first log caused by network abnormality and the like are effectively reduced, and the reliability of data reporting is improved.

Optionally, the processing the first log by the log collection tool to obtain first abnormal log data includes:

analyzing and converting the first journal based on the preset data format through the journal acquisition tool to obtain first initial data;

and combining the first initial data with the metadata associated with the client through the log acquisition tool to obtain the first abnormal log data.

In the implementation process, the log collection tool can monitor and read the first log, when the first log is sent to change, the data in the first log is analyzed and converted based on a preset data format, and metadata of a corresponding client is added to obtain first abnormal log data for supplementing and reporting. The method can perform reading processing based on the writing rule of the data in the first log, and effectively improves the efficiency and reliability of log processing.

Optionally, the configuration file includes a second log;

determining, by the development kit, a second call request for calling an application interface of the development kit in a case where the embedded point data is triggered;

And writing the second call request into the second log based on a preset data format through the development kit.

In the above implementation manner, the development kit may record only all the request data, so as to implement an asynchronous processing data reporting manner. And monitoring the embedded data by the development tool package to determine a second call request for calling an application interface of the development tool package under the condition that the embedded data is triggered, recording the second call request, and writing the second call request into a corresponding second log based on a preset data format. All the monitored requests can be recorded in the corresponding log file to ensure the integrity and reliability of the second log.

Optionally, the second log includes a normal log and an abnormal log;

the writing, by the development kit, the second call request into the second log based on a preset data format includes:

determining a second request result of the second call request through the development kit;

if the second request result is judged to be successful in calling, writing a second type of successful request into the normal log based on a preset data format through the development kit;

And if the second request result is judged to be the call failure, writing a second type failure request into the exception log based on the preset data format through the development kit.

In the above implementation process, in the data reporting manner of asynchronous processing, in order to process different types of request results respectively, the development kit may detect the second request result of the second call request first, so as to determine whether the second call request is successfully invoked. Under the two conditions of successful invoking and unsuccessful invoking of the second invoking request, the second type successful requests and the second type unsuccessful requests are written into corresponding normal logs and abnormal logs based on preset data formats respectively, so that data of different requesting results are reported respectively, and the processing effect of asynchronous reporting is effectively optimized.

processing the normal log and the abnormal log through a log acquisition tool to obtain normal log data and second abnormal log data;

and reporting the normal log data and the second abnormal log data to the server based on a preset transmission protocol through the log acquisition tool.

In the implementation process, in the data reporting mode of asynchronous processing, the development tool kit only collects and records data, and the log collection tool processes and reports the data of the normal log and the abnormal log, so that the generation and the transmission of the second log are decoupled, adverse effects on the data reporting caused by network abnormality and the like are reduced, occupation of application resources of a client is reduced, the development tool kit is suitable for reporting a plurality of requests, and the influence of log collection on service performance is reduced.

Optionally, the processing, by the log collection tool, the normal log and the abnormal log to obtain normal log data and second abnormal log data includes:

analyzing and converting the normal log based on the preset data format by the log acquisition tool to obtain second initial data; the normal log data are obtained by combining the second initial data with the metadata associated with the client through the log acquisition tool;

analyzing and converting the abnormal log based on the preset data format by the log acquisition tool to obtain third initial data; and combining the third initial data and the metadata by the log acquisition tool to obtain the second abnormal log data.

In the implementation process, the plurality of log collection tools can respectively process the normal log and the abnormal log, respectively analyze and convert the data in the normal log and the abnormal log based on a preset data format, and add metadata of a corresponding client to obtain normal log data for normal reporting and second abnormal log data for supplementary reporting. The data can be read and processed in a classified manner based on the writing rule of the data in the second log, and the efficiency and reliability of log processing are effectively improved.

Optionally, after the monitoring, by the development kit, whether the embedded data in the client is triggered, the method further includes:

determining selection parameters of all the current triggered buried point data; wherein the selection parameters include: at least one of configuration parameters, real-time requirements, and number of requests;

and selecting the corresponding development tool package for processing based on the selection parameters.

In the implementation process, development tool packages using different reporting modes can be respectively arranged in the client to evaluate from various aspects such as configuration, requirements, quantity and the like according to the selection parameters of triggered embedded point data, so as to select corresponding development tool packages and process by adopting a corresponding data reporting mode. The method can select a proper reporting mode for processing according to the actual conditions of the client and the server, and effectively improves the efficiency and the reliability of data reporting.

In a second aspect, an embodiment of the present application further provides a data reporting device, where the device includes: the system comprises a monitoring module, a writing module and a reporting module;

the monitoring module is used for monitoring whether the embedded point data in the client is triggered or not through the development kit; the embedded data comprises event data for recording user behaviors and application states;

the writing module is used for generating a configuration file based on the triggered embedded point data through the development kit;

and the reporting module is used for reporting data to the server based on the configuration file.

In the implementation process, the monitoring module monitors the triggering condition of the embedded point data in the client through the development tool package arranged in the client so as to monitor the request event of the client, the writing module records the corresponding data when triggering to obtain the configuration file, and the reporting module directly reports the data to the server based on the configuration file.

In a third aspect, an embodiment of the present application further provides an electronic device, where the electronic device includes a memory and a processor, where the memory stores program instructions, and when the processor reads and executes the program instructions, the processor executes steps in any implementation manner of the data reporting method.

In a fourth aspect, embodiments of the present application further provide a computer readable storage medium, where computer program instructions are stored, where the computer program instructions, when read and executed by a processor, perform steps in any implementation of the above data reporting method.

In summary, the embodiments of the present application provide a method and an apparatus for reporting data, which monitor, through a development kit provided in a client, a trigger condition of embedded point data in the client, so as to monitor a request event of the client, and record corresponding data to obtain a configuration file when triggering, so that data can be directly reported to a server based on the configuration file. And a proxy server does not need to be arranged between the client and the server for data reporting, and the log generation and the log transmission are decoupled, so that the integrity and the reliability of data in the configuration file are effectively improved, and the efficiency and the safety of data reporting are improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic block diagram of an electronic device according to an embodiment of the present application;

fig. 2 is a flow chart of a data reporting method provided in the embodiment of the present application;

fig. 3 is a detailed flowchart of the first step S300 provided in the embodiment of the present application;

fig. 4 is a detailed flowchart of the first step S400 provided in the embodiment of the present application;

fig. 5 is a detailed flowchart of a second step S300 provided in the embodiment of the present application;

fig. 6 is a detailed flowchart of step S340 provided in the embodiment of the present application;

fig. 7 is a detailed flowchart of the second step S400 provided in the embodiment of the present application;

fig. 8 is a flowchart of another data reporting method provided in the embodiment of the present application;

fig. 9 is a schematic structural diagram of a data reporting device provided in the embodiment of the present application.

Icon: 100-an electronic device; 111-memory; 112-a memory controller; 113-a processor; 114-a peripheral interface; 115-an input-output unit; 116-a display unit; 600-a data reporting device; 610-a monitoring module; 620-a write module; 630-reporting module.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments of the present application without making any inventive effort, are intended to be within the scope of the embodiments of the present application.

At present, middleware is usually arranged between a client and a server, for example, a proxy server performs data reporting for realizing proxy burial points. The proxy embedded point reporting mode can acquire the user interface and the interactive data without modifying the codes of the application or the website, and can realize the real-time acquisition and transmission of the data. However, the dependency and risk of reporting data by the proxy embedded point are high, and the data is easily affected by a proxy server or middleware, for example, the proxy server or middleware fails or is wrongly configured, so that data transmission failure or abnormality can be caused, and the integrity and reliability of the data are affected; the attack or tampering of the proxy server or the middleware can cause the data to be leaked or modified, and the safety and accuracy of the data are affected; the proxy server or middleware needs to additionally process the behavior data of the user, which increases the network and calculation overhead and possibly affects the response speed of the application or website, the user experience and the like, so that adverse conditions such as performance degradation, fault occurrence, configuration error and the like occur, and data transmission failure or abnormality is caused.

In order to solve the above problems, the embodiments of the present application provide a data reporting method, which is applied to a client, where the client may be an electronic device with a logic computing function, such as a server, a personal computer (Personal Computer, PC), a tablet computer, a smart phone, a personal digital assistant (Personal Digital Assistant, PDA), etc., and is capable of monitoring, collecting and reporting data in real time, and reporting data without depending on middleware, thereby improving efficiency and reliability of reporting data.

Optionally, referring to fig. 1, fig. 1 is a schematic block diagram of an electronic device according to an embodiment of the present application. The electronic device 100 may include a memory 111, a memory controller 112, a processor 113, a peripheral interface 114, an input output unit 115, and a display unit 116. Those of ordinary skill in the art will appreciate that the configuration shown in fig. 1 is merely illustrative and is not limiting of the configuration of the electronic device 100. For example, electronic device 100 may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.

The above-mentioned memory 111, memory controller 112, processor 113, peripheral interface 114, input/output unit 115 and display unit 116 are electrically connected directly or indirectly to each other to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The processor 113 is used to execute executable modules stored in the memory.

The Memory 111 may be, but is not limited to, a random access Memory (Random Access Memory, RAM), a Read Only Memory (ROM), a programmable Read Only Memory (Programmable Read-Only Memory, PROM), an erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), an electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc. The memory 111 is configured to store a program, and the processor 113 executes the program after receiving an execution instruction, and a method executed by the electronic device 100 defined by the process disclosed in any embodiment of the present application may be applied to the processor 113 or implemented by the processor 113.

The processor 113 may be an integrated circuit chip having signal processing capabilities. The processor 113 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (digital signal processor, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field Programmable Gate Arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. The general purpose processor may be a microprocessor, but in the alternative, it may be any conventional processor or the like.

The peripheral interface 114 couples various input/output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113, and the memory controller 112 may be implemented in a single chip. In other examples, they may be implemented by separate chips.

The input-output unit 115 described above is used to provide input data to a user. The input/output unit 115 may be, but is not limited to, a mouse, a keyboard, and the like.

The display unit 116 described above provides an interactive interface (e.g., a user-operated interface) between the electronic device 100 and a user or is used to display image data to a user reference. In this embodiment, the display unit may be a liquid crystal display or a touch display. In the case of a touch display, the touch display may be a capacitive touch screen or a resistive touch screen, etc. supporting single-point and multi-point touch operations. Supporting single-point and multi-point touch operations means that the touch display can sense touch operations simultaneously generated from one or more positions on the touch display, and the sensed touch operations are passed to the processor for calculation and processing. In the embodiment of the present application, the display unit 116 may display relevant situations such as data writing and data reporting results.

The electronic device in this embodiment may be configured to execute each step in each data reporting method provided in the embodiment of the present application. The implementation of the data reporting method is described in detail below by means of several embodiments.

Referring to fig. 2, fig. 2 is a flowchart of a data reporting method according to an embodiment of the present application, and the method may include steps S200-S400.

In step S200, the development kit monitors whether the embedded data in the client is triggered.

The embedded data includes event data recording user behavior and application state, such as user click, page browse, event trigger, etc. The buried data typically contains some basic information such as a time stamp, a device ID, an application version, etc., and some custom information such as an event type, an event parameter, etc. One or more different types of buried point data can be set in advance according to the actual situation and the requirement of the client. And monitoring triggering conditions of the embedded point data through a development tool package arranged in the client so as to monitor request events of the client.

Illustratively, the application of buried point data in clients is largely divided into the following categories: PC client applications including games, tools, social interactions, etc.; the mobile terminal application comprises software running on a smart phone or a tablet computer and comprises APP, applet, H5 and the like; front-end web applications, including web pages that open in a browser, such as: web sites, web forms, etc.

It should be noted that, the development kit may be integrated in the client application in advance, and the development kit may be customized according to different application platforms and languages (Software Development Kit ), for example, android sdk, iOS sdk, java sdk, and the like, and the development kit may provide corresponding high-level functions, such as data encryption, data compression, data buffering, data retry, and the like.

Alternatively, the development kit may be integrated in the client by: setting at least one buried point data in the client according to the trigger event; a development kit is set in the client based on all of the buried data. The development tool package can be connected with all the buried data, so that the monitoring and recording of the buried data are realized.

Step S300, generating a configuration file based on the triggered buried data by developing a toolkit.

The development tool kit can collect data when the embedded point data is triggered so as to record corresponding data to obtain a configuration file.

Alternatively, the configuration file may be a local file or may be stored in a remote file. For example, flexible configuration parameters can be provided for sdk, so that different real-time requirements on reported data can be realized on the premise of ensuring the reliability and the integrity of data reporting. The main configuration parameter specifications include: server. Sdk. Enable: if the HTTP calling mode of sdk is used, if false is not used, directly transferring the calling log into a log file; true is used, and an API interface is directly called to report data; server.sdk.save: whether the call log is saved to the local or not, false is not saved to the local file, and true is saved to the local file; server.sdk.eventsavepath: storing a path of the log file; server.sdk.eventsavenname: the log file name is saved.

Step S400, reporting data to the server based on the configuration file.

The client can directly report the data to the server based on the configuration file, and a proxy server is not required to be arranged between the client and the server for reporting the data.

Alternatively, the server side receiving the data may be a data processing component deployed on a cloud or other server, and configured to receive the data reported by the client side and store the data in a database or other storage system. The server may support multiple transmission protocols, such as HTTP, MQTT, kafka, and may also provide some advanced functions, such as data verification, data filtering, data conversion, and so on. It should be noted that the server may also be provided with a storage and analysis platform, where the storage platform is a database or other storage system for storing the reported data, and is used for storing the data for use by a subsequent analysis and display module. Multiple storage types may be supported, such as relational databases, non-relational databases, file systems, and the like. The analysis platform can provide various functions such as data display, data query, analysis statistics and the like, for example, statistics on browsing probability of a certain page or statistics on clicking probability of a certain advertisement.

When the data is reported, the data can be reported through a development kit, or through a log acquisition tool arranged in the client. The log collection tool can be log collection components such as a log, the log can support various log formats, such as JSON, CSV, protobuf, and the log can provide various functions such as log cutting, log backup, log cleaning and the like.

For example, to implement the corresponding function, the loggent may also be preconfigured, and the main configuration parameters of the loggent may include: loggent.inputs: inputting configuration; type input type; enabled: whether enabled or not, true is enabled, and flag is disabled; max_procs: running the maximum thread number; paths: inputting a file path and an absolute path; output. Range shttp: outputting http/kafka/file; protocol: protocols http/https; header: a request header; host: the domain name or IP of the request; loadbalance: whether load balancing is performed or not, true/false is load balancing, and false is load unbalance; works er: the number of working nodes; logging.files: logging of the loggent itself; rotateeverybytes: the log rolls and generates the byte size; keep files: the number of log files saved, etc.

In the embodiment shown in fig. 2, the integrity and reliability of the data in the configuration file are improved by solidifying the log in the file, and the log generation and the log transmission are decoupled, so that the efficiency and reliability of data acquisition are improved, and the efficiency and safety of data reporting are improved.

Optionally, the development kit may record only abnormal request data, so as to implement a data reporting manner of combining the primary and the secondary. In the primary-backup combined data reporting mode, the configuration file may include a first log. Referring to fig. 3, fig. 3 is a detailed flowchart of a first step S300 provided in the embodiment of the present application, and the step S300 may include steps S310-S320.

In step S310, a first request result of a first call request for calling an application interface of the development kit is detected by the development kit in case that the embedded data is triggered.

When the embedded point data is triggered, an application interface, such as an API interface, of the development tool package can be called to generate a corresponding first call request, and in the running process of the client application development tool package, an exception handling mechanism can be automatically set in the development tool package, and the exception handling mechanism is used for detecting a first request result of the first call request so as to judge whether the first request is successfully called.

In step S320, if the first request result is determined to be the call failure, the first type of failure request is recorded through the development kit, and the recorded first type of failure request is written into the first log based on the preset data format.

When the exception handling mechanism determines that the first request result is a call failure, the corresponding first call request is a first type failure request, the exception handling mechanism may record the first type failure request, write the recorded first type failure request into a local or remote first log based on a preset data format, for example, JSON format, and the first log file may be recorded as error1.Log.

In the embodiment illustrated in fig. 3, failed request data can be collected and processed separately, so that the integrity of the first log is effectively improved, and the adverse condition of data reporting failure or abnormality caused by network or server abnormality is reduced.

Optionally, referring to fig. 4, fig. 4 is a detailed flowchart of a first step S400 provided in the embodiment of the present application, and step S400 may include steps S410-S420.

In step S410, if the first request result is determined to be successful, the first type of successful request is reported to the server based on the asynchronous communication request through the development kit.

If the first request result is judged to be successful in calling, the corresponding first call request is a first type successful request, the development kit can monitor and capture behavior data of the first type successful request of the user, and report the data to the server for processing in the form of an HTTP asynchronous request, so that normal requests are not required to be recorded, time and calculation force required by recording are reduced, and instantaneity of data reporting is improved.

Step S420, if the first request result is judged to be call failure, the first log is processed through a log acquisition tool to obtain first abnormal log data; and reporting the first abnormal log data to the server based on a preset transmission protocol through a log acquisition tool.

When the client runs the log collection tool, the log collection tool can configure the operation parameters to monitor the data change in the first log file, read and process the first log, and supplement and report the obtained first abnormal log data to the server through a corresponding transmission protocol.

Alternatively, the transmission protocol may be a plurality of different types of transmission plug-ins, such as Kafka, redis, HTTP or a custom type transmission protocol, which may be selected and adjusted according to the actual situation and requirements.

Optionally, the manner in which the log collection tool processes the first log may include: analyzing and converting the first journal based on a preset data format through a journal acquisition tool to obtain first initial data; and obtaining first abnormal log data by combining the first initial data and the metadata associated with the client through a log acquisition tool. The log collection tool can monitor and read the first log, when the first log is changed, the data in the first log is analyzed and converted based on a preset data format, and metadata (the metadata can include a timestamp, a host name, related data of a source and the like) of a corresponding client is added, so that first abnormal log data for supplementing and reporting is obtained. The method can perform reading processing based on the writing rule of the data in the first log, and effectively improves the efficiency and reliability of log processing.

It should be noted that, the rule written by the development kit is the same as the rule read by the log collection tool, and the development kit is processed based on the same preset data format so as to ensure that the data can be read and processed normally.

For example, to implement the above-mentioned function of the active-standby reporting manner, the setting parameters of the development kit may include: enabling sdk functions, false and disable dk functions, true and sdk functions, server. Log records are not saved locally: server.sdk.save=false; asynchronous mode queue length: server.sdk.queue size=102400; saving log file path: server.sdk.eventsavepath=/app/datarange-loggent; saving the log file name: error.sdk.eventsavenme=datarange.log; the size of the single log saved at most, unit MB: server.sdk.eventsavemaxfilesize=512, etc.

For example, in order to implement the above-mentioned function of the active-standby reporting manner, the setting parameters of the log collection tool may include: loggent.inputs: type: log; enabled: true; max_procs:1, a step of; absoltates: paths: -/logs/loggent/error; output. Range shttp: enabled: true; protocol: "https", http or https, default to http; headers: { Host: "ssdk.vpc.com"; hosts: the "", "] is modified to be the server node IP/domain name; loadbalance: true; works er:1, a step of; logging.files: rotateeverybytes:1048576000 =1000 MB; keep files:10, etc.

Optionally, in the master-slave reporting mode, the real-time writing of the file and the time slicing of the file can be used to reduce the data delay, and the file is sliced according to the fixed time interval and the file size, so that the reading and sending speed of the log collecting tool is increased.

Alternatively, the sending capability of the logging tool may be enhanced by adjusting max_proc and worker.

In the embodiment shown in fig. 4, when the real-time requirement is high, a development kit (main) +log collection tool (standby) reporting mode can be adopted, and the development kit reports the normal request in an asynchronous calling mode, so that the real-time performance and transmission performance of data reporting are improved, and the sampling log collection tool monitors the first log supplement reporting mode, so that the integrity of the data is ensured.

Alternatively, the development kit may record only all the request data, so as to implement an asynchronously processed data reporting manner. In the data reporting mode of asynchronous processing, the configuration file may include a second log. Referring to fig. 5, fig. 5 is a detailed flowchart of a second step S300 provided in the embodiment of the present application, and step S300 may include steps S330-S340.

Step S330, determining, by the development kit, a second call request for calling an application interface of the development kit in a case where the embedded data is triggered.

The embedded data is monitored by the development tool package to determine a second call request for calling an application interface of the development tool package under the condition that the embedded data is triggered.

Step S340, writing the second call request into the second log based on the preset data format through the development kit.

The second call request may be recorded, and the second call request is written into a corresponding second log based on a preset data format.

In the embodiment shown in fig. 5, all the monitored requests can be recorded in the corresponding log file to ensure the integrity and reliability of the second log.

Optionally, referring to fig. 6, fig. 6 is a detailed flowchart of step S340 provided in the embodiment of the present application, and step S340 may include steps S341 to S343.

Step S341, determining a second request result of the second call request through the development kit.

In the running process of the client application development kit, the development kit can automatically detect a second request result of the second call request to judge whether the second request is successfully called, so that the request data is written into a normal log or an abnormal log according to different types of second request results.

In step S342, if the second request result is determined to be successful, the second type of successful request is written into the normal log based on the preset data format through the development kit.

When the second request result is determined to be successful in calling, the corresponding second call request is a second type successful request, the development kit can monitor and capture behavior data of the second type successful request of the user, and write the second type successful request into a local or remote normal log based on a preset data format, for example, a JSON format, wherein the normal log can be recorded as data.

In step S343, if the second request result is determined to be the call failure, the second type failure request is written into the exception log based on the preset data format through the development kit.

When it is determined that the second request results in the call failure, the corresponding second call request is a second type failure request, a log callback mechanism may be set in the development kit, and the second type failure request is written into a local or remote exception log based on a preset data format, for example, a JSON format, where the exception log may be recorded as error2.Log.

In the embodiment shown in fig. 6, different types of request data can be written in a classified manner, so that data of different request results can be reported respectively, and the processing effect of asynchronous reporting is effectively optimized.

Optionally, referring to fig. 7, fig. 7 is a detailed flowchart of a second step S400 provided in the embodiment of the present application, and step S400 may include steps S430-S440.

And step S430, processing the normal log and the abnormal log through a log acquisition tool to obtain normal log data and second abnormal log data.

Two log collection tools can be arranged in the client to run the two log collection tools, and the normal log and the abnormal log are processed separately to obtain corresponding normal log data and second abnormal log data.

Optionally, the manner in which the log collection tool processes the second log may include: analyzing and converting the normal log based on a preset data format through a log acquisition tool to obtain second initial data; the normal log data is obtained by combining the second initial data with the metadata associated with the client through a log acquisition tool; analyzing and converting the abnormal log based on a preset data format through a log acquisition tool to obtain third initial data; and combining the third initial data and the metadata through a log acquisition tool to obtain second abnormal log data. The plurality of log collection tools can respectively process the normal log and the abnormal log, respectively analyze and convert the data in the normal log and the abnormal log based on a preset data format, can also perform various processes such as caching, compression, decryption and the like, and add metadata (the metadata can comprise related data such as a time stamp, a host name, a source and the like) of the corresponding client so as to obtain normal log data for normal reporting and second abnormal log data for supplementing reporting. The data can be read and processed in a classified manner based on the writing rule of the data in the second log, and the efficiency and reliability of log processing are effectively improved.

Step S440, normal log data and second abnormal log data are reported to the server based on a preset transmission protocol through a log collection tool.

The plurality of log collection tools may report the normal log data and the second abnormal log data to the server based on a preset transmission protocol, for example, kafka, redis, HTTP or a transmission protocol of a custom type, respectively.

For example, to implement the above-described function of asynchronous reporting, the setting parameters of the development kit may include: enabling sdk function, true sdk function, false disallosdk function: server.sdk.enable=false; whether to save locally, if it needs to be used with loggent, it needs to be defined as true: server.sdk.save=true; saving log file path: server.sdk.eventsavepoth=/logs/loggent; saving the log file name: server.sdk.eventsavenname=data.log, etc.

For example, in order to implement the above-described function of the asynchronous reporting mode, the log collection tool may include a first log collection tool that processes a normal log and a second log collection tool that processes an abnormal log. The setting parameters of the first log collection tool may include: loggent.inputs: type: log; enabled: true; max_procs:1, a step of; absoltates: paths: -/logs/loggent/data.log; output. Range shttp: enabled: true; protocol: "https", http or https, default to http; headers: { Host: "shouqianba.vpc.com" }; hosts: the "", "] is modified to be the server node IP/domain name; loadbalance: true; works er:1, a step of; logging.files: rotateeverybytes:1048576000 =1000 MB; keep files:1024, etc. The distinguishing parameters of the first log collection tool of the second log collection tool may include: loggent.inputs: type: log; enabled: true; max_procs:1, a step of; absoltates: paths: -/logs/loggent/error.

In the embodiment shown in fig. 7, when the number of requests is large, the processing may be performed in an asynchronous reporting manner, so as to implement batch reporting. The generation and the sending of the second log are decoupled, the occupation of application resources of a client is reduced, the influence of log acquisition on service performance is reduced, the data acquisition flexibility is high, various data sources and destinations can be supported, and the format, the filtering rule and the buffer setting of the log can be flexibly configured; meanwhile, the data acquisition reliability is high, the reliability and the integrity of the second log can be guaranteed, the situation of data loss is reduced under the unfavorable conditions that a network terminal or a destination is unavailable, and the reliability of data reporting is improved.

Optionally, referring to fig. 8, fig. 8 is a flowchart of another data reporting method according to an embodiment of the present application, and after step S200, the method may further include steps S510 to S520.

Step S510, determining request parameters of all triggered buried point data.

Development tool packages using different reporting modes can be respectively arranged in the client, and in order to select a proper reporting mode for processing, request parameters corresponding to triggered embedded point data can be determined first. Alternatively, the selection parameters may include: configuration parameters, real-time requirements, request quantity and the like. The configuration parameters may be related parameters determined according to the resource configuration bearing capacity of the server, for example, the maximum number of data reporting allowed by the server, etc., the real-time requirement may be a real-time requirement generated by the client based on the embedded data, for example, a certain embedded data needs to be processed preferentially, etc., the number of requests may be the number parameters of the evaluated request processing, and different data reporting modes may be used in different number of request scenarios.

Step S520, selecting a corresponding development kit for processing based on the request parameter.

The client can evaluate the triggered buried point data from the aspects of configuration, demand, quantity and the like according to the selection parameters of the triggered buried point data, select a corresponding development kit and process the data by adopting a corresponding data reporting mode.

Optionally, when selecting according to multiple request parameters, weight calculation may be used to perform processing, for example, the weight of the configuration parameter is set to a first weight with the largest weight, the weight of the request number is set to a second weight with the smallest weight, and the weight of the real-time requirement is set to a third weight between the first weight and the second weight, so that a corresponding data reporting manner is selected to perform processing according to the result of the weight calculation.

For example, if the resource allocation carrying capacity of the server side is higher, when the requirement of the client side on real-time performance is lower and/or the number of requests is higher, the development kit for asynchronous reporting can be selected for wholesale reporting, or when the requirement of the client side on real-time performance is higher and/or the number of requests is lower, the development kit for master and slave reporting can be selected for quick reporting.

In the embodiment illustrated in fig. 8, a suitable reporting manner can be selected for processing according to the actual situation of the client, so that the efficiency and reliability of data reporting are effectively improved.

Referring to fig. 9, fig. 9 is a schematic structural diagram of a data reporting device provided in the embodiment of the present application, where the data reporting device 600 may include a monitoring module 610, a writing module 620, and a reporting module 630;

the monitoring module 610 is configured to monitor whether the embedded data in the client is triggered by the development kit; the embedded data comprises event data for recording user behaviors and application states;

the writing module 620 is configured to generate, by using the development kit, a configuration file based on the triggered buried data;

the reporting module 630 is configured to report data to the server based on the configuration file.

In an alternative embodiment, wherein the configuration file includes a first log; the writing module 620 is specifically configured to: detecting a first request result of a first call request for calling an application interface of the development kit under the condition that the embedded data is triggered through the development kit; if the first request result is determined to be call failure, the first type of failure request is recorded through the development kit, and the recorded first type of failure request is written into a first log based on a preset data format.

In an alternative embodiment, the reporting module 630 is specifically configured to: if the first request result is judged to be successful in calling, reporting a first type of successful request to a server side based on an asynchronous communication request through a development kit; if the first request result is judged to be the call failure, the first log is processed through a log acquisition tool to obtain first abnormal log data; and reporting the first abnormal log data to the server based on a preset transmission protocol through a log acquisition tool.

In an alternative embodiment, the reporting module 630 is specifically configured to: analyzing and converting the first journal based on a preset data format through a journal acquisition tool to obtain first initial data; and obtaining first abnormal log data by combining the first initial data and the metadata associated with the client through a log acquisition tool.

In an alternative embodiment, wherein the configuration file includes a second log; the writing module 620 is specifically configured to: determining a second call request for calling an application interface of the development kit under the condition that the embedded point data is triggered through the development kit; and writing the second call request into a second log based on a preset data format through a development kit.

In an alternative embodiment, the writing module 620 is specifically configured to: determining a second request result of the second call request through the development kit; if the second request result is judged to be successful in calling, writing a second type of successful request into a normal log based on a preset data format through a development kit; if the second request result is determined to be call failure, writing the second type failure request into an exception log based on a preset data format through a development kit.

In an alternative embodiment, the reporting module 630 is specifically configured to: processing the normal log and the abnormal log through a log acquisition tool to obtain normal log data and second abnormal log data; and reporting normal log data and second abnormal log data to the server based on a preset transmission protocol through a log acquisition tool.

In an alternative embodiment, the reporting module 630 is specifically configured to: analyzing and converting the normal log based on a preset data format through a log acquisition tool to obtain second initial data; the normal log data is obtained by combining the second initial data with the metadata associated with the client through a log acquisition tool; analyzing and converting the abnormal log based on a preset data format through a log acquisition tool to obtain third initial data; and combining the third initial data and the metadata through a log acquisition tool to obtain second abnormal log data.

In an alternative embodiment, the data reporting device 600 may further include a selection module, configured to determine selection parameters of all the triggered buried point data currently; wherein the selection parameters include: at least one of configuration parameters, real-time requirements, and number of requests; and selecting a corresponding development tool package for processing based on the selection parameters.

Since the principle of the data reporting device 600 in the embodiment of the present application for solving the problem is similar to the foregoing embodiment of the data reporting method, the implementation of the data reporting device 600 in the embodiment of the present application may refer to the description in the foregoing embodiment of the data reporting method, and the repetition is omitted.

The embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores computer program instructions, and when the computer program instructions are read and run by a processor, the steps in any one of the methods for reporting data provided in the embodiment are executed.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus may be implemented in other ways. The apparatus embodiments described above are merely illustrative, for example, block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices according to various embodiments of the present application. In this regard, each block in the block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams, and combinations of blocks in the block diagrams, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In addition, the functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.

The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing is merely exemplary embodiments of the present application and is not intended to limit the scope of the present application, and various modifications and variations may be suggested to one skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application. It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.

The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

Claims

1. A method for reporting data, the method comprising:

and reporting data to a server based on the configuration file.

2. The method of claim 1, wherein the configuration file comprises a first log;

3. The method of claim 2, wherein the reporting data to the server based on the configuration file comprises:

4. The method of claim 3, wherein the processing the first log by the log collection tool to obtain first exception log data comprises:

5. The method of claim 1, wherein the configuration file comprises a second log;

6. The method of claim 5, wherein the second log comprises a normal log and an abnormal log;

7. The method of claim 6, wherein the reporting data to the server based on the configuration file comprises:

8. The method of claim 7, wherein the processing the normal log and the abnormal log by the log collection tool to obtain normal log data and second abnormal log data comprises:

9. The method of claim 8, wherein after monitoring whether the buried data in the client is triggered by the development kit, the method further comprises:

10. A data reporting apparatus, the apparatus comprising: the system comprises a monitoring module, a writing module and a reporting module;