CN111177094B - Log data processing method and device, electronic equipment and storage medium - Google Patents
Log data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111177094B CN111177094B CN201910900896.1A CN201910900896A CN111177094B CN 111177094 B CN111177094 B CN 111177094B CN 201910900896 A CN201910900896 A CN 201910900896A CN 111177094 B CN111177094 B CN 111177094B
- Authority
- CN
- China
- Prior art keywords
- log data
- target
- log
- offline
- data transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 230000005540 biological transmission Effects 0.000 claims abstract description 157
- 238000004891 communication Methods 0.000 claims abstract description 57
- 238000012545 processing Methods 0.000 claims abstract description 37
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000012795 verification Methods 0.000 claims description 15
- 238000013480 data collection Methods 0.000 claims description 13
- 230000004044 response Effects 0.000 claims description 5
- 230000006835 compression Effects 0.000 claims description 4
- 238000007906 compression Methods 0.000 claims description 4
- 238000004140 cleaning Methods 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 11
- 238000007726 management method Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000010926 purge Methods 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000011282 treatment Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a log data processing method, a device, electronic equipment and a storage medium, wherein target log data is obtained by responding to a log data transmission request, and the target log data comprises at least part of offline log data locally stored by a log data acquisition end; creating a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end; and sending the target log data to the log data acquisition end through the target data transmission channel. The method realizes the purposes of low data processing capacity of the server and saving the performance of the server.
Description
Technical Field
The present application relates to the technical field of log data processing, and in particular, to a log data processing method, a log data processing device, an electronic device, and a storage medium.
Background
With the rapid development of the internet, internet technology has been incorporated into various industries, and network data generated every day has exhibited explosive growth. The acquisition of log data is an important link in the data processing process, and gradually becomes a focus of attention of developers.
When the log data is obtained, the log data is collected by a log data collection end, and then the log data is sent to a receiving end in real time through a server. Because the log data acquisition ends are more, if all the log data acquisition ends send the log data to the server, the server can receive more log data and the data processing pressure of the server is higher, so that the performance requirement on the server in the existing log data processing process is higher.
Disclosure of Invention
In view of the above, the present application provides a log data processing method, apparatus, electronic device and storage medium, so as to reduce the data processing amount of a server and save the performance of the server.
In order to achieve the above object, in one aspect, the present application provides a log data processing method, including:
Responding to a log data transmission request, and acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by a log data acquisition end;
creating a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel of the log data acquisition end and the log data receiving end;
And sending the target log data to the log data acquisition end through the target data transmission channel.
In one possible implementation manner, the obtaining, in response to the log data transmission request, the target log data includes:
receiving a log data transmission request sent by a server, wherein the log data transmission request is instruction information generated after the server monitors characteristic information of a log data acquisition end;
and acquiring target log data from the offline log locally stored by the log data acquisition end by using the log data transmission request.
In yet another possible implementation manner, the obtaining, in response to the log data transmission request, the target log data includes:
The generated log data transmission request is sent to a server;
and responding to receiving a reporting instruction sent by the server, and acquiring target log data from an offline log locally stored by the log data acquisition end, wherein the reporting instruction is a feedback instruction after the server verifies the log data transmission request.
In still another aspect, the present application further provides a log data processing apparatus, including:
the data acquisition unit is used for responding to the log data transmission request and acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by a log data acquisition end;
the channel creation unit is used for creating a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel of the log data acquisition end and the log data receiving end;
And the data sending unit is used for sending the target log data to the log data acquisition end through the target data transmission channel.
In yet another aspect, the present application further provides an electronic device, including:
A processor and a memory;
wherein the processor is configured to execute a program stored in the memory;
The memory is used for storing a program, and the program is used for at least:
Responding to a log data transmission request, and acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by a log data acquisition end;
creating a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel of the log data acquisition end and the log data receiving end;
And sending the target log data to the log data acquisition end through the target data transmission channel.
In yet another aspect, the present application further provides a storage medium having stored therein computer executable instructions which, when loaded and executed by a processor, implement a log data processing method as described in any one of the above.
Therefore, when the log data is expected to be processed, the target log data is obtained only by responding to the log transmission request, and the target log data is transmitted to the log data receiving end through the target data transmission channel, so that the log data receiving end can process the target log data. Because the target log data comprises at least part of offline log data locally stored by the log data acquisition end, the log data acquired by the log data acquisition end is not reported to the log data receiving end in real time, but is locally stored first, the offline log meeting the log data transmission request is sent to the log data receiving end, and is transmitted through a point-to-point communication channel between the log data acquisition end and the log data receiving end, namely, the transmission of the target log data does not pass through a third-party server, and compared with the real-time uploading, the transmission data volume is reduced, the data processing volume of a server is reduced, and the server performance is saved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only embodiments of the present application, and other drawings may be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a log data processing system according to an embodiment of the present application;
FIG. 2 is a flow diagram of one embodiment of a log data processing method according to an embodiment of the present application;
FIG. 3 illustrates an interface diagram of a supervision log in accordance with an embodiment of the present application;
FIG. 4 shows an offline log view interface of a log data receiving end according to an embodiment of the present application;
FIG. 5 is a schematic diagram illustrating a log data processing procedure performed by a log data collection end according to an embodiment of the present application;
FIG. 6 is a schematic flow chart of a feedback information verification method according to an embodiment of the present application;
FIG. 7 is a schematic diagram showing a structure of a log data processing apparatus according to an embodiment of the present application;
fig. 8 is a schematic diagram showing a composition structure of an electronic device according to an embodiment of the present application.
Detailed Description
The scheme of the application can be applied to a log data transmission and management part in the process of processing log data. The transmission of the log data needs to realize the transmission of the log data of the data sending end and the data receiving end through a communication channel. However, data transmission between the data transmitting end and the data receiving end generally requires data forwarding through an intermediate server, which results in a large data processing capacity of the server, and in particular, when log data generated by a plurality of user ends need to be managed, the performance requirement on the server is too high, so that the maintenance cost is also large. According to the application, the point-to-point data transmission of the log data acquisition end and the log data receiving end is realized through the special data transmission channel, and part of offline log data is selected for transmission, so that the log data transmission process does not pass through an intermediate server, and the requirement on the performance of the server is reduced.
In the embodiment of the application, the log data are generated in each client, specifically, the application of each client, and the log data record real-time operation data of the client or the application in the client, so that operation state of the client can be obtained or fault response can be realized by analyzing the log data by operation and management personnel. In addition, in the embodiment of the present application, the log data may be offline log data, that is, the log data acquisition end stores the acquired log data locally in real time, obtains corresponding log data in the stored offline log according to the log data transmission request, and uses the data as uploading data for subsequent transmission.
For ease of understanding, the constituent architecture of the system to which the scheme of the present application is applied will be described first. With reference now to FIG. 1, a diagram illustrating a component architecture of a log data processing system in accordance with the present application is shown, in particular, (a) of FIG. 1, illustrating one possible component architecture of a log data processing system, and (b) of FIG. 1, illustrating another possible component architecture of a log data processing system. As can be seen from fig. 1 (a), the log processing system includes: a log data acquisition end 101 and a log data receiving end 102; as can be seen from fig. 1 (b), the log system includes: a day data acquisition end 101, a log data receiving end 102 and a server 103.
The log data collection end 101 in the two log processing systems described in fig. 1 realizes collection and transmission of log data, which can be understood as a sending end of log data, and also can be a client end for generating log data. For example, if log data of the first application of the user a needs to be acquired, the corresponding log data acquisition end 101 logs in the terminal device of the first application for the user a.
The log data receiving end 102 is a receiving and processing terminal of log data, that is, it can be understood that it is a management end of log data. Such as a terminal for supervision and processing of abnormal data generated by a certain application.
In fig. 1 (a), the log data transmission request is generated by the log data receiving end 102, specifically, the log data receiving end 102 may be a monitoring end for log data, where the log data transmission request may represent a transmission request of log data to be monitored, which needs to be acquired, that is, the log data receiving end 102 may request log data of a certain period of time, or log data of a certain user. When the log data acquisition end 101 receives the log data transmission request, according to the identification information of the log data included in the log data transmission request, the log data meeting the log data transmission request is searched in the offline log data locally stored in the log data acquisition end and used as target log data, and the target log data is directly sent to the log data receiving end 102.
It should be noted that, the point-to-point data transmission channel between the log data acquisition end and the log data receiving end, that is, the target data transmission channel, may be realized based on the instant messaging tool installed in them, or may be realized by creating a dedicated information transmission channel according to their communication address or communication interface. Correspondingly, the log data acquisition end and the log data receiving end which are subjected to data transmission can send the log data to the log data receiving end through the last data transmission channel according to the communication interface of the log data receiving end when the log data acquisition end performs the next log data transmission.
The main difference between the log data processing system in (b) of fig. 1 and (a) of fig. 1 is that it includes a server 103. But the server 103 does not transmit log data in this embodiment. The method can generate a log data transmission request according to the related information sent by the log data acquisition end 101 or the log data receiving end 102, and send the log data transmission request to the log data acquisition end 101. The server 103 may also send identification information, such as communication address information, corresponding to the log data acquisition end 101 and the log data receiving end 102 to the two terminals, so that the log data acquisition end and the log data receiving end can respectively obtain specific communication information of the data transmitting end and the data receiving end, and establish a data transmission channel. Then, the log data collection terminal 101 may send the target log data to the log data receiving terminal 102 through the established target data transmission channel.
Referring to fig. 2, a flow chart of an embodiment of a log data processing method of the present application is shown, where the method of the embodiment may include:
S201, a server sends a log data transmission request and a second communication address of a log data receiving end to a log data acquisition end;
s202, a server sends a first communication address of a log data acquisition end to a log data receiving end;
S203, the log data acquisition end obtains target log data from locally stored offline log data by using a log data transmission request;
S204, the log data acquisition end creates a target data transmission channel according to the first communication address of the log data acquisition end and the second communication address of the log data receiving end;
s205, the log data acquisition end sends the target log data to the log data receiving end through the target data transmission channel.
S206, outputting target log data by the log data receiving end.
Firstly, a server sends communication identification information of a log data acquisition end to a log data receiving end, and sends communication identification information of a log data receiving end to the log data acquisition end, so that the log data acquisition end can know which terminal the log data is sent to, and the log data receiving end can know which terminal the log data sent is received, thereby realizing that a point-to-point data transmission channel can be created or matched for the log data acquisition end and the log data receiving end to serve as a target data transmission channel for subsequent target log data transmission. The log data acquisition end can also be made to establish a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end, wherein the target data transmission channel is a point-to-point communication channel between the log data acquisition end and the log data receiving end, and no third party equipment or server forwards the log data. Therefore, in the embodiment of the application, the server does not need to forward the log data, and only needs to maintain the communication identification information, such as the communication address, of the log data receiving end and the log data collecting end.
The server sends the log transmission request to the log data acquisition end, and the generated log transmission request can be sent by the server based on the supervision condition of the user of the log data acquisition end. The corresponding process of obtaining the target log data may include:
receiving a log data transmission request sent by a server, wherein the log data transmission request is instruction information generated after the server monitors characteristic information of a log data acquisition end;
and acquiring target log data from the offline log locally stored by the log data acquisition end by using the log data transmission request.
In this embodiment, the characteristic information obtained when the server monitors the log data collection end may represent information of the log data collection end and may represent information of a specific application running on the log data collection end. For example, it may be the runtime of the log data collection end. The log reporting condition indicates a condition that log data uploading is required, and the time may reach a certain fixed time. When the monitored characteristic information meets the log reporting condition, a log data transmission request is generated and sent to a log data acquisition end.
The log data acquisition end analyzes the log data transmission request to obtain the characteristic identification of the log data to be uploaded, so that the log data acquisition end can conveniently search and obtain target log data from the locally stored offline log.
The log data acquisition end stores the acquired log data into the local database in real time, and only uploads the log data meeting the conditions, so that the data pressure of uploading the log data is reduced, and bandwidth resources are saved; on the other hand, the maintenance of log data is also facilitated.
In another possible implementation manner of the embodiment of the present application, the server sends instruction information corresponding to the log data transmission request to the log data acquisition end, and the corresponding acquisition process of the target log data may include:
The generated log data transmission request is sent to a server;
and responding to receiving a reporting instruction sent by the server, and acquiring target log data from an offline log locally stored by the log data acquisition end, wherein the reporting instruction is a feedback instruction after the server verifies the log data transmission request.
In this embodiment, the log data reporting request is generated by the log data collecting end, when the log data collecting end calls the server interface under a specific condition, inquires whether the offline log data needs to be uploaded, if the server sends an instruction, the instruction instructs to report the offline log data, that is, determines the log data transmission request, and then obtains the target log data, that is, the offline log data which is partially locally stored, based on the analysis of the log data transmission request. The specific condition that the log data acquisition end calls the server interface under the specific condition can be that the running time monitored by the log data acquisition end reaches the reporting time, and also can monitor that the information is abnormal, such as when a user repeatedly logs in a certain application.
After the log data acquisition end acquires the target log data, the target log data is transmitted based on a target data transmission channel, wherein the target data transmission channel is a point-to-point communication channel of the log data acquisition end and the log data receiving end. Therefore, the log data is directly transmitted to the log data receiving end without passing through a server when being uploaded by the log data collecting end. The target data transmission channel can be realized based on the webpage instant messaging technology or can be realized through a pre-allocated communication channel, and the application is specifically described in the following embodiments.
After the log data receiving end obtains the target log data, the target log data can be output, so that a user of the log data receiving end can perform log data processing according to the target log data or perform fault handling according to the target log data.
The following describes a log data processing method according to an embodiment of the present application with log data of browser a. Referring to fig. 3, an interface schematic of a supervision log is shown. If it is desired to obtain the most commonly used browser functions of the new user of browser a by analyzing the log data of browser a. I.e. log data of the new user of browser a that is actually desired. Therefore, the log data of the browser A can be stored offline according to the user identifications of different users. Namely, log data acquired by the log data acquisition end are stored in a local target database in real time, and when offline log data are obtained, the target database can adopt an index database. The application scenario corresponding to this embodiment may be used as an index of the index data according to the identification information of the user.
Correspondingly, responding to the log data transmission request, obtaining target log data comprises the following steps:
analyzing the log data transmission request to obtain identification information corresponding to the log data to be uploaded;
and index inquiry is carried out in the index database by utilizing the identification information, so as to obtain target log data.
Referring to the interface schematic diagram of fig. 3, if the identification information of the new user of the browser a is ID12345, an input dialog box may be displayed by selecting the button "set offline log auto-pull", and "ID12345" may be input in the input dialog box, so as to achieve the acquisition of the offline log of the user of the identification information. When the user logs in the browser A, the log data acquisition end can send the offline log of the user to the log data receiving end.
Referring to fig. 4, an offline log view interface of a log data receiving end is shown. By selecting the corresponding functional plate in the interface, the user of the log data receiving end can view the log data. The offline log reported by the log data acquisition end can be checked through a functional plate of project offline log. As shown in fig. 4, the offline log data may be displayed one by one according to the reporting time, or may be displayed one by one according to the reporting sequence of the log data.
Of course, the log data receiving end can also check the real-time log and the history log, wherein the real-time log is the log which needs to be reported to the log data collecting end in real time in the project, and the difference between the real-time log and the offline log is that the log data collecting end can directly upload the log data to the log data receiving end through a server after collecting the log data. The history log refers to the log data received by the log data receiving end, so that the log data can be managed and checked conveniently.
Referring to fig. 5, a schematic diagram of a log data processing procedure performed by the log data collection end according to an embodiment of the present application is shown. The log data collection end is the user terminal 50, stores the log data generated in real time in the index database 501 local to the user terminal, and uploads the target log data in the index database 501, that is, part of the offline log data, to the log data receiving end 51 after obtaining the log data transmission request. When a log data transmission request is received, analyzing the log data transmission request to obtain identification information corresponding to log data to be uploaded; and carrying out log data query in the index database by utilizing the identification information to obtain target log data. I.e. the identification information is queried as an index in the index database.
When processing the log data of the browser, an index database (indexedDB) is used, which is a database for storing structured data in the browser, supports indexing, and completes the query operation through pointers generated by the indexing. When indexedDB is used as a storage party of the front-end offline log system, the following advantages are provided: the key value pair based on the JavaScript object is stored, so that the method is simple and easy to use; the storage space is larger; having an asynchronous API (Application Programming Interface ) so that accesses to the database do not block the calling thread; supporting transactions; the compatibility is better. It should be noted that indexedDB is only an example of a database for offline log data storage in the embodiment of the present application, other databases may be used for different application scenarios, for example, a network disk database may be used at the applet end.
Also, for browser applications, webRTC (Web Real-Time Communication, web instant messaging) may be used as a transmission method of the offline log. The system is an API supporting a web browser to conduct real-time voice conversation or video conversation, and has the following advantages; the operation and the transmission of the streaming data can be completed based on the JavaScript interface provided by the browser; the peer-to-peer communication reduces the load of the server, the server only needs to store the ip and the port of both communication parties and the content supervision of which users need to transmit the offline logs, and huge offline log source files are not needed to be stored, so that the performance pressure and the bandwidth pressure of the service can be effectively reduced; based on the browser and supporting cross-platform, the log data acquisition end can treat the user indifferently, and can acquire the log only by the unique identification of the user.
In order to ensure the security of the log data reporting, the uploaded log data may be checked, for example, token check. The log data acquisition end can be checked, and the log data can be uploaded according to the checking result. Correspondingly, the method for sending the target log data to the log data acquisition end comprises the following steps:
verifying the authority information of the log data receiving end to obtain a verification result;
And processing the target log data by using the verification result, and sending the processed target log data to a log data receiving end through a target data transmission channel.
The authority information of the log data receiving end is checked, whether the log data receiving end has the receiving authority or not is checked, different treatments can be performed according to the level of the checking result of the authority, for example, if the authority information of the log data receiving end is checked, the log data receiving end has risk, and the target log data can be encrypted and then sent. If the authority information indicates the data receiving capability of the log data receiving end, the target log data can be compressed according to the data receiving capability of the log data receiving end and then sent.
The embodiment of the application also provides a feedback information verification method to ensure that the target log data is successfully sent. Referring to fig. 6, a flow diagram of a feedback information verification method is shown. The method comprises the following steps:
S601, receiving feedback information of a receiving end of receiving log data aiming at target log data;
S602, judging whether the receiving feedback information meets the receiving failure condition, and if so, executing S603;
S603, determining retransmission time according to the network state of the target data transmission channel, and realizing retransmission of the target data.
If the transmission still fails, the process is repeated to be transmitted again until the target log data receiving end successfully receives the target log data. The failure of transmission is generally mainly due to poor network state, such as weak network or no network. Therefore, the log data acquisition end can judge the network state, and resend the network state when the network state is good or the network condition is met.
The log data processing method of the present application will be described below with a log system to which indexedDB and WebRTC are applied. The front end refers to a log data collection end, uses indexedDB as a storage repository, indexedDB is a high-performance database based on a browser, and in the embodiment of the present application, basic operations on indexedDB are encapsulated, and five interfaces are provided, including: the log interface is cleared, and because log data is continuously generated, in order to avoid infinite accumulation of data, fixed days are usually set, and the log data is checked to clear out outdated log data when the system is started each time; the log writing interface allows new logs to be written continuously through an asynchronous log data writing method, namely, allows different log data acquisition ends to write offline log data; searching related interfaces, including searching logs of the current user, logs of a fixed time period, logs of a fixed grade and the like, so as to conveniently obtain log information meeting the conditions; the data sorting and compressing interface can effectively reduce the size of reported data by sorting and compressing the data due to the large data volume of the log; and the data reporting interface is used for reporting the target log data.
The reporting process of the service end package comprises the following steps: detecting whether a database already exists, if not, initializing indexedDB, opening the database, creating the database and tables, and creating an index; detecting whether an expired log exists or not, and if so, cleaning the expired log; providing an interface for writing the log to call a service end, and continuously writing the offline log by the service end; and requesting the data of the server, detecting whether reporting is needed, and if so, retrieving, packaging and compressing the data and transmitting the data.
The server of the log system in the embodiment of the application has the following functions: maintaining a state of whether a user needs to upload an offline log; constructing a signaling message server as a WebRTC server, and maintaining addresses of a log receiving end and a reporting end; providing an interface for receiving offline logs for a browser or terminal which does not support WebRTC; and providing an interface for offline log viewing for a log management platform.
In order to accomplish the above functions, the server end designs 4 main interfaces in total:
the interface for reporting supervision is not capable of reporting the offline logs of all users due to the characteristics of the offline logs, so that a server side is provided with a plurality of supervision conditions, and only users meeting the conditions report the offline logs;
detecting whether an interface of the offline log is reported or not and informing a client whether to send a request to a server or not;
The offline log transmission interface informs the client of the WebRTC transmission address or reports the WebRTC transmission address to the interface of the server, and the interface is provided for being compatible with a browser or a terminal which does not support WebRTC;
The data display interface is used for returning the offline log of the server to the front end.
Besides the interfaces, the server side also needs to be compatible with cross-domain requests, decompresses data and stores files, stores addresses of reported users and stores information of management end users. And also to take into account the security of the interface, since we need token checking the reported data.
The front-end offline log system based on IndexedDB and WebRTC provided by the embodiment of the application is used for solving the problems of log reporting, error collection, statistical analysis and the like in front-end projects, and has wide application scenes in the aspects of front-end quality and data collection. IndexedDB is used for offline log collection at the front end, and statistics can be reported through JavaScript. After the log is reported, the log can be selectively stored in a server side, and the log can also be directly reported to a log receiving side through a WebRTC, so that the waste of bandwidth and disk resources of the server side is reduced. The WebRTC has the characteristics of high-efficiency transmission and point-to-point communication, and can also be compressed by using a deflate/gzip algorithm in the log transmission process, so that the transmission efficiency is further improved, and the user network resource request is reduced. The method can be applied to a PC web end, a mobile web end and an applet end (the applet storage replaces IndexedDB), a browser combination version and other numerous application scenes, and a plurality of operating systems of windows, linux, macOS, iOS and Android are supported.
In yet another aspect, referring to fig. 7, a schematic structural diagram of a log data processing apparatus according to an embodiment of the present application is shown, where the apparatus includes:
A data acquisition unit 701, configured to acquire target log data in response to a log data transmission request, where the target log data includes at least part of offline log data locally stored by a log data acquisition end;
A channel creation unit 702, configured to create a target data transmission channel from the first communication address of the log data acquisition end and the second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel of the log data acquisition end and the log data receiving end;
the data sending unit 703 is configured to send the target log data to the log data collecting end through the target data transmission channel.
In one possible implementation manner, the data acquisition unit includes:
the request receiving subunit is used for receiving a log data transmission request sent by a server, wherein the log data transmission request is instruction information generated after the server monitors characteristic information of the log data acquisition end;
And the log data acquisition subunit is used for acquiring target log data from the offline log locally stored by the log data acquisition end by utilizing the log data transmission request.
Optionally, the data acquisition unit further includes:
a request sending subunit, configured to send the generated log data transmission request to a server;
The log acquisition subunit is used for responding to the report instruction sent by the server and acquiring target log data from the offline log locally stored by the log data acquisition end, wherein the report instruction is a feedback instruction after the server verifies the log data transmission request.
Optionally, the data acquisition unit is further configured to:
analyzing the log data transmission request to obtain identification information corresponding to the log data to be uploaded;
and carrying out log data query in the index database by utilizing the identification information to obtain target log data.
In yet another possible implementation manner, the data sending unit is specifically configured to:
verifying the authority information of the log data receiving end to obtain a verification result;
And processing the target log data by using the verification result, and sending the processed target log data to a log data receiving end through a target data transmission channel.
Optionally, the device further includes a feedback information verification unit, where the feedback information verification unit is specifically configured to:
receiving feedback information of the log data receiving end aiming at the target log data;
And if the receiving feedback information meets the receiving failure condition, retransmitting the target log data according to the network state of the target data transmission channel until the target log data receiving end successfully receives the target log data.
Optionally, the apparatus further comprises a log management unit, wherein the log management unit is used for:
And managing the offline log data in the local index database so that the local index database meets the storage requirement of the offline log data in real time, wherein the management of the offline log data comprises one or more of cleaning of expired offline log data, asynchronous writing of the offline log data, inquiring of the offline log data or compression processing of the offline log data.
In another aspect, the present application further provides an electronic device, for example, referring to fig. 8, which shows a schematic structural diagram of the electronic device of the present application, an electronic device 800 of the present embodiment may include: a processor 801 and a memory 802.
Optionally, the electronic device may further comprise a communication interface 803, an input unit 804 and a display 805 and a communication bus 806.
The processor 801, the memory 802, the communication interface 803, the input unit 804, the display 805, and the like all perform communication with each other through the communication bus 806.
In an embodiment of the present application, the processor 801 may be a central processing unit (Central Processing Unit, CPU), an asic, a dsp, an off-the-shelf programmable gate array, or other programmable logic device.
The processor may call a program stored in the memory 802. The memory 1102 is used to store one or more programs, and the programs may include program code that includes computer operation instructions, and in an embodiment of the present application, at least the programs for implementing the following functions are stored in the memory:
Responding to a log data transmission request, and acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by a log data acquisition end;
creating a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel of the log data acquisition end and the log data receiving end;
And sending the target log data to the log data acquisition end through the target data transmission channel.
In one possible implementation, the memory 802 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, and applications required for at least one function (e.g., log purge, log query, etc.), etc.; the storage data area may store data created during use of the computer, such as receiver identification information, offline log data, and the like.
In addition, memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device or other volatile solid-state storage device.
The communication interface 803 may be an interface of a communication module, such as an interface of a GSM module.
The application may also include a display 805 and an input unit 804, etc.
Of course, the structure of the electronic device shown in fig. 8 does not limit the terminal in the embodiment of the present application, and the electronic device may include more or fewer components than those shown in fig. 8 or may combine some components in practical applications.
On the other hand, the embodiment of the application also provides a storage medium, wherein the storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by a processor, the log data processing method in any embodiment is realized.
It should be noted that, in the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described as different from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other. For the apparatus class embodiments, the description is relatively simple as it is substantially similar to the method embodiments, and reference is made to the description of the method embodiments for relevant points.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.
Claims (10)
1.A log data processing method, comprising:
Responding to a log data transmission request, acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by a log data acquisition end, the log data acquisition end is a user terminal, the log data generated by the user terminal in real time is stored into a local index database of the user terminal, the log data transmission request is analyzed to obtain identification information corresponding to the log data to be uploaded, the log data is inquired in the index database by utilizing the identification information to obtain the target log data, and the offline log data in the local index database are managed, so that the local index database meets the storage requirement of the offline log data in real time, wherein the management of the offline log data comprises one or more of removal of expired offline log data, asynchronous writing of the offline log data, inquiry of the offline log data and compression processing of the offline log data;
creating a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel of the log data acquisition end and the log data receiving end;
The target log data is sent to the log data acquisition end through the target data transmission channel, wherein partial offline log data is selected to be transmitted through the special data transmission channel, so that point-to-point data transmission of the log data acquisition end and the log data receiving end is realized, an intermediate server is not needed in the log data transmission process, and the requirement on the performance of the server is reduced;
the responding to the log data transmission request, obtaining the target log data, includes:
receiving a log data transmission request sent by a server, wherein the log data transmission request is instruction information generated after the server monitors characteristic information of a log data acquisition end;
and acquiring target log data from the offline log locally stored by the log data acquisition end by using the log data transmission request.
2. The method of claim 1, wherein the obtaining the target log data in response to the log data transfer request comprises:
The generated log data transmission request is sent to a server;
and responding to receiving a reporting instruction sent by the server, and acquiring target log data from an offline log locally stored by the log data acquisition end, wherein the reporting instruction is a feedback instruction after the server verifies the log data transmission request.
3. The method according to claim 1, wherein the sending the target log data to the log data collection end through the target data transmission channel includes:
verifying the authority information of the log data receiving end to obtain a verification result;
And processing the target log data by using the verification result, and sending the processed target log data to a log data receiving end through a target data transmission channel.
4. A method according to claim 3, characterized in that the method further comprises:
receiving feedback information of the log data receiving end aiming at the target log data;
And if the receiving feedback information meets the receiving failure condition, retransmitting the target log data according to the network state of the target data transmission channel until the target log data receiving end successfully receives the target log data.
5. A log data processing apparatus, comprising:
The system comprises a data acquisition unit, a log data transmission unit and a data query unit, wherein the data acquisition unit is used for responding to a log data transmission request to acquire target log data, the target log data comprises at least part of offline log data locally stored by a log data acquisition end, the log data acquisition end is a user terminal, the log data generated by the user terminal in real time are stored in a local index database of the user terminal, the log data transmission request is analyzed to acquire identification information corresponding to the log data to be uploaded, and the identification information is utilized to perform log data query in the index database to acquire the target log data;
the channel creation unit is used for creating a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel of the log data acquisition end and the log data receiving end;
The data transmitting unit is used for transmitting the target log data to the log data acquisition end through the target data transmission channel, wherein partial offline log data is selected to be transmitted through the special data transmission channel, so that point-to-point data transmission of the log data acquisition end and the log data receiving end is realized, an intermediate server is not needed in the log data transmission process, and the requirement on the performance of the server is reduced;
the log management unit is used for managing the offline log data in the local index database so that the local index database meets the storage requirement of the offline log data in real time, wherein the management of the offline log data comprises one or more of cleaning of the expired offline log data, asynchronous writing of the offline log data, inquiring of the offline log data or compression processing of the offline log data;
Wherein the data acquisition unit includes:
the request receiving subunit is used for receiving a log data transmission request sent by a server, wherein the log data transmission request is instruction information generated after the server monitors characteristic information of the log data acquisition end;
And the log data acquisition subunit is used for acquiring target log data from the offline log locally stored by the log data acquisition end by utilizing the log data transmission request.
6. The apparatus of claim 5, wherein the data acquisition unit further comprises:
a request sending subunit, configured to send the generated log data transmission request to a server;
The log acquisition subunit is used for responding to the report instruction sent by the server and acquiring target log data from the offline log locally stored by the log data acquisition end, wherein the report instruction is a feedback instruction after the server verifies the log data transmission request.
7. The apparatus according to claim 5, wherein the data transmission unit is specifically configured to:
verifying the authority information of the log data receiving end to obtain a verification result;
And processing the target log data by using the verification result, and sending the processed target log data to a log data receiving end through a target data transmission channel.
8. The apparatus according to claim 7, further comprising a feedback information verification unit;
The feedback information verification unit is specifically configured to receive feedback information of the log data receiving end for the target log data; and if the receiving feedback information meets the receiving failure condition, retransmitting the target log data according to the network state of the target data transmission channel until the target log data receiving end successfully receives the target log data.
9. An electronic device, comprising:
A processor and a memory;
wherein the processor is configured to execute a program stored in the memory;
The memory is used for storing a program, and the program is used for at least:
Responding to a log data transmission request, acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by a log data acquisition end, the log data acquisition end is a user terminal, the log data generated by the user terminal in real time is stored into a local index database of the user terminal, the log data transmission request is analyzed to obtain identification information corresponding to the log data to be uploaded, the log data is inquired in the index database by utilizing the identification information to obtain the target log data, and the offline log data in the local index database are managed, so that the local index database meets the storage requirement of the offline log data in real time, wherein the management of the offline log data comprises one or more of removal of expired offline log data, asynchronous writing of the offline log data, inquiry of the offline log data and compression processing of the offline log data;
creating a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel of the log data acquisition end and the log data receiving end;
The target log data is sent to the log data acquisition end through the target data transmission channel, wherein partial offline log data is selected to be transmitted through the special data transmission channel, so that point-to-point data transmission of the log data acquisition end and the log data receiving end is realized, an intermediate server is not needed in the log data transmission process, and the requirement on the performance of the server is reduced;
the responding to the log data transmission request, obtaining the target log data, includes:
receiving a log data transmission request sent by a server, wherein the log data transmission request is instruction information generated after the server monitors characteristic information of a log data acquisition end;
and acquiring target log data from the offline log locally stored by the log data acquisition end by using the log data transmission request.
10. A storage medium having stored therein computer executable instructions which when loaded and executed by a processor implement the log data processing method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910900896.1A CN111177094B (en) | 2019-09-23 | 2019-09-23 | Log data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910900896.1A CN111177094B (en) | 2019-09-23 | 2019-09-23 | Log data processing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111177094A CN111177094A (en) | 2020-05-19 |
| CN111177094B true CN111177094B (en) | 2024-08-06 |
Family
ID=70622389
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910900896.1A Active CN111177094B (en) | 2019-09-23 | 2019-09-23 | Log data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111177094B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112015707A (en) * | 2020-08-26 | 2020-12-01 | 长春创世麒麟科技有限公司 | Log data collection method and log data collection system |
| CN111953551B (en) * | 2020-08-27 | 2023-05-16 | 网易（杭州）网络有限公司 | Log data transmission method and device, electronic equipment and storage medium |
| CN114500503B (en) * | 2022-01-14 | 2024-08-30 | 广东好太太智能家居有限公司 | Intelligent device offline log uploading method, intelligent device and readable medium |
| CN114116731B (en) * | 2022-01-24 | 2022-04-22 | 北京智象信息技术有限公司 | Data separation storage display method and device based on indexedDB storage |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273266A (en) * | 2017-06-09 | 2017-10-20 | 上海艾融软件股份有限公司 | A kind of processing method and processing device of application daily record |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102891873B (en) * | 2011-07-21 | 2017-02-15 | 腾讯科技（深圳）有限公司 | Method for storing log data and log data storage system |
| CN108365971A (en) * | 2018-01-10 | 2018-08-03 | 深圳市金立通信设备有限公司 | Daily record analytic method, equipment and computer-readable medium |
-
2019
- 2019-09-23 CN CN201910900896.1A patent/CN111177094B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107273266A (en) * | 2017-06-09 | 2017-10-20 | 上海艾融软件股份有限公司 | A kind of processing method and processing device of application daily record |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111177094A (en) | 2020-05-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111177094B (en) | Log data processing method and device, electronic equipment and storage medium | |
| CN110740103A (en) | Service request processing method and device, computer equipment and storage medium | |
| US7702917B2 (en) | Data transfer using hyper-text transfer protocol (HTTP) query strings | |
| EP3386150A1 (en) | Terminal failure processing method, device and system | |
| US20180287920A1 (en) | Intercepting application traffic monitor and analyzer | |
| CN113687974B (en) | Client log processing method and device and computer equipment | |
| US11188443B2 (en) | Method, apparatus and system for processing log data | |
| CN110162512A (en) | A kind of log searching method, apparatus and storage medium | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| WO2023168926A1 (en) | Method and apparatus for determining software anomaly, storage medium, and electronic apparatus | |
| CN112765103A (en) | File analysis method, system, device and equipment | |
| CN105099769A (en) | Method, device and system for processing abnormal operations of service platform | |
| CN108228432A (en) | A kind of distributed link tracking, analysis method and server, global scheduler | |
| CN106126419A (en) | The adjustment method of a kind of application program and device | |
| CN104348661A (en) | Network failure data uploading and receiving methods and apparatus and network failure data recording method and system | |
| CN109409948B (en) | Transaction abnormity detection method, device, equipment and computer readable storage medium | |
| CN110309028B (en) | Monitoring information acquisition method, service monitoring method, device and system | |
| CN113055493A (en) | Data packet processing method, device, system, scheduling device and storage medium | |
| CN112181929A (en) | Cloud management platform log processing method and device, electronic device and storage medium | |
| CN114500348B (en) | CDN gateway testing method and system | |
| CN109684279B (en) | Data processing method and system | |
| CN113612811B (en) | Method, system, equipment and medium for client mounting in multiple channels | |
| US20140143264A1 (en) | Policy event driven remote desktop recording across a data network | |
| CN113746794B (en) | Account processing method and device, electronic equipment and computer storage medium | |
| CN112131198B (en) | Log analysis method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |