CN111177094A - Log data processing method and device, electronic equipment and storage medium - Google Patents
Log data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111177094A CN111177094A CN201910900896.1A CN201910900896A CN111177094A CN 111177094 A CN111177094 A CN 111177094A CN 201910900896 A CN201910900896 A CN 201910900896A CN 111177094 A CN111177094 A CN 111177094A
- Authority
- CN
- China
- Prior art keywords
- log data
- target
- log
- offline
- data transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 230000005540 biological transmission Effects 0.000 claims abstract description 131
- 238000004891 communication Methods 0.000 claims abstract description 56
- 238000012545 processing Methods 0.000 claims abstract description 30
- 238000000034 method Methods 0.000 claims description 36
- 230000004044 response Effects 0.000 claims description 12
- 238000012795 verification Methods 0.000 claims description 12
- 238000013480 data collection Methods 0.000 claims description 9
- 238000012546 transfer Methods 0.000 claims 2
- 230000008569 process Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 11
- 238000007726 management method Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 238000012544 monitoring process Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a log data processing method, a log data processing device, electronic equipment and a storage medium, wherein target log data are acquired by responding to a log data transmission request, and the target log data comprise at least part of offline log data locally stored by a log data acquisition terminal; establishing a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end; and sending the target log data to the log data acquisition terminal through the target data transmission channel. The data processing amount of the server is reduced, and the performance of the server is saved.
Description
Technical Field
The present application relates to the field of technologies, and in particular, to a log data processing method and apparatus, an electronic device, and a storage medium.
Background
With the rapid development of the internet, the internet technology has been integrated into various industries, and the network data generated every day shows explosive growth. The acquisition of log data is an important link in the data processing process, and becomes a focus of attention of developers gradually.
Usually, when acquiring log data, a log data acquisition end acquires the log data, and then the log data is sent to a receiving end in real time through a server. Because the number of log data acquisition ends is large, if all the log data acquisition ends send the log data to the server, the log data received by the server is large, and the data processing pressure of the server is large, so that the performance requirement on the server in the existing log data processing process is high.
Disclosure of Invention
In view of this, the present application provides a log data processing method, an apparatus, an electronic device, and a storage medium, so as to reduce data processing amount of a server and save server performance.
In order to achieve the above object, in one aspect, the present application provides a log data processing method, including:
responding to a log data transmission request, and acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by a log data acquisition end;
creating a target data transmission channel by using a first communication address of the log data acquisition end and a second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel between the log data acquisition end and the log data receiving end;
and sending the target log data to the log data acquisition terminal through the target data transmission channel.
In one possible implementation manner, the obtaining target log data in response to the log data transmission request includes:
receiving a log data transmission request sent by a server, wherein the log data transmission request is instruction information generated after the server monitors the characteristic information of a log data acquisition end;
and acquiring target log data from the offline log locally stored at the log data acquisition end by using the log data transmission request.
In another possible implementation manner, the obtaining target log data in response to the log data transmission request includes:
sending the generated log data transmission request to a server;
and in response to receiving a reporting instruction sent by the server, acquiring target log data from an offline log locally stored at the log data acquisition end, wherein the reporting instruction is a feedback instruction after the server verifies the log data transmission request.
In another aspect, the present application further provides a log data processing apparatus, including:
the data acquisition unit is used for responding to the log data transmission request and acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by the log data acquisition end;
a channel creating unit, configured to create a target data transmission channel from a first communication address of the log data acquisition end and a second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel between the log data acquisition end and the log data receiving end;
and the data sending unit is used for sending the target log data to the log data acquisition end through the target data transmission channel.
In another aspect, the present application further provides an electronic device, including:
a processor and a memory;
wherein the processor is configured to execute a program stored in the memory;
the memory is to store a program to at least:
responding to a log data transmission request, and acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by a log data acquisition end;
creating a target data transmission channel by using a first communication address of the log data acquisition end and a second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel between the log data acquisition end and the log data receiving end;
and sending the target log data to the log data acquisition terminal through the target data transmission channel.
In still another aspect, the present application further provides a storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are loaded and executed by a processor, the log data processing method as described in any one of the above is implemented.
Therefore, when log data is expected to be processed, target log data only needs to be obtained by responding to the log transmission request, and the target log data is transmitted to the log data receiving end through the target data transmission channel, so that the log data receiving end can process the target log data. The target log data comprises at least part of offline log data locally stored by the log data acquisition end, so that the log data acquired by the log data acquisition end is not reported to the log data receiving end in real time, but is locally stored firstly, the offline log meeting the log data transmission request is sent to the log data receiving end, and is transmitted through a point-to-point communication channel between the log data acquisition end and the log data receiving end, namely the target log data is not transmitted through a third-party server, and the transmission data amount is reduced compared with real-time uploading, so that the data processing amount of the server is reduced, and the performance of the server is saved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on the provided drawings without creative efforts.
FIG. 1 is a block diagram of a log data processing system according to an embodiment of the present application;
FIG. 2 is a flowchart illustrating an embodiment of a log data processing method according to an embodiment of the present application;
FIG. 3 is a schematic diagram illustrating an interface for listening to a log according to an embodiment of the present application;
fig. 4 illustrates an offline log viewing interface of a log data receiving end according to an embodiment of the present application;
fig. 5 is a schematic diagram illustrating a process of processing log data by a log data acquisition end according to an embodiment of the present application;
fig. 6 is a schematic flow chart illustrating a feedback information verification method according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a log data processing apparatus according to an embodiment of the present application;
fig. 8 is a schematic diagram illustrating a composition structure of an electronic device according to an embodiment of the present application.
Detailed Description
The scheme of the application can be applied to the transmission and management part of the log data in the process of processing the log data. The transmission of the log data needs to realize the transmission of the log data of the data sending end and the data receiving end through a communication channel. However, generally, data transmission between a data sending end and a data receiving end requires data forwarding through an intermediate server, which results in a large data processing amount of the server, and particularly, for some log data generated by a plurality of user ends to be monitored, the performance requirement on the server is too high, which results in a large maintenance cost. In the application, point-to-point data transmission between the log data acquisition end and the log data receiving end is realized through the special data transmission channel, and partial offline log data is selected for transmission, so that the log data transmission process does not pass through an intermediate server, and the requirement on the performance of the server is reduced.
In the embodiment of the application, the log data is generated at each client, specifically, the application of each client, and the log data records real-time operation data of the client or the application in the client, so that operation and maintenance and management personnel can obtain the operation state of the client or realize fault response by analyzing the log data. In addition, in the embodiment of the application, the log data may be offline log data, that is, the log data acquisition end stores the acquired log data locally in real time, obtains corresponding log data from the stored offline log according to the log data transmission request, and performs subsequent transmission by using the data as upload data.
For ease of understanding, the constituent architecture of the system to which the solution of the present application is applied will be described first. For example, referring to fig. 1, a schematic diagram of a component architecture of a log data processing system according to the present application is shown, specifically, fig. 1(a) shows one possible component architecture of the log data processing system, and fig. 1(b) shows another possible component architecture of the log data processing system. As can be seen from fig. 1(a), the log processing system includes: a log data acquisition end 101 and a log data receiving end 102; as can be seen from fig. 1(b), the log system includes: a day data acquisition end 101, a log data receiving end 102 and a server 103.
The log data collection end 101 in the two log processing systems described in fig. 1 realizes collection and transmission of log data, which can be understood as a sending end of log data, and also can be a client end that generates log data. For example, if log data of a first application of the user a needs to be acquired, the corresponding log data acquisition terminal 101 logs in the terminal device of the first application for the user a.
The log data receiving end 102 is a receiving and processing terminal of log data, i.e. it can be understood as a management end of log data. For example, a terminal for monitoring and processing abnormal data generated by a certain application.
In fig. 1(a), the log data transmission request is generated by the log data receiving end 102, specifically, the log data receiving end 102 may be a monitoring end for log data, and the log data transmission request may represent a transmission request of log data to be monitored, which needs to be acquired, that is, the log data receiving end 102 may request log data of a certain time period or log data of a certain user. When receiving the log data transmission request, the log data acquisition end 101 searches for the log data meeting the log data transmission request from the offline log data locally stored in the log data acquisition end as target log data according to the identification information of the log data contained in the log data transmission request, and directly sends the target log data to the log data receiving end 102.
It should be noted that the point-to-point data transmission channel between the log data acquisition end and the log data receiving end, i.e., the target data transmission channel, may be implemented based on an instant messaging tool installed therein, or may be implemented by creating a dedicated information transmission channel according to a communication address or a communication interface thereof. Correspondingly, the log data acquisition end and the log data receiving end which are subjected to data transmission can send the log data to the log data receiving end through the last data transmission channel according to the communication interface of the log data receiving end when the log data acquisition end carries out next log data transmission.
The main difference between the log data processing system in fig. 1(b) and fig. 1(a) is that it includes a server 103. But the server 103 does not transmit log data in this embodiment. The log data transmission request can be generated according to the related information sent by the log data acquisition end 101 or the log data receiving end 102, and the log data transmission request is issued to the log data acquisition end 101. The server 103 may also send identification information, such as communication address information, corresponding to the log data acquisition end 101 and the log data receiving end 102 to the two terminals, so that the log data acquisition end and the log data receiving end can respectively obtain specific communication information of the data sending end and the data receiving end, and establish a data transmission channel. Then, the log data collection end 101 may send the target log data to the log data receiving end 102 through the established target data transmission channel.
Corresponding to the log data processing method in fig. 1(b), please refer to fig. 2, which shows a schematic flowchart of an embodiment of the log data processing method according to the present application, and the method of the embodiment may include:
s201, the server sends the log data transmission request and a second communication address of a log data receiving end to a log data acquisition end;
s202, the server sends the first communication address of the log data acquisition end to a log data receiving end;
s203, the log data acquisition end obtains target log data from locally stored offline log data by using the log data transmission request;
s204, the log data acquisition end establishes a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end;
s205, the log data acquisition end sends the target log data to a log data receiving end through a target data transmission channel.
S206, the log data receiving end outputs target log data.
Firstly, the server sends the communication identification information of the log data acquisition end to the log data receiving end, and sends the communication identification information of the log data receiving end to the log data acquisition end, so that the log data acquisition end can know which terminal the log data is sent to, and the log data receiving end can know which terminal the log data sent by, and a point-to-point data transmission channel can be created or matched for the log data acquisition end and the log data receiving end to serve as a target data transmission channel for subsequent target log data transmission. The log data acquisition end can also establish a target data transmission channel by the first communication address of the log data acquisition end and the second communication address of the log data receiving end, wherein the target data transmission channel is a point-to-point communication channel between the log data acquisition end and the log data receiving end, and no third-party equipment or server forwards the log data. Therefore, in the embodiment of the application, the server does not need to forward the log data, and only needs to maintain the communication identification information of the log data receiving end and the log data collecting end, such as the communication address and the like.
The server sends the log transmission request to the log data acquisition end, or the server sends the log transmission request generated based on the monitoring condition of the user of the log data acquisition end. The corresponding process for acquiring the target log data may include:
receiving a log data transmission request sent by a server, wherein the log data transmission request is instruction information generated after the server monitors the characteristic information of a log data acquisition end;
and acquiring target log data from the offline log locally stored at the log data acquisition end by using the log data transmission request.
In this embodiment, the characteristic information obtained when the server monitors the log data acquisition end can represent the information of the log data acquisition end and can represent the information of the specific application run by the log data acquisition end. For example, the running time of the log data collection end may be the user characteristic information of the first application of the log data collection end. The log reporting condition indicates a condition that log data is required to be uploaded, and may be that the time reaches a certain fixed time, or when the user characteristic information indicates that the user is a specific target user. Namely, when the monitored characteristic information meets the log reporting condition, a log data transmission request is generated and sent to the log data acquisition end.
The log data transmission request is analyzed by the log data acquisition end to obtain the characteristic identification of the log data to be uploaded, so that the log data acquisition end can conveniently search and obtain target log data from locally stored offline logs.
It should be noted that the log data acquisition end stores the acquired log data into the local database in real time, and only uploads the log data meeting the conditions, so that on one hand, the data pressure of log data uploading is reduced, and the bandwidth resource is saved; and on the other hand, the maintenance of log data is also facilitated.
In another possible implementation manner of the embodiment of the application, the sending, by the server, the instruction information corresponding to the log data transmission request to the log data acquisition end, and the corresponding acquisition process of the target log data may include:
sending the generated log data transmission request to a server;
and in response to receiving a reporting instruction sent by the server, acquiring target log data from an offline log locally stored at the log data acquisition end, wherein the reporting instruction is a feedback instruction after the server verifies the log data transmission request.
In the embodiment, the log data report request is generated by a log data acquisition end, when the log data acquisition end calls a server interface under a specific condition, whether offline log data needs to be uploaded is inquired, if the server sends an indication instruction, and the indication instruction indicates that offline log data is reported, a log data transmission request is determined, and then target log data, namely partial locally stored offline log data, is obtained based on analysis of the log data transmission request. The specific condition that the log data acquisition end calls the server interface under the specific condition can be that the running time monitored by the log data acquisition end reaches the reporting time, or the abnormal information can be monitored, for example, when a user logs in a certain application repeatedly.
After the log data acquisition end obtains the target log data, the target log data is transmitted based on a target data transmission channel, and the target data transmission channel is a point-to-point communication channel between the log data acquisition end and a log data receiving end. Therefore, the log data is directly transmitted to the log data receiving end without passing through the server when the log data acquisition end uploads the log data. The target data transmission channel may be implemented based on a web page instant messaging technology, or may be implemented through a pre-allocated communication channel, which will be specifically described in the following embodiments of the present application.
After the log data receiving end obtains the target log data, the log data receiving end can output the target log data, so that a user of the log data receiving end can process the log data according to the target log data or deal with faults according to the target log data.
A log data processing method according to an embodiment of the present application will be described below with reference to log data of the browser a. Referring to fig. 3, a diagram of an interface for a snoop log is shown. If the function of the browser which is most frequently used by the new user of the browser A is expected to be obtained by analyzing the log data of the browser A. I.e. actually wishes to obtain log data for a new user of browser a. Therefore, the log data of the browser a can be stored offline according to the user identifications of different users. The log data collected by the log data collection end is stored in the local target database in real time, and when the offline log data is obtained, the target database can adopt an index database. The application scenario of this embodiment may be used as an index of the index data according to the identification information of the user.
Correspondingly, in response to the log data transmission request, obtaining the target log data includes:
analyzing the log data transmission request to obtain identification information corresponding to the log data to be uploaded;
and utilizing the identification information to perform index query in the index database to obtain target log data.
Referring to the interface diagram of fig. 3, if the identification information of the new user of the browser a is ID12345, an input dialog box may be shown by selecting the button "set offline log automatic pull", and then "ID 12345" may be input in the input dialog box, so as to implement acquisition of the offline log of the user of the identification information. That is, when the user logs in the browser a, the log data acquisition end sends the offline log of the user to the log data receiving end.
Referring to fig. 4, an offline log viewing interface at the log data receiving end is shown. By selecting the corresponding function plate in the interface, the user at the receiving end of the log data can view the log data. The off-line log reported by the log data acquisition end can be checked through a functional plate of 'project off-line log'. As shown in fig. 4, the offline log data may be displayed item by item according to the reporting time, and certainly may also be displayed item by item according to the reporting sequence of the log data, and the display form of the offline log data is not limited in the embodiment of the present application.
Of course, the log data receiving end may also check a real-time log and a historical log, where the real-time log is a log that needs to be reported to the log data acquisition end in real time in a project, and the difference between the real-time log and the offline log is that the log data acquisition end may upload the log data to the log data receiving end through a server or directly after acquiring the log data. The history log refers to log data which has been received by the log data receiving end, and management and checking of the log data are facilitated.
Referring to fig. 5, a schematic diagram of a process of processing log data by a log data collection end according to an embodiment of the present application is shown. The log data acquisition end is the user terminal 50, and stores the log data generated in real time into the local index database 501 of the user terminal, and after obtaining the log data transmission request, uploads the target log data in the index database 501, that is, part of the offline log data, to the log data receiving end 51. When a log data transmission request is received, analyzing the log data transmission request to obtain identification information corresponding to the log data to be uploaded; and performing log data query in the index database by using the identification information to obtain target log data. Namely, the identification information is used as an index in the index database for inquiring.
When log data of the browser is processed, an index database (indexedDB) is adopted, which is a database for storing structured data in the browser, supports indexes, and completes query operation through pointers generated by the indexes. When indexedDB is used as a storage side of a front-end offline log system, the following advantages are achieved: key value pair storage based on the JavaScript object is simple and easy to use; the storage space is large; an asynchronous API (Application Programming Interface) is provided, so that the access of the database can not block a calling thread; supporting the transaction; the compatibility is better. It should be noted that indexedDB in the embodiment of the present application is only an example of a database for storing offline log data, and other databases may also be used for different application scenarios, for example, a network disk database may be used at an applet end.
Also for the browser application, WebRTC (Web Real-Time Communication, Web instant messaging) may be used as a transmission mode of the offline log. It is an API supporting a web browser to perform real-time voice dialog or video dialog, which has the following advantages; the method is simple and efficient, and the operation and transmission of streaming data can be completed based on a JavaScript interface provided by a browser; point-to-point communication is adopted, the load of the server is reduced, the server only needs to store ip and ports of two communication parties and which users need to transmit the content monitoring of the offline log, and huge offline log source files do not need to be stored, so that the performance pressure and bandwidth pressure of the service can be effectively reduced; based on the browser and supporting cross-platform, the log data acquisition end can treat the user indiscriminately and can acquire the log only by the unique identification of the user.
In order to ensure the safety of log data reporting, the uploaded log data can be checked, for example, token checking. And the log data acquisition end can also be verified, and the log data can be uploaded according to the verification result. Correspondingly, the target log data are sent to a log data acquisition end, and the method comprises the following steps:
verifying the authority information of the log data receiving end to obtain a verification result;
and processing the target log data by using the verification result, and sending the processed target log data to a log data receiving end through a target data transmission channel.
The authority information of the log data receiving end is verified, not only whether the log data receiving end has the receiving authority is verified, but also different processing can be carried out according to the level of the authority verification result, for example, if the authority information of the log data receiving end is verified, the log data receiving end has risk, the target log data can be encrypted and then sent. If the authority information represents the data receiving capacity of the log data receiving end, the target log data can be compressed and then transmitted according to the data receiving capacity of the log data receiving end.
The embodiment of the application also provides a method for verifying the feedback information so as to ensure that the target log data is successfully sent. Referring to fig. 6, a flow chart of a feedback information verification method is shown. The method comprises the following steps:
s601, receiving feedback information of a receiving end of log data aiming at target log data;
s602, judging whether the receiving feedback information meets the receiving failure condition, if so, executing S603;
s603, according to the network state of the target data transmission channel, determining a resending opportunity to resend the target data.
If the sending still fails, the above process is repeated to send again until the target log data receiving end successfully receives the target log data. Generally, the failure of transmission is mainly due to the poor network status, such as weak network or no network. Therefore, the log data acquisition end can judge the network state and send the log data again when the network state is better or meets the reported network condition.
Next, a log data processing method according to the present application will be described with reference to a log system to which indexedDB and WebRTC are applied. The front end refers to a log data acquisition end, indexedDB is used as a storage warehouse, the indexedDB is a high-performance database based on a browser, basic operations on the indexedDB are packaged in the embodiment of the invention, and five interfaces are provided, including: a log clearing interface is adopted, log data are generated continuously, fixed days are set to avoid infinite accumulation of the data, and the expired log data are cleared by checking the log data every time a system is started; the log writing interface allows new logs to be written continuously by an asynchronous log data writing method, namely different log data acquisition ends are allowed to write off-line log data; searching related interfaces including searching logs of a current user, logs in a fixed time period, logs in a fixed grade and the like, so that log information meeting conditions can be conveniently obtained; the data sorting and compressing interface is used for sorting and compressing the data due to the large data volume of the log, so that the size of the reported data can be effectively reduced; and the data reporting interface is used for reporting the target log data.
The reporting process for service end encapsulation comprises the following steps: detecting whether a database already exists, initializing indexedDB if the database does not exist, opening the database, creating the database and a table, and creating an index; detecting whether an expired log exists or not, and if so, cleaning the expired log; providing an interface for writing in the log for calling a service end, and continuously writing the offline log into the service end; requesting server data, detecting whether reporting is needed, if so, retrieving the data, packing and compressing the data, and transmitting the data.
The server side of the log system in the embodiment of the application has the following functions: maintaining the state whether a user needs to upload an offline log or not; the method comprises the steps that a signaling message server is set up as a WebRTC server, and addresses of a log receiving end and a report end are maintained; providing an interface for receiving an offline log for a browser or a terminal which does not support WebRTC; and providing an interface for offline log viewing for the log management platform.
In order to complete the above functions, the server end designs 4 main interfaces:
the monitoring interface reports the offline logs of all users due to the characteristics of the offline logs, so that some monitoring conditions are set at the server end, and only the users meeting the conditions report the offline logs;
an interface for detecting whether to report the offline log is used for informing the client whether to send a request to the server;
the off-line log transmission interface informs the client of a WebRTC transmission address or reports the WebRTC transmission address to an interface of the server, and the part of interface is provided for being compatible with a browser or a terminal which does not support the WebRTC;
and the data display interface is used for returning the offline log of the server side to the front end.
Besides the above interfaces, the server side needs to be compatible with cross-domain requests, decompress data and store files, store addresses of reported users, and store information of management-side users. And also the interface security is considered, because we need to carry out token check on the reported data.
The embodiment of the application provides a front-end offline log system based on IndexDB and WebRTC, which is used for solving the problems of log reporting, error collection, statistical analysis and the like in a front-end project and has wide application scenes in the aspects of front-end quality and data collection. The IndexEDDB is used for collecting offline logs at the front end, counting and reporting through JavaScript. After the log is reported, the log can be selectively stored in a server side, and the log can also be directly reported to a log receiving end through WebRTC, so that the waste of bandwidth and disk resources of the server side is reduced. WebRTC has high-efficient transmission, the characteristics of point-to-point communication, can also use deflate/gzip algorithm to compress in the log transmission process, further improve transmission efficiency, reduce user's network resource request. The method can be applied to a PC web end, a mobile web end, an applet end (enabling an applet storage to replace an IndexDB), a browser, a combination version and other application scenes, and supports various operating systems of windows, linux, macOS, iOS and Android.
In another aspect, referring to fig. 7, a schematic structural diagram of a log data processing apparatus provided in an embodiment of the present application is shown, where the apparatus includes:
a data obtaining unit 701, configured to obtain target log data in response to a log data transmission request, where the target log data includes at least part of offline log data locally stored by a log data acquisition end;
a channel creating unit 702, configured to create a target data transmission channel from a first communication address of the log data acquisition end and a second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel between the log data acquisition end and the log data receiving end;
a data sending unit 703, configured to send the target log data to the log data acquisition end through the target data transmission channel.
In one possible implementation manner, the data obtaining unit includes:
the request receiving subunit is used for receiving a log data transmission request sent by a server, wherein the log data transmission request is instruction information generated after the server monitors the characteristic information of the log data acquisition terminal;
and the log data acquisition subunit is used for acquiring target log data from the offline log locally stored by the log data acquisition end by using the log data transmission request.
Optionally, the data acquiring unit further includes:
a request sending subunit, configured to send the generated log data transmission request to the server;
and the log obtaining subunit is configured to, in response to receiving a reporting instruction sent by the server, obtain target log data from an offline log locally stored at the log data acquisition end, where the reporting instruction is a feedback instruction after the server verifies the log data transmission request.
Optionally, the data obtaining unit is further configured to:
analyzing the log data transmission request to obtain identification information corresponding to the log data to be uploaded;
and performing log data query in the index database by using the identification information to obtain target log data.
In another possible implementation manner, the data sending unit is specifically configured to:
verifying the authority information of the log data receiving end to obtain a verification result;
and processing the target log data by using the verification result, and sending the processed target log data to a log data receiving end through a target data transmission channel.
Optionally, the apparatus further includes a feedback information verification unit, where the feedback information verification unit is specifically configured to:
receiving feedback information of the log data receiving end aiming at the target log data;
and if the receiving feedback information meets the receiving failure condition, retransmitting the target log data according to the network state of the target data transmission channel until the target log data receiving end successfully receives the target log data.
Optionally, the apparatus further comprises a log management unit, wherein the log management unit is configured to:
managing the offline log data in the local index database to enable the local index database to meet the storage requirement of the offline log data in real time, wherein the management of the offline log data comprises one or more of clearing expired offline log data, asynchronously writing the offline log data, inquiring the offline log data or compressing the offline log data.
On the other hand, the present application also provides an electronic device, as shown in fig. 8, which shows a schematic structural diagram of the electronic device of the present application, and the electronic device 800 of this embodiment may include: a processor 801 and a memory 802.
Optionally, the electronic device may further comprise a communication interface 803, an input unit 804 and a display 805 and a communication bus 806.
The processor 801, the memory 802, the communication interface 803, the input unit 804 and the display 805 all communicate with each other via a communication bus 806.
In the embodiment of the present application, the processor 801 may be a Central Processing Unit (CPU), an application specific integrated circuit, a digital signal processor, an off-the-shelf programmable gate array, or other programmable logic device.
The processor may call a program stored in the memory 802. The memory 1102 is used for storing one or more programs, which may include program codes including computer operation instructions, and in this embodiment, the memory stores at least the programs for implementing the following functions:
responding to a log data transmission request, and acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by a log data acquisition end;
creating a target data transmission channel by using a first communication address of the log data acquisition end and a second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel between the log data acquisition end and the log data receiving end;
and sending the target log data to the log data acquisition terminal through the target data transmission channel.
In one possible implementation, the memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as log clearing, log query function, etc.), and the like; the storage data area may store data created according to the use of the computer, such as receiving end identification information, offline log data, and the like.
Further, the memory 802 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device or other volatile solid state storage device.
The communication interface 803 may be an interface of a communication module, such as an interface of a GSM module.
The present application may also include a display 804 and an input unit 805, and the like.
Of course, the structure of the electronic device shown in fig. 8 does not constitute a limitation of the terminal in the embodiment of the present application, and in practical applications, the electronic device may include more or less components than those shown in fig. 8, or some components may be combined.
On the other hand, an embodiment of the present application further provides a storage medium, where computer-executable instructions are stored in the storage medium, and when the computer-executable instructions are loaded and executed by a processor, the log data processing method in any one of the above embodiments is implemented.
It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the device-like embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that it is obvious to those skilled in the art that various modifications and improvements can be made without departing from the principle of the present invention, and these modifications and improvements should also be considered as the protection scope of the present invention.
Claims (10)
1. A log data processing method, comprising:
responding to a log data transmission request, and acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by a log data acquisition end;
creating a target data transmission channel by using a first communication address of the log data acquisition end and a second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel between the log data acquisition end and the log data receiving end;
and sending the target log data to the log data acquisition terminal through the target data transmission channel.
2. The method of claim 1, wherein obtaining target log data in response to a log data transfer request comprises:
receiving a log data transmission request sent by a server, wherein the log data transmission request is instruction information generated after the server monitors the characteristic information of a log data acquisition end;
and acquiring target log data from the offline log locally stored at the log data acquisition end by using the log data transmission request.
3. The method of claim 1, wherein obtaining target log data in response to a log data transfer request comprises:
sending the generated log data transmission request to a server;
and in response to receiving a reporting instruction sent by the server, acquiring target log data from an offline log locally stored at the log data acquisition end, wherein the reporting instruction is a feedback instruction after the server verifies the log data transmission request.
4. The method of claim 1, wherein the log data collection end stores the offline log in a local index database, and wherein the obtaining the target log data in response to the log data transmission request comprises:
analyzing the log data transmission request to obtain identification information corresponding to the log data to be uploaded;
and performing log data query in the index database by using the identification information to obtain target log data.
5. The method of claim 1, wherein the sending the target log data to the log data collection end via the target data transmission channel comprises:
verifying the authority information of the log data receiving end to obtain a verification result;
and processing the target log data by using the verification result, and sending the processed target log data to a log data receiving end through a target data transmission channel.
6. The method of claim 5, further comprising:
receiving feedback information of the log data receiving end aiming at the target log data;
and if the receiving feedback information meets the receiving failure condition, retransmitting the target log data according to the network state of the target data transmission channel until the target log data receiving end successfully receives the target log data.
7. The method of claim 4, further comprising:
managing the offline log data in the local index database to enable the local index database to meet the storage requirement of the offline log data in real time, wherein the management of the offline log data comprises one or more of clearing expired offline log data, asynchronously writing the offline log data, inquiring the offline log data or compressing the offline log data.
8. A log data processing apparatus characterized by comprising:
the data acquisition unit is used for responding to the log data transmission request and acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by the log data acquisition end;
a channel creating unit, configured to create a target data transmission channel from a first communication address of the log data acquisition end and a second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel between the log data acquisition end and the log data receiving end;
and the data sending unit is used for sending the target log data to the log data acquisition end through the target data transmission channel.
9. An electronic device, comprising:
a processor and a memory;
wherein the processor is configured to execute a program stored in the memory;
the memory is to store a program to at least:
responding to a log data transmission request, and acquiring target log data, wherein the target log data comprises at least part of offline log data locally stored by a log data acquisition end;
creating a target data transmission channel by using a first communication address of the log data acquisition end and a second communication address of the log data receiving end, so that the target data transmission channel is a point-to-point communication channel between the log data acquisition end and the log data receiving end;
and sending the target log data to the log data acquisition terminal through the target data transmission channel.
10. A storage medium having stored therein computer-executable instructions that, when loaded and executed by a processor, carry out a method of processing log data according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910900896.1A CN111177094B (en) | 2019-09-23 | 2019-09-23 | Log data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910900896.1A CN111177094B (en) | 2019-09-23 | 2019-09-23 | Log data processing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111177094A true CN111177094A (en) | 2020-05-19 |
| CN111177094B CN111177094B (en) | 2024-08-06 |
Family
ID=70622389
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910900896.1A Active CN111177094B (en) | 2019-09-23 | 2019-09-23 | Log data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111177094B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111953551A (en) * | 2020-08-27 | 2020-11-17 | 网易（杭州）网络有限公司 | Log data transmission method and device, electronic equipment and storage medium |
| CN112015707A (en) * | 2020-08-26 | 2020-12-01 | 长春创世麒麟科技有限公司 | Log data collection method and log data collection system |
| CN114116731A (en) * | 2022-01-24 | 2022-03-01 | 北京智象信息技术有限公司 | Data separation storage display method and device based on indexedDB storage |
| CN114500503A (en) * | 2022-01-14 | 2022-05-13 | 广东好太太智能家居有限公司 | Intelligent device offline log uploading method, intelligent device and readable medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102891873A (en) * | 2011-07-21 | 2013-01-23 | 腾讯科技（深圳）有限公司 | Method for storing log data and log data storage system |
| CN107273266A (en) * | 2017-06-09 | 2017-10-20 | 上海艾融软件股份有限公司 | A kind of processing method and processing device of application daily record |
| CN108365971A (en) * | 2018-01-10 | 2018-08-03 | 深圳市金立通信设备有限公司 | Daily record analytic method, equipment and computer-readable medium |
-
2019
- 2019-09-23 CN CN201910900896.1A patent/CN111177094B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102891873A (en) * | 2011-07-21 | 2013-01-23 | 腾讯科技（深圳）有限公司 | Method for storing log data and log data storage system |
| CN107273266A (en) * | 2017-06-09 | 2017-10-20 | 上海艾融软件股份有限公司 | A kind of processing method and processing device of application daily record |
| CN108365971A (en) * | 2018-01-10 | 2018-08-03 | 深圳市金立通信设备有限公司 | Daily record analytic method, equipment and computer-readable medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112015707A (en) * | 2020-08-26 | 2020-12-01 | 长春创世麒麟科技有限公司 | Log data collection method and log data collection system |
| CN111953551A (en) * | 2020-08-27 | 2020-11-17 | 网易（杭州）网络有限公司 | Log data transmission method and device, electronic equipment and storage medium |
| CN114500503A (en) * | 2022-01-14 | 2022-05-13 | 广东好太太智能家居有限公司 | Intelligent device offline log uploading method, intelligent device and readable medium |
| CN114116731A (en) * | 2022-01-24 | 2022-03-01 | 北京智象信息技术有限公司 | Data separation storage display method and device based on indexedDB storage |
| CN114116731B (en) * | 2022-01-24 | 2022-04-22 | 北京智象信息技术有限公司 | Data separation storage display method and device based on indexedDB storage |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111177094B (en) | 2024-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111177094B (en) | Log data processing method and device, electronic equipment and storage medium | |
| CN107370806B (en) | HTTP status code monitoring method, device, storage medium and electronic equipment | |
| CN108900374B (en) | Data processing method and device applied to DPI equipment | |
| CN111176941B (en) | Data processing method, device and storage medium | |
| CN111367760B (en) | Log collection method and device, computer equipment and storage medium | |
| CN106101264B (en) | Content distributing network log method for pushing, device and system | |
| CN112765103B (en) | File analysis method, system, device and equipment | |
| CN111030888B (en) | Domain name system DNS capacity measuring method, device, equipment and medium | |
| CN110162512A (en) | A kind of log searching method, apparatus and storage medium | |
| CN112395156A (en) | Fault warning method and device, storage medium and electronic equipment | |
| CN112835792B (en) | Pressure testing system and method | |
| CN113472858B (en) | Buried point data processing method and device and electronic equipment | |
| WO2023168926A1 (en) | Method and apparatus for determining software anomaly, storage medium, and electronic apparatus | |
| CN109409948B (en) | Transaction abnormity detection method, device, equipment and computer readable storage medium | |
| JP6200376B2 (en) | In-vehicle information system and information processing method thereof | |
| CN112416708A (en) | Asynchronous call link monitoring method and system | |
| CN108228432A (en) | A kind of distributed link tracking, analysis method and server, global scheduler | |
| CN113055493B (en) | Data packet processing method, device, system, scheduling device and storage medium | |
| CN110309028B (en) | Monitoring information acquisition method, service monitoring method, device and system | |
| CN112181929A (en) | Cloud management platform log processing method and device, electronic device and storage medium | |
| CN114610689B (en) | Recording and analyzing method for request log in distributed environment | |
| CN109684279B (en) | Data processing method and system | |
| CN112131198B (en) | Log analysis method and device and electronic equipment | |
| CN114201659A (en) | Message track transmission query method, device and system | |
| CN113704203A (en) | Log file processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |