WO2019242148A1 - 日志处理方法、装置及存储介质、服务器 - Google Patents

日志处理方法、装置及存储介质、服务器 Download PDF

Info

Publication number
WO2019242148A1
WO2019242148A1 PCT/CN2018/108058 CN2018108058W WO2019242148A1 WO 2019242148 A1 WO2019242148 A1 WO 2019242148A1 CN 2018108058 W CN2018108058 W CN 2018108058W WO 2019242148 A1 WO2019242148 A1 WO 2019242148A1
Authority
WO
WIPO (PCT)
Prior art keywords
log
target traffic
server
traffic log
network storage
Prior art date
Application number
PCT/CN2018/108058
Other languages
English (en)
French (fr)
Inventor
韩琪
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2019242148A1 publication Critical patent/WO2019242148A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G06F11/3476—Data logging
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Definitions

  • the present application relates to the technical fields of traffic, computer network computing, data analysis, retrieval, storage, and backup, and in particular, to a log processing method, device, and computer-readable storage medium and server.
  • a log processing method provided in an embodiment of the present application includes: backing up a target traffic log to a corresponding network storage, where the network storage is assigned a network protocol address; and a server to analyze the target traffic log Link to the network protocol address, so that the server analyzes the target traffic log; compresses the analyzed target traffic log to obtain a compressed log, and compresses the compressed log according to an archive path specified by the network storage For storage.
  • An embodiment of the present application further provides a log processing device, including: a backup module for backing up a target traffic log to a corresponding network storage, where the network storage is assigned a network protocol address; a link module for: Linking a server for analyzing the target traffic log to the network protocol address, so that the server analyzes the target traffic log; a compression module, configured to compress the analyzed target traffic log to obtain The compressed log is stored according to an archive path specified by the network storage.
  • An embodiment of the present application further provides a non-volatile computer-readable storage medium.
  • a computer program is stored on the non-volatile computer-readable storage medium, and a log processing method is implemented when the program is executed by a processor.
  • the log processing method includes the steps of: backing up a target traffic log to a corresponding network storage, the network storage being assigned a network protocol address; and linking a server for analyzing the target traffic log to the A network protocol address, so that the server analyzes the target traffic log; compresses the analyzed target traffic log to obtain a compressed log, and stores the compressed log according to an archive path specified by the network storage.
  • An embodiment of the present application further provides a server including: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be provided by The one or more processors execute, the one or more application programs are configured to execute a log processing method, and the log processing method includes the following steps: backing up a target traffic log to a corresponding network storage, The network storage is assigned a network protocol address; a server for analyzing the target traffic log is linked to the network protocol address, so that the server analyzes the target traffic log; The target traffic log is compressed to obtain a compressed log, and the compressed log is stored according to an archive path specified by the network storage.
  • a log processing method provided in the embodiment of the present application The log is copied and analyzed through a private network to ensure timeliness and reduce log analysis. Excessive occupation analysis and compression both occupy CPU and other resources. Network storage, and the path of the network storage where the server that analyzes the log is backed up, so that the server that stores the log storage and the server that analyzes the log can correspond one-to-one. Analyze, implement distributed analysis and compression, and solve the performance bottleneck of analysis.
  • FIG. 1 is a schematic flowchart of an implementation manner in a typical example of a log processing method of this application;
  • FIG. 2 is a schematic flowchart of another implementation manner in a typical embodiment of a log processing method of this application;
  • FIG. 3 is an example of a log record in a typical embodiment of a log processing method of this application.
  • 5 is a database table structure of key information in a traffic log according to an embodiment of the present application.
  • FIG. 6 is a schematic flowchart of another implementation manner in a typical embodiment of a log processing method of this application.
  • FIG. 7 is a schematic structural diagram of a typical embodiment of a log processing device of this application.
  • FIG. 8 is a schematic structural diagram of an embodiment of a server of the present application.
  • a log processing method provided in the embodiment of the present application is mainly applied in the genesys environment.
  • the log is preprocessed to make abnormal traffic analysis or query traffic logic change. It is simpler, reduces the resources occupied during log analysis, and speeds up traffic analysis.
  • the embodiment of the present application provides a log processing method.
  • the method includes S100 to S300.
  • S100 Back up the target traffic log to a corresponding network storage, where the network storage is assigned a network protocol address.
  • the embodiment of the present application is mainly directed to the analysis of the log generation logic and the log content of the same product.
  • the main log is copied to the network storage (NAS) through a replication program in advance.
  • NAS Network Attached Storage
  • Each product has at least one Corresponding NAS, so you can store some logs in the designated NAS by setting.
  • NAS Network Attached Storage
  • network storage implements data transmission based on standard network protocols, and provides file sharing and data backup for computers with various operating systems such as Windows / Linux / Mac OS on the network. It is also a mechanism that uses special equipment directly connected to the network medium to implement data storage.
  • the network storage is assigned a network protocol address, that is, the device is assigned an IP address, so the client can access it through a server acting as a data gateway. Access, and in some cases, these devices can be accessed directly by clients without any intermediate media. Specifically, before the target traffic log is backed up to the NAS, other steps are included, as follows:
  • the method before the target traffic log is backed up to a corresponding network storage, the method further includes:
  • S101 Determine whether the current log is the target traffic log that needs to be backed up according to preset key information and preset matching rules;
  • the target traffic log is the log that needs to be analyzed. Because the log includes the main log and the backup log, when determining the target traffic log, determine whether the current log is the target that needs to be analyzed through preset key information and preset matching rules.
  • the traffic log for example, when the target traffic log is the main log, by analyzing the current log, it can be determined that if the log is currently the main, the tHA or sipserver log contains the keyword 'HARole: Primary', that is, 'HA Role: Primary 'is the key message. Based on this key information, the log can be determined as the main log.
  • the server that backs up the log corresponds to the main log and the backup log
  • the log is generated.
  • Role changed primary / backup
  • if the currently backed up server is not the main server corresponding to the main log determine to switch the currently backed up server to the main server corresponding to the main log; if the currently backed up server is
  • the master server corresponds to the master log, no switchover is performed.
  • the target traffic log is a standby log, the processing is the same, and details are not described here.
  • the above switching keyword does not appear in the routing service log. Instead, it is determined whether it is the main log according to the regular match request ⁇ d + and does not include the RequestRegisterAddress keyword.
  • the preset matching rule is a regular matching rule.
  • Regular matching rules are mainly matched by regular expressions.
  • Regular expressions are a matching tool used to manipulate and verify string data. It is a special string of characters that can be used to perform operations such as text matching.
  • Regular expression matching The specific syntax can be found at the URL: http: //www.regular-expressions.infoj. For example, for the log record of FIG. 3, the following regular expression can be constructed for matching.
  • Pl ["Processing ⁇ s + ( ⁇ w +) # ( ⁇ w +) ⁇ s ⁇ (for ⁇ s + (( ⁇ d + ⁇ .) ⁇ 3 ⁇ ⁇ d +) ⁇ s + at ⁇ s + ( ⁇ d +- ⁇ d +- ⁇ d + ⁇ s ⁇ d +: ⁇ d +: ⁇ d +) ⁇ ) ⁇ s + ⁇ [( ⁇ w +) ⁇ ] ⁇ n + (Parameters: ⁇ . +) ", controller, method, client_P, timestamp, http_method, content].
  • the data matched by the regular expression will be used as the data of the corresponding field position in the log profile.
  • the value of the controller field corresponds to " ⁇ w +”
  • the value of the method field corresponds to " ⁇ w +”
  • the value of the client_IP field corresponds to "( ⁇ d + ⁇ .) ⁇ 3 ⁇ ⁇ d +”
  • the value of the timestamp field corresponds to " ⁇ d +- ⁇ d +- ⁇ d + ⁇ s ⁇ d +: ⁇ d +: ⁇ d +”
  • the value of the http_method field corresponds to " ⁇ w +”
  • the value of the content field corresponds to "Parameters: ⁇ . + ".
  • the log records can be matched, and the log data in the log records can be parsed and extracted. Therefore, regular expressions can parse and extract dynamic data from log content.
  • the preset matching rule may also be implemented by using a string template, which is a template matching engine that supports languages such as Java, C #, and Python.
  • string template is a template matching engine that supports languages such as Java, C #, and Python.
  • For the matching syntax of string templates see the URL: http://www.stringtemplate.org/.
  • the matching of the string template is different from the regular expression, but both can achieve the effect of extracting the buzz records, that is, determine whether the target traffic log is in a large number of logs.
  • S200 Link a server for analyzing the target traffic log to the network protocol address, so that the server analyzes the target traffic log.
  • the target traffic log is backed up (copied) to the network storage through the server, so that the server analyzing the target traffic log can correctly find the target traffic Traffic log, linking the server that analyzes the target traffic log to the network protocol address, that is, the server that analyzes the target traffic log hangs the same network storage path as the network storage that stores the target traffic log.
  • the server that analyzes the log can determine the location of the target traffic log through this path, and analyze the log in the network storage according to the path. Specifically, the server that analyzes the log is linked with the NAS path.
  • the server that analyzes the log directly calls the target traffic log from the NAS, or the server that analyzes the log directly analyzes the target traffic log in the NAS, and then Facilitate analysis of target traffic logs.
  • the target traffic log is copied to the network storage through its corresponding private network, the target traffic log is analyzed to ensure the timeliness of the log analysis and avoid the log analysis occupying the production system. CPU and other resources.
  • S300 Compress the analyzed target traffic log to obtain a compressed log, and store the compressed log according to an archive path specified by the network storage.
  • the method further includes:
  • the analyzed logs are compressed and backed up by gzip and other methods, and then the logs that have been backed up to the NAS and have been analyzed are deleted. Since the analyzed logs have been compressed, the original data in the NAS can be compressed. After the analysis, the target traffic log is deleted. Compressing the log can reduce the log occupying the storage space of the database or the index structure. The log after the original analysis corresponding to the compressed log is deleted, further reducing the log occupying the storage space of the database and avoiding insufficient storage space. The speed of querying logs.
  • the compressed log is stored in accordance with the archive path specified by the network storage.
  • the server for analyzing the target traffic log when the server for analyzing the target traffic log is linked to the network protocol address, so that the server can After analyzing the transaction log, it also includes:
  • S210 Extract key fields in the target traffic log after analysis, and store the key segment in a database in an index manner.
  • index structure index database
  • key fields in the analysis log are extracted
  • key fields in the log are structured and stored in the database (index structure), and an association relationship is established.
  • the key fields in the log are used to build the index structure (see Figure 5 for details).
  • the data of each layer in the index structure is set according to the actual needs of the user.
  • the name can be the data at the top of the index structure, and the number and ID can be used as the top-level next-level data. Of course, the number and / or ID can also be used as the top of the top index structure. Data, with the name as the top-level data.
  • the index database can also be displayed in the form of a table. Key fields are displayed in the table. Each key field is associated with other key fields such as: object, time, event, etc.
  • tbl_files_log is used as the index structure for the entire top of the log-related data table, which is associated with the surrounding data such as tbl_app, tbl_logserver, tbl_ts_log, tbl_rels_log, etc., and each data table has a corresponding one.
  • the header of each table can be used as the top of the index structure, and the corresponding header can be found through tbl_files_log. According to the header, the data corresponding to the header can be obtained.
  • tbl_app records the master-backup relationship of the application and the corresponding relationship between the application and the server of tbl_logserver.
  • tbl_files_log records the name of the log file, the actual time point generated by the log and the type of log application, and is associated with the application of tbl_app.
  • tbl_ts_log generates an index of key information such as traffic time, event name, and node by scanning the tserver log, and associates it with the file id of tbl_files_log.
  • tbl_rels_log scans the on-hook index generated by the tserver log and associates it with the file id of tbl_files_log.
  • tbl_uuid_log scans the relationship between uuid and connid generated by the tserver log, and associates the file id of tbl_files_log.
  • tbl_connid_log scans the tserver log for the connid relationship before and after the transfer, and associates it with the file id of tbl_files_log.
  • tbl_sip_log scans the sip time, event name, ip generated by the sipserver log, and associates the file id of tbl_files_log.
  • tbl_callid_log scans the callid generated by the sipserver log and the uuid relationship, and associates the file id of tbl_files_log.
  • tbl_urs_log scans the time, event name, node generated by the urs log, and associates the file id of tbl_files_log.
  • tbl_mcp_log scans the time and content of the mcp log and associates the file id of tbl_files_log.
  • tbl_mcpsession_log scans the uuid and sessionid relationship generated by the mcp log, and associates the file id of tbl_files_log.
  • the key field is stored in a database in an index manner.
  • the content generated by querying the page for a period of time (the user queries the traffic query page for a period of time) will be based on the keywords in the above index structure Segment, showing the table related to the tbl_files_log file. Clicking on a specific traffic will show the two-dimensional actual table of the traffic by time and object. After clicking on the specific event, the remote rpc interface will be called through the file name and file pointer and Byte, get the log content, the actual interface reads the compressed log from nas through 3 parameters (source_dir, dist_path, fn2) and returns the content.
  • the key information entered and key fields in the index structure can be matched.
  • the corresponding traffic logic can be found, or the position corresponding to the traffic logic can be determined. Such as time of occurrence, event name, node, etc.
  • the method further includes:
  • S310 Construct a log query interface corresponding to the web query interface and capable of obtaining the compressed log from the network storage.
  • a log query interface corresponding to the web query interface, so that users can query the corresponding traffic logic and analyze and locate it through the web interface, so that not only can the tserver log be analyzed, but also the web query log, Get detailed data in compressed logs, making cross-platform transfer easy.
  • the key field of the web query input because it corresponds to the log query interface, enables the content entered by the web query interface to obtain detailed data of the corresponding log in the corresponding compressed log through the log query interface, that is, to query the traffic Corresponding logic achieves fast analysis and positioning.
  • a program is set for the interface.
  • the compressed file has a specific naming convention, which includes the application name time in the log.
  • the web interface can receive the log file name, file pointer, and byte size. After receiving the interface, the file can be located through the file name specification. .
  • key fields including the log file name of the tserver component, the corresponding application name of the file, the corresponding event, the time of the event, the object, the customer number, the connid of the call, queue information, if it involves transfers and conferences, and cross-platform Transfer connid, calluuid associated with sip log, client request id, file pointer, and byte Sipserver components
  • the key information of sip signaling includes signaling time, event name, direction of interaction with sip, callid , To and from, ip address, file pointer, and log file name of the Urs component, file corresponding application name, corresponding event, event time, object, customer number, call connid client request id, file pointer, and byte.
  • the interface passes in the file name, character string, file pointer, number type, byte, and number. One or more of the types.
  • the output is one or more of log content and string type.
  • One kind of interface is to obtain the compressed file of the original log.
  • the interface is as follows: the interface is passed in as the file name, and the string is passed out as the file stream, which can be kept in the form of a file.
  • the implementation details of the interface use code are as follows:
  • the compressing the analyzed target traffic log to obtain a compressed log further includes:
  • the target traffic log of the previous unit time segment includes the The content of the target traffic log previously preset in the target traffic log, and the content of the target traffic log is marked as the content of time continuity check; the target traffic log of each unit time period is compressed.
  • the content of each package includes the first paragraph of the content of the back bread, for example, a 10-point package, and the last includes 11: 0-5 points.
  • the 10-point package including the contents of 11: 00-5 points, can be collated with the 11-point package, so as to determine that the 10-point package is before 11, and then re-sort by time.
  • a log processing device is also provided in the embodiment of the present application. In one embodiment, as shown in FIG. 7, it includes:
  • a backup module 100 configured to back up a target traffic log to a corresponding network storage, where the network storage is assigned a network protocol address;
  • a linking module 200 configured to link a server for analyzing the target traffic log to the network protocol address, so that the server analyzes the target traffic log;
  • the compression module 300 is configured to compress the analyzed target traffic log to obtain a compressed log, and store the compressed log according to an archive path specified by the network storage.
  • a log processing device provided in an embodiment of the present application further includes an extraction module 210 for extracting key fields in the target traffic log after analysis, and Key fields are stored into the database by indexing.
  • the deleting module 310 is configured to delete the analyzed target traffic log in the network storage.
  • the construction module 320 is configured to construct a log query interface corresponding to a web query interface and capable of obtaining the compressed log from the network storage.
  • a judging module 101 is configured to determine whether a current log is the target traffic log that needs to be backed up according to preset key information and a preset matching rule; a switching module 102 is configured to determine that the current log is a destination that needs to be backed up.
  • the server backing up the target traffic log is switched to a server corresponding to the target traffic log.
  • the preset matching rule is a regular matching rule; the dividing unit 301 divides the target traffic log into target traffic logs in a unit time period according to time; among the target traffic logs in two adjacent unit time periods, The target traffic log of the previous unit time period includes the content of the target traffic log previously preset in the target traffic log of the next unit time period, and the content of the target traffic log is used as the time continuity check content ; Respectively compress the target traffic log of the unit time period.
  • a log processing apparatus provided in the embodiment of the present application can implement the foregoing log processing method embodiment.
  • a computer-readable storage medium provided in the embodiments of the present application.
  • a computer program is stored on the computer-readable storage medium, and when the program is executed by a processor, the log processing method according to any one of the technical solutions is implemented.
  • the computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory, read-only memory), and RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Programmable Read-Only Memory), flash memory, magnetic card or Rays card. That is, the storage device includes any medium that stores or transmits information in a readable form by a device (for example, a computer, a mobile phone), and may be a read-only memory, a magnetic disk, or an optical disk.
  • a computer-readable storage medium provided in the embodiments of the present application can intelligently implement the embodiment of the above-mentioned log processing method.
  • the logs are copied and analyzed through a private network to ensure timeliness and reduce the excessive occupation of log analysis. Both analysis and compression occupy CPU and other resources.
  • the embodiment of this application also provides a log processing method to target traffic
  • the logs are backed up to the corresponding network storage, and the network storage is assigned with a network protocol address; a server for analyzing the target traffic log is linked to the network protocol address, so that the server performs the target traffic Log analysis; compress the target traffic log after analysis to obtain compression Chi, specified according to the network path to the archival storage log storage compression.
  • logs of the same product are copied to the network storage in a timely manner, thereby reducing the log occupying the memory of the terminal product where the app is located, and by attaching the server that analyzes the product log to the storage log
  • the path of the network storage implements that the logs can be analyzed in the network storage, reducing the analysis logs occupying the CPU and other resources of the terminal product where the app is located, compressing the analyzed logs, and deleting the analyzed and compressed backup logs. Reduced the log occupies the memory of network storage, thereby reducing the impact of memory on the speed of analyzing logs.
  • Different products or different types of logs of the same product are stored in the corresponding network storage, and the log analysis and compression backup are performed in the corresponding network storage to achieve distributed analysis and compression of the backup log, thereby solving the direct
  • the bottleneck of analyzing logs at the terminal improves the speed of analyzing logs. Compressing and backing up the logs can reduce the log occupying the memory of the network storage, ensure that the corresponding logs can be found during subsequent traffic queries, and avoid the situation that the traffic query and analysis can no longer be performed after the logs in the application are deleted.
  • the computer-readable storage medium provided in the embodiment of the present application can implement the above-mentioned embodiment of the log processing method.
  • the present application further provides a server.
  • the server processor 503, the memory 505, the input unit 507, and the display unit 509 are components.
  • the memory 505 may be used to store an application program 501 and various functional modules, and the processor 503 runs the application program 501 stored in the memory 505 so as to execute various functional applications and data processing of the device.
  • the memory 505 may be an internal memory or an external memory, or include both an internal memory and an external memory.
  • the internal memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or a random access memory.
  • ROM read-only memory
  • PROM programmable ROM
  • EPROM electrically programmable ROM
  • EEPROM electrically erasable programmable ROM
  • flash memory or a random access memory.
  • External storage can include hard disks, floppy disks, ZIP disks, U disks, magnetic tapes, and so on.
  • the memory disclosed in this application includes, but is not limited to, these types of memory.
  • the memory 505 disclosed in this application is only an example and not a limitation.
  • the input unit 507 is used to receive input of signals, and key fields input by the user and operation instructions of the user to perform query retrieval.
  • the input unit 507 may include a touch panel and other input devices.
  • the touch panel can collect the touch operations of the customer on or near it (such as the operation of the customer on the touch panel or near the touch panel using any suitable object or accessory such as a finger or a stylus), and according to the preset
  • the program drives the corresponding connection device; other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as playback control keys, switch keys, etc.), a trackball, a mouse, and an joystick.
  • the display unit 509 may be used to display information input by the customer or information provided to the customer and various menus of the computer device.
  • the display unit 509 may take the form of a liquid crystal display, an organic light emitting diode, or the like.
  • the processor 503 is the control center of the computer equipment. It uses various interfaces and lines to connect various parts of the entire computer. It runs or executes the software programs and / or modules stored in the memory 503, and calls the data stored in the memory to execute Various functions and processing data.
  • One or more processors 503 shown in FIG. 8 can execute and implement the functions of the backup module 100, the function of the link module 200, the function of the compression module 300, the function of the extraction module 210, and the deletion module 310 shown in FIG. Functions of the building module 320, functions of the judging module 101, functions of the switching module 102, and functions of the dividing unit 301.
  • the server includes one or more processors 503, and one or more memories 505, one or more applications 501, wherein the one or more applications 501 are stored in the memory 505 And is configured to be executed by the one or more processors 503, and the one or more application programs 301 are configured to execute the log processing method described in the above embodiment.
  • a server provided in the embodiment of the present application can intelligently implement the embodiment of the above-mentioned log processing method.
  • the logs are copied out and analyzed through a private network, which ensures timeliness and reduces excessive log analysis.
  • the CPU and other resources are occupied.
  • the log storage server and the log analysis server can correspond one-to-one.
  • the hanging network storage path can realize log analysis, realize distributed analysis and compression, and solve the performance bottleneck of analysis.
  • a log processing method is also provided to back up the target traffic log to the corresponding
  • the network storage is assigned a network protocol address;
  • a server for analyzing the target traffic log is linked to the network protocol address, so that the server analyzes the target traffic log ;
  • Said network storage path specified in the archive stores the compaction log.
  • logs of the same product are copied to the network storage in a timely manner, thereby reducing the log occupying the memory of the terminal product where the app is located, and by attaching the server that analyzes the product log to the storage log
  • the path of the network storage implements that the logs can be analyzed in the network storage, reducing the analysis logs occupying the CPU and other resources of the terminal product where the app is located, compressing the analyzed logs, and deleting the analyzed and compressed backup logs. Reduced the log occupies the memory of network storage, thereby reducing the impact of memory on the speed of analyzing logs.
  • Different products or different types of logs of the same product are stored in the corresponding network storage, and the log analysis and compression backup are performed in the corresponding network storage to achieve distributed analysis and compression of the backup log, thereby solving the direct
  • the bottleneck of analyzing logs at the terminal improves the speed of analyzing logs. Compressing and backing up the logs can reduce the log occupying the memory of the network storage, ensure that the corresponding logs can be found during subsequent traffic queries, and avoid the situation that the traffic query and analysis can no longer be performed after the logs in the application are deleted.
  • the server provided in the embodiment of the present application can implement the embodiment of the log processing method provided above.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

本申请涉及话务、计算机网络计算、数据分析、检索、存储、备份技术领域，具体涉及本申请实施例提供的一种日志处理方法，包括：将目标话务日志备份到对应的网络存储器中，网络存储器被分配有网络协议地址；将用于分析目标话务日志的服务器链接到网络协议地址，以使得所述服务器对目标话务日志进行分析；将分析后的目标话务日志压缩获得压缩日志，依据网络存储器指定的归档路径将压缩日志进行存储。日志通过专有网络复制出去再分析，保证了及时性也降低了日志分析过多的占用分析和压缩都占用cpu等资源，通过将日志压缩备份到网络存储器，且将分析日志的服务器挂备份日志的网络存储器的路径，实现分布式分析及压缩，解决了分析的性能瓶颈。

Description

日志处理方法、装置及存储介质、服务器
本申请要求于2018年6月21日提交中国专利局、申请号为201810642922.0，发明名称为“日志处理方法、装置及存储介质、服务器”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及话务、计算机网络计算、数据分析、检索、存储、备份技术领域，具体涉及一种日志处理方法、装置及计算机可读存储介质、服务器。
背景技术
随着互联网行业的迅猛发展，尤其是移动端应用的异军突起，新一轮的数据爆炸时代的大门己经开启，各类在线应用相互间的竞争口趋自热化。其中，部分日志不仅能够反应服务器的运行状况，另一部分日志更是为业务提供了发展规划的重要指标。特别地，发明人发现在话务系统中，由于需要分析一下异常话路，或查询用户对于话务的查询，只能通过tserver日志进行分析，由于日志结构和web的完全不一样，并且有主备日志之分，并且主备都会输出日志，需要区分主备，分析变得非常困难，另外又涉及跨平台的话务转接，造成话务分析变得困难，另外由于日志会被清理，程序本身没有备份。另外在一个全平台中产生的日志量非常大，单平台的tserver日志每秒日志量在5M左右，分析和压缩都占用cpu较高，单台无法满足，从而导致单平台的tserver日志分析成为话务日志分析和定位的瓶颈。
发明内容
为克服以上技术问题，特别是单平台的tserver日志分析成为话务日志分析和定位的瓶颈的问题，特提出以下技术方案：
本申请实施例提供的一种日志处理方法，包括：将目标话务日志备份到对应的网络存储器中，所述网络存储器被分配有网络协议地址；将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析；将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。
本申请实施例中还提供了一种日志处理装置，包括：备份模块，用于将目标话务日志备份到对应的网络存储器中，所述网络存储器被分配有网络协议地址；链接模块，用于将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析；压缩模块，用于将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。
本申请实施例中还提供了一种非易失性计算机可读存储介质，所述非易失性计算机可读存储介质上存储有计算机程序，该程序被处理器执行时实现一种日志处理方法，所述日志处理方法包括以下步骤：将目标话务日志备份到对应的网络存储器中，所述网络存储器被分配有网络协议地址；将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析；将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。
本申请实施例中还提供了一种服务器包括：一个或多个处理器；存储器；一个或多个应用程序，其中所述一个或多个应用程序被存储在所述存储器中并被配置为由所述一个或多个处理器执行，所述一个或多个应用程序配置用于执行一种日志处理方法，所述日志处理方法包括以下步骤：将目标话务日志备份到对应的网络存储器中，所述网络存储器被分配有网络协议地址；将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析；将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。
本申请实施例提供的一种日志处理方法，日志通过专有网络复制出去 再分析，保证了及时性也降低了日志分析过多的占用分析和压缩都占用cpu等资源，通过将日志压缩备份到网络存储器，且将分析日志的服务器挂备份日志的网络存储器的路径，使得存储日志存储器和分析日志的服务器能够一一对应，在需要分析日志时，通过挂的网络存储器路径便能实现对日志的分析，实现分布式分析及压缩，解决了分析的性能瓶颈。
附图说明
图1为本申请日志处理方法的典型实施例中一种实施方式的流程示意图；
图2为本申请日志处理方法的典型实施例中又一种实施方式的流程示意图；
图3为本申请日志处理方法的典型实施例中日志记录示例；
图4为本申请日志处理方法的典型实施例中又一种实施方式的流程示意图；
图5为本申请实施例中话务日志中关键信息的数据库表结构；
图6为本申请日志处理方法的典型实施例中又一种实施方式的流程示意图；
图7为本申请日志处理装置的典型实施例的结构示意图；
图8为本申请服务器的一实施例结构示意图。
具体实施方式
本申请实施例中提供的一种日志处理方法，主要应用于genesys环境中，通过本申请实施例中提供的日志处理方法，通过对日志进行预处理，使得分析异常话务或者查询话务逻辑变得更为简单，降低日志分析过程中占用的资源，加快了话务分析的速度。
本申请实施例提供一种日志处理方法，在其中一种实施方式中，如图1所示，包括S100至S300。
S100：将目标话务日志备份到对应的网络存储器中，所述网络存储器被分配有网络协议地址。
本申请实施例主要针对同一产品的日志产生的逻辑与日志内容的分析。在日志的结构中，分为主日志和备日志，且两者都输出日志，为了降低分析日志的困难，提前将主日志通过复制程序复制到网络存储器(NAS)中，每个产品至少有一个对应的NAS，因此，可通过设置将部分日志存储于指定的NAS中。NAS(Network Attached Storage)网络存储基于标准网络协议实现数据传输，为网络中的Windows/Linux/Mac OS等各种不同操作系统的计算机提供文件共享和数据备份。其还是一种采用直接与网络介质相连的特殊设备实现数据存储的机制，所述网络存储器分配有网络协议地址，即该设备分配有IP地址，所以客户机通过充当数据网关的服务器可以对其进行存取访问，甚至在某些情况下，不需要任何中间介质客户机也可以直接访问这些设备。具体的，在将目标话务日志备份到NAS中之前还包括其他步骤，详情如下：
进一步地，在其中一种实施方式中，如图2所示，在所述将目标话务日志备份到对应的网络存储器中之前，还包括：
S101：依据预设关键信息和预设匹配规则确定当前日志是否为需要被备份的所述目标话务日志；
S102：当确定所述当前日志为需要被备份的所述目标话务日志时，将备份所述目标话务日志的服务器切换为与所述目标话务日志相对应的服务器。
目标话务日志为需要被分析的日志，由于日志包括主日志和备日志，因此在确定目标话务日志时，通过预设关键信息和和预设匹配规则确定当前日志是否为需要被分析的目标话务日志，如在目标话务日志为主日志时，通过分析当前日志，可以确定如果日志当前为主，tserver或sipserver日志里面存着'HA Role:Primary'关键字，也就是说，'HA Role:Primary'为关键信息。通过该关键信息就能够确定该日志为主日志，由于备份日志的服务器分别与主日志和备份日志相对应，在确定当前日志为需要被分析备份的目标话务日志(主日志)时，在产生Role changed to(primary/backup)，若当前备份的服务器不为与主日志相对应的主服务器时，确定将当前备份的服务器切换为与主日志相对应的主服务器；若当前备份的服务器为与主 日志相对应的主服务器时，则不进行切换。当目标话务日志为备日志时，其处理相同，在此不做赘述。另外在路由服务日志里面并不会出现上述切换关键字，而是根据正则匹配request to\d+并且不包括RequestRegisterAddress关键字确定是否是主日志。
优选地，所述预设匹配规则为正则匹配规则。
正则匹配规则主要通过正则表达式进行匹配，正则表达式是一种用来操作和检验字符串数据的匹配工具，其为一串特殊的字符，可以对文本进行匹配等操作，正则表达式的匹配语法可以具体参见网址:http://www.regular-expressions.infoj。例如，针对图3的日志记录，可以构建如下的正则表达式进行匹配。
Pl＝["Processing\s+(\w+)#(\w+)\s\(for\s+((\d+\.){3}\d+)\s+at\s+(\d+-\d+-\d+\s\d+:\d+:\d+)\)\s+\[(\w+)\]\n+(Parameters:\.+)”，controller,method,client_P,timestamp,http_method,content]。
由正则表达式匹配出的数据将作为日志简档中相应的字段位置的数据。在上面的正则表达式示例中，controller字段的值对应于“\w+",method字段的值对应于"\w+",client_IP字段的值对应于“(\d+\.){3}\d+",timestamp字段的值对应于“\d+-\d+-\d+\s\d+:\d+:\d+",http_method字段的值对应于“\w+”，content字段的值对应于"Parameters:\.+”。通过正则表达式的匹配规则，可以日志记录进行匹配，从而能够解析并抽取出日志记录中的日志数据。因此，正则表达式可以能够解析并抽取出日志内容中的动态数据。优选地，在其他的实施方式中，在日志为结构化数据时，所述预设匹配规则还可以通过字符串模板实现，字符串模板是一种模板匹配引擎，支持java,C#,Python等语言，字符串模板的匹配语法可以参见网址:http://www.stringtemplate.org/。字符串模板的匹配与正则表达式有所不同，然而二者均能达到对口志记录进行抽取的效果，即在大量的日志中确定是否为目标话务日志。
S200：将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析。
在确定目标话务日志以及和目标话务日志相对应的服务器后，通过该 服务器将目标话务日志备份(复制)到网络存储器中，为了使得分析目标话务日志的服务器能够正确的找到目标话务日志，链接分析所述目标话务日志的服务器到所述网络协议地址，即分析所述目标话务日志的服务器挂与存储目标话务日志网络存储器相同的网络存储器路径，在需要分析日志时，分析日志的服务器通过该路径便能够确定目标话务日志所在的位置，并依据该路径对网络存储器里面的日志进行分析。具体的将分析日志的服务器与NAS路径链接起来，在进行日志分析时，分析日志的服务器便直接从NAS中调用目标话务日志，或者分析日志的服务器直接在NAS中分析目标话务日志，进而便于对目标话务日志分析。结合前述的说明，将目标话务日志通过其对应的专有网络复制到网络存储器中后，再对目标话务日志进行分析，保证了日志分析的及时性，同时也避免了日志分析占用生产系统CPU等资源。
S300：将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。
进一步地，在其中一种实施方式中，在所述依据所述网络存储器指定的归档路径将所述压缩日志进行存储之后，还包括：
删除所述网络存储器内的分析后的所述目标话务日志。
在日志分析完成后，将分析后的日志通过gzip等方式压缩备份，然后将备份到NAS中且已被分析过的日志删除，由于已经将分析之后的日志进行了压缩，因此可以将NAS中原有的分析后目标话务日志删除，压缩日志可以降低日志占用数据库或者索引结构的存储空间，将压缩日志对应的原分析之后的日志删除，进一步降低日志占用数据库的存储空间，避免存储空间不够降低后续查询日志的速度。优选地，为了便于后续能够通过接口快速地定位查询到压缩日志，因此，压缩日志按照网络存储器指定的归档路径进行存储。
进一步地，在其中一种实施方式中，如图4所示，在所述将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析之后，还包括：
S210：提取分析后的所述目标话务日志中的关键字段，将所述关键字 段以索引方式存储到数据库中。
在分析NAS中的日志时，构建一个索引结构(索引数据库)，提取分析日志中关键字段，把日志中的关键字段结构化后存入数据库(索引结构)，并建立关联关系。将日志中的关键字段用于构建索引结构(详情如图5)。索引结构中各层的数据以用户实质需要进行设置，如可以名称为索引结构顶端的数据，以号码和ID作为顶端下一层并列的数据；当然也可以号码和/或ID作为顶端索引结构顶端的数据，将名称作为顶端下一层的数据。读取了日志中关键信息，如通话的事件的名称，号码，id，及关联信息等等，并记录了该信息在文件中的指针位置及内容字节数。不同日志顶层的端的数据类型一样。索引数据库还可以表格形式展示，在表格中显示关键字段，每一个关键字段关联有其他关键字段如：对象、时间、事件等。在图5中，以tbl_files_log作为索引结构整个与日志相关的顶端的数据表格，其分别于tbl_app、tbl_logserver、tbl_ts_log、tbl_rels_log等周边的数据相关联，在每一个数据表格中都分别具有与其相对应的数据，每个表格的表头可以作为索引结构中的顶端，通过tbl_files_log查询到对应的表头，依据该表头便可获得与该表头对应的数据。其中，在图5中，tbl_app记录应用的主备关系及该应用与tbl_logserver的服务器的对应关系。这样可以不用区分主备应用到底哪个目前是主的。都可查询。tbl_files_log记录日志文件名称及该日志产生的实际时间点和日志应用类型并关联tbl_app的应用。tbl_ts_log通过扫描tserver日志产生话务时间，事件名称，节点等关键信息的索引，并且关联tbl_files_log的文件id。tbl_rels_log通过扫描tserver日志产生的挂机索引，并且关联tbl_files_log的文件id。tbl_uuid_log通过扫描tserver日志产生的uuid与connid关系，并关联tbl_files_log的文件id。tbl_connid_log通过扫描tserver日志产生的转接前后connid关系，并关联tbl_files_log的文件id。tbl_sip_log通过扫描sipserver日志产生的sip的时间，事件名称，ip，并关联tbl_files_log的文件id。tbl_callid_log通过扫描sipserver日志产生的callid与uuid关系，并关联tbl_files_log的文件id。tbl_urs_log通过扫描urs日志产生的时间，事件名称，节点，且关联tbl_files_log的文件id。tbl_mcp_log通过 扫描mcp日志产生的时间，内容，且关联tbl_files_log的文件id。tbl_mcpsession_log通过扫描mcp日志产生的uuid与sessionid关系，且关联tbl_files_log的文件id。优选地，所述关键字段以索引方式存储在数据库中。
在前述的基础上，通过索引结构关系以及索引结构中记录的关键字段，在页面查询一段时间产生的内容(用户在话务查询页面查询一段时间)，就会按上述索引结构中的关键字段，展示与tbl_files_log文件相关的表格，点击具体的话务就会展示该话务按时间和对象的二维的实际表，点击具体的事件后就会调用远程rpc接口通过文件名字和文件指针和字节，获取日志内容，实际接口通过3个参数(source_dir、dist_path、fn2)从nas中读取压缩日志并返回内容出来。因此通过压缩就能节省90％的空间(100M压缩到10M)，还能够实现快速的查询，后续查询就直接对压缩后的日志进行提取。当然，也可以在查询页面的搜索引擎输入关键字段进行查询。同时由于原有话务系统中genesys环境下的日志无法长期保存，通过本申请将日志在网络存储器里面压缩备份实现了日志长期保持；另外，话务系统中问题上报都有延时性，通过压缩备份为分析过往的问题提供依据。需要说明的是，在本申请提供的实施例中，只是把目标话务日志中的关键内容少量的记录到索引结构中，实际还有很多内容并没记录到索引结构中，其中很大一部分还在目标话务日志文件中。因此，在后续的查询步骤中，便可以通过输入的关键信息和索引结构中的关键字段进行匹配，匹配成功时，便能够找到与其对应的话务逻辑，或者确定话务逻辑对应的位置，如发生的时间、事件名称，节点等。
进一步地，在其中一种实施方式中，如图6所示，在所述依据所述网络存储器指定的归档路径将所述压缩日志进行存储中之后，还包括：
S310：构建与web查询接口相对应且能从所述网络存储器中获取到所述压缩日志的日志查询接口。
开发构建与web查询接口相对应的日志查询接口，以便于用户可以通过web接口查询对应话务逻辑并对其进行分析定位等，使得不仅仅可以通过tserver日志进行分析，也可以通过web查询日志、获取压缩日志中的 详细数据，使得跨平台转接简便。在web查询输入关键字段，由于其与日志查询接口相对应，使得web查询接口输入的内容能够通过该日志查询接口获取到对应的压缩日志里面的相应日志的详细数据，即查询到话务的相应的逻辑，实现快速地达到快速分析定位。具体的，例如为接口设置一段程序，在web查询中个输入关键字段(关键字段为一个或者多个)时，即启动该段或者调用该段程序，通过对应的路径获取压缩包。压缩的文件有具体的命名规范，其中在日志中带了应用名称时间，web接口可以收到日志的文件名及文件指针及字节大小，接口收到后通过文件名规范就能定位到该文件。其中，关键字段、包括tserver组件的日志文件名、文件对应的应用名、对应的事件、事件的时间、对象、客户号码、通话的connid、队列信息、如果涉及转接及会议和跨平台的转接connid、关联到sip日志的calluuid、客户端请求id、文件指针及字节Sipserver组件除上述以外还有sip信令的关键信息包括信令的时间、事件名称、与sip交互的方向、callid、to和from、ip地址、文件指针和字节Urs组件的日志文件名、文件对应的应用名、对应的事件、事件的时间、对象、客户号码、通话的connid客户端请求id、文件指针及字节。在本申请提供的实施例中，根据功能定义了2种功能实现一种是获取对应事件的详细内容接口如下：接口传入的为文件名、字符串、文件指针、数字类型、字节、数字类型中的一种或者多种。传出的为日志内容、字符串类型中的一种或者多种。一种接口是获取原始日志的压缩文件，接口如下：接口传入为文件名，字符串传出的为文件流，可保持为文件的形式。具体的，结合前文的说明，在本申请的实施例中，接口使用实现代码如下的详情如下：
Figure PCTCN2018108058-appb-000001
Figure PCTCN2018108058-appb-000002
Figure PCTCN2018108058-appb-000003
进一步地，在其中一种实施方式中，在所述将分析后的目标话务日志压缩获得压缩日志之中，还包括：
将目标话务日志按时间分割成单位时间段的目标话务日志；相邻两个单位时间段的目标话务日志中，前一个单位时间段的目标话务日志，包含后一个单位时间段的目标话务日志中前面预置时长的目标话务日志内容，并标记所述目标话务日志内容作为时间连贯性校验内容；分别压缩所述单位时间段的目标话务日志。
对于多线程同时压缩包情况，比如，采集分割3个包，依次为9点一个包，10点一个包，11点一个包，三个线程同时压缩，一般为前面时间先压缩完，但有可能10点的包很大，导致11点的包先压缩完；在解压时， 可能导致11点的包排在10点之前。为解决这个问题，每个包的内容，均包括后面包内容的前面一段，比如，10点包，最后面包括11点0-5分的内容。则在解压时候，10点包，包括11点0-5分的内容，可与11点的包的进行校对，从而确定10点的包在11点之前，然后重新按时间排序。
本申请实施例中还提拱了一种日志处理装置，在其中一种实施方式中，如图7所示，包括：
备份模块100，用于将目标话务日志备份到对应的网络存储器中，所述网络存储器被分配有网络协议地址；
链接模块200，用于将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析；
压缩模块300，用于将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。
进一步地，如图7所示，本申请实施例中提供的一种日志处理装置，还包括：提取模块210，用于提取分析后的所述目标话务日志中的关键字段，将所述关键字段以索引方式存储到数据库中。删除模块310，用于删除所述网络存储器内的分析后的所述目标话务日志。构建模块320，用于构建与web查询接口相对应且能从所述网络存储器中获取到所述压缩日志的日志查询接口。判断模块101，用于依据预设关键信息和预设匹配规则确定当前日志是否为需要被备份的所述目标话务日志；切换模块102，用于当确定所述当前日志为需要被备份的所述目标话务日志时，将备份所述目标话务日志的服务器切换为与所述目标话务日志相对应的服务器。优选地，所述预设匹配规则为正则匹配规则；分割单元301，将目标话务日志按时间分割成单位时间段的目标话务日志；相邻两个单位时间段的目标话务日志中，前一个单位时间段的目标话务日志，包含后一个单位时间段的目标话务日志中前面预置时长的目标话务日志内容，并标记所述目标话务日志内容作为时间连贯性校验内容；分别压缩所述单位时间段的目标话务日志。
本申请实施例提供的一种日志处理装置可以实现上述日志处理方法的实施例，具体功能实现请参见方法实施例中的说明，在此不再赘述。
本申请实施例提供的一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，该程序被处理器执行时实现任一项技术方案所述的日志处理方法。其中，所述计算机可读存储介质包括但不限于任何类型的盘(包括软盘、硬盘、光盘、CD-ROM、和磁光盘)、ROM(Read-Only Memory，只读存储器)、RAM(Random Access Memory，随即存储器)、EPROM(Erasable Programmable Read-Only Memory，可擦写可编程只读存储器)、EEPROM(Electrically Erasable Programmable Read-Only Memory，电可擦可编程只读存储器)、闪存、磁性卡片或光线卡片。也就是，存储设备包括由设备(例如，计算机、手机)以能够读的形式存储或传输信息的任何介质，可以是只读存储器，磁盘或光盘等。
本申请实施例提供的一种计算机可读存储介质，可智能化地实现上述日志处理方法的实施例，日志通过专有网络复制出去再分析，保证了及时性也降低了日志分析过多的占用分析和压缩都占用cpu等资源，通过将日志压缩备份到网络存储器，且将分析日志的服务器挂备份日志的网络存储器的路径，使得存储日志存储器和分析日志的服务器能够一一对应，在需要分析日志时，通过挂的网络存储器路径便能实现对日志的分析，实现分布式分析及压缩，解决了分析的性能瓶颈；本申请实施例中还提拱了一种日志处理方法，将目标话务日志备份到对应的网络存储器中，所述网络存储器分配有网络协议地址；将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析；将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。本申请提供的实施例中，将同一产品(如同一app)的日志及时地复制到网络存储器中，进而降低日志占用app所在终端产品的内存，同时通过将分析该产品日志的服务器挂到存储日志的网络存储器的路径，实现了可在网络存储器中分析日志，降低分析日志占用app所在终端产品的cpu等资源，将分析后的日志进行压缩，并把已分析且被压缩的备份的日志删除，降低了日志占用网络存储器的内存，进而降低内存对分析日志速度的影响。将不同产品或者同一产品不同类型的日志分别存储到对应的网络存储器中，并在其对应的网络存储器中进行日志 的分析和压缩备份，实现了分布式的分析和压缩备份日志，进而解决了直接在终端分析日志的瓶颈，提高了分析日志的速度。将日志压缩备份，可以降低日志占用网络存储器的内存，保证了后续话务查询时可以查找到对应的日志，避免应用程序中的日志被删除后，出现不能再进行话务查询和分析的情况。
本申请实施例提供的计算机可读存储介质可以实现上述日志处理方法的实施例，具体功能实现请参见方法实施例中的说明，在此不再赘述。
此外，在又一种实施例中，本申请还提供一种服务器，如图8所示，所述服务器处理器503、存储器505、输入单元507以及显示单元509等器件。本领域技术人员可以理解，图8示出的结构器件并不构成对所有服务器的限定，可以包括比图示更多或更少的部件，或者组合某些部件。存储器505可用于存储应用程序501以及各功能模块，处理器503运行存储在存储器505的应用程序501，从而执行设备的各种功能应用以及数据处理。存储器505可以是内存储器或外存储器，或者包括内存储器和外存储器两者。内存储器可以包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦写可编程ROM(EEPROM)、快闪存储器、或者随机存储器。外存储器可以包括硬盘、软盘、ZIP盘、U盘、磁带等。本申请所公开的存储器包括但不限于这些类型的存储器。本申请所公开的存储器505只作为例子而非作为限定。
输入单元507用于接收信号的输入，以及用户输入关键字段和用户输入执行查询检索的操作指令。输入单元507可包括触控面板以及其它输入设备。触控面板可收集客户在其上或附近的触摸操作(比如客户使用手指、触笔等任何适合的物体或附件在触控面板上或在触控面板附近的操作)，并根据预先设定的程序驱动相应的连接装置；其它输入设备可以包括但不限于物理键盘、功能键(比如播放控制按键、开关按键等)、轨迹球、鼠标、操作杆等中的一种或多种。显示单元509可用于显示客户输入的信息或提供给客户的信息以及计算机设备的各种菜单。显示单元509可采用液晶显示器、有机发光二极管等形式。处理器503是计算机设备的控制中心，利用各种接口和线路连接整个电脑的各个部分，通过运行或执行存储在存 储器503内的软件程序和/或模块，以及调用存储在存储器内的数据，执行各种功能和处理数据。图8中所示的一个或多个处理器503能够执行、实现图7中所示的备份模块100的功能、链接模块200的功能、压缩模块300的功能、提取模块210的功能、删除模块310的功能、构建模块320的功能、判断模块101的功能、切换模块102的功能、分割单元301的功能。
在一种实施方式中，所述服务器包括一个或多个处理器503，以及一个或多个存储器505，一个或多个应用程序501，其中所述一个或多个应用程序501被存储在存储器505中并被配置为由所述一个或多个处理器503执行，所述一个或多个应用程序301配置用于执行以上实施例所述的日志处理方法。
本申请实施例提供的一种服务器，可智能化地实现上述日志处理方法的实施例，日志通过专有网络复制出去再分析，保证了及时性也降低了日志分析过多的占用分析和压缩都占用cpu等资源，通过将日志压缩备份到网络存储器，且将分析日志的服务器挂备份日志的网络存储器的路径，使得存储日志存储器和分析日志的服务器能够一一对应，在需要分析日志时，通过挂的网络存储器路径便能实现对日志的分析，实现分布式分析及压缩，解决了分析的性能瓶颈；本申请实施例中还提拱了一种日志处理方法，将目标话务日志备份到对应的网络存储器中，所述网络存储器被分配有网络协议地址；将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析；将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。本申请提供的实施例中，将同一产品(如同一app)的日志及时地复制到网络存储器中，进而降低日志占用app所在终端产品的内存，同时通过将分析该产品日志的服务器挂到存储日志的网络存储器的路径，实现了可在网络存储器中分析日志，降低分析日志占用app所在终端产品的cpu等资源，将分析后的日志进行压缩，并把已分析且被压缩的备份的日志删除，降低了日志占用网络存储器的内存，进而降低内存对分析日志速度的影响。将不同产品或者同一产品不同类型的日志分别存储 到对应的网络存储器中，并在其对应的网络存储器中进行日志的分析和压缩备份，实现了分布式的分析和压缩备份日志，进而解决了直接在终端分析日志的瓶颈，提高了分析日志的速度。将日志压缩备份，可以降低日志占用网络存储器的内存，保证了后续话务查询时可以查找到对应的日志，避免应用程序中的日志被删除后，出现不能再进行话务查询和分析的情况。
本申请实施例提供的服务器可以实现上述提供的日志处理方法的实施例，具体功能实现请参见方法实施例中的说明，在此不再赘述。

Claims (20)

  1. 一种日志处理方法，包括：
    将目标话务日志备份到对应的网络存储器中，所述网络存储器被分配有网络协议地址；
    将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析；
    将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。
  2. 根据权利要求1所述的日志处理方法，在所述将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析之后，还包括：
    提取分析后的所述目标话务日志中的关键字段，将所述关键字段以索引方式存储到数据库中。
  3. 根据权利要求1所述的日志处理方法，在所述依据所述网络存储器指定的归档路径将所述压缩日志进行存储之后，还包括：
    删除所述网络存储器内的分析后的所述目标话务日志。
  4. 根据权利要求1所述的日志处理方法，在所述依据所述网络存储器指定的归档路径将所述压缩日志进行存储中之后，还包括：
    构建与web查询接口相对应且能从所述网络存储器中获取到所述压缩日志的日志查询接口。
  5. 根据权利要求1所述的日志处理方法，在所述将目标话务日志备份到对应的网络存储器中之前，还包括：
    依据预设关键信息和预设匹配规则确定当前日志是否为需要被备份的所述目标话务日志；
    当确定所述当前日志为需要被备份的所述目标话务日志时，将备份所述目标话务日志的服务器切换为与所述目标话务日志相对应的服务器。
  6. 根据权利要求5所述的日志处理方法，所述预设匹配规则为正则匹配规则。
  7. 根据权利要求5所述的日志处理方法，在所述将分析后的目标话务日志压缩获得压缩日志之中，还包括：
    将所述目标话务日志按时间分割成单位时间段的所述目标话务日志；相邻两个单位时间段的目标话务日志中，前一个单位时间段的目标话务日志，包含后一个单位时间段的目标话务日志中前面预置时长的目标话务日志内容，并标记所述目标话务日志内容作为时间连贯性校验内容；分别压缩所述单位时间段的目标话务日志。
  8. 一种日志处理装置，包括：
    备份模块，用于将目标话务日志备份到对应的网络存储器中，所述网络存储器被分配有网络协议地址；
    链接模块，用于将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析；
    压缩模块，用于将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。
  9. 一种非易失性计算机可读存储介质，所述非易失性计算机可读存储介质上存储有计算机程序，该程序被处理器执行一种日志处理方法，所述日志处理方法包括以下步骤：
    将目标话务日志备份到对应的网络存储器中，所述网络存储器被分配有网络协议地址；
    将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析；
    将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。
  10. 根据权利要求9所述的非易失性计算机可读存储介质，在所述将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析之后，还包括以下步骤：
    提取分析后的所述目标话务日志中的关键字段，将所述关键字段以索引方式存储到数据库中。
  11. 根据权利要求9所述的非易失性计算机可读存储介质，在所述依 据所述网络存储器指定的归档路径将所述压缩日志进行存储之后，还包括以下步骤：
    删除所述网络存储器内的分析后的所述目标话务日志。
  12. 根据权利要求9所述的非易失性计算机可读存储介质，在所述依据所述网络存储器指定的归档路径将所述压缩日志进行存储中之后，还包括以下步骤：
    构建与web查询接口相对应且能从所述网络存储器中获取到所述压缩日志的日志查询接口。
  13. 根据权利要求9所述的非易失性计算机可读存储介质，在所述将目标话务日志备份到对应的网络存储器中之前，还包括以下步骤：
    依据预设关键信息和预设匹配规则确定当前日志是否为需要被备份的所述目标话务日志；
    当确定所述当前日志为需要被备份的所述目标话务日志时，将备份所述目标话务日志的服务器切换为与所述目标话务日志相对应的服务器。
  14. 一种服务器，包括：
    一个或多个处理器；
    存储器；
    一个或多个应用程序，其中所述一个或多个应用程序被存储在所述存储器中并被配置为由所述一个或多个处理器执行，所述一个或多个应用程序配置用于执行一种日志处理方法，所述日志处理方法包括以下步骤：
    将目标话务日志备份到对应的网络存储器中，所述网络存储器被分配有网络协议地址；
    将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析；
    将分析后的目标话务日志压缩获得压缩日志，依据所述网络存储器指定的归档路径将所述压缩日志进行存储。
  15. 根据权利要求14所述的服务器，在所述将用于分析所述目标话务日志的服务器链接到所述网络协议地址，以使得所述服务器对所述目标话务日志进行分析之后，还包括以下步骤：
    提取分析后的所述目标话务日志中的关键字段，将所述关键字段以索引方式存储到数据库中。
  16. 根据权利要求14所述的服务器，在所述依据所述网络存储器指定的归档路径将所述压缩日志进行存储之后，还包括以下步骤：
    删除所述网络存储器内的分析后的所述目标话务日志。
  17. 根据权利要求14所述的服务器，在所述依据所述网络存储器指定的归档路径将所述压缩日志进行存储中之后，还包括以下步骤：
    构建与web查询接口相对应且能从所述网络存储器中获取到所述压缩日志的日志查询接口。
  18. 根据权利要求14所述的服务器，在所述将目标话务日志备份到对应的网络存储器中之前，还包括以下步骤：
    依据预设关键信息和预设匹配规则确定当前日志是否为需要被备份的所述目标话务日志；
    当确定所述当前日志为需要被备份的所述目标话务日志时，将备份所述目标话务日志的服务器切换为与所述目标话务日志相对应的服务器。
  19. 根据权利要求18所述的服务器，所述预设匹配规则为正则匹配规则。
  20. 根据权利要求18所述的服务器，在所述将分析后的目标话务日志压缩获得压缩日志之中，还包括以下步骤：
    将所述目标话务日志按时间分割成单位时间段的所述目标话务日志；相邻两个单位时间段的目标话务日志中，前一个单位时间段的目标话务日志，包含后一个单位时间段的目标话务日志中前面预置时长的目标话务日志内容，并标记所述目标话务日志内容作为时间连贯性校验内容；分别压缩所述单位时间段的目标话务日志。
PCT/CN2018/108058 2018-06-21 2018-09-27 日志处理方法、装置及存储介质、服务器 WO2019242148A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810642922.0A CN109062774A (zh) 2018-06-21 2018-06-21 日志处理方法、装置及存储介质、服务器
CN201810642922.0 2018-06-21

Publications (1)

Publication Number Publication Date
WO2019242148A1 true WO2019242148A1 (zh) 2019-12-26

Family

ID=64821281

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/108058 WO2019242148A1 (zh) 2018-06-21 2018-09-27 日志处理方法、装置及存储介质、服务器

Country Status (2)

Country Link
CN (1) CN109062774A (zh)
WO (1) WO2019242148A1 (zh)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110536031B (zh) * 2019-08-30 2021-06-08 上海浦东发展银行股份有限公司信用卡中心 一种话务数据的关联及整合方法
CN113064752B (zh) * 2019-12-16 2023-11-21 华晨宝马汽车有限公司 用于将日志归档的方法、系统和计算机可读介质
CN111414613B (zh) * 2020-03-18 2023-12-26 杭州迪普科技股份有限公司 一种日志处理的方法和装置
CN111488314B (zh) * 2020-03-30 2023-06-30 北京中电华大电子设计有限责任公司 一种基于Python的仿真日志分析方法
CN113190726A (zh) * 2021-04-16 2021-07-30 珠海格力精密模具有限公司 读取cae模流分析数据的方法、电子设备及存储介质
CN113297008B (zh) * 2021-05-19 2023-12-12 阿里巴巴新加坡控股有限公司 数据处理方法以及系统
CN114697070B (zh) * 2021-12-31 2024-04-02 成都思维世纪科技有限责任公司 一种针对http协议流量动态压缩存储的方法及系统
CN114936615B (zh) * 2022-07-25 2022-10-14 南京大数据集团有限公司 一种基于表征一致性校对的小样本日志信息异常检测方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106649336A (zh) * 2015-10-30 2017-05-10 华为数字技术（苏州）有限公司 一种日志压缩方法及日志处理设备、日志处理系统
CN106897187A (zh) * 2017-01-23 2017-06-27 北京思特奇信息技术股份有限公司 一种终端数据分布方法及装置
US20170302758A1 (en) * 2014-10-05 2017-10-19 YScope Inc. Systems and processes for computer log analysis
CN107622084A (zh) * 2017-08-10 2018-01-23 深圳前海微众银行股份有限公司 日志管理方法、系统以及计算机可读存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1063898C (zh) * 1998-05-13 2001-03-28 广东省邮电科学技术研究院 模拟移动通信网的集中操作维护方法
US8068599B2 (en) * 2008-01-09 2011-11-29 Accenture Global Services Limited Call center application data and interoperation architecture for a telecommunication service center
CN102750196A (zh) * 2011-04-20 2012-10-24 大连兆阳软件科技有限公司 一种用于数据存储和备份系统及方法
CN103200037B (zh) * 2013-04-11 2016-07-06 深圳市共进电子股份有限公司 一种syslog日志存储方法
CN105824744B (zh) * 2016-03-21 2018-06-15 焦点科技股份有限公司 一种基于b2b平台的实时日志采集分析方法
CN107885817A (zh) * 2017-11-06 2018-04-06 余帝乾 一种基于大数据网络用户行为的方法和装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170302758A1 (en) * 2014-10-05 2017-10-19 YScope Inc. Systems and processes for computer log analysis
CN106649336A (zh) * 2015-10-30 2017-05-10 华为数字技术（苏州）有限公司 一种日志压缩方法及日志处理设备、日志处理系统
CN106897187A (zh) * 2017-01-23 2017-06-27 北京思特奇信息技术股份有限公司 一种终端数据分布方法及装置
CN107622084A (zh) * 2017-08-10 2018-01-23 深圳前海微众银行股份有限公司 日志管理方法、系统以及计算机可读存储介质

Also Published As

Publication number Publication date
CN109062774A (zh) 2018-12-21

Similar Documents

Publication Publication Date Title
WO2019242148A1 (zh) 日志处理方法、装置及存储介质、服务器
US11645183B1 (en) User interface for correlation of virtual machine information and storage information
US11860874B2 (en) Multi-partitioning data for combination operations
US10956362B1 (en) Searching archived data
US11775501B2 (en) Trace and span sampling and analysis for instrumented software
US20220121628A1 (en) Streaming synthesis of distributed traces from machine logs
US11966797B2 (en) Indexing data at a data intake and query system based on a node capacity threshold
US11755531B1 (en) System and method for storage of data utilizing a persistent queue
US11604789B1 (en) Bi-directional query updates in a user interface
US11687487B1 (en) Text files updates to an active processing pipeline
US11892976B2 (en) Enhanced search performance using data model summaries stored in a remote data store
US11599396B2 (en) Resegmenting chunks of data based on source type to facilitate load balancing
US11922222B1 (en) Generating a modified component for a data intake and query system using an isolated execution environment image
US11494408B2 (en) Asynchronous row to object enrichment of database change streams
WO2021097713A1 (zh) 分布式安全检测系统、方法、设备及存储介质
US20230073627A1 (en) Analytics database and monitoring system for structuring and storing data streams
US20220245091A1 (en) Facilitating generation of data model summaries
US20140279926A1 (en) Active archive bridge
US10354062B2 (en) System and method for simultaneous forensic, acquisition, examination and analysis of a computer readable medium at wire speed
US11836146B1 (en) Storing indexed fields per source type as metadata at the bucket level to facilitate search-time field learning
CN114422388B (zh) 一种远程数据监管系统
US20170270132A1 (en) File browser user interface
US11860869B1 (en) Performing queries to a consistent view of a data set across query engine types
CN117319349A (zh) 域名请求处理方法、装置、设备及存储介质
KR100423840B1 (ko) 분산 데이터베이스의 데이터 처리방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18923078

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18923078

Country of ref document: EP

Kind code of ref document: A1