CA3136021A1 - Log storage method and device and computer system - Google Patents

Log storage method and device and computer system Download PDF

Info

Publication number
CA3136021A1
CA3136021A1 CA3136021A CA3136021A CA3136021A1 CA 3136021 A1 CA3136021 A1 CA 3136021A1 CA 3136021 A CA3136021 A CA 3136021A CA 3136021 A CA3136021 A CA 3136021A CA 3136021 A1 CA3136021 A1 CA 3136021A1
Authority
CA
Canada
Prior art keywords
preset
data
log
key
data key
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CA3136021A
Other languages
French (fr)
Inventor
Zhiliang Geng
Yuxue Bao
Xu Chen
Xin Xu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
10353744 Canada Ltd
Original Assignee
10353744 Canada Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 10353744 Canada Ltd filed Critical 10353744 Canada Ltd
Publication of CA3136021A1 publication Critical patent/CA3136021A1/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/11File system administration, e.g. details of archiving or snapshots
    • G06F16/113Details of archiving

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The present invention discloses to a log storage method, apparatus, and system. The method comprises: obtaining to-be-processed log in preset format generated by preset service platform and configuration data corresponding to preset service platform, to-be-processed log includes key-value pair and preset identifier, key-value pair is used to store historical operation record; parsing to-be-processed log, recognizing data key and data value corresponding to the data key included in to-be-parsed log;
when included data key pair includes preset data key, modifying data value corresponding to included preset data key according to corresponding replacement rule; generating historical operation record according to both meaning and data value corresponding to each data key;
storing historical operation record in index library address corresponding to preset identifier; storing historical operation record in index library address corresponding to preset identifier improves log processing efficiency, convenient for non-operation-maintenance to deal with system problems and improves the system problem-solving efficiency.

Description

LOG STORAGE METHOD, APPARATUS, AND COMPUTER SYSTEM
Field [0001] The present disclosure relates to computer technology filed, particularly to a log storage method, apparatus, and system.
Background
[0002] During system operation, a system log is generated. In the period of small-scale system, the number of generated system logs is small, the original text of the generated logs can be stored directly, and the operation and maintenance of the system logs can be manually performed by the operation and maintenance people. However, with the rapid expansion of business requirements and the diversified development of services, the system has become more and more complex, a system can include a plurality of devices and deploy a plurality of instances, resulting in inconsistent log data structures, when system has problems, operation and maintenance people are required to manually locate the cause of the problem, which is not only very difficult but also inefficient.
Invention Content
[0003] To solve the present technical problems, the present application provides a log storage method, apparatus and system. The technical solutions are as following:
[0004] To achieve the above-mentioned purpose, the first aspect is providing a log storage method, comprising:
[0005] Obtaining a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store historical operation records of the Date Recue/Date Received 2021-12-24 preset service platform, the configuration data include meaning corresponding to each data key, replacement rule corresponding to preset data key, and index library address corresponding to each preset identifier;
[0006] Parsing the to-be-processed log, recognizing data key and data value corresponding to the data keys included in the to-be-parsed log;
[0007] When included data key pair includes preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule;
[0008] Generating historical operation record according to both meaning and data value corresponding to each data key;
[0009] Storing historical operation record in index library address corresponding to the preset identifier.
[0010]
In some implementations, storing historical operation record in index library address corresponding to the preset identifier, comprising:
[0011] Sending data storage request to preset system corresponding to the index library address, so that the preset system returns data access authority according to the data storage request;
[0012] When receiving the returned data access authority, storing the historical operation record in the index library address.
[0013] In some implementations, the method also comprises:
[0014] Receiving log reading request sent by user;

Date Recue/Date Received 2021-12-24
[0015] Generating visual log according to historical operation record corresponding to the log reading request;
[0016] Returning the visual log to the user.
[0017] In some implementations, the to-be-processed log is in Json format.
[0018] In some implementations, obtaining a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, comprising:
[0019] Using Flume log collection system to obtain the to-be-processed log in the preset format generated by the preset service platform and the configuration data corresponding to the preset service platform;
[0020] Parsing the to-be-processed log, recognizing data key and data value corresponding to the data key included in the to-be-parsed log, comprising:
[0021] Using Milk streaming computing engine to parse the to-be-processed log, recognizing the data key and the data value corresponding to the data key included in the to-be-parsed log.
[0022] The second aspect, the present invention provides a log storage method, the method comprises:
[0023] Sending generated to-be-processed log to preset processing terminal, so that the preset processing terminal receives the to-be-processed log and obtains corresponding configuration data, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store the historical operation record of the preset service platform, the configuration data includes the meaning corresponding to each data key, the replacement rule corresponding to the preset data key, and the index library address Date Recue/Date Received 2021-12-24 corresponding to each preset identifier; parsing the to-be-processed log, recognizing data key and data value corresponding to the data key included in the to-be-parsed log; when the included data key pair includes preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule; storing the historical operation record to the index library address corresponding to the preset identifier.
[0024] The third aspect, the present invention provides a log storage apparatus, the apparatus comprises:
[0025] An obtaining module configured to obtain a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store historical operation record of the preset service platform, the configuration data include meaning corresponding to each data key, replacement rule corresponding to preset data key, and index library address corresponding to each preset identifier;
[0026] A parsing module configured to parse the to-be-processed log, recognizing data key and data value corresponding to the data key included in the to-be-parsed log; when included data key pair include preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule; generating historical operation record according to both meaning and data value corresponding to each data key;
[0027] A storage module configured to store historical operation record in index library address corresponding to the preset identifier.
[0028] The fourth aspect, the present invention provides a computer system, the system comprises:
[0029] One or a plurality of processors;

Date Recue/Date Received 2021-12-24
[0030] A memory associated with one or a plurality of processors, the memory is configured to store program commands, if the program commands are executed by one or a plurality of processors, executing following operations:
[0031] Obtaining a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store historical operation record of the preset service platform, the configuration data include meaning corresponding to each data key, replacement rule corresponding to preset data key, and index library address corresponding to each preset identifier;
[0032] Parsing the to-be-processed log, recognizing data key and data value corresponding to the data key included in the to-be-parsed log;
[0033] When included data key pair include preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule;
[0034] Generating historical operation record according to both meaning and data value corresponding to each data key;
[0035] Storing historical operation record in index library address corresponding to the preset identifier.
[0036] The beneficial effects achieved by the present invention are:
[0037] The present invention discloses to a log storage method, including obtaining to-be-processed log in preset format generated by preset service platform and configuration data corresponding to preset service Date Recue/Date Received 2021-12-24 platform, to-be-processed log includes key-value pair and preset identifier, key-value pair is used to store historical operation record, the configuration data include meaning corresponding to each data key, replacement rule corresponding to preset data key, and index library address corresponding to each preset identifier; the configuration data include meaning corresponding to each data key, replacement rule corresponding to preset data key, and index library address corresponding to each preset identifier; parsing to-be-processed log, recognizing data key and data value corresponding to the data key included in to-be-parsed log; when included data key pair includes preset data key, modifying data value corresponding to included preset data key according to corresponding replacement rule;
generating historical operation record according to both meaning and data value corresponding to each data key; storing historical operation record in index library address corresponding to preset identifier;
storing historical operation record in index library address corresponding to preset identifier improves log processing efficiency, and when need to consult the log later, people can directly obtain the stored parsed log without re-analyzing the log, non-operations and maintenance people can also understand the parsed log, which is convenient for non-operations and maintenance people to deal with problems caused by the system and improve system problem-solving efficiencies.
Drawing Description
[0038] In order to describe the technical solutions clearer in the implementations of the present application or the prior art, the following are drawings that need to be used are briefly introduced.
Obviously, the drawings in the following description are only some implementations of the application, for those of ordinary skill in the art , without creative work, they can also obtain other drawings based on these drawings.
[0039] Figure 1 is a service workflow diagram of a log storage system in implementation of the present application;

Date Recue/Date Received 2021-12-24
[0040] Figure 2 is a method process diagram in implementation of the present application;
[0041] Figure 3 is a structural diagram of apparatus in implementation of the present application.
[0042] Figure 4 is a structural diagram of computer system in implementation of the present application.
Specific implementation methods
[0043] The following will describe the technical solutions of the implementations in the present application with accompanying drawings, obviously the described implementations are only a part of the implementations in the present application. Based on the implementations in the present application, all other implementations obtained by those of ordinary skilled in the art will fall in the protection scope of the present application.
[0044] As described in the background art, in order to solve the above-mentioned technical problems, this application proposes a log storage system, which includes a service platform for generating logs and a server for storing logs. The system can include a plurality of service platforms, and each service platform is preset with the same log format. Preferably, the log can be in Json format.
Implementation one [0046] As shown in Figure 1, the storage system performs process of log storage and query comprising:
[0047] Step 1, the service platform generates a log in a preset format and sends it to server;
[0048] The generated log can include prefix and Json string. Each prefix uniquely corresponds to an index library. The Json string stores historical operating record of the platform in the format of key-value pair.

Date Recue/Date Received 2021-12-24 The historical operation record can include the service platform's operation process parameter, executed data operation, etc., which are used to record the platform's operation situation and usage which is not limited in this application.
[0049] Step 2, the service terminal uses Flume log collection system to obtain the configuration data corresponding to the service platform;
[0050] Flume is a system for highly available, highly reliable, distributed massive log collection, aggregation and transmission, Flume supports customizing various data senders in the log system to collect data; at the same time, Flume provides ability to simple process data and write to various data recipients.
[0051] Each service platform has corresponding configuration data, the configuration data includes the meaning of each data key contained in the log generated by the service platform, the field type of each key-value pair, whether each data key is displayed, the enumeration value corresponding to the preset data key and the meaning corresponding to the enumeration value.
[0052] The configuration data may include the meaning of the enumerated value corresponding to the preset data key. For example, the configuration data can specify that the value corresponding to a certain data key can be 1 or 0, when the value corresponding to the data key is 0, which means the purchase has failed, and the value 1 means the purchase is successful, etc., this application does not add to this limited.
The configuration data can also include filtering rules, and the data values corresponding to the data keys contained in the filtering rules are not stored.
[0053] Step 3, the server uses Flink streaming to obtain the data value corresponding to each data key contained in the log and determines the effects of the data value corresponding to each data key according to the meaning of each data key.

Date Recue/Date Received 2021-12-24 [0054] Flink is an open-source stream processing framework whose core is a distributed stream data stream engine written in Java and Scala. Flink executes arbitrary stream data programs in data parallel and pipeline mode, Flink's pipeline nmtime system can execute batch and stream processing programs.
[0055] According to the effects of the data value corresponding to each data key, the server can obtain the historical operation record of the service platform contained in the to-be-stored log. For example, the data value corresponding to a certain data key is used to indicate the user's purchase amount on the service platform, the server can determine the user's purchase amount according to the data value corresponding to the data key in the log and generate a purchase record. The historical operation record includes the purchase record.
[0056] Preferably, the historical operation record can be generated in any format, which is not limited in this application.
[0057] Step 4: The server determines the index library corresponding to the log according to the prefix;
[0058] Since each prefix uniquely corresponds to an index library, the server can determine the corresponding to-be-stored index library according to the prefix. The index library can be deployed on any storage system or device such as ElasticSearch, which is not limited in this application.
[0059] Step 5, the server requests data access permission from the index library, when obtaining the data access permission, the historical operation record is stored in the corresponding index library.
[0060] Step 6, the server receives the user's log query request;
[0061] The server determines to-be-read log according to user's log query request and locates the index library where the to-be-read log is located.

Date Recue/Date Received 2021-12-24 [0062] Step 7, the server generates a corresponding visual log according to the obtained corresponding historical operation record and the preset template, then returns the visual log to the user.
[0063] The server requests data access permissions from the index library, and when obtaining the data access permissions, obtaining the corresponding historical operation record from the index library.
Implementation two [0065] For the above-mentioned implementation, the present invention provides a log storage method, as shown in figure 2, the method includes:
[0066] 210, obtaining a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store historical operation records of the preset service platform, the configuration data include meaning corresponding to each data key, replacement rule corresponding to preset data key, and index library address corresponding to each preset identifier;
[0067] 220, parsing the to-be-processed log, recognizing data key and data value corresponding to the data keys included in the to-be-parsed log;
[0068] Preferably, obtaining a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, comprising:
[0069] 221, using Flume log collection system to obtain the to-be-processed log in the preset format generated by the preset service platform and the configuration data corresponding to the preset service Date Recue/Date Received 2021-12-24 platform;
[0070] Parsing the to-be-processed log, recognizing data key and data value corresponding to the data key included in the to-be-parsed log, comprising:
[0071] 222, using Flink streaming computing engine to parse the to-be-processed log, recognizing the data key and the data value corresponding to the data key included in the to-be-parsed log.
[0072] 230, when included data key pair includes preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule;
[0073] 240, generating historical operation record according to both meaning and data value corresponding to each data key;
[0074] 250, storing historical operation record in index library address corresponding to the preset identifier.
[0075] Preferably, storing historical operation record in index library address corresponding to the preset identifier, comprising:
[0076] 251, Sending data storage request to preset system corresponding to the index library address, so that the preset system returns data access authority according to the data storage request;
[0077] 252, when receiving the returned data access authority, storing the historical operation record in the index library address.
[0078] Preferably, the to-be-processed log is in Json format.

Date Recue/Date Received 2021-12-24 Implementation three [0080] For the above-mentioned implementation, as shown in Figure 3, the present application provides a log storage apparatus, the apparatus comprises:
[0081] An obtaining module 310 configured to obtain a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store historical operation record of the preset service platform, the configuration data include meaning corresponding to each data key, replacement rule corresponding to preset data key, and index library address corresponding to each preset identifier;
[0082] A parsing module 320 configured to parse the to-be-processed log, recognizing data key and data value corresponding to the data key included in the to-be-parsed log; when included data key pair include preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule; generating historical operation record according to both meaning and data value corresponding to each data key;
[0083] A storage module 330 configured to store historical operation record in index library address corresponding to the preset identifier.
[0084] Preferably, the storage module 330 is also for sending data storage request to preset system corresponding to the index library address, so that the preset system returns data access authority according to the data storage request; when receiving the returned data access authority, storing the historical operation record in the index library address.

Date Recue/Date Received 2021-12-24 [0085] Preferably, the apparatus also comprises a receiving module 340 configured to receive log reading request sent by user; generate visual log according to historical operation record corresponding to the log reading request; return the visual log to the user.
[0086] Preferably, the to-be-processed log is in Json format.
[0087] Preferably, the obtaining module 310 is also configured to use Flume log collection system to obtain the to-be-processed log in the preset format generated by the preset service platform and the configuration data corresponding to the preset service platform, the parsing module 320 is also configured to use Flink streaming computing engine to parse the to-be-processed log, recognize the data key and the data value corresponding to the data key included in the to-be-parsed log.
Implementation four [0089] For the above-mentioned method, the present application provides a log storage method, the method comprise:
[0090] Sending generated to-be-processed log to preset processing terminal, so that the preset processing terminal receives the to-be-processed log and obtains corresponding configuration data, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store the historical operation record of the preset service platform, the configuration data includes the meaning corresponding to each data key, the replacement rule corresponding to the preset data key, and the index library address corresponding to each preset identifier; parsing the to-be-processed log, recognizing data key and data value corresponding to the data key included in the to-be-parsed log; when the included data key pair includes preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule; storing the historical operation record to the index library address corresponding to the preset identifier.

Date Recue/Date Received 2021-12-24 Implementation five [0092] The implementation five of the present application provides a computer system, comprising: one or a plurality of processors; A memory associated with one or a plurality of processors, the memory is configured to store program commands, if the program commands are executed by one or plurality of processors, executing the following operations:
[0093] Obtaining a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store historical operation records of the preset service platform, the configuration data include meaning corresponding to each data key, replacement rule corresponding to preset data key, and index library address corresponding to each preset identifier;
[0094] Parsing the to-be-processed log, recognizing data key and data value corresponding to the data keys included in the to-be-parsed log;
[0095] When included data key pair includes preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule;
[0096] Generating historical operation record according to both meaning and data value corresponding to each data key;
[0097] Storing historical operation record in index library address corresponding to the preset identifier.
Wherein, Figure 4 exemplarily shows the architecture of the computer system , which can specifically include a processor 1510, video display adapter 1511, disk driver 1512, input/output Date Recue/Date Received 2021-12-24 interface 1513, network interface 1514, and memory 1520. The above-mentioned processor 1510, video display adapter 1511, disk driver 1512, input/output interface 1513, network interface 1514 and memory 1520 can be connected through a communication bus 1530.
[0098] Wherein, the processor 1510 can be achieved by using a general CPU
(Central Processing Unit), Microprocessor, Application Specific Integrated Circuit (ASIC) , or one or more integrated circuits, which are used to execute some relative program to achieve the technical solutions provided in this application.
[0099]
The memory 1520 can adopt ROM (Read Only Memory), RAM (Random Access Memory), static storage devices and dynamic storage devices to achieve. The memory 1520 can store operate system 1521 used to control the running of the computer system 1500, used to control the low-level operation of the computer system 1500's Basic Input Output System (BIOS) 1522.
In addition, storing a web browser 1523, data storage management 1524, and device identity information processing system 1525 and so on. The above-mentioned device identity information processing system 1525 can be the specific application that implements the above-mentioned steps. To sum up, when achieving the technical solutions provided by this application through software or firmware, related program codes are stored in the memory 1520 and executed by a processor 1510. Input / output interface 1513 is used for connecting input / output modules to achieve the information input and output.
Input / output module can be configured in the device as a component ( not shown in the figure ), or it can be connected to the device to provide corresponding functions. Wherein, Input devices can include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices can include monitors, speakers, vibrators, lights and so on.
[0100] The network interface 1514 is used to connect a communication module (not shown in the figure) to achieve the communication interaction between this device and other devices. Wherein, the communication module can achieve communication through wired means (such as USB , network cable, etc. ) , or through wireless methods ( such as mobile network, WIFI, Bluetooth, etc.) to achieve Date Recue/Date Received 2021-12-24 communication.
[0101] The bus 1530 includes a path and transmits information among various components of the device (such as the processor 1510, the video display adapter 1511, the disk driver 1512, the input/output interface 1513, the network interface 1514, and the memory 1520).
[0102] In addition, the electronic device 1500 also can obtain information with specific receiving conditions from the virtual resource object's receiving condition information database 1541 for condition judgement and son on.
[0103] What should be noted is that although the above device only shows the processor 1510 , the video display adapter 1511, the disk driver 1512, input/output interface 1513, network interface 1514, mcmory 1520, bus 1530, etc., but in the process of the specific implementation, the device may also include other essential components for normal operation. In addition, those skilled in the art can understand that the above apparatus can comprise only the essential components of the present application to achieve the implementation, but there is no need to contain all the components as shown in figure.
[0104] Known from the description of the above implementations that those skilled in the art can clearly understand that the application can be achieved with the help of software and essential general hardware platform. Based on this understanding, the essence of the technical solution of this application, or in other words, the part that contributes to the existing technology can be implemented in the form of a software product, the computer software product can be stored in storage media, such as ROM/RAM, magnetic disks, optical disks, etc., including several commands to make a computer device (can be a personal computer, a cloud server, or a network device, etc.) to execute the methods described in each implementation or some of the implementations of the present application.
[0105] The various implementations in this description are described in a progressive manner, the same Date Recue/Date Received 2021-12-24 and similar parts among the various implementations can be referred to each other separately, and each implementation focuses on the differences compared with the other implementations. Especially for the concern of the system or the system implementations, since it is basically similar to the implementation method, the description is relatively simple. For related details, please refer to the implementation method. The system and system implementations described in the above are only illustrative, and the units described by separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, which means, it can be in one place, or it may be distributed to a plurality of network units. Some or all the modules are selected according to actual needs to achieve the implementation's solution purpose. The ordinary skill in the art can understand and implement without creative work.
[0106] The above-mentioned are only preferred implementations of the present invention, but not used to restrict the present invention, anything in the spirits and the principles of the present invention, then any modifications, equivalent replacements, improvements shall be included in the protection scope of the present invention.

Date Recue/Date Received 2021-12-24

Claims (8)

Claims:
1. A log storage method comprises:
obtaining a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store historical operation records of the preset service platform, the configuration data include meaning corresponding to each data key, replacement rule corresponding to preset data key, and index library address corresponding to each preset identifier;
parsing the to-be-processed log, recognizing data key and data value corresponding to the data keys included in the to-be-parsed log;
when included data key pair includes preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule;
generating historical operation record according to both meaning and data value corresponding to each data key; and storing historical operation record in index library address corresponding to the preset identifier.
2. The method according to claim 1, storing historical operation record in index library address corresponding to the preset identifier, comprising:
sending data storage request to preset system corresponding to the index library address, so that the preset system returns data access authority according to the data storage request; and when receiving the returned data access authority, storing the historical operation record in the index library address.
3. The method according to claim 1, wherein, comprising:
receiving log reading request sent by user;
generating visual log according to historical operation record corresponding to the log reading request; and returning the visual log to the user.
4. The method according to claim 1, wherein, the to-be-processed log is in Json format.
5. The method according to claim 1, wherein, obtaining a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, comprising:
using Flume log collection system to obtain the to-be-processed log in the preset format generated by the preset service platform and the configuration data corresponding to the preset service platform;
parsing the to-be-processed log, recognizing data key and data value corresponding to the data key included in the to-be-parsed log, comprising:
using Flink streaming computing engine to parse the to-be-processed log, recognizing the data key and the data value corresponding to the data key included in the to-be-parsed log.
6. A log's storage method, wherein, the method comprises:
sending generated to-be-processed log to preset processing terminal, so that the preset processing terminal receives the to-be-processed log and obtains corresponding configuration data, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store the historical operation record of the preset service platform, the configuration data includes the meaning corresponding to each data key, the replacement rule corresponding to the preset data key, and the index library address corresponding to each preset identifier;
parsing the to-be-processed log, recognizing data key and data value corresponding to the data key included in the to-be-parsed log; when the included data key pair includes preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule;
storing the historical operation record to the index library address corresponding to the preset identifier.
7. A log storage apparatus comprises:
an obtaining module configured to obtain a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store historical operation record of the preset service platform, the configuration data include meaning corresponding to each data key, replacement rule corresponding to preset data key, and index library address corresponding to each preset identifier;
a parsing module configured to parse the to-be-processed log, recognizing data key and data value corresponding to the data key included in the to-be-parsed log; when included data key pair include preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule; generating historical operation record according to both meaning and data value corresponding to each data key; and a storage module configured to store historical operation record in index library address corresponding to the preset identifier.
8. A computer system, wherein, the system comprises:
one or a plurality of processors; and a memory associated with one or a plurality of processors, the memory is configured to store program commands, if the program commands are executed by one or a plurality of processors, executing following operations:
obtaining a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, the to-be-processed log includes a key-value pair and a preset identifier, and the key-value pair is used to store historical operation record of the preset service platform, the configuration data include meaning corresponding to each data key, replacement rule corresponding to preset data key, and index library address corresponding to each preset identifier;
parsing the to-be-processed log, recognizing data key and data value corresponding to the data key included in the to-be-parsed log;
when included data key pair include preset data key, modifying data value corresponding to the included preset data key according to the corresponding replacement rule;

generating historical operation record according to both meaning and data value corresponding to each data key; and storing historical operation record in index library address corresponding to the preset identifier.
CA3136021A 2020-10-27 2021-10-27 Log storage method and device and computer system Pending CA3136021A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011159844.2 2020-10-27
CN202011159844.2A CN112035425B (en) 2020-10-27 2020-10-27 Log storage method and device and computer system

Publications (1)

Publication Number Publication Date
CA3136021A1 true CA3136021A1 (en) 2022-04-27

Family

ID=73573377

Family Applications (1)

Application Number Title Priority Date Filing Date
CA3136021A Pending CA3136021A1 (en) 2020-10-27 2021-10-27 Log storage method and device and computer system

Country Status (2)

Country Link
CN (1) CN112035425B (en)
CA (1) CA3136021A1 (en)

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111241078A (en) * 2020-01-07 2020-06-05 网易(杭州)网络有限公司 Data analysis system, data analysis method and device
CN111522786A (en) * 2020-04-21 2020-08-11 中国建设银行股份有限公司 Log processing system and method
CN111526060B (en) * 2020-06-16 2023-02-28 网易(杭州)网络有限公司 Method and system for processing service log

Also Published As

Publication number Publication date
CN112035425B (en) 2021-11-09
CN112035425A (en) 2020-12-04

Similar Documents

Publication Publication Date Title
US11394625B2 (en) Service level agreement based storage access
CN107979632B (en) Processing method, device and system for push information
CN111414407A (en) Data query method and device of database, computer equipment and storage medium
CN110225104B (en) Data acquisition method and device and terminal equipment
CN109359118B (en) Data writing method and device
CN112867988A (en) Implementing compliance settings by a mobile device to follow a configuration scenario
CN111597065B (en) Method and device for collecting equipment information
US11882154B2 (en) Template representation of security resources
CN111506579A (en) Method, program and equipment for generating intelligent contract code
CN111090666A (en) Data processing method, device and system and computer readable storage medium
CN110244963B (en) Data updating method and device and terminal equipment
CN109522021B (en) Parameter callback processing method and device, electronic equipment and storage medium
CN111339560A (en) Data isolation method, device and system
CN114895879B (en) Management system design scheme determining method, device, equipment and storage medium
US20110282980A1 (en) Dynamic protection of a resource during sudden surges in traffic
CA3136021A1 (en) Log storage method and device and computer system
CN114357032A (en) Data quality monitoring method and device, electronic equipment and storage medium
CN108076067B (en) Method and system for authorized crawler configuration simulation login
CN114064429A (en) Audit log acquisition method and device, storage medium and server
CN113656378A (en) Server management method, device and medium
CN113434423A (en) Interface test method and device
US20170163564A1 (en) Discovering resource availability across regions
CN112948733A (en) Interface maintenance method, device, computing equipment and medium
CN114518984A (en) Reporting method and device of embedded point information, storage medium and terminal equipment
CN111639936B (en) Transaction information acquisition method and device, electronic equipment and readable storage medium

Legal Events

Date Code Title Description
EEER Examination request

Effective date: 20220916

EEER Examination request

Effective date: 20220916

EEER Examination request

Effective date: 20220916

EEER Examination request

Effective date: 20220916

EEER Examination request

Effective date: 20220916

EEER Examination request

Effective date: 20220916

EEER Examination request

Effective date: 20220916

EEER Examination request

Effective date: 20220916

EEER Examination request

Effective date: 20220916