WO2016095329A1 - 一种日志记录系统及日志记录操作方法 - Google Patents

一种日志记录系统及日志记录操作方法 Download PDF

Info

Publication number
WO2016095329A1
WO2016095329A1 PCT/CN2015/072429 CN2015072429W WO2016095329A1 WO 2016095329 A1 WO2016095329 A1 WO 2016095329A1 CN 2015072429 W CN2015072429 W CN 2015072429W WO 2016095329 A1 WO2016095329 A1 WO 2016095329A1
Authority
WO
WIPO (PCT)
Prior art keywords
log
log record
distributed
record
unit
Prior art date
Application number
PCT/CN2015/072429
Other languages
English (en)
French (fr)
Inventor
黄媛媛
龙卉
孙向东
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2016095329A1 publication Critical patent/WO2016095329A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor

Definitions

  • the present invention relates to information maintenance technologies, and in particular, to a log recording system and a log recording operation method.
  • SDN Software Defined Network
  • OpenFlow a network switching model
  • the existing software-defined network adopts a centralized management method, and the log record data is only stored in a database of the software-defined network. Once the database is destroyed, the log record may be permanently lost and the data security is low;
  • the database is mainly Oracle (Oracle) database or Structured Query Language (SQL, Structured Query Language) service database, these two databases will occupy a large data space, and the hardware requirements are also high, increasing the amount of data in the software-defined network And maintenance difficulty.
  • the embodiment of the present invention is expected to provide a log recording system and a log recording operation method, which can at least solve the problem of low security of log records.
  • the embodiment of the invention provides a log recording system, comprising:
  • Distributed cache unit configured to cache log records, operate on log records based on log operation instructions, and synchronize log records to other distributed cache sections in a distributed environment -;
  • a log record writing unit configured to write a log record to the distributed cache unit
  • a log record query unit configured to query log records in the distributed cache unit.
  • system further includes: a log record deletion unit configured to receive a log deletion instruction, and delete log information corresponding to the deletion instruction in the distributed cache unit.
  • the distributed cache unit includes:
  • the information receiving module is configured to receive a log recording operation instruction, and operate the log record according to the log recording operation instruction;
  • a distributed storage module configured to save log records
  • An information sending module configured to synchronize logging to distributed nodes in a distributed environment.
  • the log record writing unit includes:
  • An interface module configured to receive an operation instruction of the log record information, and send the operation instruction to the distributed cache unit;
  • a write module configured to write log records to the distributed cache unit.
  • the distributed cache unit, the log record writing unit, the log record query unit, the log record deleting unit, the information receiving module, the distributed storage module, the information sending module, the interface module, and the writing module may perform central processing when performing processing.
  • CPU Central Processing Unit
  • DSP Digital Singnal Processor
  • FPGA Programmable Array
  • the embodiment of the invention further provides a log record writing method, including:
  • the receiving and storing the log record sent by the sending end of the log record comprises: receiving the log record sent by the sending end of the log record, generating the log identifier of the log record according to the current time, and saving the log identifier To the log record, put the log record into a log queue; the write module of the log record writing unit reads the log record from the log queue according to the time sequence corresponding to the log identifier, and saves the log record to the distributed storage Module.
  • the synchronizing the saved log records to other distributed cache nodes in the distributed environment includes: saving the log records to a log file according to a distributed configuration requirement, and sending the information through the distributed cache unit.
  • the module synchronizes the log records to other distributed cache nodes in a distributed environment.
  • the method further includes: sending the log identifier to the log record sender.
  • the distributed configuration requirement includes a full synchronization type or a distributed type; when it is a full synchronization type, the distributed storage module synchronizes data to all nodes in the distributed environment; when it is a distributed type, the distributed storage The module configures the number of synchronization nodes. When the log records are synchronized, a set number of nodes are randomly selected from the distributed environment as the synchronization node.
  • the embodiment of the invention further provides a log record query method, including:
  • the log record viewing request corresponding to the log record viewing request is sent to the log record querying end; if yes, the log record corresponding to the log record viewing request is sent to the log record querying end; otherwise, the null information is returned. Record the query side for the log.
  • the sending the log record corresponding to the log record viewing request to the log record querying end includes: sending, by the information sending module, the log record corresponding to the log record viewing request to the log record
  • the query unit sends the log record corresponding to the log record viewing request to the log record query end.
  • the returning null information to the log record querying end includes: sending, by the information sending module, null information to the log record query unit, where the log record query unit The null information is sent to the log record query end.
  • the logging system and the log recording operation method provided by the embodiments of the present invention implement the distributed storage of the log records through the distributed cache unit, so that the log records are saved to the nodes in the distributed environment, thereby greatly improving the security of the log records.
  • Sex also increases the storage capacity of the log record, and can also improve the write speed and query speed of the log record.
  • FIG. 1 is a schematic structural diagram of a structure of a log recording system according to Embodiment 1 of the present invention.
  • FIG. 2 is a schematic flowchart of an implementation process of a log record writing method according to Embodiment 2 of the present invention
  • FIG. 3 is a schematic flowchart of an implementation process of a log record query method according to Embodiment 3 of the present invention.
  • FIG. 4 is a structural diagram of a logging system in a software-defined network according to Embodiment 4 of the present invention.
  • FIG. 5 is a flowchart of a method for writing a log record to a distributed cache unit according to Embodiment 4 of the present invention.
  • FIG. 6 is a structural information transmission structure diagram of a method for writing a log record to a distributed cache unit according to Embodiment 4 of the present invention.
  • FIG. 7 is a structural diagram of information transmission of a method for querying a log record by a distributed cache unit according to Embodiment 4 of the present invention.
  • the embodiment provides a logging system. As shown in FIG. 1 , the system includes:
  • the distributed cache unit 101 is configured to cache log records, operate the log records according to the log record operation instructions, and synchronize the log records to other distributed caches in the distributed environment. node;
  • distributed cache nodes each of which includes a distributed cache unit; the synchronization is to synchronously store all log records to a local distributed cache unit and other distributed cache nodes.
  • Distributed buffer unit Distributed buffer unit.
  • the distributed cache unit 101 can be an open source data grid platform, and the open source data grid platform exposes a simple data structure to store objects.
  • the distributed cache unit 101 can cache clusters of software-defined networks and expose large amounts of memory. By distributing a fixed number of copies for each distributed cache node, distributed cache functionality is provided and the scalability of the software-defined network is improved.
  • the cluster of the software-defined network is equivalent to connecting a plurality of software-defined networks to process data together.
  • a log record writing unit 102 configured to write a log record to the distributed cache unit 101;
  • the log record writing unit 102 is called by other function modules in the software definition network, and saves log records sent by other function modules to the distributed cache unit 101; here, the other function modules may be security monitoring modules and event queries. Modules and more.
  • the log record query unit 103 is configured to query the log records in the distributed cache unit 101 to obtain the required log records.
  • the log record query unit 103 acquires the required log record from the distributed cache unit 101 in accordance with certain filter conditions.
  • the screening condition is preset and carried in the log record query instruction;
  • the specific filter condition may be preset according to the name, the type, the event that occurred, and the like.
  • the system can implement distributed storage of log records, so that the log records are saved on multiple distributed cache nodes, thereby improving the security of the log records.
  • the distributed cache unit 101 itself has a data clearing function.
  • the distributed cache unit 101 automatically sets the time to be the oldest. The extra log data is cleared.
  • the log recording system of the embodiment may further include a log record deleting unit 104 configured to receive a log delete command, and delete the log record corresponding to the delete command in the distributed cache unit 101. .
  • the log record deleting unit 104 can actively delete the log records that meet the deletion conditions (such as time, type, data size, and the like) according to the preset deletion rule, and avoid the log records caused by the large amount of useless data.
  • the data processing speed is slow, and the system hardware requirements are high.
  • the distributed cache unit 101 includes: an information receiving module, configured to receive a log recording operation instruction, and operate the log record according to the log recording operation instruction; the distributed storage module is configured to save the log record; A sending module configured to synchronize logging to other distributed nodes in a distributed environment.
  • the log record writing unit 102 includes: an interface module configured to receive an operation instruction of the log record, and send the operation instruction to the distributed cache unit 101; and the write module is configured to write the log record The distributed cache unit 101 is described.
  • This embodiment is a log record writing method proposed on the basis of Embodiment 1. As shown in FIG. 2, the method includes the following steps:
  • Step S201 Receive and save a log record sent by the log record sending end
  • the interface module of the log record writing unit may first receive the log record sent by the log record sending end, generate the log identifier of the log record according to the current time, and save the log identifier to the log. Record, put the log record into the log queue;
  • the write module of the log record writing unit reads the log record from the log queue according to the time sequence corresponding to the log identifier, and saves the log record to the distributed storage module;
  • the log record sending end may be each module in the application layer in the software-defined network; the content of the log record includes operation data of the software-defined network, security event data and/or system event data, etc.; the log queue is used to write the log record
  • the write order into the distributed cache unit is sorted according to the time corresponding to the log identifier.
  • the log identifier may be sent to the log record sender.
  • Step S202 Synchronizing the saved log records to other distributed cache nodes in a distributed environment
  • the distributed storage module includes a configuration file, and the configuration file may be configured to log a full synchronization type or a distributed type; when it is a full synchronization type, the distributed storage module synchronizes data to all nodes in the distributed environment, and the distributed The cache nodes belong to the nodes in the distributed environment.
  • the type is distributed, the number of synchronization nodes needs to be configured.
  • the log records are synchronized, a set number of nodes are randomly selected from the distributed environment as the synchronization node.
  • the configuration can be configured according to the network environment where the current software defines the network, or according to the storage requirements of the log records.
  • This embodiment is a logging query method proposed on the basis of Embodiment 1. As shown in FIG. 3, the method includes the following steps:
  • Step S301 Receive a log record viewing instruction sent by the query end of the log record, and send a log record viewing request according to the log record viewing instruction;
  • the receiving log record viewing instruction and the sending log record viewing request may be completed by a log record query unit;
  • the sending log record viewing request is a letter to a distributed cache unit
  • the information receiving module sends;
  • Step S302 According to the log record viewing request, check whether there is a log record corresponding to the log record viewing request, and if yes, send the log record corresponding to the log record viewing request to the log record query end; otherwise, Return null information to the log record query side.
  • the null information refers to information that is returned to the query end of the log record when the corresponding log record cannot be found according to the log record viewing request.
  • the viewing of the log record corresponding to the presence of the log record viewing request may be viewed by the information receiving module in the distributed storage module;
  • the sending by the information sending module, the log record corresponding to the log record viewing request to the log record query, by sending the log record corresponding to the log record viewing request to the log record querying end a unit, the log record query unit sends a log record corresponding to the log record viewing request to the log record query end;
  • the returning null information to the log record querying end includes: sending, by the information sending module, the null information to the log record querying unit, where the log record querying unit sends the null information to the log record querying end.
  • the method in this embodiment may further include the step of deleting the log record by the log record deletion unit.
  • the distributed cache unit 101 itself has a data clearing function. When the capacity of the saved log record exceeds the set capacity, the distributed cache unit 101 automatically clears the log data with the earliest time.
  • the log record deleting unit 104 may also actively delete the log records that meet the deletion conditions (such as time, type, data size, etc.) according to a preset deletion rule, thereby avoiding uselessness. The amount of data that is caused by the large amount of data is slow, and the system hardware requirements are high.
  • the structure of the logging system in the software-defined network in the embodiment of the present invention is as shown in FIG. 4.
  • the software-defined network in the embodiment of the present invention is divided into three layers:
  • Supporting layer at the bottom of the software-defined network, including the service loading framework and the supporting module such as the distributed cache unit described in Embodiment 1, which is the basis for loading the subsystems of the entire system, and the service layer and the application layer depend on the supporting layer. .
  • Service layer Located in the middle layer of the software-defined network, it is used to provide services to the application layer. It needs to rely on the support layer, including the log record writing unit, the log record query unit and other service units described in Embodiment 1.
  • Application layer It is mainly a business application module in a software-defined network.
  • the application layer depends on both the service layer and the support layer.
  • FIG. 5 a flowchart of a method for writing a log record to a distributed cache unit in this embodiment is as shown in FIG. 5, and includes the following steps:
  • Step S4011 Each service application module of the application layer in the software-defined network encapsulates an operation, a security event, or a system event into a log record, and invokes an interface module of the log record writing unit to transmit a log record;
  • Step S4012 The interface module generates a log identifier according to the current timestamp, sets the log identifier to the incoming log record, and then puts the log record into the log queue.
  • Step S4013 The interface module returns a log identifier to the service application module of the application layer.
  • Step S4014 The write module of the log record writing unit periodically determines whether there is data in the log queue, and if so, retrieves the log record of the first queue;
  • Step S4015 The information receiving module puts the log object into the distributed storage module
  • Step S4016 The distributed storage module synchronizes the log records to other distributed cache nodes in the distributed environment according to the distributed configuration requirements;
  • the distributed configuration requirement is configured in a configuration file of a distributed storage module
  • the distributed configuration requirement includes a full synchronization type or a distributed type; when it is a full synchronization type, the distributed storage module synchronizes data to all nodes in the distributed environment; when it is a distributed type, it also needs to configure the number of synchronization nodes.
  • a set number of nodes are randomly selected from the distributed environment as the synchronization node.
  • the specific content of the configuration can be selected according to the actual situation in the engineering environment.
  • Step S4017 The distributed cache unit writes the log record to the log record file according to the distributed configuration requirement
  • the log record information is not only placed in the cache, but also in the log record file; when the system is powered off or restarted for other reasons, the distributed cache unit can also obtain log records from the file, so that the log record will not be Lost due to system reboot.
  • the corresponding information transfer structure is shown in Figure 6.
  • Step 4021 The Hypertext Transfer Protocol (HTTP) in the software-defined network requests the distribution processor to distribute the log record viewing instruction to the information receiving module of the distributed cache unit.
  • HTTP Hypertext Transfer Protocol
  • HTTP request distribution processor is equivalent to the log record query end in the above embodiment
  • Step 4022 The information receiving module checks, according to the log record viewing request, whether the log record corresponding to the log record viewing request exists in the distributed storage module, and if yes, the log record is sent by the information sending module.
  • the log record corresponding to the request is sent to the log record query unit, and the log record query unit sends the log record corresponding to the log record view request to the log record query end; otherwise, the information sending module sends the The log record query unit sends null information, and the log record query unit sends the null information to the log record query end.
  • the corresponding information transfer structure is shown in Figure 7.
  • each distribution Logging is done on the node, so you can get log records from distributed cache units or distributed cache units of other distributed cache nodes in a distributed environment.
  • the disclosed apparatus and method may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner such as: multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored or not executed.
  • the coupling, or direct coupling, or communication connection of the components shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, and may be electrical, mechanical or other forms. of.
  • the units described above as separate components may or may not be physically separated, and the components displayed as the unit may or may not be physical units, that is, may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing module, or each unit may be separately used as one unit, or two or more units may be integrated into one unit; the above integration
  • the unit can be implemented in the form of hardware or in the form of hardware plus software functional units.
  • the foregoing program may be stored in a computer readable storage medium, and when executed, the program includes The foregoing steps of the method embodiment; and the foregoing storage medium includes: a removable storage device, a read-only memory (ROM), a magnetic disk, or an optical disk, and the like, which can store program codes.
  • ROM read-only memory
  • the logging system and the log recording operation method provided by the embodiments of the present invention implement the distributed storage of the log records through the distributed cache unit, so that the log records are saved to the nodes in the distributed environment, thereby greatly improving the security of the log records.
  • Sex also increases the storage capacity of the log record, and can also improve the write speed and query speed of the log record.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

本发明公开了一种日志记录系统，包括：分布式缓存单元，配置为缓存日志记录，根据日志记录操作指令对日志记录进行操作，并将日志记录同步到分布式环境中的其它分布式缓存节点；日志记录写入单元，配置为向所述分布式缓存单元中写入日志记录；日志记录查询单元，配置为查询所述分布式缓存单元中的日志记录。本发明还同时公开了日志记录写入方法和日志记录查询方法。

Description

一种日志记录系统及日志记录操作方法 技术领域
本发明涉及信息维护技术，尤其涉及一种日志记录系统及日志记录操作方法。
背景技术
软件定义网络(SDN，Software Defined Network)是一种网络创新架构，其核心技术OpenFlow(一种网络交换模型)通过将网络设备控制面与转发面分离开来，实现了网络流量的灵活控制，为核心网络及应用的创新提供了良好的平台。
现有软件定义网络采用集中式管理的方法，日志记录数据只保存一份在软件定义网络的数据库中，一旦数据库遭到破坏，日志记录可能永久性丢失，数据安全性低；另外，目前采用的数据库主要为甲骨文(Oracle)数据库或结构化查询语言(SQL，Structured Query Language)服务数据库，这两种数据库会占用很大的数据空间，并且对硬件要求也高，增加了软件定义网络的数据量和维护难度。
发明内容
有鉴于此，本发明实施例期望提供一种日志记录系统及日志记录操作方法，至少能解决日志记录安全性低等问题。
本发明实施例的技术方案是这样实现的：
本发明实施例提供了一种日志记录系统，包括：
分布式缓存单元，配置为缓存日志记录，根据日志记录操作指令对日志记录进行操作，并将日志记录同步到分布式环境中的其它分布式缓存节 -；
日志记录写入单元，配置为向所述分布式缓存单元中写入日志记录；
日志记录查询单元，配置为查询所述分布式缓存单元中的日志记录。
上述方案中，所述系统还包括：日志记录删除单元，配置为接收日志删除指令，并删除所述分布式缓存单元中对应所述删除指令的日志信息。
上述方案中，所述分布式缓存单元包括：
信息接收模块，配置为接收日志记录操作指令，并根据所述日志记录操作指令对日志记录进行操作；
分布式存储模块，配置为保存日志记录；
信息发送模块，配置为将日志记录同步到分布式环境中的分布式节点。
上述方案中，所述日志记录写入单元包括：
接口模块，配置为接收日志记录信息的操作指令，并将所述操作指令发送给所述分布式缓存单元；
写入模块，配置为将日志记录写入所述分布式缓存单元。
上述分布式缓存单元、日志记录写入单元、日志记录查询单元、日志记录删除单元、信息接收模块、分布式存储模块、信息发送模块、接口模块、写入模块在执行处理时，可以采用中央处理器(CPU，Central Processing Unit)、数字信号处理器(DSP，Digital Singnal Processor)或可编程逻辑阵列(FPGA，Field－Programmable Gate Array)实现。
本发明实施例还提供了一种日志记录写入方法，包括：
接收并保存日志记录发送端发来的日志记录；
将保存的日志记录同步到分布式环境下的其它分布式缓存节点。
上述方案中，所述接收并保存日志记录发送端发来的日志记录包括：接收日志记录发送端发来的日志记录，根据当前时间生成所述日志记录的日志标识，并将所述日志标识保存至所述日志记录，将所述日志记录放入 日志队列；再由日志记录写入单元的写入模块按所述日志标识对应的时间的先后顺序，从所述日志队列中读取所述日志记录，并将所述日志记录保存至分布式存储模块。
上述方案中，所述将保存的日志记录同步到分布式环境下的其它分布式缓存节点包括：根据分布式配置要求将所述日志记录保存至日志文件中，并通过分布式缓存单元的信息发送模块将所述日志记录同步到分布式环境下的其它分布式缓存节点。
上述方案中，所述将所述日志记录放入日志队列之后还包括：将所述日志标识发送给所述日志记录发送方。
上述方案中，所述分布式配置要求包括全同步类型或分布式类型；当为全同步类型时，分布式存储模块向分布式环境中所有节点同步数据；当为分布式类型时，分布式存储模块配置同步节点的数量，在日志记录同步时，从分布式环境中随机选取设定数量的节点作为同步节点。
本发明实施例又提供了一种日志记录查询方法，包括：
接收日志记录查询端发来的日志记录查看指令，并根据所述日志记录查看指令发送日志记录查看请求；
根据所述日志记录查看请求查看是否存在所述日志记录查看请求对应的日志记录，若存在，则将所述日志记录查看请求对应的日志记录发送给所述日志记录查询端；否则，返回空信息给所述日志记录查询端。
上述方案中，所述将所述日志记录查看请求对应的日志记录发送给所述日志记录查询端包括：通过所述信息发送模块将所述日志记录查看请求对应的日志记录发送给所述日志记录查询单元，所述日志记录查询单元将所述日志记录查看请求对应的日志记录发送给所述日志记录查询端。
上述方案中，所述返回空信息给所述日志记录查询端包括：通过信息发送模块向所述日志记录查询单元发送空信息，所述日志记录查询单元将 所述空信息发送给所述日志记录查询端。
本发明实施例所提供的日志记录系统及日志记录操作方法，将日志记录通过分布式缓存单元实现分布式的保存，使得日志记录保存到分布式环境下的节点中，大大提高了日志记录的安全性；同时提高了日志记录的存储容量，还能提高日志记录的写入速度和查询速度。
附图说明
图1为本发明实施例1日志记录系统的组成结构示意图；
图2为本发明实施例2日志记录写入方法的实现流程示意图；
图3为本发明实施例3日志记录查询方法的实现流程示意图；
图4为本发明实施例4日志记录系统在软件定义网络中的结构图；
图5为本发明实施例4向分布式缓存单元写入日志记录的方法的流程图；
图6为本发明实施例4向分布式缓存单元写入日志记录的方法的信息传递结构图；
图7为本发明实施例4向分布式缓存单元查询日志记录的方法的信息传递结构图。
具体实施方式
以下结合说明书附图及具体实施例对本发明的技术方案做进一步的详细阐述。
实施例1
为了解决日志记录安全性低的问题，本实施例提供了一种日志记录系统，如图1所示，所述系统包括：
分布式缓存单元101，配置为缓存日志记录，根据日志记录操作指令对日志记录进行操作，并将日志记录同步到分布式环境中的其它分布式缓存 节点；
这里，所述分布式缓存节点可以有多个，每个分布式缓存节点均包括分布式缓存单元；所述同步是将全部日志记录同步存储到本地的分布式缓存单元和其它各个分布式缓存节点的分布式缓存单元中。
分布式缓存单元101可以是一个开源的数据网格平台，开源的数据网格平台公开了一个简单的数据结构来存储对象。在分布式环境下，分布式缓存单元101可以将软件定义网络的集群缓存起来并公开大容量的内存。通过为每个分布式缓存节点分配固定数量的副本，提供了分布式缓存的功能，并且提升了软件定义网络的可伸缩性。其中，所述软件定义网络的集群相当于将多个软件定义网络连接起来共同处理数据。
日志记录写入单元102，配置为向所述分布式缓存单元101中写入日志记录；
日志记录写入单元102被软件定义网络中其它功能模块调用，并将其它功能模块发来的日志记录保存到分布式缓存单元101中；这里，所述其它功能模块可以是安全监控模块和事件查询模块等等。
日志记录查询单元103，配置为查询所述分布式缓存单元101中的日志记录，获取所需的日志记录。
日志记录查询单元103按照一定的筛选条件从分布式缓存单元101中获取所需的日志记录。
这里，所述筛选条件是预先设定的并携带在日志记录查询指令中；具体筛选条件可根据名称、类型、发生的事件等等预先设定。
本实施例系统能够实现对日志记录的分布式保存，使得日志记录保存在多个分布式缓存节点上，提高日志记录的安全性。
本实施例中，分布式缓存单元101自身具有数据清除功能，当保存的日志记录的容量超过设定容量时，分布式缓存单元101会自动对时间最早 的多余的日志记录数据进行清除。
为了对日志记录数据的进一步控制，本实施例的日志记录系统还可以包括日志记录删除单元104，配置为接收日志删除指令，并删除所述分布式缓存单元101中对应所述删除指令的日志记录。
需要说明的是，日志记录删除单元104可以根据预先设定的删除规则对符合删除条件(如时间、类型、数据大小等属性)的日志记录进行主动删除，避免由于无用数据量多导致的日志记录数据处理速度慢，对系统硬件要求高等不足。
具体的，所述分布式缓存单元101包括：信息接收模块，配置为接收日志记录操作指令，并根据所述日志记录操作指令对日志记录进行操作；分布式存储模块，配置为保存日志记录；信息发送模块，配置为将日志记录同步到分布式环境中的其它分布式节点。
所述日志记录写入单元102包括：接口模块，配置为接收日志记录的操作指令，并将所述操作指令发送给所述分布式缓存单元101；写入模块，配置为将日志记录写入所述分布式缓存单元101。
实施例2
本实施例是在实施例1的基础上提出的一种日志记录写入方法，如图2所示，该方法包括以下步骤：
步骤S201：接收并保存日志记录发送端发来的日志记录；
具体的，本步骤可先由日志记录写入单元的接口模块接收日志记录发送端发来的日志记录，根据当前时间生成所述日志记录的日志标识，并将所述日志标识保存至所述日志记录，将所述日志记录放入日志队列；
再由日志记录写入单元的写入模块按所述日志标识对应的时间的先后顺序，从所述日志队列中读取所述日志记录，并将所述日志记录保存至分布式存储模块；
这里，日志记录发送端可以是软件定义网络中应用层中的各个模块；日志记录的内容包括软件定义网络的操作数据、安全事件数据和/或系统事件数据等；日志队列用于对日志记录写入分布式缓存单元的写入顺序按日志标识对应的时间进行排序。为了让日志记录发送方了解日志记录在分布式缓存单元中的记录参数，可以将所述日志标识发送给所述日志记录发送方。
步骤S202：将保存的日志记录同步到分布式环境下的其它分布式缓存节点；
本步骤可由所述分布式存储模块根据分布式配置要求将所述日志记录保存至日志文件中，并通过分布式缓存单元的信息发送模块将所述日志记录同步到分布式环境下的其它分布式缓存节点。
这里，分布式存储模块包括配置文件，配置文件可以配置为日志记录全同步类型或分布式类型；当为全同步类型时，分布式存储模块向分布式环境中所有节点同步数据，所述分布式缓存节点均属于分布式环境下的节点；当为分布式类型时还需要配置同步节点的数量，在日志记录同步时，从分布式环境中随机选取设定数量的节点作为同步节点，其中具体如何配置可根据当前软件定义网络所在的网络环境配置，也可以根据日志记录的保存要求配置。
实施例3
本实施例是在实施例1的基础上提出的一种日志记录查询方法，如图3所示，该方法包括以下步骤：
步骤S301：接收日志记录查询端发来的日志记录查看指令，并根据所述日志记录查看指令发送日志记录查看请求；
这里，所述接收日志记录查看指令和发送日志记录查看请求可由日志记录查询单元完成；所述发送日志记录查看请求是向分布式缓存单元的信 息接收模块发送；
步骤S302：根据所述日志记录查看请求查看是否存在所述日志记录查看请求对应的日志记录，若存在，则将所述日志记录查看请求对应的日志记录发送给所述日志记录查询端；否则，返回空信息给所述日志记录查询端。
其中，所述空信息是指无法按照日志记录查看请求找到对应的日志记录时，返回给日志记录查询端的信息。
这里，所述查看是否存在日志记录查看请求对应的日志记录可由所述信息接收模块在分布式存储模块中查看；
具体的，所述将所述日志记录查看请求对应的日志记录发送给所述日志记录查询端包括：通过所述信息发送模块将所述日志记录查看请求对应的日志记录发送给所述日志记录查询单元，所述日志记录查询单元将所述日志记录查看请求对应的日志记录发送给所述日志记录查询端；
所述返回空信息给所述日志记录查询端包括：通过信息发送模块向所述日志记录查询单元发送空信息，所述日志记录查询单元将所述空信息发送给所述日志记录查询端。
另外，为了实现对日志记录的主动删除，本实施例方法还可以包括通过日志记录删除单元删除日志记录的步骤。
具体的，分布式缓存单元101自身具有数据清除功能，当保存的日志记录的容量超过了设定容量时，分布式缓存单元101会自动对时间最早的多于的日志记录数据进行清除。为了对日志记录数据的进一步控制，还可以通过日志记录删除单元104根据预先设定的删除规则对符合删除条件(如时间、类型、数据大小等属性)的日志记录进行主动删除，避免了由于无用数据量多大导致的日志记录数据处理速度慢，对系统硬件要求高等不足。
实施例4
以下通过一个实际场景对本发明进行详细说明。
本发明实施例的日志记录系统在软件定义网络中的结构如图4所示，本发明实施例的软件定义网络分为三层架构：
支撑层：处于软件定义网络的最底层，包括服务加载框架和实施例1所述分布式缓存单元等支撑模块，这部分是整个系统各个子系统加载的基础，服务层和应用层都依赖支撑层。
服务层：处于软件定义网络的中间层，用于给应用层提供服务，需要依赖支撑层，包括实施例1中所述的日志记录写入单元、日志记录查询单元和其他服务单元。
应用层：主要是软件定义网络中的业务应用模块，应用层同时依赖服务层和支撑层。
基于上述结构，本实施例向分布式缓存单元写入日志记录的方法的流程图如图5所示，包括以下步骤：
步骤S4011：软件定义网络中应用层的各业务应用模块将操作、安全事件或系统事件封装成日志记录，调用日志记录写入单元的接口模块传入日志记录；
步骤S4012：接口模块根据当前时间戳生成日志标识，将日志标识设置到传入的日志记录中，再将日志记录放入日志队列；
步骤S4013：接口模块向应用层的业务应用模块返回日志标识；
步骤S4014：日志记录写入单元的写入模块循环判断日志队列中是否有数据，如果有，则取出最先入队列的日志记录；
步骤S4015：信息接收模块将日志对象放入分布式存储模块；
步骤S4016：分布式存储模块根据分布式配置要求，将日志记录同步到分布式环境中的其它分布式缓存节点；
这里，所述分布式配置要求是在分布式存储模块的配置文件中配置， 所述分布式配置要求包括全同步类型或分布式类型；当为全同步类型时，分布式存储模块向分布式环境中所有节点同步数据；当为分布式类型时还需要配置同步节点的数量，在日志记录同步时，从分布式环境中随机选取设定数量的节点作为同步节点。配置的具体内容可以根据工程环境中实际情况来做选择。
步骤S4017：分布式缓存单元根据分布式配置要求，将日志记录写到日志记录文件中；
这里，日志记录信息不仅放在缓存中，同时也放在日志记录文件中；当系统掉电或其他原因关闭重启时，分布式缓存单元还能从文件中获取到日志记录，使日志记录不会因为系统重启而丢失。对应的信息传递结构如图6所示。
本实施例通过分布式缓存单元查询日志记录的方法包括以下步骤：
步骤4021：软件定义网络中的超文本传送协议(HTTP，Hypertext transfer protocol)请求分发处理器向所述分布式缓存单元的信息接收模块分发日志记录查看指令；
这里，所述HTTP请求分发处理器相当于上述实施例中的日志记录查询端；
步骤4022：所述信息接收模块根据所述日志记录查看请求在所述分布式存储模块中查看是否存在所述日志记录查看请求对应的日志记录，若存在，则通过信息发送模块将所述日志记录查看请求对应的日志记录发送给所述日志记录查询单元，所述日志记录查询单元将所述日志记录查看请求对应的日志记录发送给所述日志记录查询端；否则，通过信息发送模块向所述日志记录查询单元发送空信息，所述日志记录查询单元将所述空信息发送给所述日志记录查询端。对应的信息传递结构如图7所示。
其中，如果所述日志记录查看请求配置的是复制模式，那么每个分布 式节点上都有日志记录)，那么就可以从分布式缓存单元或分布式环境中的其他分布式缓存节点的分布式缓存单元中获取日志记录。
在本申请所提供的几个实施例中，应该理解到，所揭露的设备和方法，可以通过其它的方式实现。以上所描述的设备实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，如：多个单元或组件可以结合，或可以集成到另一个系统，或一些特征可以忽略，或不执行。另外，所显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口，设备或单元的间接耦合或通信连接，可以是电性的、机械的或其它形式的。
上述作为分离部件说明的单元可以是、或也可以不是物理上分开的，作为单元显示的部件可以是、或也可以不是物理单元，即可以位于一个地方，也可以分布到多个网络单元上；可以根据实际的需要选择其中的部分或全部单元来实现本实施例方案的目的。
另外，在本发明各实施例中的各功能单元可以全部集成在一个处理模块中，也可以是各单元分别单独作为一个单元，也可以两个或两个以上单元集成在一个单元中；上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：移动存储设备、只读存储器(Read-Only Memory，ROM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本发明的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本发明的保护范围之内。因此，本发明 的保护范围应以所述权利要求的保护范围为准。
工业实用性
本发明实施例所提供的日志记录系统及日志记录操作方法，将日志记录通过分布式缓存单元实现分布式的保存，使得日志记录保存到分布式环境下的节点中，大大提高了日志记录的安全性；同时提高了日志记录的存储容量，还能提高日志记录的写入速度和查询速度。

Claims (12)

  1. 一种日志记录系统，所述系统包括：
    分布式缓存单元，配置为缓存日志记录，根据日志记录操作指令对日志记录进行操作，并将日志记录同步到分布式环境中的其它分布式缓存节点；
    日志记录写入单元，配置为向所述分布式缓存单元中写入日志记录；
    日志记录查询单元，配置为查询所述分布式缓存单元中的日志记录。
  2. 根据权利要求1所述的系统，其中，所述系统还包括：
    日志记录删除单元，配置为接收日志删除指令，并删除所述分布式缓存单元中对应所述删除指令的日志信息。
  3. 根据权利要求1所述的系统，其中，所述分布式缓存单元包括：
    信息接收模块，配置为接收日志记录操作指令，并根据所述日志记录操作指令对日志记录进行操作；
    分布式存储模块，配置为保存日志记录；
    信息发送模块，配置为将日志记录同步到分布式环境中的分布式节点。
  4. 根据权利要求1所述的系统，其中，所述日志记录写入单元包括：
    接口模块，配置为接收日志记录信息的操作指令，并将所述操作指令发送给所述分布式缓存单元；
    写入模块，配置为将日志记录写入所述分布式缓存单元。
  5. 一种日志记录写入方法，所述方法包括：
    接收并保存日志记录发送端发来的日志记录；
    将保存的日志记录同步到分布式环境下的其它分布式缓存节点。
  6. 根据权利要求5所述的方法，其中，所述接收并保存日志记录发送端发来的日志记录包括：
    接收日志记录发送端发来的日志记录，根据当前时间生成所述日志记 录的日志标识，并将所述日志标识保存至所述日志记录，将所述日志记录放入日志队列；再由日志记录写入单元的写入模块按所述日志标识对应的时间的先后顺序，从所述日志队列中读取所述日志记录，并将所述日志记录保存至分布式存储模块。
  7. 根据权利要求5所述的方法，其中，所述将保存的日志记录同步到分布式环境下的其它分布式缓存节点包括：
    根据分布式配置要求将所述日志记录保存至日志文件中，并通过分布式缓存单元的信息发送模块将所述日志记录同步到分布式环境下的其它分布式缓存节点。
  8. 根据权利要求6所述的方法，其中，所述将所述日志记录放入日志队列之后还包括：
    将所述日志标识发送给所述日志记录发送方。
  9. 根据权利要求7所述的方法，其中，所述分布式配置要求包括全同步类型或分布式类型；当为全同步类型时，分布式存储模块向分布式环境中所有节点同步数据；当为分布式类型时，分布式存储模块配置同步节点的数量，在日志记录同步时，从分布式环境中随机选取设定数量的节点作为同步节点。
  10. 一种日志记录查询方法，所述方法包括：
    接收日志记录查询端发来的日志记录查看指令，并根据所述日志记录查看指令发送日志记录查看请求；
    根据所述日志记录查看请求查看是否存在所述日志记录查看请求对应的日志记录，若存在，则将所述日志记录查看请求对应的日志记录发送给所述日志记录查询端；否则，返回空信息给所述日志记录查询端。
  11. 根据权利要求10所述的方法，其中，所述将所述日志记录查看请求对应的日志记录发送给所述日志记录查询端包括：
    通过所述信息发送模块将所述日志记录查看请求对应的日志记录发送给所述日志记录查询单元，所述日志记录查询单元将所述日志记录查看请求对应的日志记录发送给所述日志记录查询端。
  12. 根据权利要求10所述的方法，其中，所述返回空信息给所述日志记录查询端包括：
    通过信息发送模块向所述日志记录查询单元发送空信息，所述日志记录查询单元将所述空信息发送给所述日志记录查询端。
PCT/CN2015/072429 2014-12-18 2015-02-06 一种日志记录系统及日志记录操作方法 WO2016095329A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201410797673.4 2014-12-18
CN201410797673.4A CN105760398A (zh) 2014-12-18 2014-12-18 一种日志记录系统及日志记录操作方法

Publications (1)

Publication Number Publication Date
WO2016095329A1 true WO2016095329A1 (zh) 2016-06-23

Family

ID=56125721

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/072429 WO2016095329A1 (zh) 2014-12-18 2015-02-06 一种日志记录系统及日志记录操作方法

Country Status (2)

Country Link
CN (1) CN105760398A (zh)
WO (1) WO2016095329A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110209969A (zh) * 2019-05-08 2019-09-06 口碑(上海)信息技术有限公司 菜谱信息的缓存预热方法及装置

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107015872B (zh) * 2016-12-09 2018-03-16 上海壹账通金融科技有限公司 监控数据的处理方法及装置
CN106649729A (zh) * 2016-12-23 2017-05-10 深圳市金证科技股份有限公司 日志运行方法及系统、客户端和服务器
CN110019514B (zh) * 2017-11-07 2023-05-09 阿里巴巴集团控股有限公司 数据同步方法、装置以及电子设备

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103177116A (zh) * 2013-04-08 2013-06-26 国电南瑞科技股份有限公司 一种基于两级索引的分布式日志处理和查询方法
CN103473258A (zh) * 2013-06-01 2013-12-25 西安邮电大学 云存储文件系统
US20140279931A1 (en) * 2013-03-15 2014-09-18 Amazon Technologies, Inc. System-wide checkpoint avoidance for distributed database systems

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102611611B (zh) * 2011-12-13 2015-08-12 北京安天电子设备有限公司 一种日志缓存系统及方法
CN102411639B (zh) * 2011-12-31 2013-05-08 曙光信息产业股份有限公司 元数据的多副本存储管理方法和系统
US8719225B1 (en) * 2012-01-17 2014-05-06 Amazon Technologies, Inc. System and method for log conflict detection and resolution in a data store
CN102970158B (zh) * 2012-11-05 2017-02-08 广东睿江云计算股份有限公司 日志存储与处理的方法及日志服务器
CN103490923B (zh) * 2013-09-11 2016-09-14 中国联合网络通信集团有限公司 日志文件的读写方法、装置及系统
CN103870570B (zh) * 2014-03-14 2018-04-27 上海艾讯云计算有限公司 一种基于远程日志备份的 HBase 数据可用性及持久性的方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140279931A1 (en) * 2013-03-15 2014-09-18 Amazon Technologies, Inc. System-wide checkpoint avoidance for distributed database systems
CN103177116A (zh) * 2013-04-08 2013-06-26 国电南瑞科技股份有限公司 一种基于两级索引的分布式日志处理和查询方法
CN103473258A (zh) * 2013-06-01 2013-12-25 西安邮电大学 云存储文件系统

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110209969A (zh) * 2019-05-08 2019-09-06 口碑(上海)信息技术有限公司 菜谱信息的缓存预热方法及装置

Also Published As

Publication number Publication date
CN105760398A (zh) 2016-07-13

Similar Documents

Publication Publication Date Title
WO2019154394A1 (zh) 分布式数据库集群系统、数据同步方法及存储介质
US11093455B2 (en) Log-structured storage systems
US11423015B2 (en) Log-structured storage systems
US11893264B1 (en) Methods and systems to interface between a multi-site distributed storage system and an external mediator to efficiently process events related to continuity
TWI737395B (zh) 日誌結構儲存系統及方法
ES2881606T3 (es) Sistema de ficheros geográficamente distribuido que usa replicación de espacio de nombres coordinada
US10885022B1 (en) Log-structured storage systems
CN103138912B (zh) 数据同步方法及系统
US11294881B2 (en) Log-structured storage systems
US9367261B2 (en) Computer system, data management method and data management program
CN105493474B (zh) 用于支持用于同步分布式数据网格中的数据的分区级别日志的系统及方法
WO2016095329A1 (zh) 一种日志记录系统及日志记录操作方法
JP5686034B2 (ja) クラスタシステム、同期制御方法、サーバ装置および同期制御プログラム
CN104506625A (zh) 一种提升云数据库元数据节点可靠性的方法
CN103458044A (zh) 一种面向广域网环境下多存储集群的元数据共享管理方法
WO2016177231A1 (zh) 基于双主控的主备倒换方法及装置
JP2016521427A (ja) ピアツーピアデータ複製用の方法、デバイス、およびシステム、ならびにマスタノード切替え用の方法、デバイス、およびシステム
CN103795801A (zh) 一种基于实时应用集群的元数据集群设计方法
US20210081320A1 (en) Log-structured storage systems
WO2017096942A1 (zh) 一种文件存储系统、数据调度方法及数据节点
WO2023103190A1 (zh) 一种多级联动人工智能平台样本模型透明共享装置
CN106855869A (zh) 一种实现数据库高可用的方法、装置和系统
WO2023138213A1 (zh) 一种基于地铁车站系统的数仓数据查询方法及装置
CN117061535A (zh) 多活构架数据同步方法、装置、计算机设备和存储介质
CN114741367B (zh) 数据同步方法、装置、电子设备、存储介质和程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15868846

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15868846

Country of ref document: EP

Kind code of ref document: A1