WO2021139280A1 - 基于zookeeper的日志处理方法、装置、计算机设备和存储介质 - Google Patents

基于zookeeper的日志处理方法、装置、计算机设备和存储介质 Download PDF

Info

Publication number
WO2021139280A1
WO2021139280A1 PCT/CN2020/119369 CN2020119369W WO2021139280A1 WO 2021139280 A1 WO2021139280 A1 WO 2021139280A1 CN 2020119369 W CN2020119369 W CN 2020119369W WO 2021139280 A1 WO2021139280 A1 WO 2021139280A1
Authority
WO
WIPO (PCT)
Prior art keywords
server
task
node
zookeeper
list
Prior art date
Application number
PCT/CN2020/119369
Other languages
English (en)
French (fr)
Inventor
严艺辉
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2021139280A1 publication Critical patent/WO2021139280A1/zh

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • This application relates to the field of cloud technology for big data. This application particularly relates to a zookeeper-based log processing method, device, computer equipment, and storage medium.
  • the log is a record used to record the operating status of the computer system.
  • the important value information extracted by analyzing the log can be used to analyze human behavior, prevent safety and repair defects, etc. Because logs are usually massive and complex, and the logs generated by each computer system are different. Therefore, it is extremely technically difficult to manage and process logs efficiently. Traditionally, the development of a log management platform is dedicated to the management and processing of logs.
  • this application provides a zookeeper-based log processing method, the method includes: the zookeeper master node monitors the failed task list slave node, and when the failed task list is captured, the server list slave node receives the registration in the server list slave node When the server submits the failure log, query the registered server with the lowest current load among the slave nodes of the server list; the zookeeper master node assigns the failure log to the task of the task implementation list slave node corresponding to the registered server with the lowest current load Among the child nodes, when the registration server with the lowest current load determines that the failure log belongs to a local task, the failure log is processed.
  • this application provides a zookeeper-based log processing device, the device includes: a server list slave node, used to manage registration servers; a failed task list slave node, used to receive all of the server list slave nodes The failure log submitted by the registration server; the zookeeper master node is used to monitor the failed task list slave node, and when the failed task list slave node receives the failure log, query the server list from the current node The registration server with the lowest load; and, assigning the failure log to the task subnode corresponding to the registration server with the lowest current load in the task implementation list, so that the registration server with the lowest current load determines that the failure log belongs to a local task When processing the failure log; the task implementation list slave node is used to receive the failure log distributed by the zookeeper master node.
  • the present application also provides a zookeeper-based log processing system, the system includes a zookeeper server and a registration server, the registration server is a log processing server registered to the slave node of the server list in the zookeeper server;
  • the registration server is used to submit the failed log processing to the failed task list slave node in the zookeeper server through the blocking queue when the processing log fails;
  • the zookeeper server is configured to use the zookeeper master node to monitor the failed task list slave node, and when the zookeeper master node captures the failed task list slave node and receives the failure log, query the server list from the current slave node The registration server with the lowest load; and, using the zookeeper master node to distribute the failure log to the task sub-nodes corresponding to the registration server with the lowest current load in the task implementation list slave node;
  • the registration server is further configured to process the failure log when it is determined that the failure log allocated by the zookeeper server to the corresponding task subnode belongs to a local task.
  • this application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, a zookeeper-based log processing method is implemented; wherein, the zookeeper-based log
  • the processing method includes: the zookeeper master node listens to the failed task list slave node, and when the failed task list slave node receives the failure log submitted by the server list slave node registration server, query the current load of the server list slave node The lowest registration server; the zookeeper master node assigns the failure log to the task subnodes in the task implementation list slave node corresponding to the registration server with the lowest current load, so that the registration server with the lowest current load determines that the failure log belongs to In the case of a local task, process the failure log.
  • the above-mentioned zookeeper-based log processing method, device, computer equipment and storage medium monitor and capture the failed task list through the zookeeper master node.
  • the query server list is current from the node.
  • the registration server with the lowest load then assigns the failure log to the task sub-node in the task implementation list from the node corresponding to the registration server with the lowest current load, so that the registration server with the lowest current load determines that the failure log belongs to the local task, and the failure log is processed .
  • This method implements distributed processing of failed logs through the master and slave nodes of the zookeeper service, so that the failed logs are quickly sent to zookeeper, and then the server with the lowest load is selected through the zookeeper service to process the failed logs, avoiding doing it on a machine with a higher load. Needless waiting and retrying can effectively improve load balancing and improve processing efficiency.
  • Figure 1 is a zookeeper-based log processing system in an embodiment of the application
  • FIG. 2 is a schematic flowchart of a zookeeper-based logging method in an embodiment of this application;
  • FIG. 3 is a schematic flowchart of a zookeeper-based logging method in another embodiment of the application.
  • FIG. 4 is a schematic diagram of the work flow of the master and slave nodes of the zookeeper service in an embodiment of the application;
  • FIG. 5 is a structural block diagram of a zookeeper-based logging device in an embodiment of this application.
  • Figure 6 is a structural block diagram of a zookeeper-based logging device in another embodiment of the application.
  • Figure 7 is a diagram of the internal structure of the zookeeper server in an embodiment of the application.
  • the zookeeper-based log processing method provided in this application relates to the field of cloud technology for big data, and can be applied to the zookeeper-based log processing system as shown in FIG. 1.
  • the registration server 104 is a log processing server registered to the zookeeper server 102 for processing logs, and the zookeeper server 102 communicates with the registration server 104 through the network.
  • a log processing failure occurs when the registration server 104 processes the log, it submits the failed log of the processing failure to the failed task list slave node of the zookeeper server 102 through the blocking queue.
  • the zookeeper master node in the zookeeper server 102 listens to the failed task list slave node.
  • the failed task list slave node receives the failure log submitted by the server list slave registration server 104 in the node, query the server list from the node with the lowest current load Registration server; the zookeeper master node of the zookeeper server 102 distributes the failure log to the task sub-nodes corresponding to the registration server with the lowest current load among the slave nodes of the task implementation list.
  • the registration server 104 with the lowest current load determines that the failure log in the task child node belongs to the local task, it processes the failure log.
  • the zookeeper server 102 and the registration server 104 can be implemented by independent servers or a server cluster composed of multiple servers. It should be understood that a zookeeper server cluster composed of multiple zookeeper servers 102, each zookeeper server 102 in the zookeeper server cluster includes a zookeeper master node, a failed task list slave node, a task implementation list slave node, a server list slave node, and Task status list slave node.
  • the failed task list slave node, the task implementation list slave node, the server list slave node, and the task status list slave node are slave nodes in the zookeeper service.
  • the registration server 104 submits the failure log of the processing failure to the failed task list slave node of the zookeeper server 102 through the blocking queue, including: writing the failure log of the processing failure to the blocking queue; when it is determined that the blocking queue is satisfied When submitting the request, submit the failure log in the blocking queue to the failed task list slave node of the zookeeper server 102.
  • the registration server 104 determines that the log processing fails, it submits the failure log to the local blocking queue LinkedBlockingQueue. Then, when the blocking queue is full or exceeds the preset time threshold, it is determined that the blocking queue meets the submission requirement. The registration server 104 submits the failure log in the blocking queue to the failed task list slave node of the zookeeper server 102.
  • the registration server 104 determines whether the failure log in the task subnode belongs to a local task, including: monitoring and determining that the zookeeper server assigns the failure log to the corresponding task subnode, obtaining the task identifier corresponding to the failure log; Generated by the zookeeper master node of the zookeeper server according to the determined registration server with the lowest current load; when the failure log is determined to belong to the local task according to the task identifier, the failure log is processed.
  • the registration server 104 monitors the task implementation list in the zookeeper server 102 from the corresponding task subnodes of the nodes.
  • the registration server 104 monitors that the corresponding task subnode is assigned to the failure log, it acquires the task identifier of the failure log.
  • the registration server 104 is generated by the registration server 104 selected according to the query and associated with the failure log.
  • the registration server 104 determines that the task belongs to its own task according to the task identifier of the failure log, it implements the processing of the task.
  • the task ID can be the IP address, name, etc. of the registration server, as long as the task ID is unique.
  • the registration server further judges that the failure log assigned to its corresponding task child node belongs to its own task before processing, so as to avoid the failure log being delayed due to the allocation error of the zookeeper master node, and by ensuring the allocation Accuracy and improve the efficiency of processing.
  • the zookeeper server is further configured to use the zookeeper master node to monitor the task status list slave node, and obtain processing information received by the task status list slave node through monitoring; the processing information is provided by the server The list is generated and sent from the registration server in the node according to the processing result of processing the failure log;
  • the zookeeper master node determines that the processing information is processing failure information, it queries the registered server with the lowest current load among the slave nodes in the server list.
  • the zookeeper master node when the zookeeper master node determines that the processing information is processing success information, it marks the failure log in the task implementation list slave node as processed, and slaves the failed task list Delete the failure log from the node.
  • the zookeeper server is further configured to query the server list from each of the task subnodes in the task implementation list slave node for the current task amount of each of the registration servers in the slave node;
  • the registration server corresponding to the task child node with the least amount of tasks serves as the registration server with the lowest current load on the server list slave node.
  • any registration server is abnormal or processing a timeout when processing the log, it is determined that the log is a failure log.
  • this application provides a zookeeper-based log processing method. Taking the method applied to the zookeeper server in FIG. 1 as an example, the method includes the following steps:
  • Step S202 the zookeeper master node monitors the failed task list slave node, and when the failed task list slave node receives the failure log submitted by the server list slave registration server in the node, it queries the registration server with the lowest current load in the server list slave node.
  • the zookeeper server is the server deployed with the zookeeper service
  • the zookeeper master node is the master node of the zookeeper service in the zookeeper server, and plays a leading role in the entire zookeeper service.
  • the zookeeper service in the zookeeper server also includes failed task list slave nodes, server list slave nodes, task implementation list slave nodes, and task status list slave nodes.
  • the registration server submits the failed log of the processing failure to the slave node of the failed task list in the zookeeper service of the zookeeper server for receiving the failed log.
  • the zookeeper master node in the zookeeper server monitors the failed task list slave nodes in real time.
  • the listener captures the failed task list slave node through the getChildren event and receives the failed log submitted by the registration server, it starts to coordinate the registration server to reprocess the failed log.
  • the zookeeper server queries the registration server with the lowest current load from each registration server managed by the node through the zookeeper master node slave server list. The registered server with the lowest load is selected as the server selected to reprocess the failure log.
  • the registration server with the lowest current load refers to the registration server with the lowest amount of tasks currently to be processed.
  • Step S204 the zookeeper master node distributes the failure log to the task subnodes in the task implementation list slave node corresponding to the registration server with the lowest current load, so that when the registration server with the lowest current load determines that the failure log belongs to a local task, the process will be processed.
  • the failure log is described.
  • the task implementation list slave node is used to receive the failure log assigned by the zookeeper master node that needs to be reprocessed.
  • the task implementation list slave node includes task subnodes corresponding to each registered server in the server list slave node. Each task child node is used to receive the failure log assigned to the corresponding registration server for processing.
  • the failure log is assigned to the task implementation list from the node with the lowest load.
  • the task embodiment table includes the task sub-node 1, task sub-node 2 and task sub-node 3 corresponding to the registration server 1, the registration server 2 and the registration server 3.
  • the failure log is assigned to the task child node 2, which means that the registration server 2 is responsible for reprocessing the currently received failure log.
  • the registration server 2 determines that the failure log assigned to the task child node 2 belongs to the local task, the registration server 2 starts to process the failure log.
  • the local task can be understood as the failure log task that belongs to its own processing.
  • the above-mentioned zookeeper-based log processing method uses the zookeeper master node to monitor and capture the failed task list.
  • the server list is queried from the registration server with the lowest current load in the node, and then the The failure log is assigned to the task sub-nodes of the task implementation list node corresponding to the registration server with the lowest current load, so that the registration server with the lowest current load determines that the failure log belongs to the local task, and the failure log is processed.
  • This method implements distributed processing of failed logs through the master and slave nodes of the zookeeper service, so that the failed logs are quickly sent to zookeeper, and then the server with the lowest load is selected through the zookeeper service to process the failed logs, avoiding doing it on a machine with a higher load. Needless waiting and retrying can effectively improve load balancing and improve processing efficiency.
  • step S204 the method further includes the following steps:
  • Step S206 the zookeeper master node monitors the task status list slave node, and obtains the processing information received by the task status list from the node through monitoring; the processing information is generated and sent by the registration server in the server list from the node according to the processing result of the processing failure log.
  • Step S208 Determine whether the processing information is processing success information.
  • the zookeeper master node determines that the processing information is processing failure information, it returns to step S202 to re-query the server list slave node for the registration server with the lowest current load.
  • Step S210 When the zookeeper master node determines that the processing information is the processing success information, it marks the failure log in the task implementation list slave node as processed, and deletes the failure log from the failed task list slave node.
  • the task status list slave node is used to receive and store the reprocessing completion status of each failure log from the registration server, and the processing information is the information used to indicate the processing completion status, which is generated by the registration server according to the actual processing status.
  • the zookeeper master node in the zookeeper server obtains the processing completion status of each failure log by monitoring the task status list, and obtains the processing information.
  • the zookeeper master node further judges whether the processing information is information indicating that the processing is successful.
  • the zookeeper master node determines that the processing information corresponding to the failure log is processing success information, it indicates that the failure log is successfully reprocessed by the selected registration server, and zookeeper marks the failure log as processed to mark that the failure log has been processed. And, delete the task from the failed task list node.
  • the zookeeper master node determines that the processing information corresponding to the failure log is processing failure information, it means that the failure log has failed to be processed again by the selected registration server.
  • the zookeeper master node queries the registered server with the lowest current load again, and processes the failure log by querying and selecting a server again until the failure log is processed successfully.
  • the failure log assigned to the task implementation list node may be a task to be implemented for processing the failure log, instead of assigning the actual failure log to the task subnode. Therefore, when the zookeeper master node assigns the task to be implemented corresponding to the failed log to the task child node, the registration server obtains the task to be implemented from the node according to the assigned task to be implemented from the failed task list when processing the failure log. The corresponding failure log is processed. At the same time, after the task to be implemented in the task child node is successfully processed, the task to be implemented in the task child node is marked as processed, and the failure log corresponding to the task to be implemented is deleted from the failure list from the node.
  • the processing information may be processed through different signals or different marks, and different signals or marks may indicate whether the processed information is processing success information or processing failure information.
  • the processing of the failure log is monitored in real time, and different responses are quickly made according to different processing conditions, and the next registration is selected as soon as possible for the log that fails to be processed again.
  • the server performs processing to ensure the orderliness of processing and to ensure that the failure log can be reprocessed successfully. For failed logs that have been successfully processed, mark processed from the node in the task implementation list and delete from the node from the failed task list to avoid repeated call processing after the failed log is processed successfully.
  • querying the server list from the registered server with the lowest current load in the node includes: querying the server list from the task implementation list from the task sub-nodes in the node, from the current task amount of each registration server in the node; The registration server corresponding to the child node with the least amount of tasks serves as the registration server with the lowest current load from the server list.
  • the task implementation list in the zookeeper service of this application deploys multiple task subnodes that store failure logs to be processed by the registration server from the nodes according to the number of registration servers. Therefore, when the zookeeper master node queries the server with the lowest current load from the registered server in the server list, the zookeeper master node obtains the amount of tasks stored in the task child node from each task child node in the task implementation list node, and obtains each registration The workload of the server. Then, since the registration server with the least amount of tasks has the lowest load, the zookeeper master node selects the registration server with the least amount of tasks as the registration server with the lowest current load.
  • failure log stored in the task sub-node or the task to be implemented is successfully processed, the failure log or the task to be implemented in the task sub-node is marked as processed, so as to avoid the storage in the task sub-node. Failure logs or tasks to be implemented are processed repeatedly.
  • the registration server with the lowest load is selected through the task amount to ensure the rationality and accuracy of the registration server selection, thereby improving the processing efficiency.
  • FIG. 4 a working flow chart of the master-slave node of the zookeeper service is provided, and the log processing method based on zookeeper is explained in detail according to the schematic diagram of the master-slave node shown in FIG. 4.
  • the zookeeper service includes a master node (zookeeper master node), a failed task list slave node, a server machine list slave node, a task implementation list slave node, and a task status list slave node.
  • the master node separately monitors the failed task list slave node, the server machine list slave node, the task implementation list slave node, and the task status list slave node.
  • the registration server (server) registered to the slave node of the server machine list encounters an exception or timeout when processing the log, it is determined that the log processing has failed.
  • the server machine list writes the failure log to the local blocking queue from the registration server of the node.
  • the registration server of the server machine list slave node submits the failure log in the blocking queue to the failed task list slave node.
  • the failed task list includes sub-nodes of failed tasks corresponding to the number of registered servers from the nodes. Therefore, the server machine list is stored in the failed task sub-node (tasklist) corresponding to the registration server from the failure log submitted by the registration server of the node.
  • the zookeeper master node listens to the failed task list slave nodes.
  • the failed task list slave node receives the failure log submitted by the registration server from any failed task child node in the failed task list, the registration server with the smallest amount of current tasks is selected as the registration server with the lowest current load .
  • the zookeeper master node assigns the failure log to the task implementation list slave node. Since the task implementation list slave node also includes the task child node (assing) corresponding to the registration server, the failure log is allocated to the selected task with the lowest current load. Register the task child node corresponding to the server.
  • the zookeeper master node assigns the failure log to the task implementation list slave node according to the name and IP address of the registration server with the lowest current load. Any one or more generate task identification. Then, the zookeeper master node associates the task identifier with the failure log and assigns it to the task subnodes corresponding to the registration server with the lowest current load in the task implementation list slave node. And, before assigning the failure log to the registration server with the lowest current load, the operation logic signal of the registration server with the lowest current load may be obtained from the task status list node. Determine whether the registered server with the lowest current load is down by judging the running logic signal. When the current registration server with the lowest load is down, it means that the registration server cannot perform log processing. The zookeeper master node will re-query from the remaining registration servers to determine the current registration server with the lowest load.
  • the zookeeper master node also monitors the task status list slave nodes, and obtains the processing information of the failure log from the task status list slave nodes in real time.
  • the processing information is processing success information, mark the failure log in the task implementation list from the node as processed, and delete the failed task list from the failure log in the node.
  • the server list monitors the corresponding task sub-nodes from the registration server in the node, and when a failure log is assigned to the task sub-node, obtain the task ID of the failure log.
  • the task identifier of the failure log is assigned to the task child node by the zookeeper master node, it is generated according to the selected registration server in the query and is associated with the failure log.
  • the registration server determines that the task belongs to its own task according to the task identifier in the failure log, it implements the processing of the task.
  • the task ID can be the IP address, name, etc. of the registration server, as long as the task ID is unique.
  • the registration server After the registration server finishes processing the failure log, it generates processing information according to the processing situation and sends it to the slave node of the task status list.
  • the zookeeper master node performs subsequent processing according to the processing information.
  • the task status list slave node also has multiple status subnodes (status) corresponding to each registration server one-to-one, so the registration server submits the processing information to the corresponding status subnode.
  • this application provides a zookeeper-based log processing device, including: a server list slave node 502, a failed task list slave node 504, a zookeeper master node 506, and a task implementation list slave node 508, of which:
  • the server list slave node 502 is used to manage registered servers.
  • the failed task list slave node 504 is used to receive the failure log submitted by the registration server in the server list slave node.
  • the zookeeper master node 506 is used to monitor the failed task list slave nodes. When the failed task list is captured and the slave node receives the failure log, query the server list from the registered server with the lowest current load in the node; and assign the failed log to the task implementation The list is selected from the task child nodes corresponding to the registration server with the lowest current load among the nodes, so that the registration server with the lowest current load determines that the failure log belongs to the local task, and the failure log is processed.
  • the task implementation list slave node 508 is used to receive the failure log allocated by the zookeeper master node.
  • the zookeeper-based log processing apparatus further includes a task status list slave node 510.
  • the task status list slave node 510 is configured to receive processing information generated and sent by the registration server in the server list slave node according to the processing result of processing the failure log;
  • the zookeeper master node 506 is also used to monitor the task status list to obtain the task status list from the node and the processing information received from the node; when it is determined that the processing information is processing failure information, return to the step of querying the server list from the registered server with the lowest current load from the node .
  • the zookeeper master node 506 is also used to mark the failure log in the task implementation list slave node as processed when determining that the processing information is the processing success information, and to mark the failure log from the failed task list slave node delete.
  • the zookeeper master node 506 is also used to query the server list from each task subnode in the task implementation list and the current task amount of each registration server in the slave node; and the task subnode with the least amount of tasks corresponds to The registration server serves as the registration server with the lowest current load from the server list.
  • Each module in the above-mentioned zookeeper-based log processing device can be implemented in whole or in part by software, hardware, and a combination thereof.
  • the above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.
  • a zookeeper-based server is provided, and the internal structure diagram of the zookeeper server may be as shown in FIG. 7.
  • the zookeeper server includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the zookeeper server is used to provide computing and control capabilities.
  • the memory of the computer device includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system, a computer program, and a database.
  • the internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium.
  • the zookeeper server database is used to store failure log data.
  • the network interface of the computer device is used to communicate with the registered registration server through a network connection.
  • the computer program is executed by the processor to realize a zookeeper-based log processing method.
  • FIG. 7 is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the zookeeper server to which the solution of this application is applied.
  • the specific zookeeper server can be Including more or fewer parts than shown in the figure, or combining some parts, or having a different arrangement of parts.
  • the present application provides a computer-readable storage medium.
  • the computer-readable storage medium may be non-volatile or volatile.
  • a computer program is stored thereon, and the computer program is executed by a processor. The following steps are implemented during execution:
  • the zookeeper master node listens to the failed task list slave node.
  • the failed task list slave node receives the failure log submitted by the server list slave node registration server, it queries the server list for the registration server with the lowest current load in the slave node;
  • the zookeeper master node assigns the failure log to the task subnode corresponding to the registration server with the lowest current load in the task implementation list, so that when the registration server with the lowest current load determines that the failure log belongs to the local task, the failure log is processed.
  • the zookeeper master node monitors the task status list from the node, and obtains the processing information received by the task status list from the node through monitoring; the processing information is generated and sent by the server list from the registered server in the node according to the processing result of the processing failure log;
  • the zookeeper master node determines that the processing information is processing failure information, it returns to the step of querying the server list of the registered server with the lowest current load among the slave nodes.
  • the zookeeper master node determines that the processing information is processing success information, it marks the failure log in the task implementation list slave node as processed, and deletes the failure log from the failed task list slave node.
  • the registration server corresponding to the task child node with the least amount of tasks is used as the registration server with the lowest current load on the server list slave node.
  • the zookeeper master node when the registration server with the lowest current load is in a down state, the zookeeper master node re-query from the remaining registration servers to determine the registration server with the lowest current load.
  • Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory may include random access memory (RAM) or external cache memory.
  • RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Debugging And Monitoring (AREA)

Abstract

本申请涉及大数据的云技术领域，提供一种基于zookeeper的日志处理方法、装置、计算机设备和存储介质。所述方法包括：zookeeper主节点监听失败任务列表从节点，当捕捉到所述失败任务列表从节点接收到服务器列表从节点中的注册服务器提交的失败日志时，查询所述服务器列表从节点中当前负载最低的注册服务器；所述zookeeper主节点将所述失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定所述失败日志属于本地任务时，处理所述失败日志。采用本方法能够提高效率。

Description

基于zookeeper的日志处理方法、装置、计算机设备和存储介质
本申请要求于2020年7月20日提交中国专利局、申请号为202010698574.6，发明名称为“基于zookeeper的日志处理方法、装置、计算机设备和存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及大数据的云技术领域，本申请特别是涉及一种基于zookeeper的日志处理方法、装置、计算机设备和存储介质。
 背景技术
日志是用于记录计算机系统的运行状态的记录，通过分析日志提取的重要价值信息可以用于分析人类行为、安全防患和修复缺陷等。由于日志通常都是海量和复杂的，并且每一个计算机系统所产生的日志又各不相同。所以，高效地对日志进行管理处理都是极大的技术难度，传统大多通过开发日志管理平台专用于对日志进行管理处理。
然而，现有的日志管理平台虽然在日志处理和存储都有自己的特点，但是发明人意识到对于处理失败日志消息大多都是通过队列存储重试或重新发送至输入端。但是，由于这种处理方式只是单纯的进行重试动作，容易导致效率低下并且影响后续消息的处理，进而降低了整个平台的效率。
 技术问题
现有的日志管理平台虽然在日志处理和存储都有自己的特点，但是对于处理失败日志消息大多都是通过队列存储重试或重新发送至输入端。但是，由于这种处理方式只是单纯的进行重试动作，容易导致效率低下并且影响后续消息的处理，进而降低了整个平台的效率。
 技术解决方案
基于此，有必要针对上述技术问题，提供一种能够提高效率的基于zookeeper方法、装置、计算机设备和存储介质。
第一方面，本申请提供一种基于zookeeper的日志处理方法，所述方法包括：zookeeper主节点监听失败任务列表从节点，当捕捉到所述失败任务列表从节点接收到服务器列表从节点中的注册服务器提交的失败日志时，查询所述服务器列表从节点中当前负载最低的注册服务器；所述zookeeper主节点将所述失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定所述失败日志属于本地任务时，处理所述失败日志。
第二方面，本申请提供一种基于zookeeper的日志处理装置，所述装置包括：服务器列表从节点，用于管理注册服务器；失败任务列表从节点，用于接收所述服务器列表从节点中的所述注册服务器提交的失败日志；zookeeper主节点，用于监听所述失败任务列表从节点，当捕捉到所述失败任务列表从节点接收到所述失败日志时，查询所述服务器列表从节点中当前负载最低的注册服务器；以及，将所述失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定所述失败日志属于本地任务时，处理所述失败日志；所述任务实施列表从节点，用于接收zookeeper主节点分配的失败日志。
第三方面，本申请还提供一种基于zookeeper的日志处理系统，所述系统包括zookeeper服务器和注册服务器，所述注册服务器为注册到所述zookeeper服务器中服务器列表从节点的日志处理服务器；
所述注册服务器，用于当处理日志失败时，将处理失败的日志通过阻塞队列提交到zookeeper服务器中的失败任务列表从节点；
所述zookeeper服务器，用于利用zookeeper主节点监听所述失败任务列表从节点，当zookeeper主节点捕捉到所述失败任务列表从节点接收到所述失败日志时，查询所述服务器列表从节点中当前负载最低的注册服务器；以及，利用所述zookeeper主节点将所述失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中；
所述注册服务器还用于，当确定所述zookeeper服务器分配至对应的所述任务子节点中的所述失败日志属于本地任务时，处理所述失败日志。
第四方面，本申请还提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现一种基于zookeeper的日志处理方法；其中，所述基于zookeeper的日志处理方法包括：zookeeper主节点监听失败任务列表从节点，当捕捉到所述失败任务列表从节点接收到服务器列表从节点中的注册服务器提交的失败日志时，查询所述服务器列表从节点中当前负载最低的注册服务器；所述zookeeper主节点将所述失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定所述失败日志属于本地任务时，处理所述失败日志。
 有益效果
上述基于zookeeper的日志处理方法、装置、计算机设备和存储介质，通过zookeeper主节点监听捕捉失败任务列表从节点接收到服务器列表从节点中的注册服务器提交的失败日志时，查询服务器列表从节点中当前负载最低的注册服务器，进而将失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定失败日志属于本地任务时，处理失败日志。该方法通过zookeeper服务的主从节点实现分布式处理失败日志，使得将处理失败的日志快速发送到zookeeper，进而通过zookeeper服务选择负载最低的服务器去处理失败日志，避免在负载较高的机器上做无谓的等待和重试，有效提升负载均衡，提高处理的效率。
 附图说明
图1为本申请一个实施例中基于zookeeper的日志处理系统；
图2为本申请一个实施例中基于zookeeper的日志方法的流程示意图；
图3为本申请另一个实施例中基于zookeeper的日志方法的流程示意图；
图4为本申请一个实施例中zookeeper服务的主从节点工作流程示意图；
图5为本申请一个实施例中基于zookeeper的日志装置的结构框图；
图6为本申请另一个实施例中基于zookeeper的日志装置的结构框图；
图7为本申请一个实施例中zookeeper服务器的内部结构图。
 本发明的最佳实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
本申请提供的基于zookeeper的日志处理方法，涉及大数据的云技术领域，可以应用于如图1所示的基于zookeeper的日志处理系统中。其中，注册服务器104是注册至zookeeper服务器102中用于处理日志的日志处理服务器，zookeeper服务器102通过网络与注册服务器104通过网络进行通信。
具体地，注册服务器104在处理日志时发生日志处理失败时，将处理失败的失败日志通过阻塞队列提交到zookeeper服务器102的失败任务列表从节点中。zookeeper服务器102中的zookeeper主节点监听失败任务列表从节点，当捕捉到失败任务列表从节点接收到服务器列表从节点中的注册服务器104提交的失败日志时，查询服务器列表从节点中当前负载最低的注册服务器；zookeeper服务器102的zookeeper主节点将失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中。当前负载最低的注册服务器104确定任务子节点中的失败日志属于本地任务时，处理失败日志。其中，zookeeper服务器102和注册服务器104可以用独立的服务器或者是多个服务器组成的服务器集群来实现。应当理解的是，由多个zookeeper服务器102组成的zookeeper服务器集群，该zookeeper服务器集群中各个zookeeper服务器102中均包括zookeeper主节点、失败任务列表从节点、任务实施列表从节点、服务器列表从节点以及任务状态列表从节点。失败任务列表从节点、任务实施列表从节点、服务器列表从节点以及任务状态列表从节点是zookeeper服务中的从节点。
在一个实施例中，注册服务器104将处理失败的失败日志通过阻塞队列提交到zookeeper服务器102的失败任务列表从节点中，包括：将处理失败的失败日志写入阻塞队列中；当确定阻塞队列满足提交要求时，将阻塞队列中的失败日志提交至zookeeper服务器102的失败任务列表从节点中。
具体地，当注册服务器104确定日志处理失败时，将失败日志提交到本地的阻塞队列LinkedBlockingQueue中。然后，当阻塞队列中写满或者超过预设的时间阈值时，确定阻塞队列满足提交要求。注册服务器104将阻塞队列中的失败日志提交到zookeeper服务器102的失败任务列表从节点中。
在一个实施例中，注册服务器104确定任务子节点中的失败日志是否属于本地任务，包括：监听确定zookeeper服务器将失败日志分配至对应的任务子节点时，获取失败日志对应的任务标识；任务标识由zookeeper服务器的zookeeper主节点根据确定的当前负载最低的注册服务器生成；根据任务标识确定失败日志属于本地任务时，处理失败日志。
具体地，注册服务器104监听zookeeper服务器102中任务实施列表从节点中与各自对应的任务子节点，当注册服务器104监听到对应的任务子节点中分配到失败日志时，获取失败日志的任务标识。失败日志的任务标识由zookeeper主节点分配至任务子节点之前，根据查询选定的注册服务器104生成并与失败日志关联。然后，注册服务器104根据失败日志的任务标识判断该任务属于自己的任务时，实施处理该任务。其中，任务标识可以是注册服务器的IP地址、名称等，只要确保任务标识唯一即可。而当注册服务器104确定分配到对应的任务子节点的失败日志不属于本地任务时，通知zookeeper服务器102重新进行分配。
本实施例中，注册服务器通过进一步判断分配到自身对应的任务子节点的失败日志属于自身的任务后再进行处理，避免因zookeeper主节点分配出错而导致失败日志迟迟不被处理，通过保证分配的准确性而提高处理的效率。
在一实施例中，所述zookeeper服务器，还用于利用所述zookeeper主节点监听任务状态列表从节点，通过监听获取所述任务状态列表从节点接收的处理信息；所述处理信息由所述服务器列表从节点中注册服务器根据处理所述失败日志的处理结果生成和发送；
当所述zookeeper主节点确定所述处理信息为处理失败信息时，查询所述服务器列表从节点中当前负载最低的注册服务器。
在一实施例中，当所述zookeeper主节点确定所述处理信息为处理成功信息时，将所述任务实施列表从节点中的所述失败日志标记为已处理，并从所述失败任务列表从节点中将所述失败日志删除。
在一实施例中，所述zookeeper服务器，还用于从所述任务实施列表从节点中的各所述任务子节点中查询所述服务器列表从节点中各所述注册服务器当前的任务量；将所述任务量最少的所述任务子节点对应的注册服务器作为所述服务器列表从节点当前负载最低的注册服务器。
在一实施例中，当任一台注册服务器在处理日志时发生异常或处理超时，确定该日志为失败日志。
在一个实施例中，如图2所示，本申请提供了一种基于zookeeper的日志处理方法，以该方法应用于图1中的zookeeper服务器为例进行说明，包括以下步骤：
步骤S202，zookeeper主节点监听失败任务列表从节点，当捕捉到失败任务列表从节点接收到服务器列表从节点中的注册服务器提交的失败日志时，查询服务器列表从节点中当前负载最低的注册服务器。
其中，zookeeper服务器是部署了zookeeper服务的服务器，zookeeper主节点是zookeeper服务器中zookeeper服务的主节点，在整个zookeeper服务中起领导作用。zookeeper服务器中的zookeeper服务还包括失败任务列表从节点、服务器列表从节点、任务实施列表从节点以及任务状态列表从节点。
具体地，当某一台注册服务器在处理日志时发生异常或处理超时，可以确定该日志处理失败。然后，注册服务器将处理失败的失败日志提交到zookeeper服务器中zookeeper服务中用于接收失败日志的失败任务列表从节点。而zookeeper服务器中的zookeeper主节点实时监听失败任务列表从节点，当监听通过getChildren事件捕捉到失败任务列表从节点从注册服务器接收到其提交的失败日志时，开始协调注册服务器重新处理该失败日志。zookeeper服务器通过zookeeper主节点从服务器列表从节点所管理的各个注册服务器中查询当前负载最低的注册服务器。将查询到的负载最低的注册服务器作为选定重新处理该失败日志的服务器。
在一个实施例中，当前负载最低的注册服务器是指当前需要处理的任务量最低的注册服务器。
步骤S204，zookeeper主节点将失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定所述失败日志属于本地任务时，处理所述失败日志。
其中，任务实施列表从节点用于接收zookeeper主节点分配的需要重新处理的失败日志。任务实施列表从节点中包括与服务器列表从节点中各注册服务器对应的任务子节点。每个任务子节点用于接收分配给对应注册服务器处理的失败日志。
具体地，当zookeeper服务器中zookeeper主节点在捕捉到提交的失败日志，以及查询确定处理该失败日志的当前负载最低的注册服务器之后，将该失败日志分配至任务实施列表从节点中与该负载最低的注册服务器对应的任务子节点中。例如，任务实施例表从节点中包括注册服务器1、注册服务器2和注册服务器3对应的任务子节点1、任务子节点2和任务子节点3，当确定注册服务器2为当前负载最低的注册服务器，则将失败日志分配至任务子节点2中，表示由注册服务器2负责重新处理当前接收到的失败日志。在注册服务器2确定分配至任务子节点2的失败日志属于本地任务时，该注册服务器2开始处理该失败日志。本地任务可以理解为是属于自己处理的失败日志任务。
上述基于zookeeper的日志处理方法，通过zookeeper主节点监听捕捉失败任务列表从节点接收到服务器列表从节点中的注册服务器提交的失败日志时，查询服务器列表从节点中当前负载最低的注册服务器，进而将失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定失败日志属于本地任务时，处理失败日志。该方法通过zookeeper服务的主从节点实现分布式处理失败日志，使得将处理失败的日志快速发送到zookeeper，进而通过zookeeper服务选择负载最低的服务器去处理失败日志，避免在负载较高的机器上做无谓的等待和重试，有效提升负载均衡，提高处理的效率。
在一个实施例中，如图3所示，提供另一种基于zookeeper的日志处理方法，在步骤S204之后，还包括步骤：
步骤S206，zookeeper主节点监听任务状态列表从节点，通过监听获取任务状态列表从节点接收的处理信息；处理信息由服务器列表从节点中注册服务器根据处理失败日志的处理结果生成和发送。
步骤S208，判断处理信息是否为处理成功信息，当zookeeper主节点确定处理信息为处理失败信息时，返回步骤S202，重新查询服务器列表从节点中当前负载最低的注册服务器的步骤。
步骤S210，当zookeeper主节点确定处理信息为处理成功信息时，将任务实施列表从节点中的失败日志标记为已处理，并从失败任务列表从节点中将所述失败日志删除。
其中，任务状态列表从节点用于从注册服务器处接收并存储各失败日志的重新处理完成情况，处理信息即用于表示处理完成情况的信息，由注册服务器根据实际处理情况生成。
具体地，zookeeper服务器中的zookeeper主节点通过监听任务状态列表，获取各失败日志的处理完成情况，得到处理信息。zookeeper主节点进一步判断处理信息是否为表示处理成功的信息。当zookeeper主节点确定失败日志对应的处理信息是处理成功信息，表示该失败日志被选定的注册服务器重新处理成功，zookeeper将该失败日志标记为已处理，用于标记该失败日志已处理完成。并且，将该任务从失败任务列表节点中删除。而当zookeeper主节点确定失败日志对应的处理信息是处理失败信息，则表示该失败日志被所选定的注册服务器再次处理失败了。zookeeper主节点则再次查询当前负载最低的注册服务器，通过再次查询再次选择一个服务器来处理该失败日志，直至该失败日志被处理成功为止。
在一个实施例中，被分配至任务实施列表节点中的失败日志可以是一个处理该失败日志的待实施任务，而并非将实际的失败日志分配至任务子节点中。因此，当zookeeper主节点分配至任务子节点中的是失败日志对应的待实施任务时，注册服务器在处理该失败日志时就根据分配的待实施任务从失败任务列表从节点中获取该待实施任务对应的失败日志进行处理。同时，在该任务子节点中的待实施任务处理成功之后，将任务子节点中的待实施任务标记为已处理，并且从失败列表从节点中删除该待实施任务对应的失败日志。
在一个实施例中，处理信息可以是通过不同的信号或者不同的标记，通过不同的信号或者标记表示该处理信息是处理成功信息还是处理失败信息。
本实施例中，通过监听用于获取任务处理状态的任务状态列表从节点，实时监控失败日志的处理情况，根据不同的处理情况快速进行不同的响应，对于再次处理失败的日志尽快选择下一个注册服务器进行处理，从而确保处理的有序性，以及确保失败日志能够被重新处理成功。而对于已处理成功的失败日志，在任务实施列表从节点中标记已处理和从失败任务列表从节点中删除，避免失败日志被处理成功之后还重复被调用处理。
在一个实施例中，查询服务器列表从节点中当前负载最低的注册服务器，包括：从任务实施列表从节点中的各任务子节点中查询服务器列表从节点中各注册服务器当前的任务量；将任务量最少的任务子节点对应的注册服务器作为服务器列表从节点当前负载最低的注册服务器。
具体地，由于本申请zookeeper服务中的任务实施列表从节点中根据注册服务器的数量部署了多个存储该注册服务器所要处理的失败日志的任务子节点。因此，当zookeeper主节点从服务器列表中注册服务器查询当前负载最低的服务器时，zookeeper主节点从任务实施列表节点中的各个任务子节点中获取存储在该任务子节点中的任务量，得到各个注册服务器的任务量。然后，由于任务量最少的注册服务器负载为最低，因此zookeeper主节点选取任务量最少的注册服务器为当前负载最低的注册服务器。应当理解的是，当存储在任务子节点中的失败日志或者待实施任务被处理成功之后，将该任务子节点中的失败日志或待实施任务标记为已处理，避免存储在任务子节点中的失败日志或者待实施任务被重复处理。
本实施例中，通过任务量选取负载最低的注册服务器，确保注册服务器选取的合理性和准确性，从而提高处理的效率。
在一个实施例中，如图4所示，提供一种zookeeper服务的主从节点工作流程图，根据图4所示的主从节点示意图对基于zookeeper的日志处理方法进行详细解释说明。
其中，参考图4，zookeeper服务包括master节点（zookeeper主节点）、失败任务列表从节点、服务器机器列表从节点、任务实施列表从节点以及任务状态列表从节点。master节点分别监控失败任务列表从节点、服务器机器列表从节点、任务实施列表从节点以及任务状态列表从节点。
具体地，①当注册到服务器机器列表从节点的注册服务器（server）在处理日志时发生异常或超时，确定该日志处理失败。服务器机器列表从节点的注册服务器将失败日志写入到本地的阻塞队列中。然后，当阻塞队列写满或者超过预设的时间阈值时，服务器机器列表从节点的注册服务器将阻塞队列中的失败日志提交至失败任务列表从节点中。其中，失败任务列表从节点中包括与注册服务器数量对应的失败任务子节点。因此，服务器机器列表从节点的注册服务器所提交的失败日志被存储至该注册服务器对应的失败任务子节点（tasklist）中。
②zookeeper主节点监听失败任务列表从节点，当捕捉到失败任务列表从节点中任意一个失败任务子节点接收到注册服务器提交的失败日志时，选取当前任务量最小的注册服务器作为当前负载最低的注册服务器。然后，zookeeper主节点将失败日志分配到任务实施列表从节点中，由于任务实施列表从节点中同样包括与注册服务器对应的任务子节点（assing），因此将失败日志分配至选取的当前负载最低的注册服务器对应的任务子节点。
另外，为了确保分配的准确性以及注册服务器能够验证被分配的失败日志属于本地任务，zookeeper主节点将失败日志分配到任务实施列表从节点之前，根据当前负载最低的注册服务器的名称、IP地址中任一项或多项生成任务标识。然后，zookeeper主节点将任务标识与所述失败日志关联后分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中。以及，还可以在给当前负载最低的注册服务器分配失败日志之前，从任务状态列表节点中获取该当前负载最低的注册服务器的运行逻辑信号。通过判断运行逻辑信号确定当前负载最低的注册服务器是否处于宕机状态。而当前负载最低的注册服务器处于宕机状态时，表示该注册服务器无法进行日志处理工作，zookeeper主节点则从剩余的注册服务器中重新查询确定当前负载最低的注册服务器。
③zookeeper主节点同时监控任务状态列表从节点，实时从任务状态列表从节点中获取失败日志的处理信息。当处理信息为处理成功信息时，将任务实施列表从节点中的失败日志标记为已处理，将失败任务列表从节点中的失败日志删除。
④服务器列表从节点中的注册服务器监听各自对应的任务子节点，当任务子节点中分配到失败日志时，获取失败日志的任务标识。失败日志的任务标识由zookeeper主节点分配至任务子节点之前，根据查询选定的注册服务器生成并与失败日志关联。然后，注册服务器根据失败日志的任务标识判断该任务属于自己的任务时，实施处理该任务。其中，任务标识可以是注册服务器的IP地址、名称等，只要确保任务标识唯一即可。
⑤当注册服务器对该失败日志处理完成之后，根据处理的情况生成处理信息给到任务状态列表从节点。由zookeeper主节点根据处理信息进行后续处理。同样的，任务状态列表从节点中同样有多个与各注册服务器一一对应的状态子节点（status），因此注册服务器将处理信息提交到对应的状态子节点中。
应该理解的是，虽然图2-3的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，图2-3中的至少一部分步骤可以包括多个子步骤或者多个阶段，这些子步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些子步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤的子步骤或者阶段的至少一部分轮流或者交替地执行。
在一个实施例中，如图5所示，本申请提供了一种基于zookeeper的日志处理装置，包括：服务器列表从节点502、失败任务列表从节点504、zookeeper主节点506和任务实施列表从节点508，其中：
服务器列表从节点502，用于管理注册服务器。
失败任务列表从节点504，用于接收服务器列表从节点中的注册服务器提交的失败日志。
zookeeper主节点506，用于监听失败任务列表从节点，当捕捉到失败任务列表从节点接收到失败日志时，查询服务器列表从节点中当前负载最低的注册服务器；以及，将失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定失败日志属于本地任务时，处理失败日志。
任务实施列表从节点508，用于接收zookeeper主节点分配的失败日志。
在一个实施例中，如图6所示，基于zookeeper的日志处理装置还包括任务状态列表从节点510。
任务状态列表从节点510，用于接收服务器列表从节点中注册服务器根据处理所述失败日志的处理结果生成和发送的处理信息；
zookeeper主节点506还用于，监听任务状态列表从节点获取任务状态列表从节点接收的处理信息；当确定处理信息为处理失败信息时，返回查询服务器列表从节点中当前负载最低的注册服务器的步骤。
在一个实施例中，zookeeper主节点506还用于确定处理信息为处理成功信息时，将任务实施列表从节点中的失败日志标记为已处理，并从失败任务列表从节点中将所述失败日志删除。
在一个实施例中，zookeeper主节点506还用于从任务实施列表从节点中的各任务子节点中查询服务器列表从节点中各注册服务器当前的任务量；将任务量最少的任务子节点对应的注册服务器作为服务器列表从节点当前负载最低的注册服务器。
关于基于zookeeper的日志处理装置的具体限定可以参见上文中对于基于zookeeper的日志处理方法的限定，在此不再赘述。上述基于zookeeper的日志处理装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
在一个实施例中，提供了一种基于zookeeper服务器，该zookeeper服务器其内部结构图可以如图7所示。该zookeeper服务器包括通过系统总线连接的处理器、存储器、网络接口和数据库。其中，该zookeeper服务器的处理器用于提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统、计算机程序和数据库。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该zookeeper服务器的数据库用于存储失败日志数据。该计算机设备的网络接口用于与注册的注册服务器通过网络连接通信。该计算机程序被处理器执行时以实现一种基于zookeeper的日志处理方法。
本领域技术人员可以理解，图7中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的zookeeper服务器的限定，具体的zookeeper服务器可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
在一个实施例中，本申请提供了一种计算机可读存储介质，所述计算机可读存储介质可以是非易失性，也可以是易失性，其上存储有计算机程序，计算机程序被处理器执行时实现以下步骤：
zookeeper主节点监听失败任务列表从节点，当捕捉到失败任务列表从节点接收到服务器列表从节点中的注册服务器提交的失败日志时，查询服务器列表从节点中当前负载最低的注册服务器；
zookeeper主节点将失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定失败日志属于本地任务时，处理失败日志。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：
zookeeper主节点监听任务状态列表从节点，通过监听获取任务状态列表从节点接收的处理信息；处理信息由服务器列表从节点中注册服务器根据处理失败日志的处理结果生成和发送；
当zookeeper主节点确定处理信息为处理失败信息时，返回查询服务器列表从节点中当前负载最低的注册服务器的步骤。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：
当zookeeper主节点确定处理信息为处理成功信息时，将任务实施列表从节点中的失败日志标记为已处理，并从失败任务列表从节点中将失败日志删除。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：
从任务实施列表从节点中的各任务子节点中查询服务器列表从节点中各注册服务器当前的任务量；
将任务量最少的任务子节点对应的注册服务器作为服务器列表从节点当前负载最低的注册服务器。
在一实施例中，当前负载最低的注册服务器处于宕机状态时，zookeeper主节点则从剩余的注册服务器中重新查询确定当前负载最低的注册服务器。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器（ROM）、可编程ROM（PROM）、电可编程ROM（EPROM）、电可擦除可编程ROM（EEPROM）或闪存。易失性存储器可包括随机存取存储器（RAM）或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM（SRAM）、动态RAM（DRAM）、同步DRAM（SDRAM）、双数据率SDRAM（DDRSDRAM）、增强型SDRAM（ESDRAM）、同步链路（Synchlink） DRAM（SLDRAM）、存储器总线（Rambus）直接RAM（RDRAM）、直接存储器总线动态RAM（DRDRAM）、以及存储器总线动态RAM（RDRAM）等。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。

Claims (20)

  1. 一种基于zookeeper的日志处理方法，其中，所述方法包括：
    zookeeper主节点监听失败任务列表从节点，当捕捉到所述失败任务列表从节点接收到服务器列表从节点中的注册服务器提交的失败日志时，查询所述服务器列表从节点中当前负载最低的注册服务器；
    所述zookeeper主节点将所述失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定所述失败日志属于本地任务时，处理所述失败日志。
  2. 根据权利要求1所述的方法，其中，所述方法还包括：
    所述zookeeper主节点监听任务状态列表从节点，通过监听获取所述任务状态列表从节点接收的处理信息；所述处理信息由所述服务器列表从节点中注册服务器根据处理所述失败日志的处理结果生成和发送；
    当所述zookeeper主节点确定所述处理信息为处理失败信息时，返回查询所述服务器列表从节点中当前负载最低的注册服务器的步骤。
  3. 根据权利要求2所述的方法，其中，所述方法还包括：
    当所述zookeeper主节点确定所述处理信息为处理成功信息时，将所述任务实施列表从节点中的所述失败日志标记为已处理，并从所述失败任务列表从节点中将所述失败日志删除。
  4. 根据权利要求1所述的方法，其中，所述查询所述服务器列表从节点中当前负载最低的注册服务器，包括：
    从所述任务实施列表从节点中的各所述任务子节点中查询所述服务器列表从节点中各所述注册服务器当前的任务量；
    将所述任务量最少的所述任务子节点对应的注册服务器作为所述服务器列表从节点当前负载最低的注册服务器。
  5. 根据权利要求1所述的方法，其中，当任一台注册服务器在处理日志时发生异常或处理超时，确定该日志为失败日志。
  6. 根据权利要求1所述的方法，其中，当前负载最低的注册服务器处于宕机状态时，zookeeper主节点则从剩余的注册服务器中重新查询确定当前负载最低的注册服务器。
  7. 一种基于zookeeper的日志处理装置，其中，所述装置包括：
    服务器列表从节点，用于管理注册服务器；
    失败任务列表从节点，用于接收所述服务器列表从节点中的所述注册服务器提交的失败日志；
    zookeeper主节点，用于监听所述失败任务列表从节点，当捕捉到所述失败任务列表从节点接收到所述失败日志时，查询所述服务器列表从节点中当前负载最低的注册服务器；以及，将所述失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定所述失败日志属于本地任务时，处理所述失败日志；
    所述任务实施列表从节点，用于接收zookeeper主节点分配的失败日志。
  8. 根据权利要求7所述的装置，其中，所述装置还包括：
    任务状态列表从节点，用于接收所述服务器列表从节点中注册服务器根据处理所述失败日志的处理结果生成和发送的处理信息；
    所述zookeeper主节点还用于，监听所述任务状态列表从节点获取所述任务状态列表从节点接收的处理信息；当确定所述处理信息为处理失败信息时，返回查询所述服务器列表从节点中当前负载最低的注册服务器的步骤。
  9. 一种基于zookeeper的日志处理系统，其中，所述系统包括zookeeper服务器和注册服务器，所述注册服务器为注册到所述zookeeper服务器中服务器列表从节点的日志处理服务器；
    所述注册服务器，用于当处理日志失败时，将处理失败的日志通过阻塞队列提交到zookeeper服务器中的失败任务列表从节点；
    所述zookeeper服务器，用于利用zookeeper主节点监听所述失败任务列表从节点，当zookeeper主节点捕捉到所述失败任务列表从节点接收到所述失败日志时，查询所述服务器列表从节点中当前负载最低的注册服务器；以及，利用所述zookeeper主节点将所述失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中；
    所述注册服务器还用于，当确定所述zookeeper服务器分配至对应的所述任务子节点中的所述失败日志属于本地任务时，处理所述失败日志。
  10. 根据权利要求9所述的系统，其中，所述注册服务器还用于，
    将处理失败的失败日志写入阻塞队列中；
    当确定所述阻塞队列满足提交要求时，将所述阻塞队列中的所述失败日志提交至所述zookeeper服务器的所述失败任务列表从节点中。
  11. 根据权利要求9所述的系统，其中，所述zookeeper服务器，还用于利用所述zookeeper主节点监听任务状态列表从节点，通过监听获取所述任务状态列表从节点接收的处理信息；所述处理信息由所述服务器列表从节点中注册服务器根据处理所述失败日志的处理结果生成和发送；
    当所述zookeeper主节点确定所述处理信息为处理失败信息时，查询所述服务器列表从节点中当前负载最低的注册服务器。
  12. 根据权利要求11所述的系统，其中，当所述zookeeper主节点确定所述处理信息为处理成功信息时，将所述任务实施列表从节点中的所述失败日志标记为已处理，并从所述失败任务列表从节点中将所述失败日志删除。
  13. 根据权利要求9所述的系统，其中，所述zookeeper服务器，还用于从所述任务实施列表从节点中的各所述任务子节点中查询所述服务器列表从节点中各所述注册服务器当前的任务量；将所述任务量最少的所述任务子节点对应的注册服务器作为所述服务器列表从节点当前负载最低的注册服务器。
  14. 根据权利要求9所述的系统，其中，当任一台注册服务器在处理日志时发生异常或处理超时，确定该日志为失败日志。
  15. 根据权利要求9所述的系统，其中，所述注册服务器还用于，
    监听确定所述zookeeper服务器将所述失败日志分配至对应的任务子节点时，获取所述失败日志对应的任务标识；所述任务标识由所述zookeeper服务器的zookeeper主节点根据确定的当前负载最低的注册服务器生成；
    根据所述任务标识确定所述失败日志属于本地任务时，处理所述失败日志。
  16. 一种计算机可读存储介质，其中，其上存储有计算机程序，所述计算机程序被处理器执行一种基于zookeeper的日志处理方法；其中，所述基于zookeeper的日志处理方法包括：
    zookeeper主节点监听失败任务列表从节点，当捕捉到所述失败任务列表从节点接收到服务器列表从节点中的注册服务器提交的失败日志时，查询所述服务器列表从节点中当前负载最低的注册服务器；
    所述zookeeper主节点将所述失败日志分配至任务实施列表从节点中与当前负载最低的注册服务器对应的任务子节点中，使得当前负载最低的注册服务器确定所述失败日志属于本地任务时，处理所述失败日志。
  17. 根据权利要求16所述的计算机可读存储介质，其中，所述方法还包括：
    所述zookeeper主节点监听任务状态列表从节点，通过监听获取所述任务状态列表从节点接收的处理信息；所述处理信息由所述服务器列表从节点中注册服务器根据处理所述失败日志的处理结果生成和发送；
    当所述zookeeper主节点确定所述处理信息为处理失败信息时，返回查询所述服务器列表从节点中当前负载最低的注册服务器的步骤。
  18. 根据权利要求17所述的计算机可读存储介质，其中，所述方法还包括：
    当所述zookeeper主节点确定所述处理信息为处理成功信息时，将所述任务实施列表从节点中的所述失败日志标记为已处理，并从所述失败任务列表从节点中将所述失败日志删除。
  19. 根据权利要求16所述的计算机可读存储介质，其中，所述查询所述服务器列表从节点中当前负载最低的注册服务器，包括：
    从所述任务实施列表从节点中的各所述任务子节点中查询所述服务器列表从节点中各所述注册服务器当前的任务量；
    将所述任务量最少的所述任务子节点对应的注册服务器作为所述服务器列表从节点当前负载最低的注册服务器。
  20. 根据权利要求16所述的计算机可读存储介质，其中，当前负载最低的注册服务器处于宕机状态时，zookeeper主节点则从剩余的注册服务器中重新查询确定当前负载最低的注册服务器。
PCT/CN2020/119369 2020-07-20 2020-09-30 基于zookeeper的日志处理方法、装置、计算机设备和存储介质 WO2021139280A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010698574.6 2020-07-20
CN202010698574.6A CN111866130B (zh) 2020-07-20 2020-07-20 基于zookeeper的日志处理方法、装置、计算机设备和存储介质

Publications (1)

Publication Number Publication Date
WO2021139280A1 true WO2021139280A1 (zh) 2021-07-15

Family

ID=73001119

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/119369 WO2021139280A1 (zh) 2020-07-20 2020-09-30 基于zookeeper的日志处理方法、装置、计算机设备和存储介质

Country Status (2)

Country Link
CN (1) CN111866130B (zh)
WO (1) WO2021139280A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102025630A (zh) * 2010-12-14 2011-04-20 成都市华为赛门铁克科技有限公司 负载均衡方法及负载均衡系统
US8387064B2 (en) * 2008-10-09 2013-02-26 International Business Machines Corporation Balancing a data processing load among a plurality of compute nodes in a parallel computer
CN107341051A (zh) * 2016-05-03 2017-11-10 北京京东尚科信息技术有限公司 集群任务协调方法、系统和装置
CN109327509A (zh) * 2018-09-11 2019-02-12 武汉魅瞳科技有限公司 一种主/从架构的低耦合的分布式流式计算框架

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110661844A (zh) * 2019-08-16 2020-01-07 北京旷视科技有限公司 自动发布调度系统、方法和存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8387064B2 (en) * 2008-10-09 2013-02-26 International Business Machines Corporation Balancing a data processing load among a plurality of compute nodes in a parallel computer
CN102025630A (zh) * 2010-12-14 2011-04-20 成都市华为赛门铁克科技有限公司 负载均衡方法及负载均衡系统
CN107341051A (zh) * 2016-05-03 2017-11-10 北京京东尚科信息技术有限公司 集群任务协调方法、系统和装置
CN109327509A (zh) * 2018-09-11 2019-02-12 武汉魅瞳科技有限公司 一种主/从架构的低耦合的分布式流式计算框架

Also Published As

Publication number Publication date
CN111866130B (zh) 2023-04-18
CN111866130A (zh) 2020-10-30

Similar Documents

Publication Publication Date Title
US10838777B2 (en) Distributed resource allocation method, allocation node, and access node
WO2021129367A1 (zh) 一种监控分布式存储系统的方法及装置
CN107688496B (zh) 任务分布式处理方法、装置、存储介质和服务器
CN107547595B (zh) 云资源调度系统、方法及装置
CN111190753B (zh) 分布式任务处理方法、装置、存储介质和计算机设备
WO2022105138A1 (zh) 去中心化的任务调度方法、装置、设备及介质
US11743113B2 (en) Fault rectification operation recommendation method and apparatus, and storage medium
US8031637B2 (en) Ineligible group member status
CN113658351A (zh) 一种产品生产的方法、装置、电子设备及存储介质
WO2021218619A1 (zh) 任务分配方法及装置、任务处理系统
CN117453357A (zh) 节点任务调度方法、装置、计算机设备及存储介质
WO2021139280A1 (zh) 基于zookeeper的日志处理方法、装置、计算机设备和存储介质
US10951732B2 (en) Service processing method and device
CN113064732A (zh) 一种分布式系统及其管理方法
US20170026464A1 (en) Allocation of service endpoints to servers
CN111324668B (zh) 数据库数据同步处理方法、装置及存储介质
CN114024819B (zh) 一种事件信息上报方法及装置
CN114490003A (zh) 大规模数据的分布式作业调度方法及相关设备
CN109634848B (zh) 一种银行大型测试环境管理方法及系统
TW202315360A (zh) 微服務分配方法、電子設備及儲存介質
CN108763291B (zh) 一种数据管理方法、装置及电子设备
WO2015125225A1 (ja) データ処理システム及びデータ処理方法
CN111639089B (zh) 事务处理方法、装置、电子设备和计算机可读存储介质
CN110764882A (zh) 分布式管理方法、分布式管理系统及装置
CN113609199B (zh) 数据库系统、服务器及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20911745

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20911745

Country of ref document: EP

Kind code of ref document: A1