CN117093406A - Log center maintenance method and system - Google Patents
Log center maintenance method and system Download PDFInfo
- Publication number
- CN117093406A CN117093406A CN202311347656.6A CN202311347656A CN117093406A CN 117093406 A CN117093406 A CN 117093406A CN 202311347656 A CN202311347656 A CN 202311347656A CN 117093406 A CN117093406 A CN 117093406A
- Authority
- CN
- China
- Prior art keywords
- log
- request
- target
- response
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000012423 maintenance Methods 0.000 title claims abstract description 36
- 230000004044 response Effects 0.000 claims description 111
- 230000000903 blocking effect Effects 0.000 claims description 15
- 238000003825 pressing Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 4
- 238000004891 communication Methods 0.000 abstract description 8
- 238000007726 management method Methods 0.000 description 31
- 238000012545 processing Methods 0.000 description 10
- 230000004888 barrier function Effects 0.000 description 7
- 238000012795 verification Methods 0.000 description 5
- 238000000605 extraction Methods 0.000 description 4
- 238000013139 quantization Methods 0.000 description 4
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 239000006185 dispersion Substances 0.000 description 2
- 238000013021 overheating Methods 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007257 malfunction Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0766—Error or fault reporting or storing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G06F11/1453—Management of the data involved in backup or backup restore using de-duplication of the data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a maintenance method and a maintenance system for a log center, and belongs to the technical field of computers. The maintenance method of the log center is applied to the log center, the log center comprises a log backup server, a plurality of log storage servers, a key log management server, a log request concurrency management server and a log request router, the log backup server is in communication connection with each log storage server, the key log management server is in communication connection with the log backup server, and the log request concurrency management server is in communication connection with each log storage server through the log request router.
Description
Technical Field
The application relates to the technical field of computers, in particular to a maintenance method and system of a log center.
Background
In the related art, the maintenance of the log center has the problems of high storage pressure, untimely log request response, low capacity expansion safety, local response overheating and the like, and the system architecture of the log center is imperfect, so that the problems can be solved by the independent parts respectively in different technical schemes, but the problems in consideration of the problems exist, and a log center maintenance system capable of completely solving the problems from the system architecture level is not available at present.
Disclosure of Invention
The embodiment of the application provides a maintenance method and a maintenance system for a log center, which completely solve the problems from the system architecture level.
According to an aspect of the embodiment of the present application, there is provided a log center maintenance method, which is applied to a log center, the log center including a log backup server, a plurality of log storage servers, a key log management server, a log request concurrency management server, and a log request router, the log backup server being communicatively connected to each of the log storage servers, the key log management server being communicatively connected to the log backup server, the log request concurrency management server being communicatively connected to each of the log storage servers through the log request router, the method comprising:
under the condition that a new log is obtained, carrying out backup processing on the new log, and transmitting backup results to log storage servers in a one-to-one correspondence mode, so that each log storage server stores the new log, all backup results in the log backup server are deleted, and one new log is reserved;
the log request concurrency management server receives log query requests from a plurality of clients in a concurrency mode, and the following operation is executed for each log query request: inquiring a log request sequence based on the sequence number of the log inquiry request, and deleting the log inquiry request if the sequence number exists in the log request sequence; if the sequence of the log request does not exist the sequence number, inquiring the sequence of the log request based on a request body of the log request, if a target request corresponding to the request body exists in the sequence of the log request, judging the state of the target request, if the target request is in a to-be-processed state, deleting the log request, if the target request is in a blocking state, sending a single inquiry failure to a client corresponding to the log request, automatically continuing inquiry notification, and deleting the target request in the sequence of the log request; generating a target log query request according to the current response node corresponding to the target request and the log query request, and pushing the target log query request into the log request sequence; if the target request corresponding to the request body does not exist in the log request sequence, directly pressing the log query request into the log request sequence; each log storage server points to a corresponding response node;
each log storage server sends state information of the log storage server to the log request router at regular time, the log request router maintains a routing table according to the state information, requests in the log request sequence are sequentially extracted, and corresponding response nodes are distributed for the extracted requests based on the routing table.
In one embodiment, the allocating a corresponding reply node to the extracted request based on the routing table includes:
when the extracted request does not contain response node information, determining a target response node in an effective time interval in the routing table, routing the extracted request to a log storage server corresponding to the target response node, triggering the log storage server to respond to the extracted request, and changing the state corresponding to the extracted request from unprocessed to processed;
when the extracted request contains response node information, determining a target response node in an effective time interval in the routing table, wherein the target response node and a node pointed by the response node information are different nodes, routing the extracted request to a log storage server corresponding to the target response node, triggering the log storage server to respond to the extracted request, and changing the state corresponding to the extracted request from unprocessed to processed;
the method further comprises the steps of: after the log storage server completes the response to the extracted request, the log storage server triggers the deletion of the extracted request in the log request sequence.
In one embodiment, the method further comprises:
if a certain request exists in the log request sequence, the state of the request is in processing, and the duration of the state of the request in processing is greater than a first preset threshold value, modifying the state corresponding to the request into a blocking state;
if a certain request exists in the log request sequence and the state of the request is in a blocking state, and the time length of the request in the blocking state is larger than a second preset threshold, deleting the request in the log request sequence, sending a query failure notification to a client corresponding to the request, and generating a complaint record, wherein the complaint record comprises a response node corresponding to the request.
In one embodiment, after the sending of the single query failure to the client corresponding to the log query request and the automatic continuation of the query notification, the method further includes:
generating a complaint record, wherein the complaint record comprises a response node corresponding to the target request;
counting the complaint records regularly to obtain the number of the complaint records corresponding to each response node in a preset time interval, and deleting the information of the response nodes with the complaint number larger than a third preset threshold value from the routing table.
In one embodiment, the method further comprises:
under the condition that a client acquires a response to a log query request sent by the client, verifying validity of a response node according to response node identity information in the response, and under the condition that the validity meets the requirement, trusting data in the response;
and under the condition that the legality does not meet the requirement, generating a response node deleting request, and sending the response node deleting request to the log request router so as to trigger the log request router to delete the response node of which the legality does not meet the requirement in the routing table.
In one embodiment, the log request router further performs the following operations:
judging whether records of response nodes corresponding to a log storage server exist in the routing table or not under the condition that self state information sent by the log storage server is acquired and the self state information indicates that the state is healthy;
if not, adding a record, and updating the effective time interval of the record; if yes, the corresponding recorded effective time interval is directly updated.
In one embodiment, the method further comprises:
extracting key logs from the total logs stored in the log backup server, and storing the key logs in the key log management server;
and the key log management server receives key log retrieval requests from a plurality of clients in parallel and feeds back retrieval results in real time.
In one embodiment, the extracting the key log from the full log stored in the log backup server includes:
generating a log record form based on the full log, wherein each record in the log record form comprises a log event form item, a log source form item and a log additional information form item; the content of each form item is quantized data;
generating an empty key log record form;
sequentially extracting log records in the log record form, and judging that the target log record is processed if key records exist in the key log record form for each extracted target log record, wherein the key records are log records with the contents of the corresponding log event form, log source form and log additional information form not smaller than the contents of the corresponding form in the target log record and different from the target log record; if a record to be deleted exists in the key log record form, deleting the record to be deleted from the key log record form, and adding the target log record to the key log record form, wherein the record to be deleted is a log record which is smaller than the corresponding list item in the target log record in content of the corresponding log event list item, the log source list item and the log additional information list item and is different from the target log record; and if the target log record is the other condition, directly adding the target log record into the key log record form.
Another aspect also provides a log center maintenance system, the system including a log center including a log backup server, a plurality of log storage servers, a key log management server, a log request concurrency management server, and a log request router, the log backup server being communicatively connected to each of the log storage servers, the key log management server being communicatively connected to the log backup server, the log request concurrency management server being communicatively connected to each of the log storage servers through the log request router, the system being configured to:
under the condition that a new log is obtained, carrying out backup processing on the new log, and transmitting backup results to log storage servers in a one-to-one correspondence mode, so that each log storage server stores the new log, all backup results in the log backup server are deleted, and one new log is reserved;
the log request concurrency management server receives log query requests from a plurality of clients in a concurrency mode, and the following operation is executed for each log query request: inquiring a log request sequence based on the sequence number of the log inquiry request, and deleting the log inquiry request if the sequence number exists in the log request sequence; if the sequence of the log request does not exist the sequence number, inquiring the sequence of the log request based on a request body of the log request, if a target request corresponding to the request body exists in the sequence of the log request, judging the state of the target request, if the target request is in a to-be-processed state, deleting the log request, if the target request is in a blocking state, sending a single inquiry failure to a client corresponding to the log request, automatically continuing inquiry notification, and deleting the target request in the sequence of the log request; generating a target log query request according to the current response node corresponding to the target request and the log query request, and pushing the target log query request into the log request sequence; if the target request corresponding to the request body does not exist in the log request sequence, directly pressing the log query request into the log request sequence; each log storage server points to a corresponding response node;
each log storage server sends state information of the log storage server to the log request router at regular time, the log request router maintains a routing table according to the state information, requests in the log request sequence are sequentially extracted, and corresponding response nodes are distributed for the extracted requests based on the routing table.
Another aspect also provides a computer readable storage medium having stored therein at least one instruction, at least one program, code set, or instruction set loaded and executed by a processor to implement a log center maintenance method as mentioned in the foregoing.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a system architecture diagram of a journal center maintenance system provided in one embodiment of the present application;
FIG. 2 is a flow chart of a log center maintenance method according to an embodiment of the present application;
FIG. 3 is a flow chart of a method for allocating a response node in a log center maintenance method according to an embodiment of the present application;
fig. 4 is a flow chart illustrating a method for executing a key log extraction by a key log management server in a log center maintenance method according to an embodiment of the present application.
Detailed Description
Before describing the method embodiments of the present application, related terms or nouns that may be involved in the method embodiments of the present application are briefly described, so as to be understood by those skilled in the art of the present application.
The embodiment of the application provides a log center maintenance method, please refer to fig. 1, which shows a log center maintenance system architecture diagram, wherein the log center maintenance system comprises a log center, and each execution subject in the log center participates in the log center maintenance method. Specifically, the log center comprises a log backup server, a plurality of log storage servers, a key log management server, a log request concurrency management server and a log request router, wherein the log backup server is in communication connection with each log storage server, the key log management server is in communication connection with the log backup server, the log request concurrency management server is in communication connection with each log storage server through the log request router, and the log storage server can be in communication connection with a corresponding client in a response link aiming at a specific request and is not in communication connection in other cases. The log center interacts with multiple clients in parallel, and is not limited to the number of clients.
Referring to fig. 2, a flow chart of a log center maintenance method is shown, and the method includes:
s101, under the condition that a new log is obtained, the log backup server performs backup processing on the new log, and transmits backup results to the log storage servers in a one-to-one correspondence mode, so that each log storage server stores one part of the new log, all backup results in the log backup server are deleted, and one part of the new log is reserved.
That is, each of the log storage server and the log backup server retains a newly added log each time, which is operated such that each of the log storage server and the log backup server stores the full log, through step S101. The embodiment of the application is not limited to the source of the newly added log, and can be derived from the plurality of clients.
S102, the log request concurrency management server receives log query requests from a plurality of clients.
Each client can independently request a log to a log request concurrency management server to inquire the log, so that a log inquiry request is sent, and the log inquiry request inquires about the full log. The embodiments of the present application do not emphasize which data is present in the log query request, as it does not constitute an implementation barrier to the embodiments of the present application and may be determined by one skilled in the art based on actual situations and/or disclosure in connection with the embodiments of the present application.
S103, executing the following operations for each log query request:
s1031, inquiring a log request sequence based on the sequence number of the log inquiry request;
the log request sequence is maintained by the log request concurrency management server, and the log request concurrency management server has the effects that concurrent log requests are converted and stored in the sequence, so that the concurrent requests are sequenced, the condition that the response speed of the log requests is possibly influenced is largely avoided by combining request filtering and route distribution on the basis of sequencing, and the response speed of the log query requests is remarkably improved by considering load dispersion to avoid local response overheating and realizing concurrent response on the basis of sequencing, namely by the idea of concurrency-sequencing-secondary concurrency. In the case of concurrent log query requests, the embodiment of the present application does not limit the order in which S103 is performed, as it does not constitute an implementation barrier to the embodiment of the present application.
S1032, deleting the log query request if the sequence number exists in the log request sequence;
each log query request has a sequence number, if the sequence number exists in the log request sequence, the log query request is transferred to the sequence before, and the log query request is naturally deleted directly without further processing, and only the log query request stored in the sequence is saved.
S1033, inquiring the log request sequence based on a request body of the log inquiry request if the log request sequence does not have the sequence number;
if the sequence number does not exist in the log request sequence, which indicates that the log query request is not transferred to the sequence before, but the embodiment of the application indicates that the client may repeatedly send out a certain log query request, if both processes are performed, the response speed of the log query request may be reduced, so the embodiment of the application designs S1033 a step based on the request body query, and the purpose of this step is to query whether the same client repeatedly sends out the same log query request. The embodiments of the present application do not emphasize which data is present in the message body, as it does not constitute an implementation obstacle for the embodiments of the present application, and may be determined by one skilled in the art according to the actual situation and/or disclosure in connection with the embodiments of the present application.
S1034, judging the state of a target request corresponding to the request body if the target request exists in the log request sequence;
if the request is repeatedly sent out, checking the repeated log query request sent out by the same client which has been received before, namely the state of the target request;
s1035, deleting the log query request if the target request is in a to-be-processed state;
in this case, the target request is not responded yet, and only needs to wait for responding to the target request, and does not need to respond to the same repeated log query request as the target request again, so that the log query request is deleted directly;
s1036, if the target request is in a blocking state, sending a single query failure and automatically continuing to query a notification to a client corresponding to the log query request, and deleting the target request in the log request sequence; generating a target log query request according to the current response node corresponding to the target request and the log query request, and pushing the target log query request into the log request sequence;
if the target request is in a blocking state, it is stated that the same log query request which has been received before may be difficult to respond, then it may be considered that a responding node which specifically executes the same log query request which has been received before, i.e. the responding node of the target request is likely to be down or malfunction or refused to provide a response to the target request, in which case the target request may be a useless request, then the target request is deleted in the log request sequence, and a notification is sent to the client corresponding to the log query request, informing the client that a single query failure occurs and automatically continuing the query for the client. In this case, when considering that the answer node of the same log query request (target request) that has been received is likely to be down or faulty or refused to provide a response to the target request, the processing of the current log query request is performed again, and the re-query is automatically performed, and the answer node that is likely to be down or faulty or refused to provide a response to the target request is bypassed, so that the query success rate is improved, and the client can still obtain the response result without any processing. The method realizes that a stable response result is still output for the user under the condition that the response node fails or is down automatically by adjusting the internal execution flow.
Specifically, by generating the target log query request according to the current response node corresponding to the target request and the log query request, and pushing the target log query request into the log request sequence, the target log query request can be responded by other response nodes which do not belong to the response node where downtime or faults occur or response is refused to provide response for the target request (i.e. the response node corresponding to the target request), so that a stable response result is output for the user.
In one embodiment, after the sending of the single query failure to the client corresponding to the log query request and the automatic continuation of the query notification, the method further includes: and generating a complaint record, wherein the complaint record comprises a response node corresponding to the target request.
S1037, if the target request corresponding to the request body does not exist in the log request sequence, directly pressing the log query request into the log request sequence; each of the log storage servers points to a corresponding answering node.
To reduce data storage pressure, a chaining operation may be performed for each of the log storage servers, thereby reducing local storage pressure with the strong storage capacity of the blockchain. The relevant content of the blockchain is not limited, and does not form an implementation barrier of the embodiment of the application.
S104, each log storage server sends state information of the log storage server to the log request router at regular time, the log request router maintains a routing table according to the state information, sequentially extracts requests in the log request sequence, and distributes corresponding response nodes for the extracted requests based on the routing table, so that response to the corresponding requests is completed.
Specifically, the log request router further performs the following operations: judging whether records of response nodes corresponding to a log storage server exist in the routing table or not under the condition that self state information sent by the log storage server is acquired and the self state information indicates that the state is healthy; if not, adding the record, and updating the effective time interval of the added record; if yes, the corresponding recorded effective time interval is directly updated. The effective time interval updating method is not limited, and does not form an implementation barrier of the embodiment of the application. For example, if the self state information sent by a certain log storage server is acquired and the state health is indicated by the self state information, the valid time interval of the record corresponding to the self state information is updated to the time indicated by the time stamp corresponding to the self state information until 30 seconds thereafter.
Referring to fig. 3, a flow chart of a method for allocating a response node to an extracted request based on the routing table is shown, which includes:
s301, under the condition that the extracted request does not contain response node information, determining a target response node in an effective time interval in the routing table, routing the extracted request to a log storage server corresponding to the target response node, triggering the log storage server to respond to the extracted request, and changing the state corresponding to the extracted request from unprocessed to processed;
that is, a set of response nodes in an effective time interval at the current time is determined, and a target response node is determined in the set. The state corresponding to the extracted request is recorded in the log request sequence.
S302, under the condition that the extracted request contains response node information, determining a target response node in an effective time interval in the routing table, wherein the target response node and a node pointed by the response node information are different nodes, routing the extracted request to a log storage server corresponding to the target response node, triggering the log storage server to respond to the extracted request, and changing the state corresponding to the extracted request from unprocessed to processed;
the reply node information may point to the reply node to which the aforementioned target request corresponds, that is, other reply nodes than such reply nodes are selected, because such reply nodes may have in fact failed or crashed or refused to provide a response, although they may still be within a valid time interval, and embodiments of the present application bypass such nodes to complete the response process.
In one embodiment, the method further comprises: after the log storage server completes the response to the extracted request, the log storage server triggers the deletion of the extracted request in the log request sequence.
If a certain request exists in the log request sequence, the state of the request is in processing, and the duration of the state of the request in processing is greater than a first preset threshold value, modifying the state corresponding to the request into a blocking state; if a certain request exists in the log request sequence and the state of the request is in a blocking state, and the time length of the request in the blocking state is larger than a second preset threshold, deleting the request in the log request sequence, sending a query failure notification to a client corresponding to the request, and generating a complaint record, wherein the complaint record comprises response nodes corresponding to the corresponding request.
In one embodiment, the log request router counts each complaint record at regular time to obtain the number of complaint records corresponding to each response node in a preset time interval, and deletes the information of response nodes with the number of complaints greater than a third preset threshold value from the routing table. In the embodiment of the application, specific values of the first preset threshold, the second preset threshold, the third preset threshold and the preset time interval are not limited, and do not form an implementation barrier of the embodiment of the application.
The reason why the load dispersion is difficult to realize in the blockchain environment in the related art is that the response nodes in the blockchain cannot be fully trusted, but only the self-trusted nodes are allowed to respond to the self-request, so that the response is locally overheated. Specifically, under the condition that a client acquires a response to a log query request sent by the client, verifying validity of a response node according to response node identity information in the response, and under the condition that the validity meets the requirement, trusting data in the response; and under the condition that the legality does not meet the requirement, generating a response node deleting request, and sending the response node deleting request to the log request router so as to trigger the log request router to delete the response node of which the legality does not meet the requirement in the routing table. In this way, if the validity does not meet the requirement, the corresponding response node can be judged to be malicious, so that the record in the routing table is deleted, and the response service can not be provided for all clients, thereby avoiding the bad results of the malicious response node. The embodiment of the application is innovative in that a scattered identity verification mechanism is designed, each client can carry out identity verification when obtaining a response, malicious response nodes are deleted in time, verification pressure is scattered, and the pressure is reduced
Loss by malicious reply nodes. Of course, the embodiment of the present application is not limited to the specific content of the identity information, the validity verification and the validity requirement, and it does not form an implementation barrier of the embodiment of the present application, because there are many specific ways of validity verification.
In some scenarios, the embodiment of the present application may also provide another service other than the aforementioned full log query service, that is, a query service for the key log, where the query service is implemented on the premise that the key log is extracted from the full log. Extracting key logs from the total logs stored in the log backup server, and storing the key logs in the key log management server; and the key log management server receives key log retrieval requests from a plurality of clients in parallel and feeds back retrieval results in real time.
Referring to fig. 4, a flow chart of a method for executing the key log extraction by the key log management server is shown, where the method includes:
s401, generating a log record form based on the full log, wherein each record in the log record form comprises a log event form item, a log source form item and a log additional information form item; the content of each form item is quantized data;
the key log management server acquires the full log through the access log backup server, and records the full log in a log record form. The meaning of quantized data is that the contents related to the log event list item, the log source list item and the log additional information list item in each log record in the whole log are quantized, then the quantized result is stored in the log event list item, the log source list item and the log additional information list item of the corresponding record in the log record list, namely the contents in the log event list item, the log source list item and the log additional information list item are quantized data, and the data can be compared in size. The contents of the total log corresponding to the log event list item, the log source list item and the log additional information list item are common information of the log, do not form implementation barriers, and do not need to explain which contents can refer to the prior art. The embodiment of the present application is not limited to the quantization mode, and there are many methods in the related art, which do not form an implementation obstacle of the present application. In a possible embodiment, the present application may maintain a text information library that records some words that are important for describing the log content, such as adverbs that represent severity, such as "quite", "very", etc., adverbs that represent importance, such as "important", "key", etc., adverbs that represent urgency, such as "urgent", "quick", etc., adjectives that represent serious results, such as "broken", "damaged", etc., verbs that represent serious results, such as "alarm", "serious alarm", etc., nouns that represent alarm categories, such as "intrusion", "attack", etc., and so forth. The text information library can be set by a person skilled in the art based on specific conditions and an application scene of the log center maintenance method, weights are set for the text in the text information library according to the importance degrees of the text, and on the basis of carrying out quantization based on related technologies on the content in the log event list item, the log source list item and the log additional information list item, the quantization results can be corrected based on the weights, so that the importance degrees of the content corresponding to the list item can be more accurately represented by the quantization results corresponding to each list item.
S402, generating an empty key log record form;
s403, sequentially extracting log records in the log record form, and judging that the target log record is processed if key records exist in the key log record form for each extracted target log record, wherein the key records are log records which are different from the target log records and have the contents of the corresponding log event form, log source form and log additional information form; if a record to be deleted exists in the key log record form, deleting the record to be deleted from the key log record form, and adding the target log record to the key log record form, wherein the record to be deleted is a log record which is smaller than the corresponding list item in the target log record in content of the corresponding log event list item, the log source list item and the log additional information list item and is different from the target log record; and if the target log record is the other condition, directly adding the target log record into the key log record form.
On the basis of quantification, the content of each form item can be directly larger and smaller, through the size relation of each form item, an important log record form can be generated based on the method of the steps S401-S403, and the important log record form corresponds to important log records.
Based on the key log record form, the log records falling into the key log record form can be extracted from the full log, namely the key log records are extracted. The principle of extraction is that the corresponding relation between each record in the form and the record corresponding to the whole log is pre-stored, and the extraction result is stored in the key log management server, so that the key log management server can support the inquiry of the key log. This part of the details will not be described in detail, because this part of the content does not form an implementation obstacle in the embodiment of the present application on the basis of obtaining the key log record form.
In one embodiment, a log center maintenance system is further provided, where the system includes a log center, where the log center includes a log backup server, a plurality of log storage servers, a key log management server, a log request concurrency management server, and a log request router, where the log backup server is communicatively connected to each log storage server, the key log management server is communicatively connected to the log backup server, and the log request concurrency management server is communicatively connected to each log storage server through the log request router, where the system is configured to perform the foregoing log center maintenance method.
In one embodiment, a computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions loaded and executed by a processor to implement the log center maintenance method as described above is also provided.
The foregoing is illustrative of the present application and is not to be construed as limiting thereof, but rather, any modification, equivalent replacement, improvement or the like which comes within the spirit and principles of the present application are intended to be included within the scope of the present application.
Claims (10)
1. A log center maintenance method, the method comprising:
the log request concurrency management server queries a log request sequence based on the log query request;
if the log request sequence already has a target request corresponding to the request body of the log query request and the target request is in a blocking state, executing step 1 and step 2:
step 1: sending a corresponding notification to a corresponding client and deleting the target request;
step 2: generating a target log query request according to a current response node corresponding to the target request and the log query request, and pressing the target log query request into the log request sequence, wherein the target log query request is not allowed to be responded by the current response node;
each log storage server sends state information of the log storage servers to the log request router at regular time, the log request router maintains a routing table according to the state information, requests in the log request sequence are sequentially extracted, corresponding response nodes are distributed for the extracted requests based on the routing table, and each log storage server points to the corresponding response node.
2. The log center maintenance method according to claim 1, wherein the assigning the corresponding reply node to the extracted request based on the routing table includes:
determining a target response node in a valid time interval in the routing table under the condition that the extracted request does not contain response node information;
and routing the request to a log storage server corresponding to the target response node, triggering the log storage server to respond, and changing the state corresponding to the request from unprocessed to processed.
3. The log center maintenance method according to claim 1, wherein the assigning the corresponding reply node to the extracted request based on the routing table includes:
determining a target response node in a valid time interval in the routing table under the condition that response node information is contained in the extracted request; wherein the target reply node is not the node to which the reply node information is directed;
and routing the request to a log storage server corresponding to the target response node, triggering the log storage server to respond, and changing the state corresponding to the request from unprocessed to processed.
4. A log center maintenance method according to claim 2 or 3, characterized in that the method further comprises:
if a certain request exists in the log request sequence and the state of the request is in process, and the duration of the state of the request in process is greater than a first preset threshold value, modifying the state corresponding to the request into a blocking state;
if a certain request exists in the log request sequence and the state of the request is in a blocking state, and the time length of the request in the blocking state is larger than a second preset threshold, deleting the request in the log request sequence, and generating a complaint record, wherein the complaint record comprises a response node corresponding to the request.
5. The log center maintenance method of claim 4, wherein the method further comprises:
counting each complaint record at regular time to obtain the number of the complaint records corresponding to each response node in a preset time interval;
and deleting the information of the response nodes with the complaint quantity larger than a third preset threshold value from the routing table.
6. The log center maintenance method of claim 1, wherein the log request router further performs the following operations:
when self state information sent by a certain log storage server is obtained, and the self state information indicates that the state is healthy, judging whether records of response nodes corresponding to the log storage server exist in the routing table or not;
if not, adding a record, and updating the effective time interval of the record;
if yes, the corresponding recorded effective time interval is directly updated.
7. The log center maintenance method of claim 5, wherein the method further comprises:
the client verifies the validity of the response node according to the acquired response node identity information in the response, and trusts the data in the response under the condition that the validity meets the requirement;
and under the condition that the legality does not meet the requirement, sending a response node deleting request to the log request router, and triggering the log request router to delete the response node with the legality which does not meet the requirement in the routing table.
8. The log center maintenance method of claim 1, wherein the method further comprises:
extracting key logs from the full logs, and storing the key logs in a key log management server;
and the key log management server receives key log retrieval requests from a plurality of clients in parallel and feeds back retrieval results in real time.
9. The log center maintenance system is characterized by comprising a log request concurrency management server, a log storage server and a log request router;
the log request concurrency management server is used for executing the following operations:
querying a log request sequence based on the log query request;
if the log request sequence already has a target request corresponding to the request body of the log query request and the target request is in a blocking state, executing step 1 and step 2:
step 1: sending a corresponding notification to a corresponding client and deleting the target request;
step 2: generating a target log query request according to a current response node corresponding to the target request and the log query request, and pressing the target log query request into the log request sequence, wherein the target log query request is not allowed to be responded by the current response node;
each log storage server is used for sending self state information to the log request router at regular time;
the log request router is used for maintaining a routing table according to each state information, sequentially extracting requests in the log request sequence, distributing corresponding response nodes for the extracted requests based on the routing table, and each log storage server points to the corresponding response node.
10. A computer readable storage medium having stored therein at least one instruction, at least one program, code set, or instruction set, the at least one instruction, the at least one program, the code set, or instruction set being loaded and executed by a processor to implement the log center maintenance method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311347656.6A CN117093406B (en) | 2023-10-18 | 2023-10-18 | Log center maintenance method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311347656.6A CN117093406B (en) | 2023-10-18 | 2023-10-18 | Log center maintenance method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117093406A true CN117093406A (en) | 2023-11-21 |
| CN117093406B CN117093406B (en) | 2024-02-09 |
Family
ID=88781465
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311347656.6A Active CN117093406B (en) | 2023-10-18 | 2023-10-18 | Log center maintenance method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117093406B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102368222A (en) * | 2011-10-25 | 2012-03-07 | 曙光信息产业(北京)有限公司 | Online repairing method of multiple-copy storage system |
| CN109120366A (en) * | 2018-07-09 | 2019-01-01 | 平安科技（深圳）有限公司 | Logging time bearing calibration, device and storage medium based on Session Initiation Protocol |
| CN112650753A (en) * | 2020-12-23 | 2021-04-13 | 平安普惠企业管理有限公司 | Log management method, device, system, equipment and readable storage medium |
| CN113206858A (en) * | 2021-05-13 | 2021-08-03 | 南京邮电大学 | Mobile target defense method based on internet of things DDoS attack |
| WO2023280053A1 (en) * | 2021-07-07 | 2023-01-12 | 阿里云计算有限公司 | Data processing method and system, and electronic device and storage medium |
| CN115905151A (en) * | 2022-11-11 | 2023-04-04 | 银清科技有限公司 | Method, system and device for querying circulation information based on backup log |
| WO2023142543A1 (en) * | 2022-01-27 | 2023-08-03 | 中国银联股份有限公司 | Active-standby switching method and apparatus for distributed database, and readable storage medium |
-
2023
- 2023-10-18 CN CN202311347656.6A patent/CN117093406B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102368222A (en) * | 2011-10-25 | 2012-03-07 | 曙光信息产业(北京)有限公司 | Online repairing method of multiple-copy storage system |
| CN109120366A (en) * | 2018-07-09 | 2019-01-01 | 平安科技（深圳）有限公司 | Logging time bearing calibration, device and storage medium based on Session Initiation Protocol |
| WO2020010718A1 (en) * | 2018-07-09 | 2020-01-16 | 平安科技（深圳）有限公司 | Sip-based log time correction method and device, and storage medium |
| CN112650753A (en) * | 2020-12-23 | 2021-04-13 | 平安普惠企业管理有限公司 | Log management method, device, system, equipment and readable storage medium |
| CN113206858A (en) * | 2021-05-13 | 2021-08-03 | 南京邮电大学 | Mobile target defense method based on internet of things DDoS attack |
| WO2023280053A1 (en) * | 2021-07-07 | 2023-01-12 | 阿里云计算有限公司 | Data processing method and system, and electronic device and storage medium |
| WO2023142543A1 (en) * | 2022-01-27 | 2023-08-03 | 中国银联股份有限公司 | Active-standby switching method and apparatus for distributed database, and readable storage medium |
| CN115905151A (en) * | 2022-11-11 | 2023-04-04 | 银清科技有限公司 | Method, system and device for querying circulation information based on backup log |
Non-Patent Citations (3)
| Title |
|---|
| IBRAHIM JALUTA: "Highly concurrent multidimensional index for spatial database management systems", IEEE * |
| 王雪莲;郑晓薇;刘青昆;: "LVS集群负载调度机制的研究", 中国科技信息, no. 05 * |
| 陈庭平;沈丽娟;曾鹏;: "日志服务器建设和应用", 网络安全技术与应用, no. 09 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117093406B (en) | 2024-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102577139B1 (en) | Smart contract-based data processing methods, devices, and storage media | |
| US7188366B2 (en) | Distributed denial of service attack defense method and device | |
| US8640218B2 (en) | System, method and program for managing firewalls | |
| KR101143250B1 (en) | System and method for disaster recovery and management of an email system | |
| US20090262724A1 (en) | Proxy server, communication system, communication method and program | |
| CN108683668B (en) | Resource checking method, device, storage medium and equipment in content distribution network | |
| US20080228890A1 (en) | System and method for pushing activated instant messages | |
| CN110336848B (en) | Scheduling method, scheduling system and scheduling equipment for access request | |
| EP2362594A1 (en) | Data storage method and mail relay method of storage system in mail system | |
| JP5119844B2 (en) | File transfer system, file transfer method, file transfer program, and index server | |
| CN108696831B (en) | Short message sending method and device | |
| CN1647482B (en) | System and method for network communication management | |
| CN111240892A (en) | Data backup method and device | |
| CN110839023B (en) | Electric power marketing multi-channel customer service system | |
| CN112632375A (en) | Session information processing method, server and storage medium | |
| CN110225109B (en) | Multi-queue data transmission method based on 'industrial and commercial connection' platform | |
| CN117093406B (en) | Log center maintenance method and system | |
| CN109684109B (en) | Data sharing system and data sharing method | |
| CN113765871B (en) | Method and device for managing fort machine | |
| KR101758337B1 (en) | Method for preventing overlap notice of message, storing medium and user device implementing the same | |
| KR100583177B1 (en) | Method of processing charging message to prevent double charging, and server therefor | |
| CN113824738A (en) | Method and system for node communication management in block chain | |
| KR102537533B1 (en) | Apparatus and method for controlling the sending of roaming text messages | |
| CN108809986A (en) | A kind of privately owned cloud system of enterprise | |
| KR20180073003A (en) | Apparatus and method for syncronizing of object |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |