CN109218401A - Log collection method, system, computer equipment and storage medium - Google Patents
Log collection method, system, computer equipment and storage medium Download PDFInfo
- Publication number
- CN109218401A CN109218401A CN201810894320.4A CN201810894320A CN109218401A CN 109218401 A CN109218401 A CN 109218401A CN 201810894320 A CN201810894320 A CN 201810894320A CN 109218401 A CN109218401 A CN 109218401A
- Authority
- CN
- China
- Prior art keywords
- log
- log collection
- server
- software
- configuration information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/34—Network arrangements or protocols for supporting network services or applications involving the movement of software or configuration parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/06—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols the encryption apparatus using shift registers or memories for block-wise or stream coding, e.g. DES systems or RC4; Hash functions; Pseudorandom sequence generators
- H04L9/0643—Hash functions, e.g. MD5, SHA, HMAC or f9 MAC
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/32—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials
- H04L9/3236—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions
- H04L9/3239—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD
Abstract
The present invention relates to log collection method, system, computer equipment and storage mediums, it is that label is arranged in destination server that method, which includes: by unified log management platform, and Batch sending log collection software, destination server are installed after receiving log collection software；Log collection software sends HTTP request to server-side, server-side transfers log configuration information from log config set, log collection software generates log collection file according to log configuration information, and log collection configuration file is applied by reloading log collection process, carries out log collection；Log collection software judges whether log collection process is normal according to hash value is contained in HTTP message by calling stl status reporting functions to send HTTP message to server-side.The technical program is by calling log collection software realization to acquire log on a large scale in distributed environment, and whether normal, reduce artificial running cost if detecting using hash value log collection process.
Description
Technical field
The present invention relates to computer communication technology field, more particularly to log collection method, system, computer equipment and
Storage medium.
Background technique
Log collection refers to that generating log file to each system and application program is acquired, and log file contains currently
Program operating status, error information and the operation information of user etc..Log collection method and system include being based at present
The acquisition frame of Scribe, the acquisition frame and Flume-OG of Chukwa acquire frame.
The acquisition frame of Scribe is from each source by centrally stored to one central storage system after log collection, then
Easily there is loss of data due to the fault tolerant mechanism not responded between agent and coIIector in the statistical analysis concentrated
Situation, meanwhile, the acquisition frame of Scribe be based on thrift, rely on it is complex, environment it is invasive stronger.Chukwa
Acquisition frame primarily directed to the acquisition of various data, it contains many powerful and flexible tool sets, and can be simultaneously
Analyze collected data, favorable expandability.Compared to the acquisition frame of Scribe, the acquisition frame of Chukwa can periodically be remembered
It is high to provide the integration of fault tolerant mechanism and hadoop to record the data that have sent, but due to the acquisition frame version of Chukwa compared with
Newly, and the main original intention that designs is to cause on log collection for the acquisition of various data there is no what specific quotient
Industry is expanded production.Flume-OG acquisition frame is also a kind of Log Collect System of three layer states, agent, collector and store
Three-decker, wherein agent is responsible for reading, and collector is responsible for acquisition filter, and store is accumulation layer.Meanwhile passing through
Zookeeper provides load, so that it is safer relative to first two frame, however since frame is excessively lengthy and jumbled, it operates
It is not very convenient to come, and development amount is huge.In addition, existing three kinds of log collection frames cannot achieve in distributed environment
In, log is acquired on a large scale.
Summary of the invention
Based on this, it is necessary to for when carrying out log collection, cannot achieve and acquire day on a large scale in distributed environment
The problems such as will, provides log collection method, system, computer equipment and storage medium.
A kind of log collection method, the log collection method, specifically comprises the following steps:
Unified log management platform classifies to destination server, and is one mark of the setting of destination server described in same class
Label, the unified log management platform according to the label to the destination server Batch sending log collection software, it is described
After destination server receives and the log collection software is installed；
The log collection software sends the HTTP request for obtaining log configuration information to server-side, and the server-side receives
After the HTTP request after transferring log configuration information in the log config set being preset in the unified log management platform
It is sent to the log collection software, the log collection software generates log collection configuration text according to the log configuration information
Part applies the log collection configuration file by reloading log collection process, carries out log collection, the server-side
Setting is in the unified log management platform；
The log collection software by call the stl status reporting functions that are preset in the log collection software to
The server-side sends HTTP message, includes hash value in the HTTP message, the server-side according to the hash value come
Whether normal judge log collection process, the log collection process is normal if judging, continues log collection, if judgement
There is exception in the log collection process, and the server-side then reacquires log configuration information from the log config set,
And the log configuration information is sent to the log collection software, the log collection software is then configured according to the log
Information regenerates log collection configuration file, and the log collection configuration text is applied by reloading log collection process
Part, to carry out log collection.
In one embodiment, after the destination service receives and installation log acquisition software includes:
Unified log management platform is according to destination server application IT system title subjected to the destination server
Classify, and by the label for being set as same class destination server using IT system title, the unified log management
For platform by calling configuration deployment program to the destination server Batch sending log collection software, the destination server is logical
It crosses data-interface and receives the log collection software, by calling the installation software being arranged in the destination server to described
Log collection software is installed automatically, and the configuration deployment program setting is on the unified log management platform, the day
Will acquisition software has log reporting functions.
In one embodiment, the log collection includes:
The log collection software sends the HTTP request for obtaining log configuration information to the server-side, and the HTTP is asked
It include log collection node ID in asking；
After the server-side receives the HTTP request, out of, log config set that be preset in unified log management platform
The log configuration information consistent with the log collection node ID is transferred, the log configuration information includes the day for needing to acquire
The file path of will, log collection method and increase field information, the server-side passes through tune according to the log configuration information
Hash value is calculated with the MD5 value-based algorithm being preset in the server-side, and the hash value is put into a memory
It is stored, the log configuration information and the hash value are fed back to the log collection software by the server-side；
After the log collection software receives the log configuration information, according to the need for including in the log configuration information
The file path of the log of acquisition, log collection method and increase field to generate log collection configuration file, the log is adopted
Collection software applies the log collection file by reloading log collection process, carries out log collection, meanwhile, the day
Will acquisition software saves the hash value to a buffer.
In one embodiment, the server-side according to the hash value judge log collection process whether normally include:
The log collection software is pressed by calling the stl status reporting functions being preset in the log collection software
Send HTTP message to the server-side according to preset time interval, include in the HTTP message log collection node ID,
Information including hash value examines log collection process according to the hash value after server-side receives the HTTP message
It surveys, if the hash value is consistent with the hash value in the memory being stored in the server-side, continues log collection,
If the hash value and the hash value in the memory being stored in the server-side are inconsistent, the server-side is according to
Log collection node ID included in HTTP message transfers corresponding log configuration information from the log config set, and
The log configuration information is sent to the log collection software, the log collection software then matches confidence according to the log
Breath regenerates log collection configuration file, and the log collection configuration text is applied by reloading log collection process
Part, to carry out log collection.
In one embodiment, the installation software includes 360 sofeware managements.
In one embodiment, the memory includes NAS memory, DAS memory or SAN memory；
The buffer includes Redis.
In one embodiment, the preset time interval includes with per minute or per hour for time interval.
Based on the same technical idea, the present invention also provides a kind of Log Collect System, the system comprises installation unit,
Acquisition unit and judging unit；
The installation unit classifies to destination server for unified log management platform, and for described in same class
Destination server is arranged a label, and the unified log management platform is according to the label to the destination server Batch sending
Log collection software after the destination server receives and installs the log collection software；
The acquisition unit is asked for the log collection software to the HTTP that server-side sends acquisition log configuration information
It asks, the server-side is adjusted from the log config set being preset in the unified log management platform after receiving the HTTP request
It is sent to the log collection software after taking log configuration information, the log collection software is raw according to the log configuration information
At log collection configuration file, the log collection configuration file is applied by reloading log collection process, carries out day
Will acquisition, the server-side setting is in the unified log management platform；
The judging unit, for the log collection software by calling the day being preset in the log collection software
Will state reporting function sends HTTP message to the server-side, includes hash value, the server-side in the HTTP message
Whether normal judge log collection process according to the hash value, if judging, the log collection process is normal, continue into
Row log collection, if it is abnormal to judge that the log collection process occurs, the server-side from the log config set again
Log configuration information is obtained, and the log configuration information is sent to the log collection software, the log collection software
Log collection configuration file is then regenerated according to the log configuration information, is applied by reloading log collection process
The log collection configuration file, to carry out log collection.
Based on the same technical idea, described the present invention also provides a kind of computer equipment, including memory and processor
Computer-readable instruction is stored in memory, when the computer-readable instruction is executed by the processor, so that the place
Manage the step of device executes above-mentioned log collection method.
Based on the same technical idea, the present invention also provides a kind of storage medium for being stored with computer-readable instruction, institutes
When stating computer-readable instruction and being executed by one or more processors, so that one or more processors are executed as above-mentioned log is adopted
The step of set method.
Above-mentioned log collection method, system, computer equipment and storage medium are same by unified log management platform
Label is arranged in class destination server, and according to the label to the destination server Batch sending log collection software, described
Destination server is installed after receiving the log collection software；The log collection software sends HTTP to server-side and asks
It asks, after the server-side receives the HTTP request, log configuration information is transferred from log config set, the log collection is soft
Part generates log collection file according to the log configuration information, applies the log by reloading log collection process
Acquisition configuration file carries out log collection；The log collection software is by calling stl status reporting functions to the service
End sends HTTP message, and the H does not contain hash value in P message, whether judges log collection process according to the hash value
Normally.The technical program is utilized by calling log collection software realization to acquire log on a large scale in distributed environment
Hash value detects whether log collection process is normal, improves in distributed environment to the collecting efficiency of extensive log,
Reduce artificial running cost.
Detailed description of the invention
By reading the following detailed description of the preferred embodiment, various other advantages and benefits are common for this field
Technical staff will become clear.The drawings are only for the purpose of illustrating a preferred embodiment, and is not considered as to the present invention
Limitation.
Fig. 1 is a kind of flow chart of log collection method in one embodiment of the invention；
Fig. 2 is a kind of schematic diagram of Log Collect System in one embodiment of the invention.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and
It is not used in the restriction present invention.
Those skilled in the art of the present technique are appreciated that unless expressly stated, singular " one " used herein, " one
It " also may include plural form that a ", " described " and ", which is somebody's turn to do,.It is to be further understood that used in specification of the invention
Wording " including " refers to that there are the feature, program, step, operation, element and/or component, but it is not excluded that in the presence of or add
Add other one or more features, program, step, operation, element, component and/or their group.
Fig. 1 is a kind of flow chart of log collection method in one embodiment of the invention, as shown in Figure 1, the log is adopted
Set method specifically comprises the following steps:
Step S1: unified log management platform classifies to destination server, and is destination server described in same class
One label is set, and the unified log management platform is soft to the destination server Batch sending log collection according to the label
Part after the destination server receives and installs the log collection software；
In the present embodiment, after the destination server receives and the log collection software is installed includes:
Unified log management platform is according to destination server application IT system title subjected to the destination server
Classify, and by the label for being set as same class destination server using IT system title, the unified log management
For platform by calling configuration deployment program to the destination server Batch sending log collection software, the destination server is logical
It crosses data-interface and receives the log collection software, by calling the installation software being arranged in the destination server to described
Log collection software is installed automatically, and the configuration deployment program setting is on the unified log management platform, the day
Will acquisition software has log reporting functions.
In the present embodiment, the installation software includes 360 sofeware managements, and 360 sofeware management is in 360 security guards
One provided integrates the tool of software download, update, unloading, optimization.It is actively mentioned from software vendor to 360 security centres
The software of friendship is announced after 360 staff audit, and 360 users can update within first time to most when these software upgradings
New version.In the present embodiment, the destination server is by calling 360 sofeware management to can be realized to the log collection
The automatic installation of software.
In the present embodiment, the log reporting functions refer to log collection software according to the preset time interval to the system
Server-side transmission in one log management platform includes the HTTP report including log collection node ID, log collection process status
Text.
It include multiple destination servers in the application IT system in the present embodiment, unified log management platform
Label is set according to the entitled destination server using IT system, the label is for indicating that same class target takes
Be engaged in device, the unified log management platform by call the configuration deployment program being arranged in the unified log management platform to
The destination server Batch sending log collection software, the destination server receive the log collection by data-interface
Software is realized by calling 360 sofeware managements being arranged in the destination server to the automatic of the log collection software
Installation.
Step S2: the log collection software sends the HTTP request for obtaining log configuration information, the clothes to server-side
Business end, which receives, to be transferred log from the log config set being preset in the unified log management platform after the HTTP request and matches
The log collection software is sent to after confidence breath, the log collection software generates log according to the log configuration information and adopts
Collect configuration file, the log collection configuration file is applied by reloading log collection process, carries out log collection, institute
Server-side setting is stated in the unified log management platform.
In the present embodiment, the log collection includes:
The log collection software sends the HTTP request for obtaining log configuration information to the server-side, and the HTTP is asked
It include log collection node ID in asking；
After the server-side receives the HTTP request, out of, log config set that be preset in unified log management platform
The log configuration information consistent with the log collection node ID is transferred, the log configuration information includes the day for needing to acquire
The file path of will, log collection method and increase field information, the server-side passes through tune according to the log configuration information
Hash value is calculated with the MD5 value-based algorithm being preset in the server-side, and the hash value is put into a memory
It is stored, the log configuration information and the hash value are fed back to the log collection software by the server-side；
After the log collection software receives the log configuration information, according to the need for including in the log configuration information
The file path of the log of acquisition, log collection method and increase field to generate log collection configuration file, the log is adopted
Collection software applies the log collection file by reloading log collection process, carries out log collection, meanwhile, the day
Will acquisition software saves the hash value to a buffer.
In the present embodiment, the memory includes NAS memory, DAS memory or SAN memory.In the present embodiment, institute
It states memory and is chosen to be NAS memory, the NAS memory (Network Attached Storage: network attached storage),
A kind of special data storage server, be it is data-centered, will storage equipment and server be completely separated, manage concentratedly number
According to discharge bandwidth, improve performance, reduce total cost of ownership, protection investment.In the present embodiment, the server-side passes through tune
Hash value is saved with the NAS memory.
The buffer includes Redis, the Redis be being write using ANSI C an of open source, support network,
It is memory-based also can persistence high-performance key-value storage system, and the API of multilingual can be provided.The log
Acquisition software saves hash value by calling the Redis.
In the present embodiment, the MD5 value-based algorithm (Message Digest Algorithm MD5) is computer security neck
A kind of widely used hash function in domain has high-compressibility, is easy to calculate, resists and repair to provide the integrity protection of message
The advantages that modified and strong impact resistant.
Step S3: the log collection software is by calling the stl status being preset in the log collection software to report
Function sends H not P message to the server-side, and the H does not include hash value in P message, and the server-side is according to
Whether hash value is normal to judge log collection process, if judging, the log collection process is normal, continues log and adopts
Collection, if it is abnormal to judge that the log collection process occurs, the server-side reacquires log from the log config set
Configuration information, and the log configuration information is sent to the log collection software, the log collection software is then according to institute
It states log configuration information and regenerates log collection configuration file, apply the log by reloading log collection process
Acquisition configuration file, to carry out log collection.
In the present embodiment, the server-side judged according to the hash value log collection process whether normally include:
The log collection software is pressed by calling the stl status reporting functions being preset in the log collection software
Send H not P message to the server-side according to preset time interval, include in the HTTP message log collection node ID,
Information including hash value examines log collection process according to the hash value after server-side receives the HTTP message
It surveys, if the hash value is consistent with the hash value in the memory being stored in the server-side, continues log collection,
If the hash value and the hash value in the memory being stored in the server-side are inconsistent, the server-side is according to
Log collection node ID included in HTTP message transfers corresponding log configuration information from the log config set, and
The log configuration information is sent to the log collection software, the log collection software then matches confidence according to the log
Breath regenerates log collection configuration file, and the log collection configuration text is applied by reloading log collection process
Part, to carry out log collection.
In the present embodiment, the preset time interval include with per minute or per hour for time interval, the present embodiment
In, interval is set one minute for the preset time interval.
In the present embodiment, the hash value, also known as hash function are that one kind creates small number from any kind of data
The method of word " fingerprint ".Hash function message or data compression at abstract so that data volume becomes smaller, will data format it is fixed under
Come.The hash function upsets data, re-creates the fingerprint for being called hashed value.The hashed value is usually using one
Short random letters are represented with the digital character string formed.In the present embodiment, server-side judges day according to the hash value
Whether will acquisition process is normal, if the hash value is consistent with the hash value in the memory being stored in the server-side,
Continue log collection, if the hash value and the hash value in the memory being stored in the server-side are inconsistent,
Server-side log collection node ID according to included in the HTTP message transfers correspondence from the log config set
Log configuration information, and the log configuration information is sent to the log collection software, the log collection software is then
Log collection configuration file is regenerated according to the log configuration information, by reloading log collection process to apply
Log collection configuration file is stated, to carry out log collection.
Above-described embodiment, unified log management platform are same class destination server setting label, and according to the label
To the destination server Batch sending log collection software, the destination server carries out after receiving the log collection software
Installation；The log collection software sends HTTP request to server-side, after the server-side receives the HTTP request, from log
Log configuration information is transferred in config set, the log collection software generates log collection text according to the log configuration information
Part applies the log collection configuration file by reloading log collection process, carries out log collection；The log is adopted
Collect software by calling stl status reporting functions to send HTTP message to the server-side, contains hash in the HTTP message
Value, judges whether log collection process is normal according to the hash value.The technical program is by calling log collection software real
Log is acquired on a large scale in present distributed environment, and whether normal, improve if detecting using hash value log collection process
To the collecting efficiency of extensive log in distributed environment, artificial running cost is reduced.
Based on the same technical idea, the embodiment of the invention also provides a kind of Log Collect Systems, as shown in Fig. 2, should
System includes installation unit, acquisition unit and judging unit；
The installation unit classifies to destination server for unified log management platform, and for described in same class
Destination server is arranged a label, and the unified log management platform is according to the label to the destination server Batch sending
Log collection software after the destination server receives and installs the log collection software；
The acquisition unit is asked for the log collection software to the HTTP that server-side sends acquisition log configuration information
It asks, the server-side is adjusted from the log config set being preset in the unified log management platform after receiving the HTTP request
It is sent to the log collection software after taking log configuration information, the log collection software is raw according to the log configuration information
At log collection configuration file, the log collection configuration file is applied by reloading log collection process, carries out day
Will acquisition, the server-side setting is in the unified log management platform；
The judging unit, for the log collection software by calling the day being preset in the log collection software
Will state reporting function sends HTTP message to the server-side, includes hash value, the server-side in the HTTP message
Whether normal judge log collection process according to the hash value, if judging, the log collection process is normal, continue into
Row log collection, if it is abnormal to judge that the log collection process occurs, the server-side from the log config set again
Log configuration information is obtained, and the log configuration information is sent to the log collection software, the log collection software
Log collection configuration file is then regenerated according to the log configuration information, is applied by reloading log collection process
The log collection configuration file, to carry out log collection.
Above-described embodiment, it is same class destination server setting mark that the installation unit, which passes through unified log management platform,
Label, and according to the label to the destination server Batch sending log collection software, described in the destination server receives
It is installed after log collection software；The acquisition unit sends HTTP request to server-side by the log collection software,
After the server-side receives the HTTP request, log configuration information, the log collection software are transferred from log config set
Log collection file is generated according to the log configuration information, applies the log to adopt by reloading log collection process
Collect configuration file, carries out log collection；The judging unit is by the log collection software by calling stl status to report
Function sends HTTP message to the server-side, contains hash value in the HTTP message, judges day according to the hash value
Whether will acquires process normal.The technical program is by calling log collection software realization to acquire on a large scale in distributed environment
Log, and whether normal, improve in distributed environment to extensive log if detecting using hash value log collection process
Collecting efficiency, reduce artificial running cost.
Based on the same technical idea, the invention also provides a kind of computer equipment, the computer equipment includes depositing
Reservoir, processor and it is stored in the computer-readable instruction that can be run on the memory and on the processor, the place
It is same class destination service that reason device, which is performed the steps of when executing the computer-readable instruction through unified log management platform,
Label is arranged in device, and according to the label to the destination server Batch sending log collection software, the destination server
It is installed after receiving the log collection software；The log collection software sends HTTP request, the service to server-side
After end receives the HTTP request, log configuration information is transferred from log config set, the log collection software is according to
Log configuration information generates log collection file, and the log collection configuration text is applied by reloading log collection process
Part carries out log collection；The log collection software is by calling stl status reporting functions to send H not P to the server-side
Message, the H do not contain hash value in P message, judge whether log collection process is normal according to the hash value.
Based on the same technical idea, the present invention also provides a kind of storage medium for being stored with computer-readable instruction,
When the computer-readable instruction is executed by one or more processors, so that one or more processors execute following steps:
By unified log management platform it is that same class destination server is arranged label, and according to the label to the destination server
Batch sending log collection software, the destination server are installed after receiving the log collection software；The log is adopted
Collect software and transfers day from log config set after the server-side receives the HTTP request to server-side transmission HTTP request
Will configuration information, the log collection software generates log collection file according to the log configuration information, by reloading
Log collection process applies the log collection configuration file, carries out log collection；The log collection software passes through calling
Stl status reporting functions send HTTP message to the server-side, contain hash value in the HTTP message, according to described
Hash value judges whether log collection process is normal.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, which can be stored in a computer-readable storage and be situated between
In matter, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, storage medium above-mentioned can be
The non-volatile memory mediums such as magnetic disk, CD, read-only memory (Read-Only Memory, ROM) or random storage note
Recall body (Random Access Memory, RAM) etc..
Each technical characteristic of embodiment described above can be combined arbitrarily, for simplicity of description, not to above-mentioned reality
It applies all possible combination of each technical characteristic in example to be all described, as long as however, the combination of these technical characteristics is not deposited
In contradiction, all should be considered as described in this specification.
The embodiments described above only express several embodiments of the present invention, and the description thereof is more specific and detailed, but simultaneously
Limitations on the scope of the patent of the present invention therefore cannot be interpreted as.It should be pointed out that for those of ordinary skill in the art
For, without departing from the inventive concept of the premise, various modifications and improvements can be made, these belong to guarantor of the invention
Protect range.Therefore, the scope of protection of the patent of the invention shall be subject to the appended claims.
Claims (10)
1. a kind of log collection method, which is characterized in that the log collection method includes:
Unified log management platform classifies to destination server, and a label is arranged for destination server described in same class,
The unified log management platform is according to the label to the destination server Batch sending log collection software, the target
After server receives and the log collection software is installed；
The log collection software sends the HTTP request for obtaining log configuration information to server-side, described in the server-side receives
It is sent after transferring log configuration information in the log config set being preset in the unified log management platform after HTTP request
To the log collection software, the log collection software generates log collection configuration file according to the log configuration information,
The log collection configuration file is applied by reloading log collection process, carries out log collection, and the server-side is set
It sets in the unified log management platform；
The log collection software is by calling the stl status reporting functions that are preset in the log collection software to described
Server-side sends HTTP message, includes hash value in the HTTP message, the server-side judges according to the hash value
Whether log collection process is normal, if judging, the log collection process is normal, continues log collection, if described in judgement
There is exception in log collection process, and the server-side then reacquires log configuration information from the log config set, and will
The log configuration information is sent to the log collection software, and the log collection software is then according to the log configuration information
Log collection configuration file is regenerated, the log collection configuration file is applied by reloading log collection process,
To carry out log collection.
2. a kind of log collection method according to claim 1, which is characterized in that after the destination server receives and install
The log collection software includes:
Unified log management platform carries out the destination server according to destination server application IT system title subjected
Classification, and by the label for being set as same class destination server using IT system title, the unified log management platform
By calling configuration deployment program to the destination server Batch sending log collection software, the destination server passes through number
According to log collection software described in interface, by calling the installation software being arranged in the destination server to the log
Acquisition software is installed automatically, and on the unified log management platform, the log is adopted for the configuration deployment program setting
Collecting software has log reporting functions.
3. a kind of log collection method according to claim 1, which is characterized in that the log collection includes:
The log collection software sends the HTTP request for obtaining log configuration information to the server-side, in the HTTP request
It include log collection node ID；
After the server-side receives the HTTP request, transferred out of log config set that be preset in unified log management platform
The log configuration information consistent with the log collection node ID, the log configuration information include the log for needing to acquire
File path, log collection method and increase field information, the server-side are pre- by calling according to the log configuration information
The MD5 value-based algorithm being located in the server-side is calculated hash value, and the hash value is put into carrying out in a memory
The log configuration information and the hash value are fed back to the log collection software by storage, the server-side；
After the log collection software receives the log configuration information, need to acquire according to include in the log configuration information
The file path of log, log collection method and increase field to generate log collection configuration file, the log collection is soft
Part applies the log collection file by reloading log collection process, carries out log collection, meanwhile, the log is adopted
Collection software saves the hash value to a buffer.
4. a kind of log collection method according to claim 1, which is characterized in that the server-side according to the hash value come
Judge log collection process whether normally include:
The log collection software is by calling the stl status reporting functions being preset in the log collection software, according to pre-
If time interval send HTTP message to the server-side, include log collection node ID, hash in the HTTP message
Information including value detects log collection process according to the hash value after server-side receives the HTTP message, if
The hash value is consistent with the hash value in the memory being stored in the server-side, then continues log collection, if institute
It states hash value and the hash value in the memory that is stored in the server-side is inconsistent, then the server-side is according to the HTTP
Log collection node ID included in message, transfers corresponding log configuration information from the log config set, and by institute
It states log configuration information and is sent to the log collection software, the log collection software is then according to the log configuration information weight
Newly-generated log collection configuration file applies the log collection configuration file by reloading log collection process, with
Carry out log collection.
5. a kind of log collection method according to claim 2, which is characterized in that the installation software includes 360 software pipes
Family.
6. a kind of log collection method according to claim 3, which is characterized in that the memory include NAS memory,
DAS memory or SAN memory；
The buffer includes Redis.
7. a kind of log collection method according to claim 4, which is characterized in that the preset time interval includes with every
Minute is time interval per hour.
8. a kind of Log Collect System, which is characterized in that the system comprises installation unit, acquisition unit and judging units；
The installation unit classifies to destination server for unified log management platform, and is target described in same class
Server is arranged a label, and the unified log management platform is according to the label to the destination server Batch sending log
Acquisition software after the destination server receives and installs the log collection software；
The acquisition unit sends the HTTP request for obtaining log configuration information for the log collection software to server-side,
The server-side is transferred from the log config set being preset in the unified log management platform after receiving the HTTP request
The log collection software is sent to after log configuration information, the log collection software is generated according to the log configuration information
Log collection configuration file applies the log collection configuration file by reloading log collection process, carries out log
Acquisition, the server-side setting is in the unified log management platform；
The judging unit, for the log collection software by calling the log shape being preset in the log collection software
State reporting functions send HTTP message to the server-side, include hash value in the HTTP message, the server-side according to
Whether the hash value is normal to judge log collection process, if judging, the log collection process is normal, continues day
Will acquisition, if it is abnormal to judge that the log collection process occurs, the server-side is reacquired from the log config set
Log configuration information, and the log configuration information is sent to the log collection software, the log collection software then root
Log collection configuration file is regenerated according to the log configuration information, is applied by reloading log collection process described
Log collection configuration file, to carry out log collection.
9. a kind of computer equipment, including database and processor, it is stored with computer-readable instruction in the database, it is described
When computer-readable instruction is executed by the processor, so that the processor executes such as any one of claims 1 to 7 right
It is required that the step of log collection method.
10. a kind of storage medium for being stored with computer-readable instruction, the computer-readable instruction is handled by one or more
When device executes, so that one or more processors execute the log collection side as described in any one of claims 1 to 7 claim
The step of method.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810894320.4A CN109218401B (en) | 2018-08-08 | 2018-08-08 | Log collection method, system, computer device and storage medium |
| PCT/CN2018/106405 WO2020029376A1 (en) | 2018-08-08 | 2018-09-19 | Log acquisition method and system, and computer device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810894320.4A CN109218401B (en) | 2018-08-08 | 2018-08-08 | Log collection method, system, computer device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109218401A true CN109218401A (en) | 2019-01-15 |
| CN109218401B CN109218401B (en) | 2021-08-31 |
Family
ID=64988214
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810894320.4A Active CN109218401B (en) | 2018-08-08 | 2018-08-08 | Log collection method, system, computer device and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109218401B (en) |
| WO (1) | WO2020029376A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110968478A (en) * | 2019-11-21 | 2020-04-07 | 掌阅科技股份有限公司 | Log collection method, server and computer storage medium |
| CN111367760A (en) * | 2020-02-28 | 2020-07-03 | 平安医疗健康管理股份有限公司 | Log collection method and device, computer equipment and storage medium |
| CN111475390A (en) * | 2020-04-01 | 2020-07-31 | 深圳Tcl数字技术有限公司 | Log collection system deployment method, device, equipment and storage medium |
| CN117290190A (en) * | 2023-11-27 | 2023-12-26 | 博为科技有限公司 | Remote serial port log acquisition method, device and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102780726A (en) * | 2011-05-13 | 2012-11-14 | 中兴通讯股份有限公司 | Log analysis method and log analysis system based on WEB platform |
| CN102946320A (en) * | 2012-10-10 | 2013-02-27 | 北京邮电大学 | Distributed supervision method and system for user behavior log forecasting network |
| CN103178982A (en) * | 2011-12-23 | 2013-06-26 | 阿里巴巴集团控股有限公司 | Method and device for analyzing log |
| CN103617287A (en) * | 2013-12-12 | 2014-03-05 | 用友软件股份有限公司 | Log management method and device in distributed environment |
| WO2017131774A1 (en) * | 2016-01-29 | 2017-08-03 | AppDynamics, Inc. | Log event summarization for distributed server system |
| CN108052675A (en) * | 2017-12-28 | 2018-05-18 | 惠州Tcl家电集团有限公司 | Blog management method, system and computer readable storage medium |
| CN108306771A (en) * | 2018-02-09 | 2018-07-20 | 腾讯科技（深圳）有限公司 | Log reporting method, apparatus and system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317610B (en) * | 2014-10-11 | 2017-05-03 | 福建新大陆软件工程有限公司 | Method and device for automatic installation and deployment of hadoop platform |
| CN104486107A (en) * | 2014-12-05 | 2015-04-01 | 曙光信息产业(北京)有限公司 | Log collection device and method |
| CN105278996A (en) * | 2015-11-03 | 2016-01-27 | 亚信科技（南京）有限公司 | Log collection method and device and log service system |
-
2018
- 2018-08-08 CN CN201810894320.4A patent/CN109218401B/en active Active
- 2018-09-19 WO PCT/CN2018/106405 patent/WO2020029376A1/en active Application Filing
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102780726A (en) * | 2011-05-13 | 2012-11-14 | 中兴通讯股份有限公司 | Log analysis method and log analysis system based on WEB platform |
| CN103178982A (en) * | 2011-12-23 | 2013-06-26 | 阿里巴巴集团控股有限公司 | Method and device for analyzing log |
| CN102946320A (en) * | 2012-10-10 | 2013-02-27 | 北京邮电大学 | Distributed supervision method and system for user behavior log forecasting network |
| CN103617287A (en) * | 2013-12-12 | 2014-03-05 | 用友软件股份有限公司 | Log management method and device in distributed environment |
| WO2017131774A1 (en) * | 2016-01-29 | 2017-08-03 | AppDynamics, Inc. | Log event summarization for distributed server system |
| CN108052675A (en) * | 2017-12-28 | 2018-05-18 | 惠州Tcl家电集团有限公司 | Blog management method, system and computer readable storage medium |
| CN108306771A (en) * | 2018-02-09 | 2018-07-20 | 腾讯科技（深圳）有限公司 | Log reporting method, apparatus and system |
Non-Patent Citations (1)
| Title |
|---|
| 陈飞 等: "《基于Flume的分布式日志采集分析系统设计与实现》", 《软件》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110968478A (en) * | 2019-11-21 | 2020-04-07 | 掌阅科技股份有限公司 | Log collection method, server and computer storage medium |
| CN111367760A (en) * | 2020-02-28 | 2020-07-03 | 平安医疗健康管理股份有限公司 | Log collection method and device, computer equipment and storage medium |
| CN111367760B (en) * | 2020-02-28 | 2022-07-19 | 深圳平安医疗健康科技服务有限公司 | Log collection method and device, computer equipment and storage medium |
| CN111475390A (en) * | 2020-04-01 | 2020-07-31 | 深圳Tcl数字技术有限公司 | Log collection system deployment method, device, equipment and storage medium |
| CN117290190A (en) * | 2023-11-27 | 2023-12-26 | 博为科技有限公司 | Remote serial port log acquisition method, device and storage medium |
| CN117290190B (en) * | 2023-11-27 | 2024-02-13 | 博为科技有限公司 | Remote serial port log acquisition method, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109218401B (en) | 2021-08-31 |
| WO2020029376A1 (en) | 2020-02-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10664499B2 (en) | Content delivery network analytics management via edge stage collectors | |
| CN106878064B (en) | Data monitoring method and device | |
| CN109218401A (en) | Log collection method, system, computer equipment and storage medium | |
| CN111831548B (en) | Dependency relationship topological graph drawing method and device | |
| CN107544832B (en) | Method, device and system for monitoring process of virtual machine | |
| CN108351806A (en) | Database trigger of the distribution based on stream | |
| CN105653425A (en) | Complicated event processing engine based monitoring system | |
| CN105610648A (en) | Operation and maintenance monitoring data collection method and server | |
| US20130111018A1 (en) | Passive monitoring of virtual systems using agent-less, offline indexing | |
| CN106101213A (en) | Information-distribution type storage method | |
| CN108092936A (en) | A kind of Host Supervision System based on plug-in architecture | |
| US11934972B2 (en) | Configuration assessment based on inventory | |
| US10305738B2 (en) | System and method for contextual clustering of granular changes in configuration items | |
| CN114553953A (en) | Event pushing method for JAVA intelligent contract of block chain | |
| CN112416708A (en) | Asynchronous call link monitoring method and system | |
| US20240070123A1 (en) | Using Machine Learning to Provide a Single User Interface for Streamlined Deployment and Management of Multiple Types of Databases | |
| US20090177953A1 (en) | Method and system for updating topology changes of a computer network | |
| CN109818785A (en) | A kind of data processing method, server cluster and storage medium | |
| US9443196B1 (en) | Method and apparatus for problem analysis using a causal map | |
| CN109510730A (en) | Distributed system and its monitoring method, device, electronic equipment and storage medium | |
| CN107562435A (en) | A kind of batch upgrading method and system based on snapshot | |
| CN106506647A (en) | A kind of client has the intelligence community cloud storage system of data backup device | |
| CN116975102A (en) | Sensitive data monitoring method, system, electronic equipment and storage medium | |
| CN114186874A (en) | Flow playback-based wind control strategy configuration method, device, equipment and medium | |
| CN116094925B (en) | Data hierarchical identification method, system and storage medium for micro-service architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |