CN105635301A - Access log merging method and log processing server and system - Google Patents
Access log merging method and log processing server and system Download PDFInfo
- Publication number
- CN105635301A CN105635301A CN201610023241.7A CN201610023241A CN105635301A CN 105635301 A CN105635301 A CN 105635301A CN 201610023241 A CN201610023241 A CN 201610023241A CN 105635301 A CN105635301 A CN 105635301A
- Authority
- CN
- China
- Prior art keywords
- access log
- log
- web server
- server
- daily record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides an access log merging method and a log processing server and system. The method comprises the following steps: copying an access log recorded by each web server, wherein the suffix of the log name of each access log carries a server identifier of the corresponding web server; determining an access log set in which the log names are the same but the server identifiers carried by the suffixes of the log names are different; merging the access logs in the access log set to obtain a merged access log corresponding to the access log set. The access log merging method provided by the embodiment of the invention can be used for automatically collecting and merging the access logs recorded by a plurality of web servers.
Description
Technical field
The present invention relates to journaling technique field, be specifically related to a kind of access log and merge method, log processing server and system.
Background technology
Access log is user when accessing website, and the user that web server records accesses the log information of website; Maintenance in website and using, often can relate to the analytical work of access log, by access log is analyzed, for determining website fault, exception, and adjusts website operation and optimisation strategy is significant.
But, when website uses load balancing, the analytical work of access log will become complicated; Website uses the configuration diagram of load balancing as shown in Figure 1, for shunting net website access pressure, proxied server is diverted in multiple web server by user by the access of website, user access the access log corresponding to website will by multiple web servers record.
Therefore, when website uses load balancing, the analytical work of access log need to be based upon under the premise merged by the access log of multiple web server records; Based on this, how to collect the access log of multiple web server records automatically and to merge, thinking that the analytical work of access log provides basis, become the problem that those skilled in the art need to consider.
Summary of the invention
In view of this, the embodiment of the present invention provides a kind of access log to merge method, log processing server and system, to realize the access log collecting multiple web server records automatically the purpose merged.
For achieving the above object, the embodiment of the present invention provides following technical scheme:
A kind of access log merges method, is applied to log processing server, and described access log merges method and includes:
Replicating the access log that each web server records, wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server;
From the access log replicated, it is determined that daily record name is identical but access log set that server identification that suffix of daily record name carries is different;
Access log in each described access log set is merged mutually, obtains the merging access log that each described access log set is corresponding.
Optionally, the access log that each web server of described duplication records includes:
Determine the server identification of each web server, the daily record name of the access log that need to replicate, and the login password of each web server;
Server identification according to each web server and login password, each web server of Telnet, replicate corresponding for the daily record name with the described access log that need to replicate of each web server record access log;
Suffix in the daily record name of the access log replicated adds the server identification of corresponding web server.
Optionally, described being merged mutually by access log in each described access log set, the merging access log obtaining each described access log set corresponding includes:
For each described access log set, the log content of each access log in access log set is exported to setting in download directory, the log content of each access log of output is carried out duplicate removal and sequence, obtain the log content merged, renaming has the file of the log content of merging, obtains merging access log.
Optionally, the described server identification determining each web server, the daily record name of the access log that need to replicate, and the login password of each web server includes:
Perform the first script and call the second script by the first script, passing to described second script by first script server identification by each web server, the daily record name of the access log that need to replicate, and the login password of each web server;
The described server identification according to each web server and login password, each web server of Telnet, corresponding for the daily record name with the described access log that need to replicate of each web server record access log is carried out duplication and includes:
By crus secunda this according to the server identification of each web server and login password, the each web server of Telnet, and perform remote copy order by described second script, corresponding for the daily record name with the described access log that need to replicate of each web server record access log is replicated;
The suffix of the described daily record name at the access log replicated adds the server identification of corresponding web server and includes:
Added the server identification of corresponding web server at the suffix of the daily record name of the access log replicated by the second script.
Optionally, by described first script from the access log replicated, it is determined that daily record name is identical but access log set that server identification that suffix of daily record name carries is different; And by the first script, the access log in each described access log set is merged mutually, obtain the merging access log that each described access log set is corresponding.
The embodiment of the present invention also provides for a kind of log processing server, including:
First script, replicates, for calling the second script, the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server; And from the access log replicated, determine that daily record name is identical but the different access log set of server identification that suffix of daily record name carries, access log in each described access log set is merged mutually, obtains the merging access log that each described access log set is corresponding;
Second script, for being called by the first script, replicates the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server.
Optionally, described first script includes:
Transfer unit, for passing to described second script by the server identification of each web server, the daily record name of the access log that need to replicate, and the login password of each web server;
Described second script includes:
Telnet copied cells, for the server identification according to each web server and login password, the each web server of Telnet, and perform remote copy order, corresponding for the daily record name with the described access log that need to replicate of each web server record access log is replicated;
Mark adding device, for the server identification of the corresponding web server of suffix interpolation of the daily record name at the access log replicated.
Optionally, described first script includes:
Merge unit, for for each described access log set, the log content of each access log in access log set is exported to setting in download directory, the log content of each access log of output is carried out duplicate removal and sequence, obtain the log content merged, renaming has the file of the log content of merging, obtains merging access log.
The embodiment of the present invention also provides for a kind of access log combination system, including: proxy server, multiple web servers, log processing server;
Wherein, described proxy server branches to the plurality of web server for user being accessed;
Described web server, accesses the access log of correspondence for recording user;
Described log processing server, for replicating the access log that each web server records, wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server; From the access log replicated, it is determined that daily record name is identical but access log set that server identification that suffix of daily record name carries is different; Access log in each described access log set is merged mutually, obtains the merging access log that each described access log set is corresponding.
Optionally, described log processing server has the first script and the second script;
Described first script, replicates, for calling the second script, the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server; And from the access log replicated, determine that daily record name is identical but the different access log set of server identification that suffix of daily record name carries, access log in each described access log set is merged mutually, obtains the merging access log that each described access log set is corresponding;
Described second script, for being called by the first script, replicates the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server.
Based on technique scheme, the access log that the embodiment of the present invention provides merges method, can be applicable in log processing server; The access log that the reproducible each web server of log processing server records, wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server, and from the access log replicated, determine that daily record name is identical but the different access log set of server identification that suffix of daily record name carries, and then the access log in each described access log set is merged mutually, obtain the merging access log that each described access log set is corresponding. Can be seen that, the access log that the embodiment of the present invention provides merges method, can on the suffix of the daily record name of the access log collected from web server, carry the server identification of corresponding web server, so that it is determined that daily record name is identical but access log that server identification that suffix of daily record name carries is different needs to merge, and then realizes the automatic merging of access log. The access log that the embodiment of the present invention provides merges method, it may be achieved the access log collecting multiple web server records automatically the purpose merged.
Accompanying drawing explanation
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, the accompanying drawing used required in embodiment or description of the prior art will be briefly described below, apparently, accompanying drawing in the following describes is only embodiments of the invention, for those of ordinary skill in the art, under the premise not paying creative work, it is also possible to obtain other accompanying drawing according to the accompanying drawing provided.
Fig. 1 is the configuration diagram that website uses load balancing;
The structured flowchart of the access log combination system that Fig. 2 provides for the embodiment of the present invention;
Fig. 3 merges the flow chart of method for the access log that the embodiment of the present invention provides;
Fig. 4 merges another flow chart of method for the access log that the embodiment of the present invention provides;
Fig. 5 merges another flow chart of method for the access log that the embodiment of the present invention provides;
The structured flowchart of the log processing server that Fig. 6 provides for the embodiment of the present invention;
The functional frame composition of the first script that Fig. 7 provides for the embodiment of the present invention;
The functional frame composition of the second script that Fig. 8 provides for the embodiment of the present invention.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is only a part of embodiment of the present invention, rather than whole embodiments. Based on the embodiment in the present invention, the every other embodiment that those of ordinary skill in the art obtain under not making creative work premise, broadly fall into the scope of protection of the invention.
The structured flowchart of the access log combination system that Fig. 2 provides for the embodiment of the present invention, the access log that the embodiment of the present invention provides merges method can based on system shown in Figure 2, with reference to Fig. 2, the access log combination system that the embodiment of the present invention provides may include that proxy server 1, multiple web servers 2, log processing server 3;
Wherein, proxy server 1 is arranged at Website front-end, user can be accessed and branch to multiple web server 2;
Each web server 2 is arranged at rear end, website, can process user and access, and record user accesses the access log of correspondence;
The server for merging access log that log processing server 3 is arranged for the embodiment of the present invention, log processing server 3 can collect the access log replicating each web server 2 record, and carry out the merging of access log, obtain merging access log, in order to merging access log can be downloaded and analyze;
Optionally, log processing server 3 can belong to same cluster with multiple web servers 2, can pass through interior Network Communication.
In conjunction with access log combination system shown in Fig. 2, below with the angle of log processing server, the access log that the embodiment of the present invention is provided merges method and is introduced.
Fig. 3 merges the flow chart of method for the access log that the embodiment of the present invention provides, and the method can be applicable to log processing server, and with reference to Fig. 3, the method may include that
Step S100, replicating the access log that each web server records, wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server;
Access log is that user is when accessing website, the user that web server records accesses the log information of website, its content includes but not limited to: access the IP of user, access chained address, the mode (browser of use) accessed, web server response time, the response time etc. of the whole browsing process of user. Under normal circumstances, access log all can take daily record round, according to the date, access log carries out segmentation storage;
The embodiment of the present invention is by multiple web servers, user's access to be tapped, therefore for a certain user being split being accessed, the daily record name of the access log that the plurality of web server records is identical, only is undertaken processing record by different web servers; Therefore for a certain user being split being accessed, the embodiment of the present invention is have identical daily record name and journal format from the access log that each web server is collected, and it is the source distinguishing access log, the suffix of the daily record name of each access log after collecting the access log that each web server records, can be carried the server identification (such as server ip) of upper corresponding web server by the embodiment of the present invention;
Optionally, when replicating the access log that each web server records, the embodiment of the present invention can determine that the server identification of each web server, the information such as the daily record name of the access log that need to replicate, server identification and each web server thereby through each web server communicate, and collect access log corresponding with the daily record name of the access log that need to replicate from each web server;
Optionally, the access log replicated can leave the memory area specified in.
Step S110, from the access log replicated, it is determined that daily record name is identical but access log set that server identification that suffix of daily record name carries is different;
Optionally, for a certain user being split is accessed, the access log that server identification that same log name in the access log replicated but suffix carry is different, constitute the overall access log that this user being split accesses, it is necessary to merge; The embodiment of the present invention can in the access log replicated, and daily record name is identical but multiple access logs that server identification that suffix of daily record name carries is different are gathered, and assembles access log set; Access log in access log set is regarded as needs and is merged.
Step S120, the access log in each described access log set is merged mutually, obtain the merging access log that each described access log set is corresponding.
For each access log set, the content of each access log in access log set can be carried out duplicate removal and sequence by the embodiment of the present invention, obtains the log content merged, and renaming has the file of log content of this merging, obtains merging access log;
The content of each access log in access log set be carried out the purpose of duplicate removal, be the information in order to remain with, it is to avoid the information of repetition repeatedly occurs, reduce data volume; The mode that the content of access log is ranked up, it is possible to carry out according to the concrete time of log content record;
Optionally, the embodiment of the present invention can be placed in download directory by merging access log, in order to staff can download merging access log, carries out the work such as follow-up analysis.
The access log that the embodiment of the present invention provides merges method, can be applicable in log processing server; The access log that the reproducible each web server of log processing server records, wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server, and from the access log replicated, determine that daily record name is identical but the different access log set of server identification that suffix of daily record name carries, and then the access log in each described access log set is merged mutually, obtain the merging access log that each described access log set is corresponding. Can be seen that, the access log that the embodiment of the present invention provides merges method, can on the suffix of the daily record name of the access log collected from web server, carry the server identification of corresponding web server, so that it is determined that daily record name is identical but access log that server identification that suffix of daily record name carries is different needs to merge, and then realizes the automatic merging of access log. The access log that the embodiment of the present invention provides merges method, it may be achieved the access log collecting multiple web server records automatically the purpose merged.
Optionally, log processing server can be determined by the server identification of each web server, and the login password of each web server, it is achieved each web server of Telnet, and then the daily record name based on the access log that need to replicate copies corresponding access log from each web server;
Corresponding, Fig. 4 illustrates that the access log that the embodiment of the present invention provides merges another flow process of method, and the method can be applicable to log processing server, and with reference to Fig. 4, the method may include that
Step S200, determine the server identification of each web server, the daily record name of the access log that need to replicate, and the login password of each web server;
Step S210, according to the server identification of each web server and login password, each web server of Telnet, corresponding for the daily record name with the described access log that need to replicate of each web server record access log is replicated;
Step S220, add the server identification of corresponding web server at the suffix of the daily record name of the access log replicated;
Step S230, from the access log replicated, it is determined that daily record name is identical but access log set that server identification that suffix of daily record name carries is different;
Step S240, the access log in each described access log set is merged mutually, obtain the merging access log that each described access log set is corresponding.
Optionally, the log content of each access log in access log set can be exported to setting in download directory by the embodiment of the present invention, and then the log content of each access log of output is carried out duplicate removal and sequence, obtain the log content merged, and renaming has the file of log content of merging, obtain merging access log.
Optionally, the embodiment of the present invention can be passed through to arrange the first script and the second script realizes the merging of access log, second script is called by the first script, the Telnet of each web server can be realized, and copying access log from each web server, the access log replicated can be merged by the first script;
Optionally, in method shown in Fig. 3, the second script can be called by the first script and perform to replicate the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the flow process of the server identification of corresponding web server; First script can perform from the access log replicated, determine that daily record name is identical but the different access log set of server identification that suffix of daily record name carries, and the access log in each described access log set is merged mutually, obtain the flow process merging access log that each described access log set is corresponding;
Accordingly, Fig. 5 illustrates that the access log that the embodiment of the present invention provides merges another flow process of method, and the method can be applicable to log processing server, and with reference to Fig. 5, the method may include that
Step S300, perform the first script and call the second script by the first script, passing to described second script by first script server identification by each web server, the daily record name of the access log that need to replicate, and the login password of each web server;
Step S310, by crus secunda this according to the server identification of each web server and login password, the each web server of Telnet, and perform remote copy order by described second script, corresponding for the daily record name with the described access log that need to replicate of each web server record access log is replicated;
Step S320, added the server identification of corresponding web server at the suffix of the daily record name of the access log replicated by the second script;
Step S330, by the first script from the access log replicated, it is determined that daily record name is identical but access log set that server identification that suffix of daily record name carries is different;
Step S340, for each described access log set, by the first script, the log content of each access log in access log set is exported to setting in download directory, the log content of each access log of output is carried out duplicate removal and sequence, obtain the log content merged, renaming has the file of the log content of merging, obtains merging access log.
Optionally, the first script can as main code, and filename can be log_collect.sh, and the code of the first script can be such that
Optionally, the second script can as auxiliary code, and for Telnet web server and replicate access log, File name can be scp_expect.sh, and the code of the second script can be such that
Optionally, the access log method that the embodiment of the present invention provides can plan to perform, as the first script and the second script can every day 4:00 AM perform once, copy daily record also merges, and is placed on and can be used in the web catalogue downloaded, the catalogue that web downloads as:
14***sh/home/shell/log_dispose/log_collect.sh��
First script (log_collect.sh script), the execution flow process of the second script (scp_expect.sh script) can be such that
When performing log_collect.sh script, log_collect.sh will call scp_expect.sh script, and by web server list the server identification of each web server (record have), the log list daily record name of the access log that need to replicate (record have), date by the daily record of round, and the position deposited after copy daily record, the variable such as password of web server is transferred to scp_expect.sh script;
The scp_expect.sh script server identification according to the web server received, it is necessary to the daily record name of the daily record of duplication, uses remote copy scp order by daily record reproduction to the machine; Can under normal circumstances, during First Contact Connections, all point out Areyousureyouwanttocontinueconnecting (yes/no)? input yes is here needed to continue; Scp_expect.sh processes already by expect, it is ensured that the automatization of script;
After scp_expect.sh has performed, journal file is downloaded to/data/nginx_log/ catalogue under, and have changed filename and deposit (server ip that the suffix in daily record name adds corresponding web server) respectively;
The access log that server identification that identical for daily record name but daily record name suffix can be carried by log_collect.sh script is different, uses the cat order of linux to check log content; The log content checked all exports to standard output, then redirection " > is used " symbol, and content is carried out duplicate removal and sequence, and then the content of duplicate removal and the standard output of sequence is all exported to being available in the web catalogue downloaded, and again rename, obtain merging access log; Optionally, duplicate removal and sequence are before content exports new file (namely merging access log), namely carry out when log content all exports to standard output.
Finally give authority by the file merging access log so that it is he user has permission access.
It should be noted that, in the embodiment of the present invention, user accesses the pattern of website is adopt 1 to the access module more than 1 pair, namely 1 crowd of user accesses 1 IP and is tapped by multiple web servers, and this IP is front-end proxy agent server, and rear end has multiple web server shunting user to access; The web server of rear end has identical program, therefore accesses for the user of a certain shunting, and the access log of the web server record of rear end is not various, and they have identical daily record name, identical form; So the access log collected of the embodiment of the present invention needs the IP that the daily record name suffix of daily record simply adds server that changes as mark; Also it is that the daily record of different server IP suffix is merged into a file, it is ensured that the accuracy of merging by same log name during merging.
Meanwhile, although the embodiment of the present invention uses remote copy, but log processing server and web server are in same cluster, are use Intranet to copy, and therefore the copy stability of access log and speed can be protected.
Additionally, it is perform in a serial fashion that the access log that the embodiment of the present invention provides merges the process logic of method, although arranging log processing server to carry out daily record merging (4:00 AM as previously described) in certain time period, although consumption has instantaneous increasing, but overall consumption is in a state comparatively smoothly, and operation efficiency and speed on log processing server there is no too big impact.
The log processing the server below embodiment of the present invention provided is introduced, and log processing server described below can merge method with above-described access log mutually to should refer to.
The structured flowchart of the log processing server that Fig. 6 provides for the embodiment of the present invention, with reference to Fig. 6, this log processing server may include that
First script 100, replicates, for calling the second script, the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server; And from the access log replicated, determine that daily record name is identical but the different access log set of server identification that suffix of daily record name carries, access log in each described access log set is merged mutually, obtains the merging access log that each described access log set is corresponding;
Second script 200, for being called by the first script, replicates the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server.
Optionally, Fig. 7 illustrates the functional frame composition of the first script that the embodiment of the present invention provides, and with reference to Fig. 7, the first script may include that
Transfer unit 110, for passing to described second script by the server identification of each web server, the daily record name of the access log that need to replicate, and the login password of each web server;
Unit 120 is determined in set, for from the access log replicated, it is determined that daily record name is identical but access log set that server identification that suffix of daily record name carries is different;
Merge unit 130, for for each described access log set, the log content of each access log in access log set is exported to setting in download directory, the log content of each access log of output is carried out duplicate removal and sequence, obtain the log content merged, renaming has the file of the log content of merging, obtains merging access log.
Accordingly, Fig. 8 illustrates the functional frame composition of the second script that the embodiment of the present invention provides, and with reference to Fig. 7, this second script may include that
Telnet copied cells 210, for the server identification according to each web server and login password, the each web server of Telnet, and perform remote copy order, corresponding for the daily record name with the described access log that need to replicate of each web server record access log is replicated;
Mark adding device 220, for the server identification of the corresponding web server of suffix interpolation of the daily record name at the access log replicated.
The embodiment of the present invention also provides for a kind of access log combination system, and the structure of this access log combination system can as in figure 2 it is shown, include: proxy server, multiple web servers, log processing server;
Wherein, described proxy server branches to the plurality of web server for user being accessed;
Described web server, accesses the access log of correspondence for recording user;
Described log processing server, for replicating the access log that each web server records, wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server; From the access log replicated, it is determined that daily record name is identical but access log set that server identification that suffix of daily record name carries is different; Access log in each described access log set is merged mutually, obtains the merging access log that each described access log set is corresponding.
Optionally, described log processing server can have the first script and the second script;
Described first script, replicates, for calling the second script, the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server; And from the access log replicated, determine that daily record name is identical but the different access log set of server identification that suffix of daily record name carries, access log in each described access log set is merged mutually, obtains the merging access log that each described access log set is corresponding;
Described second script, for being called by the first script, replicates the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server.
The embodiment of the present invention can realize collecting the access log of multiple web server records and merging automatically so that the merging of access log can be concisely and efficiently and carry out.
In this specification, each embodiment adopts the mode gone forward one by one to describe, and what each embodiment stressed is the difference with other embodiments, between each embodiment identical similar portion mutually referring to. For device disclosed in embodiment, owing to it corresponds to the method disclosed in Example, so what describe is fairly simple, relevant part illustrates referring to method part.
Professional further appreciates that, the unit of each example described in conjunction with the embodiments described herein and algorithm steps, can with electronic hardware, computer software or the two be implemented in combination in, in order to clearly demonstrate the interchangeability of hardware and software, generally describe composition and the step of each example in the above description according to function. These functions perform with hardware or software mode actually, depend on application-specific and the design constraint of technical scheme. Professional and technical personnel specifically can should be used for using different methods to realize described function to each, but this realization is it is not considered that beyond the scope of this invention.
The method described in conjunction with the embodiments described herein or the step of algorithm can directly use the software module that hardware, processor perform, or the combination of the two is implemented. Software module can be placed in any other form of storage medium known in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable ROM, depositor, hard disk, moveable magnetic disc, CD-ROM or technical field.
Described above to the disclosed embodiments, makes professional and technical personnel in the field be capable of or uses the present invention. The multiple amendment of these embodiments be will be apparent from for those skilled in the art, and generic principles defined herein can without departing from the spirit or scope of the present invention, realize in other embodiments. Therefore, the present invention is not intended to be limited to the embodiments shown herein, and is to fit to the widest scope consistent with principles disclosed herein and features of novelty.
Claims (10)
1. an access log merges method, it is characterised in that be applied to log processing server, and described access log merges method and includes:
Replicating the access log that each web server records, wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server;
From the access log replicated, it is determined that daily record name is identical but access log set that server identification that suffix of daily record name carries is different;
Access log in each described access log set is merged mutually, obtains the merging access log that each described access log set is corresponding.
2. access log according to claim 1 merges method, it is characterised in that the access log that each web server of described duplication records includes:
Determine the server identification of each web server, the daily record name of the access log that need to replicate, and the login password of each web server;
Server identification according to each web server and login password, each web server of Telnet, replicate corresponding for the daily record name with the described access log that need to replicate of each web server record access log;
Suffix in the daily record name of the access log replicated adds the server identification of corresponding web server.
3. access log according to claim 1 merges method, it is characterised in that described merged mutually by access log in each described access log set, and the merging access log obtaining each described access log set corresponding includes:
For each described access log set, the log content of each access log in access log set is exported to setting in download directory, the log content of each access log of output is carried out duplicate removal and sequence, obtain the log content merged, renaming has the file of the log content of merging, obtains merging access log.
4. access log according to claim 2 merges method, it is characterised in that the described server identification determining each web server, the daily record name of the access log that need to replicate, and the login password of each web server include:
Perform the first script and call the second script by the first script, passing to described second script by first script server identification by each web server, the daily record name of the access log that need to replicate, and the login password of each web server;
The described server identification according to each web server and login password, each web server of Telnet, corresponding for the daily record name with the described access log that need to replicate of each web server record access log is carried out duplication and includes:
By crus secunda this according to the server identification of each web server and login password, the each web server of Telnet, and perform remote copy order by described second script, corresponding for the daily record name with the described access log that need to replicate of each web server record access log is replicated;
The suffix of the described daily record name at the access log replicated adds the server identification of corresponding web server and includes:
Added the server identification of corresponding web server at the suffix of the daily record name of the access log replicated by the second script.
5. access log according to claim 4 merges method, it is characterised in that by described first script from the access log replicated, it is determined that daily record name is identical but access log set that server identification that suffix of daily record name carries is different; And by the first script, the access log in each described access log set is merged mutually, obtain the merging access log that each described access log set is corresponding.
6. a log processing server, it is characterised in that including:
First script, replicates, for calling the second script, the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server; And from the access log replicated, determine that daily record name is identical but the different access log set of server identification that suffix of daily record name carries, access log in each described access log set is merged mutually, obtains the merging access log that each described access log set is corresponding;
Second script, for being called by the first script, replicates the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server.
7. log processing server according to claim 6, it is characterised in that described first script includes:
Transfer unit, for passing to described second script by the server identification of each web server, the daily record name of the access log that need to replicate, and the login password of each web server;
Described second script includes:
Telnet copied cells, for the server identification according to each web server and login password, the each web server of Telnet, and perform remote copy order, corresponding for the daily record name with the described access log that need to replicate of each web server record access log is replicated;
Mark adding device, for the server identification of the corresponding web server of suffix interpolation of the daily record name at the access log replicated.
8. log processing server according to claim 6, it is characterised in that described first script includes:
Merge unit, for for each described access log set, the log content of each access log in access log set is exported to setting in download directory, the log content of each access log of output is carried out duplicate removal and sequence, obtain the log content merged, renaming has the file of the log content of merging, obtains merging access log.
9. an access log combination system, it is characterised in that including: proxy server, multiple web servers, log processing server;
Wherein, described proxy server branches to the plurality of web server for user being accessed;
Described web server, accesses the access log of correspondence for recording user;
Described log processing server, for replicating the access log that each web server records, wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server; From the access log replicated, it is determined that daily record name is identical but access log set that server identification that suffix of daily record name carries is different; Access log in each described access log set is merged mutually, obtains the merging access log that each described access log set is corresponding.
10. access log combination system according to claim 9, it is characterised in that described log processing server has the first script and the second script;
Described first script, replicates, for calling the second script, the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server; And from the access log replicated, determine that daily record name is identical but the different access log set of server identification that suffix of daily record name carries, access log in each described access log set is merged mutually, obtains the merging access log that each described access log set is corresponding;
Described second script, for being called by the first script, replicates the access log that each web server records, and wherein the suffix of the daily record name of each access log carries the server identification of corresponding web server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610023241.7A CN105635301B (en) | 2016-01-14 | 2016-01-14 | A kind of access log merging method, log processing server and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610023241.7A CN105635301B (en) | 2016-01-14 | 2016-01-14 | A kind of access log merging method, log processing server and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105635301A true CN105635301A (en) | 2016-06-01 |
| CN105635301B CN105635301B (en) | 2018-12-21 |
Family
ID=56049772
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610023241.7A Active CN105635301B (en) | 2016-01-14 | 2016-01-14 | A kind of access log merging method, log processing server and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105635301B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107704357A (en) * | 2017-08-28 | 2018-02-16 | 北京小度信息科技有限公司 | Daily record generation method and device |
| CN109088764A (en) * | 2018-08-15 | 2018-12-25 | 郑州悉知信息科技股份有限公司 | Access request processing method and relevant device |
| CN109672546A (en) * | 2017-10-16 | 2019-04-23 | 比亚迪股份有限公司 | Log generation method, application server, log server and log generating system |
| CN111737090A (en) * | 2020-08-25 | 2020-10-02 | 北京志翔科技股份有限公司 | Log simulation method and device, computer equipment and storage medium |
| CN111913913A (en) * | 2020-08-07 | 2020-11-10 | 星辰天合（北京）数据科技有限公司 | Access request processing method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6760763B2 (en) * | 1999-08-27 | 2004-07-06 | International Business Machines Corporation | Server site restructuring |
| CN101163046A (en) * | 2007-11-22 | 2008-04-16 | 北京金山软件有限公司 | Distributed website log data acquisition method and distributed website system |
| CN101197700A (en) * | 2006-12-05 | 2008-06-11 | 阿里巴巴公司 | Method and system for providing log service |
| US20110040811A1 (en) * | 2009-08-17 | 2011-02-17 | International Business Machines Corporation | Distributed file system logging |
| CN102480489A (en) * | 2010-11-30 | 2012-05-30 | 北京千橡网景科技发展有限公司 | Logging method and device used in distributed environment |
-
2016
- 2016-01-14 CN CN201610023241.7A patent/CN105635301B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6760763B2 (en) * | 1999-08-27 | 2004-07-06 | International Business Machines Corporation | Server site restructuring |
| CN101197700A (en) * | 2006-12-05 | 2008-06-11 | 阿里巴巴公司 | Method and system for providing log service |
| CN101163046A (en) * | 2007-11-22 | 2008-04-16 | 北京金山软件有限公司 | Distributed website log data acquisition method and distributed website system |
| US20110040811A1 (en) * | 2009-08-17 | 2011-02-17 | International Business Machines Corporation | Distributed file system logging |
| CN102480489A (en) * | 2010-11-30 | 2012-05-30 | 北京千橡网景科技发展有限公司 | Logging method and device used in distributed environment |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107704357A (en) * | 2017-08-28 | 2018-02-16 | 北京小度信息科技有限公司 | Daily record generation method and device |
| CN107704357B (en) * | 2017-08-28 | 2020-12-15 | 北京星选科技有限公司 | Log generation method and device |
| CN109672546A (en) * | 2017-10-16 | 2019-04-23 | 比亚迪股份有限公司 | Log generation method, application server, log server and log generating system |
| CN109088764A (en) * | 2018-08-15 | 2018-12-25 | 郑州悉知信息科技股份有限公司 | Access request processing method and relevant device |
| CN109088764B (en) * | 2018-08-15 | 2022-04-22 | 郑州悉知信息科技股份有限公司 | Access request processing method and related equipment |
| CN111913913A (en) * | 2020-08-07 | 2020-11-10 | 星辰天合（北京）数据科技有限公司 | Access request processing method and device |
| CN111913913B (en) * | 2020-08-07 | 2024-02-13 | 北京星辰天合科技股份有限公司 | Access request processing method and device |
| CN111737090A (en) * | 2020-08-25 | 2020-10-02 | 北京志翔科技股份有限公司 | Log simulation method and device, computer equipment and storage medium |
| CN111737090B (en) * | 2020-08-25 | 2020-12-01 | 北京志翔科技股份有限公司 | Log simulation method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105635301B (en) | 2018-12-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105635301A (en) | Access log merging method and log processing server and system | |
| JP6280155B2 (en) | Techniques for network replication | |
| US9135257B2 (en) | Technique for implementing seamless shortcuts in sharepoint | |
| CN106484828B (en) | Distributed internet data rapid acquisition system and acquisition method | |
| US11403262B2 (en) | Local networked storage linked to remote networked storage system | |
| JP5160556B2 (en) | Log file analysis method and system based on distributed computer network | |
| Zdziarski | iPhone forensics: recovering evidence, personal data, and corporate assets | |
| CN104714878B (en) | A kind of method and device of collector journal data | |
| CN109359263B (en) | User behavior feature extraction method and system | |
| TW201351264A (en) | System and method for storing distributed documents | |
| CN101330406A (en) | System and method for monitoring WAP imperfect picture | |
| CN103095530A (en) | Method and system for sensitive information monitoring and leakage prevention based on front-end gateway | |
| CN107368331A (en) | The configuration file update method and device of application program | |
| US20190080016A1 (en) | Systems and methods for updating data pipelines | |
| US20160321140A1 (en) | Methods and systems of a dedupe storage network for image management | |
| CN104361007B (en) | The processing method of browser and its collection | |
| CN107181626B (en) | Method and system for monitoring network bandwidth of distributed storage cluster system | |
| JP4422742B2 (en) | Full-text search system | |
| CN107844542A (en) | A kind of distributed document storage method and device | |
| CN106815248B (en) | Website analysis method and device | |
| CN111048164A (en) | Medical big data long-term storage system | |
| CN103581343A (en) | Http (hyper text transport protocol) request recording method | |
| Sah | A New Architecture for Managing Enterprise Log Data. | |
| CN114978963A (en) | Network system monitoring analysis method and device, electronic equipment and storage medium | |
| CN113704659A (en) | Equipment terminal access marking method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |