CN105635301B - A kind of access log merging method, log processing server and system - Google Patents
A kind of access log merging method, log processing server and system Download PDFInfo
- Publication number
- CN105635301B CN105635301B CN201610023241.7A CN201610023241A CN105635301B CN 105635301 B CN105635301 B CN 105635301B CN 201610023241 A CN201610023241 A CN 201610023241A CN 105635301 B CN105635301 B CN 105635301B
- Authority
- CN
- China
- Prior art keywords
- log
- access log
- access
- server
- web server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the present invention provides a kind of access log merging method, log processing server and system, method includes: the access log that each web server of duplication is recorded, wherein the suffix of the log name of each access log carries the server identification of corresponding web server；From the access log replicated, the different access log set of the server identification that log name is identical but the suffix of log name carries is determined；Access log in each access log set is mutually merged, the corresponding merging access log of each access log set is obtained.The embodiment of the present invention can realize the purpose automatically collecting the access log of multiple web server records and merging.
Description
Technical field
The present invention relates to journaling technique fields, and in particular to a kind of access log merging method, log processing service
Device and system.
Background technique
When access log is that user accesses website, the user that web server is recorded accesses the log information of website；In net
That stands safeguards and in use, can often be related to the analysis work of access log, by analyzing access log, for determination
Website failure, exception, and the operation of adjustment website and optimisation strategy are of great significance.
However, the analysis work of access log will become complicated when website uses load balancing；As shown in Figure 1
Website uses the configuration diagram of load balancing, and to shunt website visiting pressure, user will be proxied to the access of website
Server is diverted in multiple web servers, and user, which will access access log corresponding to website, to be remembered by multiple web servers
Record.
Therefore, when website uses load balancing, the analysis work of access log need to be established by multiple web services
Under the premise of the access log of device record merges；Based on this, the access of multiple web server records how is automatically collected
Log simultaneously merges, and to provide basis for the analysis work of access log, it is in need of consideration to become those skilled in the art
Problem.
Summary of the invention
In view of this, the embodiment of the present invention provides a kind of access log merging method, log processing server and system, with
Realize the purpose automatically collecting the access log of multiple web server records and merging.
To achieve the above object, the embodiment of the present invention provides the following technical solutions:
A kind of access log merging method, is applied to log processing server, and the access log merging method includes:
The access log that each web server is recorded is replicated, wherein the suffix of the log name of each access log carries pair
Answer the server identification of web server；
From the access log replicated, determine that the server identification that log name is identical but the suffix of log name carries is different
Access log set；
Access log in each access log set is mutually merged, the corresponding conjunction of each access log set is obtained
And access log.
Optionally, the access log that each web server of the duplication is recorded includes:
Determine the server identification of each web server, the log name of the access log that need to be replicated and each web server
Login password；
According to the server identification and login password of each web server, each web server of Telnet, by each web services
The access log corresponding with the log name of the access log that need to be replicated of device record is replicated；
The server identification of corresponding web server is added in the suffix of the log name of the access log replicated.
Optionally, the access log by each access log set mutually merges, and obtains each access log
Gathering corresponding merging access log includes:
For each access log set, the log content of each access log in access log set is exported to setting
Determine in download directory, duplicate removal and sequence, combined log content, weight are carried out to the log content of each access log of output
The file with combined log content is named, obtains merging access log.
Optionally, the server identification of each web server of the determination, the log name for the access log that need to be replicated, and it is each
The login password of web server includes:
It executes the first script and the second script is called by the first script, by the first script by the clothes of each web server
Business device mark, the log name for the access log that need to be replicated and the login password of each web server pass to second script；
The server identification and login password according to each web server, each web server of Telnet, by each web
The access log corresponding with the log name of the access log that need to be replicated of server record carries out duplication
Pass through this server identification and login password according to each web server of crus secunda, each web services of Telnet
Device, and by the second script execution remote copy order is recording each web server with the access day that need to be replicated
The corresponding access log of log name of will is replicated；
The suffix of the log name in the access log replicated adds the server identification packet of corresponding web server
It includes:
The server of corresponding web server is added in the suffix of the log name of the access log replicated by the second script
Mark.
Optionally, through first script from the access log replicated, determine that log name is identical but log name
The different access log set of the server identification that suffix carries；And it will be in each access log set by the first script
Access log mutually merges, and obtains the corresponding merging access log of each access log set.
The embodiment of the present invention also provides a kind of log processing server, comprising:
First script, for calling the second script to replicate the access log that each web server is recorded, wherein respectively access day
The suffix of the log name of will carries the server identification of corresponding web server；And from the access log replicated, day is determined
The different access log set of the server identification that will name is identical but the suffix of log name carries, by each access log set
In access log mutually merge, obtain the corresponding merging access log of each access log set；
Second script replicates the access log that each web server is recorded, wherein each visit for being called by the first script
Ask that the suffix of the log name of log carries the server identification of corresponding web server.
Optionally, first script includes:
Transfer unit, for by the server identification of each web server, the log name for the access log that need to be replicated, and it is each
The login password of web server passes to second script；
Second script includes:
Telnet copied cells, for the server identification and login password according to each web server, Telnet is each
Web server, and remote copy order is executed, the log with the access log that need to be replicated that each web server is recorded
The corresponding access log of name is replicated；
Adding unit is identified, the suffix for the log name in the access log replicated adds corresponding web server
Server identification.
Optionally, first script includes:
Combining unit is used for for each access log set, by the day of each access log in access log set
Will content is exported into setting download directory, is carried out duplicate removal and sequence to the log content of each access log of output, is closed
And log content, rename have combineds log content file, obtain merging access log.
The embodiment of the present invention also provides a kind of access log combination system, comprising: proxy server, multiple web servers,
Log processing server；
Wherein, the proxy server is used to user's access branching to the multiple web server；
The web server accesses corresponding access log for recording user；
The log processing server, the access log recorded for replicating each web server, wherein each access log
The suffix of log name carry the server identification of corresponding web server；From the access log replicated, log name is determined
The different access log set of the server identification that identical but log name suffix carries；It will be in each access log set
Access log mutually merges, and obtains the corresponding merging access log of each access log set.
Optionally, the log processing server has the first script and the second script；
First script, for calling the second script to replicate the access log that each web server is recorded, wherein each visit
Ask that the suffix of the log name of log carries the server identification of corresponding web server；And from the access log replicated, really
The different access log set of the server identification that settled date will name is identical but the suffix of log name carries, by each access log
Access log in set mutually merges, and obtains the corresponding merging access log of each access log set；
Second script, for replicating the access log that each web server is recorded by the first script calling, wherein
The suffix of the log name of each access log carries the server identification of corresponding web server.
Based on the above-mentioned technical proposal, access log merging method provided in an embodiment of the present invention, can be applied to log processing
In server；The access log that the reproducible each web server of log processing server is recorded, wherein the log of each access log
The suffix of name carries the server identification of corresponding web server, and from the access log replicated, determines that log name is identical
But the different access log set of the server identification that the suffix of log name carries, and then will be in each access log set
Access log mutually merges, and obtains the corresponding merging access log of each access log set.It can be seen that the embodiment of the present invention
The access log merging method of offer can be carried and be corresponded on the suffix of the log name for the access log collected from web server
The server identification of web server, so that it is determined that the server identification that log name is identical but the suffix of log name carries is different
Access log needs to merge, and then realizes the automatic merging of access log.Access log merging method provided in an embodiment of the present invention,
The purpose automatically collecting the access log of multiple web server records and merging can be achieved.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of invention for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is the configuration diagram that website uses load balancing；
Fig. 2 is the structural block diagram of access log combination system provided in an embodiment of the present invention；
Fig. 3 is the flow chart of access log merging method provided in an embodiment of the present invention；
Fig. 4 is another flow chart of access log merging method provided in an embodiment of the present invention；
Fig. 5 is another flow chart of access log merging method provided in an embodiment of the present invention；
Fig. 6 is the structural block diagram of log processing server provided in an embodiment of the present invention；
Fig. 7 is the functional frame composition of the first script provided in an embodiment of the present invention；
Fig. 8 is the functional frame composition of the second script provided in an embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
Fig. 2 is the structural block diagram of access log combination system provided in an embodiment of the present invention, provided in an embodiment of the present invention
Access log merging method can be based on system shown in Figure 2, referring to Fig. 2, access log combination system provided in an embodiment of the present invention
It may include: proxy server 1, multiple web servers 2, log processing server 3；
Wherein, proxy server 1 is set to Website front-end, can access user and branch to multiple web servers 2；
Each web server 2 is set to website rear end, can handle user's access, and record user accesses corresponding access log；
Log processing server 3 is setting of the embodiment of the present invention for merging the server of access log, log processing
Server 3 collects the access log that each web server 2 of duplication records, and carries out the merging of access log, obtains merging access
Log can be downloaded and analyze to merge access log；
Optionally, log processing server 3 can belong to same cluster with multiple web servers 2, can pass through interior Network Communication.
Access log combination system as shown in connection with fig. 2, below with the angle of log processing server, to the embodiment of the present invention
The access log merging method of offer is introduced.
Fig. 3 is the flow chart of access log merging method provided in an embodiment of the present invention, and this method can be applied at log
Server is managed, referring to Fig. 3, this method may include:
Step S100, the access log that each web server is recorded is replicated, wherein the suffix of the log name of each access log
Carry the server identification of corresponding web server；
When access log is that user accesses website, the user that web server is recorded accesses the log information of website, in
Appearance includes but is not limited to: accessing the IP of user, accesses chained address, the mode (browser used) of access, web server is rung
Between seasonable, the response time etc. of the entire browsing process of user.Under normal conditions, access log can all take log to rotate, according to
Access log is split storage by the date；
The embodiment of the present invention is to be shunted by multiple web servers to user's access, therefore be split for a certain
User access for, the log name for the access log that multiple web server is recorded be it is identical, only by difference
Web server carry out processing record；Therefore for the user's access being split for a certain, the embodiment of the present invention is from each web
The access log that server is collected is log name having the same and journal format, and is the source for distinguishing access log, this
Inventive embodiments can be after being collected into the access log that each web server is recorded, by the suffix of the log name of each access log
The server identification (such as server ip) of web server is corresponded in carrying；
Optionally, when replicating the access log that each web server is recorded, the embodiment of the present invention can determine each web clothes
The server identification of business device, the information such as log name for the access log that need to be replicated, thus the server mark for passing through each web server
Knowledge is communicated with each web server, is collected into from each web server corresponding with the log name of access log that need to be replicated
Access log；
Optionally, the access log replicated can be stored in specified storage region.
Step S110, from the access log replicated, the service that log name is identical but the suffix of log name carries is determined
Device identifies different access log set；
Optionally, for the user's access being split for a certain, in the access log that is replicated same log name but after
Sew the different access log of the server identification of carrying, constitutes the whole access log of the user being split access, need
It merges；The embodiment of the present invention can be in the access log replicated, and log name is identical but the suffix of log name carries
The different multiple access logs of server identification are gathered, and access log set is assembled；Access in access log set
Log is regarded as needing to merge.
Step S120, the access log in each access log set is mutually merged, obtains each access log collection
Close corresponding merging access log.
For each access log set, the embodiment of the present invention can by the content of each access log in access log set into
Row duplicate removal and sequence, combined log content, and the file with the log content of the merging is renamed, it obtains merging and visit
Ask log；
The purpose that the content of each access log in access log set is carried out to duplicate removal, is to retain useful letter
Breath, avoids duplicate information from repeatedly occurring, and reduces data volume；It, can foundation to the mode that the content of access log is ranked up
The specific time of log content record carries out；
Optionally, the embodiment of the present invention can by merge access log be placed in download directory, so as to staff can under
It carries and merges access log, carry out the work such as subsequent analysis.
Access log merging method provided in an embodiment of the present invention, can be applied in log processing server；Log processing
The access log that the reproducible each web server of server is recorded, wherein the suffix of the log name of each access log carries pair
The server identification of web server is answered, and from the access log replicated, determines that log name is identical but the suffix of log name is taken
The different access log set of the server identification of band, and then the access log in each access log set is mutually merged,
Obtain the corresponding merging access log of each access log set.It can be seen that access log provided in an embodiment of the present invention
Merging method can carry the clothes of corresponding web server on the suffix of the log name for the access log collected from web server
It is engaged in device mark, so that it is determined that the different access log of the server identification that log name is identical but the suffix of log name carries needs to close
And and then realize access log automatic merging.Access log merging method provided in an embodiment of the present invention is, it can be achieved that automatically
The purpose collecting the access log of multiple web server records and merging.
Optionally, log processing server can server identification by each web server of determination and each web server
Login password, realize each web server of Telnet, and then the log name based on the access log that need to be replicated takes from each web
Corresponding access log is copied in business device；
Corresponding, Fig. 4 shows another process of access log merging method provided in an embodiment of the present invention, and this method can
Applied to log processing server, referring to Fig. 4, this method may include:
Step S200, the server identification of each web server is determined, the log name for the access log that need to be replicated and each web
The login password of server；
Step S210, according to the server identification and login password of each web server, each web server of Telnet will
The access log corresponding with the log name of the access log that need to be replicated of each web server record is replicated；
Step S220, the server mark of corresponding web server is added in the suffix of the log name of the access log replicated
Know；
Step S230, from the access log replicated, the service that log name is identical but the suffix of log name carries is determined
Device identifies different access log set；
Step S240, the access log in each access log set is mutually merged, obtains each access log collection
Close corresponding merging access log.
Optionally, the embodiment of the present invention can export the log content of each access log in access log set to setting
In download directory, and then duplicate removal and sequence carried out to the log content of each access log of output, combined log content,
And the file with combined log content is renamed, it obtains merging access log.
Optionally, the embodiment of the present invention can realize the merging of access log by the first script of setting and the second script, the
Two scripts are called the Telnet, it can be achieved that each web server by the first script, and visit is replicated from each web server
Ask log, the first script can merge the access log replicated；
Optionally, in method shown in Fig. 3, the second script can be called each web server of execution duplication to be remembered by the first script
The access log of record, wherein the suffix of the log name of each access log carries the stream of the server identification of corresponding web server
Journey；First script is executable from the access log replicated, determines the service that log name is identical but the suffix of log name carries
Device identifies different access log set, and the access log in each access log set is mutually merged, and obtains each described
The corresponding process for merging access log of access log set；
Correspondingly, Fig. 5 shows another process of access log merging method provided in an embodiment of the present invention, this method can
Applied to log processing server, referring to Fig. 5, this method may include:
Step S300, it executes the first script and the second script is called by the first script, taken each web by the first script
It is engaged in the server identification of device, the log name for the access log that need to be replicated and the login password of each web server pass to described the
Two scripts；
Step S310, by crus secunda this according to the server identification and login password of each web server, Telnet is each
Web server, and by the second script execution remote copy order, what each web server was recorded needs to replicate with described
The corresponding access log of log name of access log replicated；
Step S320, corresponding web services are added in the suffix of the log name of the access log replicated by the second script
The server identification of device；
Step S330, through the first script from the access log replicated, after determining that log name is identical but log name
Sew the different access log set of the server identification of carrying；
Step S340, for each access log set, by the first script by each access in access log set
The log content of log is exported into setting download directory, carries out duplicate removal and row to the log content of each access log of output
Sequence, combined log content rename the file with combined log content, obtain merging access log.
Optionally, the first script can be used as main code, and filename can be log_collect.sh, the generation of the first script
Code can be such that
Optionally, the second script can be used as auxiliary code, for Telnet web server and replicate access log,
File name can be scp_expect.sh, and the code of the second script can be such that
Optionally, access log method provided in an embodiment of the present invention can plan to execute, such as the first script and crus secunda
Originally it 4:00 AM can execute once daily, copy log and simultaneously merge, and be placed in the catalogue that can be used for web downloading, web downloading
Catalogue such as:
14***sh/home/shell/log_dispose/log_collect.sh。
The execution process of first script (log_collect.sh script), the second script (scp_expect.sh script) can
With as follows:
When executing log_collect.sh script, log_collect.sh will call scp_expect.sh script, and will
Web server list (server identification that record has each web server), (record has the access log that need to be replicated to log list
Log name), the date of the log rotated, and copy log after store position, the variables such as password of web server pass
It is defeated by scp_expect.sh script；
Server identification of the scp_expect.sh script according to the web server received, the day for the log for needing to replicate
Will name, using remote copy scp order by daily record reproduction to the machine；Under normal circumstances, Are can be all prompted when First Contact Connections
You sure you want to continue connecting (yes/no)? need to input yes continuation herein；scp_
It is handled by expect in expect.sh, ensure that the automation of script；
After the completion of scp_expect.sh execution, journal file has been downloaded to/data/nginx_log/ catalogue under,
And it has changed filename and stores (joined the server ip of corresponding web server in the suffix of log name) respectively；
Log_collect.sh script can the server identification that carries of log name is identical but log name suffix it is different
Access log checks log content using the cat order of linux；All output is to standard output for the log content checked, then
Duplicate removal and sequence are carried out using redirection " > " symbol, and to content, and then duplicate removal and the content of the standard output of sequence is whole
Output renames again into the catalogue for web downloading, obtains merging access log；Optionally, duplicate removal and sequence be
Before content is output to new file (i.e. merging access log), that is, in log content all output to standard outputs
It carries out.
It finally assigns the file for merging access log to permission, has permission to access other users.
It should be noted that it is using 1 pair of access mould more than 1 pair that user, which accesses the mode of website, in the embodiment of the present invention
Formula, i.e. 1 crowd of user access 1 IP and are shunted by multiple web servers, this IP is front-end proxy agent server, and there are multiple web in rear end
Server shunts user's access；The web server of rear end program having the same, therefore the user of a certain shunting is accessed,
The access log of the web server record of rear end is not multiplicity, they have identical log name, identical format；So this hair
Access log that bright embodiment is collected into need it is to be changed only using the log name suffix of log added the IP of server as
Mark；It is also by same log name when merging, the log of different server IP suffix is merged into a file, guarantees combined standard
True property.
Meanwhile although the embodiment of the present invention uses remote copy, but log processing server and web server are same
It is to be copied using Intranet, therefore the copy stability and rate of access log can be protected in one cluster.
In addition, the processing logic of access log merging method provided in an embodiment of the present invention be execute in a serial fashion, though
So setting log processing server carries out log merging (4:00 AM as previously described) in certain time period, although consumption
Have it is instantaneous increase, but overall consumption is in a more stable state, to the operation efficiency and speed of log processing server
Degree has no too big influence.
Log processing server provided in an embodiment of the present invention is introduced below, log processing service described below
Device can correspond to each other reference with above-described access log merging method.
Fig. 6 is the structural block diagram of log processing server provided in an embodiment of the present invention, referring to Fig. 6, log processing clothes
Business device may include:
First script 100, for calling the second script to replicate the access log that each web server is recorded, wherein each visit
Ask that the suffix of the log name of log carries the server identification of corresponding web server；And from the access log replicated, really
The different access log set of the server identification that settled date will name is identical but the suffix of log name carries, by each access log
Access log in set mutually merges, and obtains the corresponding merging access log of each access log set；
Second script 200 replicates the access log that each web server is recorded, wherein respectively for being called by the first script
The suffix of the log name of access log carries the server identification of corresponding web server.
Optionally, Fig. 7 shows the functional frame composition of the first script provided in an embodiment of the present invention, referring to Fig. 7, the first foot
Originally may include:
Transfer unit 110, for by the server identification of each web server, the log name for the access log that need to be replicated, and
The login password of each web server passes to second script；
Gather determination unit 120, for from the access log replicated, determining that log name is identical but the suffix of log name
The different access log set of the server identification of carrying；
Combining unit 130 is used for for each access log set, by each access log in access log set
Log content is exported into setting download directory, is carried out duplicate removal and sequence to the log content of each access log of output, is obtained
Combined log content renames the file with combined log content, obtains merging access log.
Correspondingly, Fig. 8 shows the functional frame composition of the second script provided in an embodiment of the present invention, referring to Fig. 7, this second
Script may include:
Telnet copied cells 210 are remotely stepped on for the server identification and login password according to each web server
Record each web server, and execute remote copy order, each web server is recorded with the access log that need to be replicated
The corresponding access log of log name is replicated；
Adding unit 220 is identified, the suffix for the log name in the access log replicated adds corresponding web server
Server identification.
The embodiment of the present invention also provides a kind of access log combination system, and the structure of the access log combination system can be such as figure
Shown in 2, comprising: proxy server, multiple web servers, log processing server；
Wherein, the proxy server is used to user's access branching to the multiple web server；
The web server accesses corresponding access log for recording user；
The log processing server, the access log recorded for replicating each web server, wherein each access log
The suffix of log name carry the server identification of corresponding web server；From the access log replicated, log name is determined
The different access log set of the server identification that identical but log name suffix carries；It will be in each access log set
Access log mutually merges, and obtains the corresponding merging access log of each access log set.
Optionally, the log processing server can have the first script and the second script；
First script, for calling the second script to replicate the access log that each web server is recorded, wherein each visit
Ask that the suffix of the log name of log carries the server identification of corresponding web server；And from the access log replicated, really
The different access log set of the server identification that settled date will name is identical but the suffix of log name carries, by each access log
Access log in set mutually merges, and obtains the corresponding merging access log of each access log set；
Second script, for replicating the access log that each web server is recorded by the first script calling, wherein
The suffix of the log name of each access log carries the server identification of corresponding web server.
The embodiment of the present invention can realize the access log for automatically collecting multiple web server records and merge, and make
The merging for obtaining access log can be concisely and efficiently progress.
Each embodiment in this specification is described in a progressive manner, the highlights of each of the examples are with other
The difference of embodiment, the same or similar parts in each embodiment may refer to each other.For device disclosed in embodiment
For, since it is corresponded to the methods disclosed in the examples, so being described relatively simple, related place is said referring to method part
It is bright.
Professional further appreciates that, unit described in conjunction with the examples disclosed in the embodiments of the present disclosure
And algorithm steps, can be realized with electronic hardware, computer software, or a combination of the two, in order to clearly demonstrate hardware and
The interchangeability of software generally describes each exemplary composition and step according to function in the above description.These
Function is implemented in hardware or software actually, the specific application and design constraint depending on technical solution.Profession
Technical staff can use different methods to achieve the described function each specific application, but this realization is not answered
Think beyond the scope of this invention.
The step of method described in conjunction with the examples disclosed in this document or algorithm, can directly be held with hardware, processor
The combination of capable software module or the two is implemented.Software module can be placed in random access memory (RAM), memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of storage medium well known in field.
The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present invention.
Various modifications to these embodiments will be readily apparent to those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, of the invention
It is not intended to be limited to the embodiments shown herein, and is to fit to and the principles and novel features disclosed herein phase one
The widest scope of cause.
Claims (10)
1. a kind of access log merging method, which is characterized in that be applied to log processing server, the access log merging side
Method includes:
The access log that each web server is recorded is replicated, wherein the suffix of the log name of each access log carries corresponding web
The server identification of server；
From the access log replicated, the different visit of the server identification that log name is identical but the suffix of log name carries is determined
Ask log set；
Access log in each access log set is mutually merged, the corresponding merging of each access log set is obtained and visits
Ask log.
2. access log merging method according to claim 1, which is characterized in that each web server of duplication is remembered
The access log of record includes:
Determine the server identification of each web server, the login of the log name and each web server of the access log that need to be replicated
Password；
According to the server identification and login password of each web server, each web server of Telnet remembers each web server
The access log corresponding with the log name of the access log that need to be replicated of record is replicated；
The server identification of corresponding web server is added in the suffix of the log name of the access log replicated.
3. access log merging method according to claim 1, which is characterized in that described by each access log set
In access log mutually merge, obtaining the corresponding merging access log of each access log set includes:
For each access log set, the log content of each access log in access log set is exported to setting
It carries in catalogue, duplicate removal and sequence, combined log content, renaming is carried out to the log content of each access log of output
File with combined log content obtains merging access log.
4. access log merging method according to claim 2, which is characterized in that the clothes of each web server of determination
Business device mark, the log name for the access log that need to be replicated and the login password of each web server include:
It executes the first script module and the second script module is called by first script module, pass through the first script mould
Block is by the server identification of each web server, the log name for the access log that need to be replicated and the login password of each web server
Pass to second script module；
The server identification and login password according to each web server, each web server of Telnet, by each web services
The access log corresponding with the log name of the access log that need to be replicated of device record carries out duplication
By the second script module according to the server identification and login password of each web server, each web services of Telnet
Device, and remote copy order is executed by second script module, it is that each web server is recorded with the visit that need to be replicated
Ask that the corresponding access log of log name of log is replicated；
The server identification that the suffix of the log name in the access log replicated adds corresponding web server includes:
The clothes of corresponding web server are added in the suffix of the log name of the access log replicated by second script module
Business device mark.
5. access log merging method according to claim 4, which is characterized in that by first script module from institute
In the access log of duplication, the different access log collection of the server identification that log name is identical but the suffix of log name carries is determined
It closes；And mutually merged the access log in each access log set by first script module, obtain each visit
Ask log set corresponding merging access log.
6. a kind of log processing server characterized by comprising
First script module, for calling the second script module to replicate the access log that each web server is recorded, wherein each visit
Ask that the suffix of the log name of log carries the server identification of corresponding web server；And from the access log replicated, really
The different access log set of the server identification that settled date will name is identical but the suffix of log name carries, by each access log
Access log in set mutually merges, and obtains the corresponding merging access log of each access log set；
Second script module replicates the access day that each web server is recorded for being called by first script module
Will, wherein the suffix of the log name of each access log carries the server identification of corresponding web server.
7. log processing server according to claim 6, which is characterized in that first script module includes:
Transfer unit, for by the server identification of each web server, the log name for the access log that need to be replicated and each web to be taken
The login password of business device passes to second script module；
Second script module includes:
Telnet copied cells, for the server identification and login password according to each web server, each web of Telnet
Server, and remote copy order is executed, the log famous prime minister with the access log that need to be replicated that each web server is recorded
The access log answered is replicated；
Adding unit is identified, the suffix for the log name in the access log replicated adds the service of corresponding web server
Device mark.
8. log processing server according to claim 6, which is characterized in that first script module includes:
Combining unit is used for for each access log set, will be in the log of each access log in access log set
Hold output into setting download directory, duplicate removal and sequence are carried out to the log content of each access log of output, merged
Log content renames the file with combined log content, obtains merging access log.
9. a kind of access log combination system characterized by comprising proxy server, multiple web servers, log processing
Server；
Wherein, the proxy server is used to user's access branching to the multiple web server；
The web server accesses corresponding access log for recording user；
The log processing server, the access log recorded for replicating each web server, wherein the day of each access log
The suffix of will name carries the server identification of corresponding web server；From the access log replicated, determine that log name is identical
But the different access log set of the server identification that the suffix of log name carries；By the access in each access log set
Log mutually merges, and obtains the corresponding merging access log of each access log set.
10. access log combination system according to claim 9, which is characterized in that the log processing server has
First script module and the second script module；
First script module, for calling second script module to replicate the access log that each web server is recorded,
Wherein the suffix of the log name of each access log carries the server identification of corresponding web server；And from the access replicated
In log, the different access log set of the server identification that log name is identical but the suffix of log name carries, Jiang Gesuo are determined
The access log stated in access log set mutually merges, and obtains the corresponding merging access log of each access log set；
Second script module replicates the access day that each web server is recorded for being called by first script module
Will, wherein the suffix of the log name of each access log carries the server identification of corresponding web server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610023241.7A CN105635301B (en) | 2016-01-14 | 2016-01-14 | A kind of access log merging method, log processing server and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610023241.7A CN105635301B (en) | 2016-01-14 | 2016-01-14 | A kind of access log merging method, log processing server and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105635301A CN105635301A (en) | 2016-06-01 |
| CN105635301B true CN105635301B (en) | 2018-12-21 |
Family
ID=56049772
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610023241.7A Active CN105635301B (en) | 2016-01-14 | 2016-01-14 | A kind of access log merging method, log processing server and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105635301B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107704357B (en) * | 2017-08-28 | 2020-12-15 | 北京星选科技有限公司 | Log generation method and device |
| CN109672546A (en) * | 2017-10-16 | 2019-04-23 | 比亚迪股份有限公司 | Log generation method, application server, log server and log generating system |
| CN109088764B (en) * | 2018-08-15 | 2022-04-22 | 郑州悉知信息科技股份有限公司 | Access request processing method and related equipment |
| CN111913913B (en) * | 2020-08-07 | 2024-02-13 | 北京星辰天合科技股份有限公司 | Access request processing method and device |
| CN111737090B (en) * | 2020-08-25 | 2020-12-01 | 北京志翔科技股份有限公司 | Log simulation method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6760763B2 (en) * | 1999-08-27 | 2004-07-06 | International Business Machines Corporation | Server site restructuring |
| CN101163046A (en) * | 2007-11-22 | 2008-04-16 | 北京金山软件有限公司 | Distributed website log data acquisition method and distributed website system |
| CN101197700A (en) * | 2006-12-05 | 2008-06-11 | 阿里巴巴公司 | Method and system for providing log service |
| CN102480489A (en) * | 2010-11-30 | 2012-05-30 | 北京千橡网景科技发展有限公司 | Logging method and device used in distributed environment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8868601B2 (en) * | 2009-08-17 | 2014-10-21 | International Business Machines Corporation | Distributed file system logging |
-
2016
- 2016-01-14 CN CN201610023241.7A patent/CN105635301B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6760763B2 (en) * | 1999-08-27 | 2004-07-06 | International Business Machines Corporation | Server site restructuring |
| CN101197700A (en) * | 2006-12-05 | 2008-06-11 | 阿里巴巴公司 | Method and system for providing log service |
| CN101163046A (en) * | 2007-11-22 | 2008-04-16 | 北京金山软件有限公司 | Distributed website log data acquisition method and distributed website system |
| CN102480489A (en) * | 2010-11-30 | 2012-05-30 | 北京千橡网景科技发展有限公司 | Logging method and device used in distributed environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105635301A (en) | 2016-06-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105635301B (en) | A kind of access log merging method, log processing server and system | |
| JP6280155B2 (en) | Techniques for network replication | |
| Luotonen | Web proxy servers | |
| Abd Wahab et al. | Data pre-processing on web server logs for generalized association rules mining algorithm | |
| US7472132B2 (en) | Attributing memory usage by individual software components | |
| Zdziarski | iPhone forensics: recovering evidence, personal data, and corporate assets | |
| US20090083714A1 (en) | Remote monitoring of local behavior of network applications | |
| CN108141380A (en) | Network-based resource distribution finds service | |
| CN104331369B (en) | Page detection method and device, server based on browser | |
| MX2008008509A (en) | Method and apparatus for a distributed file storage and indexing service. | |
| CN107885658B (en) | Test preposition implementation method, device, terminal device and storage medium | |
| CN109617924A (en) | A kind of account usage behavior detection method and device | |
| CN102739423B (en) | Method, apparatus and network management system for performance data acquisition | |
| Sah | A New Architecture for Managing Enterprise Log Data. | |
| Simoneau | SNMP network management | |
| Singh et al. | Hierarchical multi-log cloud-based search engine | |
| JP2002526858A (en) | Software system development framework | |
| Spillner et al. | RAFT-REST-A client-side framework for reliable, adaptive and fault-tolerant restful service consumption | |
| Hassan et al. | Analyzing digital evidence | |
| US20090083363A1 (en) | Remote monitoring of local behavior of network applications | |
| Manesh et al. | Forensic investigation framework for P2P protocol | |
| Bai | Developing Java Web Services to Access Databases | |
| Huawei Technologies Co., Ltd. | Introduction to Cloud Computing Computing | |
| Kelly | An extensible framework for creating personal archives of web resources requiring authentication | |
| Singh | Monitoring Hadoop |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |