CN107239382A - The log processing method and system of a kind of container application - Google Patents
The log processing method and system of a kind of container application Download PDFInfo
- Publication number
- CN107239382A CN107239382A CN201710487826.9A CN201710487826A CN107239382A CN 107239382 A CN107239382 A CN 107239382A CN 201710487826 A CN201710487826 A CN 201710487826A CN 107239382 A CN107239382 A CN 107239382A
- Authority
- CN
- China
- Prior art keywords
- log
- daily record
- record storage
- information
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a kind of log processing method of container application and system, the log processing system application server for solving prior art causes the problem of over-burden.Present invention method includes：The Log Directory that multiple containers are applied in log collection subsystem application server carries out content detection, and Log Directory is used for the log information that storage container is applied；When detecting Log Directory and there is daily record renewal, log collection subsystem obtains the log information updated；Log collection subsystem sends the log information of renewal to daily record storage subsystem；Daily record storage subsystem parse obtaining log recording to the log information of renewal, and stores log recording.
Description
Technical field
The present invention relates to field of computer technology, and in particular to the log processing method and system of a kind of container application.
Background technology
The application program disposed on one server will be hundreds of, and the application program run on server will be recorded and applied
The relevant information and error message of the related daily record of program, i.e. application program operation, and the daily record storage of record is arrived specified
Log Directory, so that programmer searches and analyzes daily record, application program is accordingly debugged, meanwhile, also allow for O＆M people
Member carries out wrong investigation to system based on the daily record found and handled.Big operation system includes a large amount of servers, and every
Extensive application program is run on server, programmer or operation maintenance personnel need the day to the extensive application program on a large amount of servers
Will is searched and analyzed, it is therefore desirable to is put into very big manpower and is carried out log processing work, and time-consuming, and efficiency is low.
For reduction human cost, the efficiency of log processing work is improved, prior art proposes some log collection instruments,
Such as log4j, is the system for application log collection that Apache increases income, collections of the log4j using application program as daily record
Object, it is necessary in each application program of exploitation integrated log4j log collection client modules, collect application program day
Will, and log4j log pattern is called, daily record is written on a unified log server.
Under container environment, application program highly dense is distributed on the server, and each application program is required for integrated
Log4j client modules are simultaneously each worked, and the daily record of collection is write to unified log server, and each server will be disposed
The log4j client modules of quantity identical with application program, make server burden, meanwhile, each application program is respectively required for
Calling and programming for log4j client modules is carried out, the burden of programmer has been aggravated.
The content of the invention
The present invention provides the log processing method and system of a kind of container application, the log processing for solving prior art
System application server causes the problem of over-burden.
The one side of the embodiment of the present invention provides a kind of log processing method of container application, including：
The Log Directory that multiple containers are applied in log collection subsystem application server carries out content detection, the day
Will catalogue is used for the log information for storing the container application；
When detecting the Log Directory and there is daily record renewal, the log collection subsystem obtains the daily record letter updated
Breath；
The log collection subsystem sends the log information of the renewal to daily record storage subsystem；
The daily record storage subsystem parse obtaining log recording to the log information of the renewal, and stores described
Log recording.
Optionally, the log recording includes one in daily record generation time, application name, container name and specific log content
Item or multinomial log parameters information.
Optionally, after the daily record storage subsystem stores the log recording, methods described also includes：
The daily record storage subsystem sets up daily record index to the log recording with the log parameters information, and stores
The daily record index.
Optionally, after the daily record storage subsystem stores the daily record index, methods described also includes：
Log query server receives the log query instruction that user inputs, the daily record by log query web interface
Query statement includes target journaling parameter information；
The log query server is described by calling log query interface to be obtained from the daily record storage subsystem
Log query instructs corresponding target journaling record, and returns to Query Result to user.
Optionally, the log collection subsystem sends the log information of the renewal to daily record storage subsystem bag
Include：
The log collection subsystem sends the log information of the renewal to Message Queuing server；
The Message Queuing server caches to the log information of the renewal；
The daily record storage subsystem obtains the log information of the renewal from the Message Queuing server.
Optionally, the daily record storage subsystem includes many daily record storage servers, and the daily record storage subsystem is deposited
Storing up the log recording or daily record index includes：
The daily record storage subsystem carries out data fragmentation to the log recording or daily record index, obtains daily record
Burst or index burst；
The daily record storage subsystem stores the scattered write-in many daily records of the daily record burst or index burst
Server.
The second aspect of the embodiment of the present invention provides a kind of log processing method system of container application, including daily record is received
Subsystem and daily record storage subsystem；
Log collection subsystem is used for：
The Log Directory that multiple containers are applied in application server carries out content detection, and the Log Directory is used to store
The log information of the container application；
When detecting the Log Directory and there is daily record renewal, the log information updated is obtained；
The log information of the renewal is sent to daily record storage subsystem；
The daily record storage subsystem is used for：
The log information of the renewal parse obtaining log recording, and stores the log recording.
Optionally, the log recording includes one in daily record generation time, application name, container name and specific log content
Item or multinomial log parameters information.
Optionally, the daily record storage system is additionally operable to：
Daily record index is set up to the log recording with the log parameters information, and stores the daily record index.
Optionally, the log processing system also includes log query server, and the log query server is used for：
The log query instruction that user inputs is received by log query web interface, the log query instruction includes mesh
Mark log parameters information；
It is corresponding by calling log query interface to obtain the log query instruction from the daily record storage subsystem
Target journaling is recorded, and returns to Query Result to user.
Optionally, the log processing system also includes Message Queuing server, and the Message Queuing server is used for：
Receive the log information for the renewal that the log collection subsystem is sent；
The log information of the renewal is cached；
The daily record storage subsystem is additionally operable to obtain the log information of the renewal from the Message Queuing server.
Optionally, the daily record storage subsystem includes many daily record storage servers, the daily record storage subsystem tool
Body is used for：
Data fragmentation is carried out to the log recording or daily record index, daily record burst or index burst is obtained；
By the daily record burst or the scattered write-in many daily record storage servers of index burst.
As can be seen from the above technical solutions, the embodiment of the present invention has advantages below：
The embodiment of the present invention can utilize the daily record mesh that multiple containers are applied in log collection subsystem application server
Record carries out content detection, and when detecting Log Directory and there is daily record renewal, log collection subsystem can get renewal
Log information, afterwards log collection subsystem the log information of renewal can be sent to daily record storage subsystem, daily record storage
Subsystem can parse obtain log recording to the log information of renewal, and store log recording.Because the present invention is only needed
The Log Directory that multiple containers are applied in application server carries out content detection, just can obtain multiple appearances in the application server
The log information that device application updates, and daily record storage subsystem is uniformly uploaded to, therefore, the present invention is on an application server
A log collection subsystem only need to be installed, the Log Directory applied for detection container compared to the prior art, can be with
Mitigate the burden of server, mitigate the burden of programmer and operation maintenance personnel.
Brief description of the drawings
Fig. 1 is log processing method one embodiment schematic diagram of inventive container application；
Fig. 2 is another embodiment schematic diagram of the log processing method of inventive container application；
Fig. 3 is log query web interface one embodiment schematic diagram that log query server of the present invention is provided；
Fig. 4 is another embodiment schematic diagram of the log query web interface of log query server offer of the present invention；
Fig. 5 is another embodiment schematic diagram of the log processing method of inventive container application；
Fig. 6 is log processing system one embodiment schematic diagram of inventive container application；
Fig. 7 is another embodiment schematic diagram of the log processing system of inventive container application.
Embodiment
The embodiments of the invention provide a kind of log processing method of container application and system, in log processing system
Deployment on reduce application server burden.
In order that those skilled in the art more fully understand the present invention program, below in conjunction with the embodiment of the present invention
Accompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is only
The embodiment of a part of the invention, rather than whole embodiments.Based on the embodiment in the present invention, ordinary skill people
The every other embodiment that member is obtained under the premise of creative work is not made, should all belong to the model that the present invention is protected
Enclose.
Term " first ", " second ", " the 3rd " in description and claims of this specification and above-mentioned accompanying drawing, "
The (if present)s such as four " are for distinguishing similar object, without for describing specific order or precedence.It should manage
The data that solution is so used can be exchanged in the appropriate case, so that the embodiments described herein can be with except illustrating herein
Or the order beyond the content of description is implemented.In addition, term " comprising " and " having " and their any deformation, it is intended that
Covering is non-exclusive to be included, for example, containing process, method, system, product or the equipment of series of steps or unit need not limit
In those steps or unit for clearly listing, but may include not list clearly or for these processes, method, production
Product or the intrinsic other steps of equipment or unit.
For ease of understanding, the idiographic flow in the embodiment of the present invention is described below, referring to Fig. 1, the present invention is real
Applying log processing method one embodiment of container application in example includes：
101st, the Log Directory that multiple containers are applied in log collection subsystem application server carries out content detection；
Log processing system can include log collection subsystem, and log collection subsystem can be with more in application server
The Log Directory of individual container application carries out content detection, and the log recording of container application is used for the daily record letter that storage container is applied
Breath.
102nd, when detecting Log Directory and there is daily record renewal, log collection subsystem obtains the log information updated；
When daily record collection subsystem, which detects Log Directory, there is daily record renewal, log collection subsystem can be obtained more
New log information.
103rd, log collection subsystem sends the log information of renewal to daily record storage subsystem；
Log collection subsystem is got after the log information of container application renewal, can be by the day of the renewal got
Will information is sent to daily record storage subsystem.
104th, daily record storage subsystem parse obtaining log recording to the log information of renewal, and stores daily record note
Record.
Daily record storage subsystem can receive the log information of the renewal of log collection subsystem transmission, afterwards can be right
The log information of the renewal received is parsed, and obtains log recording, and obtained log recording is stored.
The embodiment of the present invention can utilize the daily record mesh that multiple containers are applied in log collection subsystem application server
Record carries out content detection, and when detecting Log Directory and there is daily record renewal, log collection subsystem can get renewal
Log information, afterwards log collection subsystem the log information of renewal can be sent to daily record storage subsystem, daily record storage
Subsystem can parse obtain log recording to the log information of renewal, and store log recording.Because the present invention is only needed
The Log Directory that multiple containers are applied in application server carries out content detection, just can obtain multiple appearances in the application server
The log information that device application updates, and daily record storage subsystem is uniformly uploaded to, therefore, the present invention is on an application server
A log collection subsystem only need to be installed, the Log Directory applied for detection container compared to the prior art, can be with
Mitigate the burden of server, mitigate the burden of programmer.
Log processing system can also include Message Queuing server and log query server, referring to Fig. 2, of the invention
Another embodiment of the log processing method of container application includes in embodiment：
201st, log collection subsystem collects the log information of container application in application server；
The present invention can dispose log collection module in units of server, that is to say, that on each application server
Dispose a log collection module, the log collection module composition log collection subsystem of many application server deployments.Daily record
Collection module will observe the Log Directory specified, when specified Log Directory has daily record renewal, and log collection module will be automatic
Collector journal.Wherein, the Log Directory specified can be used for storing the daily record of all or part of container application in application server
Information.By step 201, day is collected in multiple applications that log collection subsystem can be simultaneously from the multiple servers of multiple clusters
Will information.
202nd, log collection subsystem sends log information to Message Queuing server；
Log collection subsystem is collected into after the log information of container application, can be sent log information to message team
Row server.
203rd, the log information that Message Queuing server collects to log collection subsystem is cached；
Message Queuing server can cache to the log information received.In actual use, daily record storage
System is deployed in different servers from log collection subsystem, it is generally the case that can provide one or more server in addition,
For disposing daily record storage subsystem, if log collection subsystem directly sends log information into daily record storage subsystem,
Because the speed of log collection subsystem and daily record storage subsystem is mismatched log information may be caused to lose.Therefore, step
202 and step 203 log collection subsystem and daily record storage subsystem can be decoupled, realize asynchronous between two subsystems
Change.
It is preferred that, Message Queuing server can be formed a team to dispose two-by-two, that is to say, that two-server is matched somebody with somebody as one group
Master-slave synchronisation pattern is set to, is realized when main Message Queuing server delays machine, Message Queuing server still can be at another
Normal work on server.
204th, daily record storage subsystem obtains the log information cached in Message Queuing server；
Log information is buffered in after Message Queuing server, daily record storage subsystem can obtain Message Queuing Services
The log information cached in device is stored into daily record memory module.It is preferred that, daily record storage subsystem can be with monitoring information queue
Server, when Message Queuing server stores new log information, daily record storage subsystem will be from Message Queuing server
Obtain new log information.
205th, daily record storage subsystem storage log information, and be parsed into log recording；
Daily record storage subsystem is got after new log information, can store the log information of renewal, and by daily record
Information is parsed into the log recording of unified form, writes daily record storage server.
One log recording can include the fields such as daily record generation time, application name, container name and specific log content.1)
The daily record generation time can be yy-mm-dd h:m:S form, such as 2017-01-09 01:22:During 12 expression log recording
Generated 12 seconds January 9 day 1 point 22 minute in 2017.2) the entitled maximum length of application is the character string of 100 characters, example
Such as " t3 " or " xcmyt ".3) container name is that a maximum length is 100 characters and has the character string of certain naming rule,
It is " -5 Arabic numerals of character string and 26 lowercases of -10 Arabic numerals compositions of application name that rule, which can be set,
Constitute character string ", such as " t3-3486809255-oh7t0 " or " xcmyt-1422840873-9n43w ".4) specific daily record
Content can include daily record rank, log recording where program segment program name, program line number, daily record description etc. subfield,
Wherein,
A) daily record rank can define five ranks such as " tests, information, warning, mistake, fatal ", and test grades are used to adjust
Try to record fine granularity message event during application program；Levels of information is used for coarse grain information event in application running；Alarm
Rank is used for the record of latent fault；Error level is used for misregistration event, but still does not influence continuing to run with for system；Cause
Life rank is used to record grave error event, it will cause exiting for application program.
B) program name of program segment represents the place program filename that log recording occurs, such as one java program
Example is：com.mysql.jdbc.MySQL；The example of such as one C++ program is：Cbox.
C) program line number：How much rows in the program occur for log, and form can be n line numbers, and such as n200 is represented
Occur in the 200th row.
D) daily record is described：Specific daily record description is recorded, for example, " have read the 1 to 23rd from/opt/data/1.txt files
Row data ".
1 specific example of log recording described above is：
“2017-01-09 01:22:12 | xcmyt | xcmyt-1422840873-9n43w | (information,
Com.mysql.jdbc.MySQL, n123, the 1 to 23rd row data are have read from/opt/data/1.txt files) ", represent 2017
12 seconds January 9 day 1 point 22 minute of year, the xcmyt-1422840873-9n43w containers of xcmyt applications
123 rows of com.mysql.jdbc.MySQL programs generate a daily record rank for " information ", are specifically described as " from/opt/
Data/1.txt files have read the 1 to 23rd row data " log recording.
206th, daily record storage subsystem sets up daily record index to the log recording of storage with log parameters information, and stores；
After step 205, programmer or operation maintenance personnel can search the daily record of container application in daily record storage subsystem
Record, carries out log analysis.For faster procedure person or the log query speed of operation maintenance personnel, daily record storage subsystem can be right
The log recording of storage sets up daily record index on demand, and the daily record index of foundation is stored.Such as, can with cluster name,
The log parameters information such as application name, container name, logging time, keyword, with inquire about the modes such as binary tree set up daily record index, will
Full record traversing operation during log query is changed into the rapid search operations of binary tree.If log recording bar number is N, by building
The time complexity of user's inquiry operation from original O (N) can be reduced to O (logN) by vertical daily record index.Likewise it is preferred that,
Daily record storage subsystem can store daily record index using distributed memory framework mode, so as to further reduce daily record index
Read access time.
Daily record of the query argument that daily record storage subsystem can be inputted according to user to being stored in daily record storage subsystem
Record is quickly searched, and is searched and is determined log recording corresponding with query statement, and Query Result is returned into user.In order to
Facilitate the separation of log query interface and daily record storage subsystem so that user is not used in disposing the clothes of daily record storage subsystem
Log query is carried out on business device, log processing system can set log query server and queried access interface server, with
The mode of web service sets up log query access interface, and the input parameter of interface can include cluster name, application name, container
The fields such as name, daily record initial time, daily record end time and keyword, are output as 1 or a plurality of log recording, provide the user
Easy-to-use web interface, quick search log recording reduces log processing workload.Specifically, refer to following steps 207 to
Step 209.
207th, log query server provides the user log query web interface, and receives the log query of user's input
Instruction；
Log query server can provide the user log query web interface, corresponding to being deposited in daily record storage subsystem
The log recording of storage and the input parameter of queried access interface server, log query web interface can provide the user collection
The Filter Options such as group's name, application name, container name, daily record initial time, daily record end time, and support user to carry out self-defined close
Keyword is inquired about, and user can assign log query instruction, day after log query web interface selection target log parameters information
Will inquiry server can receive the log query instruction of user's input, and the log query instruction received includes requiring to look up
Target journaling record target journaling parameter information.
208th, log query server calls log query interface obtains log query instruction in daily record storage subsystem
Corresponding target journaling record；
Log query server is received after the log query instruction of user's input, and queried access interface can be called to take
The log query interface of business device, data read request is sent to daily record storage subsystem, and data read request includes interface
The value of input parameter, the target journaling parameter information during the value of input parameter is instructed with log query is corresponding, and daily record is deposited
Store up input parameter of the subsystem in data read request and search corresponding target journaling index, pass through target journaling rope afterwards
Draw the corresponding target journaling record of inquiry, and target journaling returned to log query server by log query interface and record,
Log query server can obtain log query in daily record storage subsystem and instruct corresponding target journaling record afterwards, will
The one or more target journaling record display got is on log query web interface.
209th, log query server returns to Query Result to user.
Log query server is got after the corresponding target journaling record of log query instruction, can be returned to user
Query Result, by the one or more target journaling record display got on log query web interface.
In order to illustrate step 207 to step 209, it is assumed that the log query web interface that log query server is provided
As shown in figure 3, being log query web interface in frame 1, frame 2, frame 3, frame 4, frame 5, frame 6 are respectively cluster name, application name, container
Name, logging time scope, the Filter Options of keyword, specific Filter Options can be inserted by User Defined, or
User, which provides, includes the drop-down menu of multiple options, is selected by user to select screening parameter, on log query interface, leads to
Cross combobox and select a nearest time range.For example, the Filter Options of logging time scope, can provide the user 2 points
Optional time range in clock, in 5 minutes, in 30 minutes etc., can be by User Defined time range, self defined time scope
The random time section of input up to the present is supported, the second is accurate to.Frame 7 is inquiry button, when user chooses above-mentioned screening bar
, can be with choice box 7 after part, now log query server can be by server cluster selected above, application example name, appearance
Device name, time range, keyword are packaged into request bag, by log query interface interchange, request bag is sent to daily record and stored
Subsystem.Afterwards log query server can obtain in daily record storage server with looking into that filter information in request bag matches
Result bag is ask, and Query Result bag is parsed, parsing obtains log query result, and is illustrated in the frame at log query interface
In 8.
Deposited because the container application in application server is continuously generated in new log information, therefore daily record storage subsystem
The log recording of storage is being constantly updated, and user sees the log query result of real-time update, log query server for convenience
Auto Scroll refresh function option can be provided a user, as shown in frame 9, when user's choice box 9, log query server can
Just to call log query interface in real time or not every certain time length, request bag is sent to daily record storage subsystem, to obtain
The log query result of renewal.
Log query web interface is returned after log query result, and user can be to the log recording in log query result
Analyzed, when user finds the log recording for needing selective analysis, wish to check and the log recording in many cases
The front and rear multirow log recording of other associated log recordings, such as this log recording, therefore, log query net of the present invention
In the frame 1 in page boundary face, i.e. Fig. 3, the frame 10 in correlation inquiry option, such as Fig. 4 can be included, user wishes to check certain daily record note
During the association log recording of record, it can select to deploy this log recording, now, log query server can be according to user's
Frame 10 in instruction display association log recording window interface, such as Fig. 4, can support the row of user's selection association log recording
Number, can be by information such as the line numbers of the log recording of selection and front/rear log recording after the query statement of user is received
Request bag is packaged into, by log query interface interchange, request bag is sent to daily record storage subsystem.Log query takes afterwards
Business device can obtain the Query Result bag matched in daily record storage server with filter information in request bag, and to Query Result
Bag is parsed, and parsing obtains log query result, and is illustrated in log query interface.
In actual use, it is preferred that log processing system can set many log interface servers, it is preferred that can
To dispose a log query interface on every log interface server, due to the ill-mannered step response of log query interface, i.e.,
Do not associated between multiple inquiry operation, when partial log interface server delays machine, other log interface servers can continue
Log query service is carried, so as to realize the high reliability of log query interface.
It is preferred that, log processing system can set many log query servers, and in every log query server
One log query interface of upper deployment, because log query interface also has ill-mannered step response, when partial log inquires about server
Delay machine when, other log query servers can continue carrying log query interface, so as to realize that the height at log query interface can
By property.
On the corresponding embodiments of Fig. 2, daily record storage subsystem can store log recording and daily record index, in order to realize
High-performance, high reliability and the expansibility of daily record storage subsystem, referring to Fig. 5, the Card read/write of daily record storage subsystem
Method one embodiment includes：
501st, host node is elected from least two daily record storage servers；
High performance daily record storage subsystem generally includes many daily record storage servers, and many daily record storage servers can
Think non-stop layer framework, can therefrom elect a host node to manage other daily record storage servers by election mechanism.It is many
Heartbeat checking mechanism can be retained between platform daily record storage server, such as, every the heartbeat checking mechanism of 10~30 seconds, work as election
Host node out delay machine when, other daily record storage servers in daily record storage subsystem can be sent out by heartbeat checking mechanism
Existing host node is delayed machine event, and re-elects new host node.
502nd, when receiving the write instruction indexed to log recording or daily record, using host node to log recording or daily record
Index carries out data fragmentation and obtains daily record burst or index burst；
Daily record storage subsystem can be provided with daily record storage write service device, when daily record storage subsystem is received to daily record
During the write instruction of record or daily record index, daily record storage write service device can be with the main section in access log storage server
Point, host node can control log recording or daily record to index the storage in daily record storage server.
It is preferred that, host node can carry out data fragmentation to log recording to be written or daily record index, obtain daily record point
Piece or index burst.Log recording and index record can carry out burst according to the burst quantity of a setting, and such as daily record is remembered
Record has a 10GB, and daily record index has 20GB, and daily record storage server number of units is m (m >=1), can set log recording burst number as
2m, daily record index burst is 4m, then log recording will have 2 bursts in each daily record storage server, and daily record index will be
Each daily record storage server has 4 bursts.
503rd, by scattered write-in each daily record storage server of daily record burst or index burst；
Host node is obtained after daily record burst or index burst, daily record burst or index burst can be write into each daily record
Storage server, it is preferred that data fragmentation can be disperseed to be stored in multiple daily record storage servers, carry out data query
When, query actions will occur in many daily record storage servers simultaneously, concurrent operations while realizing multiple servers, so that
Accelerate data query speed.
In addition, the storage indexed for daily record, during due to data query, according to sixteen principles, that is, important minority
With trifling majority, user is frequently necessary to the sub-fraction that log recording is only accounted in the log recording of all storages checked, because
This, it is preferred that the corresponding daily record index of log recording that can often inquire about this part is stored in internal memory, and other daily records
Index is stored in disk, because the read-write efficiency of internal memory is generally higher than disk 1000 times, so in this way, Ke Yi great
Amplitude lifts the inquiry velocity of significant data.
504th, the daily record burst or index burst stored to daily record storage server is replicated, and will replicate obtained day
Will burst or index burst write other daily record storage servers；
Host node is carried out after data fragmentation, and fragment data can be replicated.It can be deposited to data fragmentation
Duplication operation is carried out before storage, duplication operation can also be carried out after the storage of data fragmentation.In embodiments of the present invention, with
Illustrated exemplified by the latter.Host node writes daily record burst or index burst after each daily record storage server, can be right
The daily record burst or index burst of each daily record storage server storage are replicated, and will replicate obtained daily record burst or rope
Draw burst to write in other daily record storage servers, to cause same data fragmentation content backup in multiple daily record storage servers
In.Duplication number is for example set as n (n<M) (if here, m=1, n=0, then 0) number of copies can only take, then each burst
Will replicate n part, and according to an equilibrium strategy try one's best disperse be stored in many daily record storage servers, by daily record burst
Or index burst is replicated and scattered storage, can accelerate the speed of data query, while data storage can also be improved
Reliability, when certain part of data fragmentation (such as server delay machine or data exception) loss because system reason, it is possible to use
Backup Data in other daily record storage servers carries out the recovery of data fragmentation.
505th, when receiving the reading instruction indexed to target journaling, each of host node reading target journaling index is utilized
Burst is indexed；
Daily record storage subsystem can be provided with daily record storage reading service device, when daily record storage subsystem is received to target
During the reading instruction of daily record index, daily record storage reading service device can be with the host node in access log storage server, main section
Point can read target journaling and index corresponding all index bursts.
506th, by the corresponding daily record burst of target journaling index search, so that obtaining target indexes corresponding target journaling
Record.
Host node is found after target journaling index, can be recorded by the corresponding target journaling of target journaling index search
All daily record bursts so that get target journaling record.
In the log processing method of the embodiment of the present invention, solving conventional art needs to be multiple applications on a server
The problem of program disposes the log collection client modules of respective amount, while alleviating programmer needs to compile for log collection
Write the workload of code.Secondly, a kind of method and system of Integrated query is proposed, it is many for server and number of applications
Container application scene, realize a centrally stored daily record and inquiry log processing system, realize a similar search engine
Daily record Integrated query interface, support based on cluster, carry out daily records using multiple dimensions such as, container, time, keyword and check quickly soon
Ask, settlement procedure person or operation maintenance personnel need the server and daily record substantial amounts checking or analyze, it is necessary to put into very big people
Power carries out the workload problem of log processing.
The log processing method of the container application in the embodiment of the present invention is described above, below to of the invention real
The log processing system for applying the container application in example is described.
Referring to Fig. 6, one embodiment of log processing system includes in the embodiment of the present invention：
Log collection subsystem 601 and daily record storage subsystem 602；
Log collection subsystem 601 is used for：
The Log Directory that multiple containers are applied in application server carries out content detection, and Log Directory is used for storage container
The log information of application；
When detecting Log Directory and there is daily record renewal, the log information updated is obtained；
The log information of renewal is sent to daily record storage subsystem；
Daily record storage subsystem 602 is used for：
The log information of renewal parse obtaining log recording, and stores log recording.
Referring to Fig. 7, another embodiment of log processing system includes in the embodiment of the present invention：
Log collection subsystem 701, Message Queuing server 702, daily record storage subsystem 703, log query server
704 and interface server 705.
Wherein, log collection subsystem 701 is integrated in application server 7011, and log collection subsystem 701 is used for：
The Log Directory that multiple containers are applied in application server carries out content detection, and Log Directory is used for storage container
The log information of application；
When detecting Log Directory and there is daily record renewal, the log information updated is obtained；
The log information of renewal is sent to daily record storage subsystem.
Message Queuing server 702 is used for：
Receive the log information for the renewal that log collection subsystem is sent；
The log information of renewal is cached.
Daily record storage subsystem 703 is used for：
The log information of renewal is obtained from Message Queuing server；
The log information of renewal parse obtaining log recording, and stores log recording；
Daily record index is set up to log recording with log parameters information, and stores daily record index.
Daily record storage subsystem is included many daily record storage servers 7031 and (stored in the embodiment of the present invention with three daily records
Exemplified by server), daily record storage subsystem 703 specifically for：
Data fragmentation is carried out to log recording or daily record index, daily record burst or index burst is obtained；
By daily record burst or the scattered many daily record storage servers of write-in of index burst.
Log query subsystem 704 is used for：
The log query instruction that user inputs is received by log query web interface, log query instruction includes target day
Will parameter information；
Corresponding target journaling is instructed by calling log query interface to obtain log query from daily record storage subsystem
Record, and return to Query Result to user.
Interface server 705 is used to provide log query interface for log query subsystem 704.
It is apparent to those skilled in the art that, for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and unit, may be referred to the corresponding process in preceding method embodiment, will not be repeated here.
In described several embodiments, it should be understood that disclosed system, apparatus and method can pass through others
Mode is realized.For example, device embodiment described above is only schematical, for example, the division of unit, is only one kind
Division of logic function, can there is other dividing mode when actually realizing, such as multiple units or component can combine or can
To be integrated into another system, or some features can be ignored, or not perform.It is another, it is shown or discussed each other
Coupling direct-coupling or communication connection can be by some interfaces, the INDIRECT COUPLING or communication connection of device or unit,
Can be electrical, machinery or other forms.
The unit illustrated as separating component can be or may not be physically separate, be shown as unit
Part can be or may not be physical location, you can with positioned at a place, or can also be distributed to multiple networks
On unit.Some or all of unit therein can be selected to realize the purpose of this embodiment scheme according to the actual needs.
In addition, each functional unit in each embodiment of the invention can be integrated in a processing unit, can also
That unit is individually physically present, can also two or more units it is integrated in a unit.Above-mentioned integrated list
Member can both be realized in the form of hardware, it would however also be possible to employ the form of SFU software functional unit is realized.
If integrated unit is realized using in the form of SFU software functional unit and is used as independent production marketing or in use, can
To be stored in a computer read/write memory medium.Based on it is such understand, technical scheme substantially or
Saying all or part of the part contributed to prior art or the technical scheme can be embodied in the form of software product
Out, the computer software product is stored in a storage medium, including some instructions are to cause a computer equipment
(can be personal computer, server, or network equipment etc.) performs all or part of each embodiment method of the invention
Step.And foregoing storage medium includes：USB flash disk, mobile hard disk, read-only storage (ROM, Read-OnlyMemory), deposit at random
Access to memory (RAM, Random Access Memory), magnetic disc or CD etc. are various can be with the medium of store program codes.
More than, the above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although with reference to foregoing reality
Example is applied the present invention is described in detail, it will be understood by those within the art that：It still can be to foregoing each
Technical scheme described in embodiment is modified, or carries out equivalent substitution to which part technical characteristic；And these are changed
Or replace, the essence of appropriate technical solution is departed from the spirit and scope of various embodiments of the present invention technical scheme.
Claims (12)
1. a kind of log processing method of container application, it is characterised in that including：
The Log Directory that multiple containers are applied in log collection subsystem application server carries out content detection, the daily record mesh
Employ in the log information for storing the container application；
When detecting the Log Directory and there is daily record renewal, the log collection subsystem obtains the log information updated；
The log collection subsystem sends the log information of the renewal to daily record storage subsystem；
The daily record storage subsystem parse obtaining log recording to the log information of the renewal, and stores the daily record
Record.
2. the log processing method of container application according to claim 1, it is characterised in that the log recording includes day
One or more of log parameters information in will generation time, application name, container name and specific log content.
3. the log processing method of container application according to claim 2, it is characterised in that the daily record storage subsystem
Store after the log recording, methods described also includes：
The daily record storage subsystem sets up daily record index to the log recording with the log parameters information, and stores described
Daily record is indexed.
4. the log processing method of container application according to claim 3, it is characterised in that the daily record storage subsystem
Store after the daily record index, methods described also includes：
Log query server receives the log query instruction that user inputs, the log query by log query web interface
Instruction includes target journaling parameter information；
The log query server is by calling log query interface to obtain the daily record from the daily record storage subsystem
The corresponding target journaling record of query statement, and return to Query Result to user.
5. the log processing method of container application according to any one of claim 1 to 4, it is characterised in that the day
Will collection subsystem, which sends the log information of the renewal to daily record storage subsystem, to be included：
The log collection subsystem sends the log information of the renewal to Message Queuing server；
The Message Queuing server caches to the log information of the renewal；
The daily record storage subsystem obtains the log information of the renewal from the Message Queuing server.
6. the log processing method of container application according to claim 5, it is characterised in that the daily record storage subsystem
Including many daily record storage servers, the daily record storage subsystem stores the log recording or daily record index bag
Include：
The daily record storage subsystem carries out data fragmentation to the log recording or daily record index, obtains daily record burst
Or index burst；
The write-in many daily record storage services that the daily record storage subsystem disperses the daily record burst or index burst
Device.
7. a kind of log processing system of container application, it is characterised in that including log collection subsystem and daily record storage subsystem
System；
Log collection subsystem is used for：
The Log Directory that multiple containers are applied in application server carries out content detection, and the Log Directory is used to store described
The log information of container application；
When detecting the Log Directory and there is daily record renewal, the log information updated is obtained；
The log information of the renewal is sent to daily record storage subsystem；
The daily record storage subsystem is used for：
The log information of the renewal parse obtaining log recording, and stores the log recording.
8. the log processing system of container application according to claim 7, it is characterised in that the log recording includes day
One or more of log parameters information in will generation time, application name, container name and specific log content.
9. the log processing system of container application according to claim 8, it is characterised in that the daily record storage system is also
For：
Daily record index is set up to the log recording with the log parameters information, and stores the daily record index.
10. the log processing system of container application according to claim 9, it is characterised in that the log processing system
Also include log query server, the log query server is used for：
The log query instruction that user inputs is received by log query web interface, the log query instruction includes target day
Will parameter information；
Corresponding target is instructed by calling log query interface to obtain the log query from the daily record storage subsystem
Log recording, and return to Query Result to user.
11. the log processing system of the container application according to any one of claim 7 to 10, it is characterised in that described
Log processing system also includes Message Queuing server, and the Message Queuing server is used for：
Receive the log information for the renewal that the log collection subsystem is sent；
The log information of the renewal is cached；
The daily record storage subsystem is additionally operable to obtain the log information of the renewal from the Message Queuing server.
12. the log processing system of container application according to claim 11, it is characterised in that the daily record stores subsystem
System includes many daily record storage servers, the daily record storage subsystem specifically for：
Data fragmentation is carried out to the log recording or daily record index, daily record burst or index burst is obtained；
By the daily record burst or the scattered write-in many daily record storage servers of index burst.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710487826.9A CN107239382A (en) | 2017-06-23 | 2017-06-23 | The log processing method and system of a kind of container application |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710487826.9A CN107239382A (en) | 2017-06-23 | 2017-06-23 | The log processing method and system of a kind of container application |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107239382A true CN107239382A (en) | 2017-10-10 |
Family
ID=59987954
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710487826.9A Pending CN107239382A (en) | 2017-06-23 | 2017-06-23 | The log processing method and system of a kind of container application |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107239382A (en) |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107766512A (en) * | 2017-10-23 | 2018-03-06 | 中国联合网络通信集团有限公司 | A kind of daily record data storage method and log data storage system |
| CN107766497A (en) * | 2017-10-19 | 2018-03-06 | 北京京东尚科信息技术有限公司 | The method and terminal of Data Collection based on container |
| CN108132868A (en) * | 2018-01-15 | 2018-06-08 | 政采云有限公司 | A kind of data monitoring method, device, computing device and storage medium |
| CN108170573A (en) * | 2017-12-15 | 2018-06-15 | 珠海金山网络游戏科技有限公司 | The cycle erasing-writing method and system of a kind of mobile terminal daily record |
| CN108446203A (en) * | 2018-03-20 | 2018-08-24 | 万帮充电设备有限公司 | Server transaction log processing method and processing device |
| CN108762984A (en) * | 2018-05-23 | 2018-11-06 | 杭州宏杉科技股份有限公司 | A kind of method and device of continuity data backup |
| CN110096485A (en) * | 2019-03-18 | 2019-08-06 | 平安普惠企业管理有限公司 | Log inquiring method, device, computer equipment and storage medium |
| WO2019161551A1 (en) * | 2018-02-24 | 2019-08-29 | 黄裕才 | Method for processing log based on logrotate technology |
| CN110245059A (en) * | 2019-05-20 | 2019-09-17 | 平安普惠企业管理有限公司 | A kind of data processing method, equipment and storage medium |
| CN110287154A (en) * | 2019-06-06 | 2019-09-27 | 郑州阿帕斯科技有限公司 | The querying method and device of container log |
| CN110399357A (en) * | 2019-06-28 | 2019-11-01 | 苏州浪潮智能科技有限公司 | A kind of method, apparatus and system for managing the log of big data component concentratedly |
| CN110611592A (en) * | 2019-09-20 | 2019-12-24 | 中国联合网络通信集团有限公司 | Log recording method and device |
| CN110688355A (en) * | 2018-06-19 | 2020-01-14 | 北京京东尚科信息技术有限公司 | Method and device for changing container state |
| CN111046011A (en) * | 2019-11-27 | 2020-04-21 | 中科曙光国际信息产业有限公司 | Log collection method, system, node, electronic device and readable storage medium |
| CN111061721A (en) * | 2018-10-16 | 2020-04-24 | 成都鼎桥通信技术有限公司 | Data processing method and device |
| CN111324606A (en) * | 2020-01-23 | 2020-06-23 | 北京恒华伟业科技股份有限公司 | Data fragmentation method and device |
| CN111694793A (en) * | 2020-06-12 | 2020-09-22 | 北京金山云网络技术有限公司 | Log storage method and device and log query method and device |
| CN112363905A (en) * | 2021-01-13 | 2021-02-12 | 北京云真信科技有限公司 | Application log collection system |
| CN113094348A (en) * | 2021-03-19 | 2021-07-09 | 北京达佳互联信息技术有限公司 | Log management method and device |
| US20220004480A1 (en) * | 2019-06-28 | 2022-01-06 | Boe Technology Group Co., Ltd. | Log data collection method, log data collection device, storage medium, and log data collection system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101197700A (en) * | 2006-12-05 | 2008-06-11 | 阿里巴巴公司 | Method and system for providing log service |
| WO2016134072A1 (en) * | 2015-02-20 | 2016-08-25 | Threatstop, Inc. | Normalization and extraction of log data |
-
2017
- 2017-06-23 CN CN201710487826.9A patent/CN107239382A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101197700A (en) * | 2006-12-05 | 2008-06-11 | 阿里巴巴公司 | Method and system for providing log service |
| WO2016134072A1 (en) * | 2015-02-20 | 2016-08-25 | Threatstop, Inc. | Normalization and extraction of log data |
Non-Patent Citations (2)
| Title |
|---|
| 仇臣: "Docker容器的性能监控和日志服务的设计与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 奥凯恩M.杰森: "《机器人操作系统浅析》", 30 September 2016 * |
Cited By (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107766497A (en) * | 2017-10-19 | 2018-03-06 | 北京京东尚科信息技术有限公司 | The method and terminal of Data Collection based on container |
| CN107766512A (en) * | 2017-10-23 | 2018-03-06 | 中国联合网络通信集团有限公司 | A kind of daily record data storage method and log data storage system |
| CN107766512B (en) * | 2017-10-23 | 2021-02-02 | 中国联合网络通信集团有限公司 | Log data storage method and log data storage system |
| CN108170573A (en) * | 2017-12-15 | 2018-06-15 | 珠海金山网络游戏科技有限公司 | The cycle erasing-writing method and system of a kind of mobile terminal daily record |
| CN108132868A (en) * | 2018-01-15 | 2018-06-08 | 政采云有限公司 | A kind of data monitoring method, device, computing device and storage medium |
| WO2019161551A1 (en) * | 2018-02-24 | 2019-08-29 | 黄裕才 | Method for processing log based on logrotate technology |
| CN108446203A (en) * | 2018-03-20 | 2018-08-24 | 万帮充电设备有限公司 | Server transaction log processing method and processing device |
| CN108762984A (en) * | 2018-05-23 | 2018-11-06 | 杭州宏杉科技股份有限公司 | A kind of method and device of continuity data backup |
| CN110688355A (en) * | 2018-06-19 | 2020-01-14 | 北京京东尚科信息技术有限公司 | Method and device for changing container state |
| CN111061721A (en) * | 2018-10-16 | 2020-04-24 | 成都鼎桥通信技术有限公司 | Data processing method and device |
| CN110096485A (en) * | 2019-03-18 | 2019-08-06 | 平安普惠企业管理有限公司 | Log inquiring method, device, computer equipment and storage medium |
| CN110245059A (en) * | 2019-05-20 | 2019-09-17 | 平安普惠企业管理有限公司 | A kind of data processing method, equipment and storage medium |
| WO2020233013A1 (en) * | 2019-05-20 | 2020-11-26 | 平安普惠企业管理有限公司 | Data processing method and device, and storage medium |
| CN110287154A (en) * | 2019-06-06 | 2019-09-27 | 郑州阿帕斯科技有限公司 | The querying method and device of container log |
| US20220004480A1 (en) * | 2019-06-28 | 2022-01-06 | Boe Technology Group Co., Ltd. | Log data collection method, log data collection device, storage medium, and log data collection system |
| CN110399357A (en) * | 2019-06-28 | 2019-11-01 | 苏州浪潮智能科技有限公司 | A kind of method, apparatus and system for managing the log of big data component concentratedly |
| US11755452B2 (en) * | 2019-06-28 | 2023-09-12 | Boe Technology Group Co., Ltd. | Log data collection method based on log data generated by container in application container environment, log data collection device, storage medium, and log data collection system |
| CN110611592A (en) * | 2019-09-20 | 2019-12-24 | 中国联合网络通信集团有限公司 | Log recording method and device |
| CN110611592B (en) * | 2019-09-20 | 2023-04-07 | 中国联合网络通信集团有限公司 | Log recording method and device |
| CN111046011A (en) * | 2019-11-27 | 2020-04-21 | 中科曙光国际信息产业有限公司 | Log collection method, system, node, electronic device and readable storage medium |
| CN111046011B (en) * | 2019-11-27 | 2024-02-23 | 中科曙光国际信息产业有限公司 | Log collection method, system, device, electronic equipment and readable storage medium |
| CN111324606A (en) * | 2020-01-23 | 2020-06-23 | 北京恒华伟业科技股份有限公司 | Data fragmentation method and device |
| CN111694793A (en) * | 2020-06-12 | 2020-09-22 | 北京金山云网络技术有限公司 | Log storage method and device and log query method and device |
| CN112363905B (en) * | 2021-01-13 | 2021-08-24 | 北京云真信科技有限公司 | Application log collection system |
| CN112363905A (en) * | 2021-01-13 | 2021-02-12 | 北京云真信科技有限公司 | Application log collection system |
| CN113094348A (en) * | 2021-03-19 | 2021-07-09 | 北京达佳互联信息技术有限公司 | Log management method and device |
| CN113094348B (en) * | 2021-03-19 | 2024-02-23 | 北京达佳互联信息技术有限公司 | Log management method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107239382A (en) | The log processing method and system of a kind of container application | |
| Kuenning et al. | Automated hoarding for mobile computers | |
| Gousios | The GHTorent dataset and tool suite | |
| CN104657459B (en) | A kind of mass data storage means based on file granularity | |
| CN110166285B (en) | Network security experiment platform building method based on Docker | |
| Tang et al. | Deferred lightweight indexing for log-structured key-value stores | |
| CN105787093B (en) | A kind of construction method of the log file system based on LSM-Tree structure | |
| CN108463817A (en) | Personalised entity library | |
| CN105069703A (en) | Mass data management method of power grid | |
| CN109308329A (en) | A kind of log collecting method and device based on cloud platform | |
| CN104424287B (en) | Data query method and apparatus | |
| CN102597966B (en) | Operation management device and operation management method | |
| CN110377595A (en) | A kind of vehicle data management system | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN104182482B (en) | A kind of news list page determination methods and the method for screening news list page | |
| CN108595664A (en) | A kind of agricultural data monitoring method under hadoop environment | |
| CN103631922A (en) | Hadoop cluster-based large-scale Web information extraction method and system | |
| US8019765B2 (en) | Identifying files associated with a workflow | |
| CN106991177A (en) | The nondestructive collection method and its system of real-time logs under a kind of distributed environment | |
| CN110175206A (en) | Intellectual analysis operational approach, system and medium for multiple database separation | |
| CN110515958A (en) | Data consistency method, apparatus, equipment and storage medium based on big data | |
| CN107122238A (en) | Efficient iterative Mechanism Design method based on Hadoop cloud Computational frame | |
| CN113377292A (en) | Single machine storage engine | |
| CN106682061A (en) | Distributed system for collection and storage of origin data | |
| Chen et al. | Optimal algorithms for finding user access sessions from very large Web logs |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20171010 |