CN104301360B - A kind of method of logdata record, log server and system - Google Patents

A kind of method of logdata record, log server and system Download PDF

Info

Publication number
CN104301360B
CN104301360B CN201310306476.3A CN201310306476A CN104301360B CN 104301360 B CN104301360 B CN 104301360B CN 201310306476 A CN201310306476 A CN 201310306476A CN 104301360 B CN104301360 B CN 104301360B
Authority
CN
China
Prior art keywords
data
log
storage server
tables
daily record
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201310306476.3A
Other languages
Chinese (zh)
Other versions
CN104301360A (en
Inventor
刘宾
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201310306476.3A priority Critical patent/CN104301360B/en
Publication of CN104301360A publication Critical patent/CN104301360A/en
Priority to HK15102899.1A priority patent/HK1202722A1/en
Application granted granted Critical
Publication of CN104301360B publication Critical patent/CN104301360B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the present application discloses method, log server and the system of a kind of logdata record, which includes: that the daily record data got according to log acquisition condition is formed log set；Determine the data storage server for storing the log set；The log set is stored to the data storage server, and generates the tables of data of the log set on the data storage server；It is that the tables of data of the log set creates index after all daily record datas of the log set all import the tables of data.Using the method for the embodiment of the present application, log server or system, the storage speed of daily record data can be improved, so that the network platform can record a large amount of daily record datas generated in the short time in time, prevents loss of data and shorten the delay time of user query.

Description

A kind of method of logdata record, log server and system
Technical field
This application involves network data processing fields, method, log services more particularly to a kind of logdata record Device and system.
Background technique
As internet communication is more more and more universal, between the network platform increasingly by the number of users of network interaction More, the number interacted between user and platform is also more and more.Having for completed interaction can be inquired for the ease of user Information is closed, the mode generallyd use is the daily record data that interactive information is generated user by the network platform, then records each user The daily record data that generates of each interaction can be directly in network in this way, when user needs to inquire completed interactive information The daily record data that platform is recorded always searches daily record data corresponding to the interaction, then again returns to the daily record data found To user, such inquiry is obtained with required interactive information.
In existing logdata record method, as soon as whenever generating a user journal data, log service system will The insertion storage of this daily record data is in a certain storage equipment into system, in this way, for the log services in user query System can find the storage location of daily record data in log storage equipment, will be right while being inserted into every daily record data Index in log storage equipment is once updated.
By the prior art come log data, since one log of every insertion requires the rope for storing equipment to log The primary update of row is introduced, so the speed of logdata record is slower, causes the efficiency of logdata record lower.And it is further , currently due to the increase of network platform portfolio, the daily record data quantity generated in the network platform is more and more, by existing Technology carrys out log data, it is difficult to a large amount of daily record datas for recording generation in time, in this way, not only resulting in daily record data It loses, can also cause user inquiry prolonging less than interactive information namely user query within the long period after interaction is completed The slow time is longer.
Summary of the invention
Technical problems to be solved in this application are to provide the method, apparatus and system of a kind of logdata record, with solution The lower technical problem of the efficiency of logdata record caused by certainly in the way of logdata record in the prior art, into one Step, moreover it is possible to solve the network platform that generation daily record data reaches hundred million number of levels in the short time and be difficult to record generation in time A large amount of daily record datas and caused by daily record data lose and the delay time longer technical problem of user query.
In order to solve the above technical problems, the embodiment of the present application provides a kind of method of logdata record, this method packet It includes:
The daily record data got according to log acquisition condition is formed into log set；
Determine the data storage server for storing the log set；
Generate the tables of data of the log set on the data storage server, and by the institute in the log set There is daily record data to import the tables of data；
It is the number of the log set after all daily record datas of the log set all import the tables of data It creates and indexes according to table.
It is preferably, described that the daily record data got according to log acquisition condition is formed into log set, comprising:
It is primary before obtaining to form the daily record data generated after the log collection conjunction from distributed file system；
Judge whether the item number of the daily record data reaches preset acquisition data bulk threshold value；
If it is not, then when once forming log set before returning to the judgement acquisition after waiting for the first waiting time The step of daily record data generated after carving；
If it is, the daily record data is formed log set；The log set corresponding period is preceding primary At the time of forming log set to this at the time of forming log set.
It is preferably, described that the daily record data got according to log acquisition condition is formed into log set, comprising:
Judge to obtain whether the waiting time reaches preset waiting time threshold value, the acquisition waiting time is current time With it is preceding it is primary composition log set at the time of between time；
Described judge whether the acquisition waiting time reaches default if it is not, then returning after waiting second waiting time Waiting time threshold value the step of；
If it is, from distributed file system, it is primary before obtaining to form the log generated after the log collection conjunction Data, and the daily record data is formed into a log set；The log set corresponding period is preceding primary composition day At the time of forming log set to this at the time of will set.
Preferably, the log acquisition condition is to obtain log, the log set according to the time sequencing of daily record data In time of all daily record datas belong to the same period；
Correspondingly, by after the daily record data got according to log acquisition condition composition log set, further includes: by institute The daily record data stated in log set is classified according to type of service, forms at least one log subclass；
Correspondingly, the tables of data for generating the log set on the data storage server, comprising:
According to the daily record data in log subclass each in the log set, the corresponding each log subclass of generation Subdatasheet；
The subdatasheet of log subclass all in the log set is combined into the tables of data of the log set.
Preferably, the determination is used to store the data storage server of the log set, specifically: it determines for depositing Store up the disk that writing task is currently not carried out in the data storage server of the log set.
Preferably, after the tables of data that the log set is generated on the data storage server, further includes:
Judge whether the data storage server returns to storage in preset storage time and complete information；The storage Completing information is that the storage server is completed all daily record datas in the log set importing the tables of data simultaneously For the information returned after tables of data creation index；
If it is not, then being never determined for redefining in the storage server for storing the log set for depositing The data storage server of the log set is stored up, and is returned described by log set storage to the data storage service The step of device.
Preferably, the method also includes:
The log set that storage is not completed in the data storage server is periodically deleted according to preset erasing time.
Preferably, after the tables of data that the log set is generated on the data storage server, further includes:
Judge on the data storage server according to preset merging time cycle property with the presence or absence of daily record data quantity Less than the tables of data of preset storage quantity threshold value；If it is, the daily record data quantity is less than storage quantity threshold value Tables of data merges.
Preferably, after the tables of data creation index for the log set, further includes:
The tables of data storage of the log set is independent data list file set；
Replicate the data list file in the corresponding data list file set of the log set；
The number obtained for storing the duplication is determined from the storage server of the not stored data list file According to the backup storage server of list file；
The data list file that the duplication obtains is stored correspondingly to the backup storage server and is not held currently In the disk of row write task.
Preferably, described to determine from the storage server of the not stored data list file for storing described replicate The backup storage server of the data list file set arrived, comprising:
The data list file intersection of sets not stored in the storage server connected is changed planes and is determined as target friendship It changes planes；
The number obtained for storing the duplication is determined from the storage server that the target switch is connected According to the backup storage server of list file.11, according to the method described in claim 9, it is it is characterized in that, described by the duplication After obtained data list file is stored into the backup storage server, further includes:
Monitor the copy number of the data list file of the log set；
Judge the size relation between the copy number of the data list file and preset copy number threshold values；
If the copy number of the data list file is greater than the copy number threshold value, by the tables of data of difference quantities File is deleted；The difference quantities are the difference between the copy number and the copy number threshold value of the data list file；
If the copy number of the data list file is less than the copy number threshold value, the duplication day is returned The step of data list file of the corresponding data list file set of will set.
Disclosed herein as well is a kind of methods of inquiry log data, store log number by the method for aforementioned any one According to the time of all daily record datas belongs to the same period in the log set；Pass through method backup log number above-mentioned According to；It records between the log set and the data storage server, the log set and the backup storage server Between and the log set and the corresponding relationship between the period；
The described method includes:
Receive daily record data inquiry request；
Period and the log set and time where the time of target journaling data in the inquiry request Corresponding relationship between section, determines the target journaling set where the target journaling data；
According to the corresponding relationship and/or the log set between the log set and data storage server and back up Corresponding relationship between storage server, from be stored with the target journaling collective data list file data storage server and In backup storage server, the inquiry storage server for inquiring the target journaling is determined；
The tables of data of the target journaling set is searched in the inquiry storage server, and in the tables of data found It is middle to search the target journaling data；
The target journaling data found are merged and are exported.
Disclosed herein as well is a kind of log servers of log data, comprising:
Log set forms module, and the daily record data for will be got according to log acquisition condition forms log set；
Data storage server determining module, for determining the data storage server for storing the log set；
Tables of data generation module, for generating the tables of data of the log set on the data storage server；
Module is inserted into log, for all daily record datas in the log set to be imported the tables of data；
Index creation module, for after all daily record datas of the log set all import the tables of data, Index is created for the tables of data of the log set.
Preferably, the log set formation module includes:
First daily record data acquisition submodule, for once forming log set before obtaining from distributed file system The daily record data generated after moment；
Quantity judging submodule is obtained, for judging whether the item number of the daily record data reaches preset acquisition data number Measure threshold value；
Trigger acquisition submodule, for it is described obtain quantity judging submodule judging result be it is no in the case where, etc. Module is obtained to trigger the daily record data after the first waiting time；
Submodule is combined in first log, for the case where the judging result for obtaining quantity judging submodule is to be Under, the daily record data is formed into log set；The log set corresponding period is preceding primary composition log set At the time of moment forms log set to this.
Preferably, the log set formation module includes:
Acquisition time judging submodule obtains whether the waiting time reaches preset waiting time threshold value, institute for judging State obtain the waiting time be current time and it is preceding it is primary composition log set at the time of between time；
Triggered time judging submodule, the situation for being no in the judging result of the acquisition time judging submodule Under, wait second waiting time to trigger the acquisition time judging submodule later；
Second log data acquisition submodule, the feelings for being yes in the judging result of the acquisition time judging submodule It is primary before obtaining to form the daily record data generated after the log collection conjunction from distributed file system under condition；
Submodule is combined in second log, for the daily record data to be formed a log set；The log set pair At the time of forming log set to this at the time of the period answered is preceding primary composition log set.
Preferably, the log server further include:
Log subclass forms module, for the daily record data in the log set to be classified according to type of service, shape At at least one log subclass；
Correspondingly, the tables of data generation module includes:
Subdatasheet generates submodule, for according to the daily record data in log subclass each in the log set, The corresponding subdatasheet for generating each log subclass；
Subdatasheet combines submodule, for by the subdatasheet group of the log subclass all in the log set Synthesize the tables of data of the log set.
Preferably, the data storage server determining module, specifically for determining for storing the log set The disk of writing task is currently not carried out in data storage server.
Preferably, the log server further include:
Tables of data completes judgment module, for judging whether the data storage server returns in preset storage time It restores storage and completes information；It is that the storage server was completed all days in the log set that information is completed in the storage Will data import the tables of data and the information to return after tables of data creation index；
Data storage redefines module, the situation that the judging result for completing judgment module in the tables of data is no Under, it is never determined for redefining in the storage server for storing the log set for storing the log set Data storage server；
Memory module is triggered, for triggering log set in the case where the data storage redefines module starting Memory module.
Preferably, the log server further include:
Collection modules are deleted, it is not complete in the data storage server for periodically being deleted according to preset erasing time At the log set of storage.
Preferably, the log server further include:
Data bulk judgment module, for judging the data storage server according to preset merging time cycle property It is less than the tables of data of preset storage quantity threshold value with the presence or absence of daily record data quantity；
Gather merging module, for the judging result of the data bulk be in the case where, by the daily record data The tables of data that quantity is less than storage quantity threshold value merges.
Preferably, the log server further include:
Backup generation module, for the tables of data storage of the log set to be independent data list file set；
Backup copy module, for replicating the data list file of the corresponding data list file set of the log set；
Backup server determining module, for determining from the storage server of the not stored data list file for depositing Store up the backup storage server of the data list file for replicating and obtaining；
Backup memory module, the data list file for obtaining the duplication are stored correspondingly to the backup and are deposited Storage server is currently not carried out in the disk of writing task.
Preferably, the backup server determining module, comprising:
First determines submodule, for by the exchange of the data list file not stored in the storage server connected Machine is determined as target switch；
Second determines submodule, for determining from the storage server that the target switch is connected for storing State the backup storage server for the data list file that duplication obtains.
Preferably, the log server further include:
Tables of data monitoring modular, the copy number of the data list file for monitoring the log set；
Tables of data quantity judgment module, for judge data list file copy number and preset copy number threshold values it Between size relation；
Tables of data removing module is the data list file for the judging result in the tables of data quantity judgment module Copy number be greater than the copy number threshold value in the case where, the data list files of difference quantities is deleted；The difference number Amount is the difference between the copy number and preset copy number threshold value of the data list file；
Replication module is triggered, is the data list file for the judging result in the tables of data quantity judgment module In the case that copy number is less than the copy number threshold value, the backup copy module is triggered.
Disclosed herein as well is a kind of systems of log data, which is characterized in that the system comprises log services Device and storage server；The log server is the log server of aforementioned any one；The storage server includes number According to storage server and/or backup storage server.
Disclosed herein as well is a kind of system of inquiry log data, the system comprises:
Log storage unit owns in the log set for passing through log server storing daily record data above-mentioned The time of daily record data belongs to the same period；
Log backup unit, for passing through log server backup log data above-mentioned；
Corresponding relationship recording unit, for recording between the log set and the data storage server, the day Between will set and the backup storage server and the log set and the corresponding relationship between the period；
Inquiry request receiving unit, for receiving daily record data inquiry request；
Target collection determination unit, for according to the period where the time of target journaling in the inquiry request, really The target journaling set to set the goal where daily record data；
Query service device determination unit, for according to the corresponding relationship between the log set and data storage server And/or the corresponding relationship between the log set and backup storage server, from being stored with the target journaling collective data In the data storage server and backup storage server of list file, determine that the inquiry for inquiring the target journaling stores clothes Business device；
Searching unit, for searching the tables of data of the target journaling set in the inquiry storage server；
Target journaling searching unit, for searching the target journaling data in the tables of data found；
Target journaling output module, the target journaling data for will find merge output.
It can be seen from the above technical scheme that logdata record method disclosed in the embodiment of the present application, because adopting The daily record data composition log that in such a way that set is stored, first will acquire with will acquire daily record data formation log set Set, then determines the data storage server for storing log set, thus can be first in the data storage server All daily record datas in the log set are simultaneously all imported on data storage server by the upper tables of data for generating the log set, It is again log collective data table creation index after all daily record datas of the log set all import the tables of data, Log is remembered caused by being thus avoided that the index for requiring Update log storage equipment because of one daily record data of every record Record slow phenomenon so that the writing speed of log recording is accelerated so that the network platform can record in time it is short The a large amount of daily record datas generated in time, prevent loss of data and shorten the delay time of user query, allow users to Completed interactive information is inquired more in time.
Further, it can be determined in the embodiment of the present application when determining the data storage server of storage log set It is currently not carried out in data storage server in the disk of writing task, in this way, the disk of storage log set is not held in storage Other writing tasks of row, can be improved the storage speed of daily record data in log set.
Further, when being also provided with storage to data storage server completion log set storage in the embodiment of the present application Between threshold value, when data storage server by storage time also do not return completion storage message, will just redefine data storage Server stores the log set, and log set is Sino-Japan caused by can there is mistake in this way to avoid data storage server Will data cannot be stored as completely tables of data.In addition, can also be provided with and delete on the basis of storage time threshold value is arranged Time does not complete the log set of storage periodically to delete, and can reduce the junk data in data storage server in this way, Increase available space, improves utilization efficiency.
Further, storage quantity threshold value is also provided with to the quantity of daily record data in tables of data in the embodiment of the present application, When in data storage server there are daily record data quantity be less than storage quantity threshold value tables of data when, just by these tables of data into Row merges, and can reduce the quantity of tables of data in this way, the tables of data quantity required to look up when saving user query daily record data, from And save query time.
Further, in the embodiment of the present application in the log set of backup storage, due to the data of each log set Table is the data list file set independently saved, therefore can be directly in the corresponding data list file set of replication log set Data list file is simultaneously stored into backup storage server, can save the time required for backup.In addition, in data table memory When file, in this way can may be used in the disk for being currently not carried out writing task in data list file storage to backup storage server To improve the speed of data list file storage, the time required for backing up further is saved.
Further, can make to store each of same log set when determining backup server in the embodiment of the present application Each backup storage server of data list file set is located under different interchangers, in this way can be to avoid by same log set Each data list file set be stored in caused by the storage server under same switch after the interchanger damage day Daily record data in will set will be unable to the problem of inquiry.
Further, the copy number threshold value of data list file, the same day are provided in the embodiment of the present application to log set When the copy number of the data list file of will set is greater than preset copy number threshold value, extra data list file is just deleted, When the copy number of data list file is less than copy number threshold value, just replicate data list file again, can guarantee to deposit in this way The copy number for storing up the data list file of log set in server maintains to stablize, thus in the feelings for being not take up excessive memory space Guarantee that daily record data will not be lost under condition.
Further, in the embodiment of the present application when the target journaling data of record are completed in inquiry, due to storage service Each log set correspondence generates different tables of data on device, thus can first determine the target collection where target journaling, The target journaling data are searched in the tables of data of the target journaling set in inquiry storage server again, without inquiring All tables of data are searched in storage server, to improve the speed of log query, reduce the delay time of inquiry.
Certainly, any product for implementing the application does not necessarily require achieving all the advantages described above at the same time.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this The some embodiments recorded in application, for those of ordinary skill in the art, without creative efforts, It is also possible to obtain other drawings based on these drawings.
Fig. 1 is the flow chart of the embodiment of the method 1 of the application logdata record；
The flow chart of the embodiment 1 of step 101 in Fig. 2 the application embodiment of the method 1；
The flow chart of the embodiment 2 of step 101 in Fig. 3 the application embodiment of the method 1；
Fig. 4 be the application embodiment of the method 1 can Additional embodiments 1 flow chart；
Fig. 5 be the application embodiment of the method 1 can Additional embodiments 2 flow chart；
Fig. 6 is the flow chart of the embodiment of the method 2 of the application logdata record；
Fig. 7 is the flow chart of the embodiment of the method 3 of the application logdata record；
Fig. 8 is the flow chart of the embodiment of the method 4 of the application logdata record；
Fig. 9 is the flow chart of the embodiment of the method 1 of the application inquiry log data；
Figure 10 is the flow chart of the method scene embodiment of the application logdata record；
Figure 11 is the flow chart of the method scene embodiment of the application inquiry log data；
Figure 12 is the structure chart of the log server Installation practice 1 of the application logdata record；
Figure 13 is the knot for the embodiment 1 that log set forms module 1201 in the application log server Installation practice 1 Composition；
Figure 14 is the knot for the embodiment 2 that log set forms module 1201 in the application log server Installation practice 1 Composition；
Figure 15 is the structure chart of the log server Installation practice 2 of the application logdata record；
Figure 16 is the another structure chart of the log server Installation practice 2 of the application logdata record；
Figure 17 is the structure chart of the log server Installation practice 3 of the application logdata record；
Figure 18 is the structure chart of the log server Installation practice 4 of the application logdata record；
Figure 19 is the structure chart of the log server Installation practice 5 of the application logdata record；
Figure 20 is the structure chart of the log server Installation practice 6 of the application logdata record；
Figure 21 is the structure chart of the system embodiment 1 of the application logdata record；
Figure 22 is the structure chart of the system embodiment 1 of the application inquiry log data.
Specific embodiment
In order to make those skilled in the art more fully understand application scheme, below in conjunction in the embodiment of the present application Attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described embodiment is only this Apply for a part of the embodiment, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art exist Every other embodiment obtained under the premise of creative work is not made, shall fall in the protection scope of this application.
The application can be used in numerous general or special purpose computing system environments or configuration.Such as: personal computer, service Device computer, handheld device or portable device, laptop device, multicomputer system, microprocessor-based system, top set Box, programmable consumer-elcetronics devices, network PC, minicomputer, mainframe computer, including any of the above system or equipment Distributed computing environment etc..
The application can describe in the general context of computer-executable instructions executed by a computer, such as program Module.Generally, program module includes routines performing specific tasks or implementing specific abstract data types, programs, objects, group Part, data structure etc..The application can also be practiced in distribution meter environment, in these distributed computing environments, by leading to Cross communication network and connected remote processing devices execute task.In a distributed computing environment, program module can position In the local and remote computer storage media including storage equipment.
One of main thought of the application may include, by the way of the log set comprising multiple daily record datas, first Multiple daily record datas for needing to record are formed into log set, and the log set is stored completely to data storage server life All daily record datas at tables of data, then in log set all import after tables of data to be created for the tables of data of the log set It indexes.Due to no longer needing one daily record data of every record all to update index, day is can be improved in the method for this record log The speed of will data storage, and then save the time of storage.
Referring to Fig. 1, a kind of flow chart of the embodiment of the method 1 of logdata record of the application, the present embodiment are shown May include:
The daily record data got according to log acquisition condition is formed log set by step 101.
For the embodiment of the present application when implementing, acquired daily record data is the daily record data generated.Daily record data Source is the operation system of network platform application cluster.A large number of users and operation system have carried out a large amount of interaction, operation system A large amount of interactive information is produced with regard to correspondence, these interactive information are sent to log buffer server by operation system immediately, so The corresponding daily record data of each interactive information is generated by log buffer server afterwards, and caches the daily record data, with etc. it is to be recorded The daily record data.In order to make the daily record data of caching be not easy to be damaged or lost, the present embodiment can be in log buffer server After generating daily record data, daily record data is cached in distributed file system.Wherein, distributed file system can be placed in Log buffer server local can also be placed in operation system.In addition, after daily record data in the buffer is acquired, usually It needs to be removed from the cache, caches newly-generated daily record data to keep having enough memory spaces in caching.
It should be noted that as daily record data be interacted as operation system with user caused by, and operation system with Time is constantly largely interacted with a large number of users, so, usual daily record data is all generating all the time.Therefore, it is obtaining When forming the daily record data of log set, log acquisition condition can be successively to obtain sequentially in time.In this way, primary It obtains in the log set that daily record data is formed, the daily record data in the log set is all once to get this upper It generates and caches in period of acquisition, and time for usually being generated with log in daily record data, so, the log set In time of all daily record datas belong in this period, it can be seen that, successively obtain log number sequentially in time According to log set is formed by, the time of all daily record datas belongs to section at the same time in the same log set.
For the above-mentioned log acquisition condition for successively obtaining daily record data sequentially in time, two kinds are present embodiments provided The specific embodiment of step 101.
Referring to fig. 2, the flow chart of the embodiment 1 of step 101 is shown, step 101 can specifically include:
Step 201, from distributed file system, the log number that is generated after the primary composition log collection conjunction before obtaining According to；
Step 202 judges whether the item number of the daily record data reaches preset acquisition data bulk threshold value；If not, into Enter step 203, if so, entering step 204；
Step 203 waits return step 201 after the first waiting time；
Wherein, the first waiting time was a preset time value.Setting to the first waiting time needs to guarantee first The daily record data item number that can be got in waiting time no more than obtain data bulk threshold value, so, usual first wait when Between be arranged it is all smaller.
The daily record data is formed log set by step 204；The log set corresponding period is first primary group At the time of at log set is formed to this at the time of log set.
It should be noted that step 204 executes after completing, the composition of a log set is just completed, forms one A log set.After step 204 completion, for the log set, 102 completions can be entered step to day in log set The record of will data；Meanwhile the embodiment 1 of Fig. 2 can be carried out again with return step 201, to carry out daily record data next time Acquisition and form next log set.
It is understood that in the embodiment 1 of step 101 shown in Fig. 2, in the log set that forms every time, log The item number of data is identical, is all preset acquisition data bulk threshold value.
Referring to Fig. 3, the flow chart of the embodiment 2 of step 101 is shown, step 101 can specifically include:
Step 301 judges to obtain whether the waiting time reaches preset waiting time threshold value, and the acquisition waiting time is Time between at the time of current time and preceding primary composition log set；If not, 302 are entered step, if so, entering step Rapid 303；
Step 302 waits return step 301 after second waiting time；
Wherein, the second waiting time was a preset time.When being configured to the second waiting time, second wait when Between can be directly disposed as waiting time threshold value, may be set to be less than waiting time threshold value.Usual second waiting time sets That sets is all very small, uninterruptedly carries out in this way, being equivalent to step 301 pair and obtaining the judgement of waiting time, once judgement knot Fruit be it is yes, then enter step 303.
Step 303, from distributed file system, the log number that is generated after the primary composition log collection conjunction before obtaining According to, and the daily record data is formed into a log set；The log set corresponding period is preceding primary composition log At the time of forming log set to this at the time of set.
It should be noted that step 303 executes after completing, the composition of a log set is just completed, forms one A log set.After step 303 completion, for the log set, 102 completions can be entered step to day in log set The record of will data；Meanwhile the embodiment 2 of Fig. 3 can be carried out again with return step 301, to carry out daily record data next time Acquisition and form next log set.
It is understood that in the embodiment 2 of step 101 shown in Fig. 3, in the log set that forms every time, log The time of period is all equal where the time of data, is all preset waiting time threshold value.
It is then returned to Fig. 1, after step 101 is finished, enters step 102.
Step 102 determines data storage server for storing the log set.
Due to needing to store a large amount of user journal data, so being commonly used in the log service system of record log all With multiple storage servers, and there are multiple disks to store to daily record data in each storage server.Therefore, exist In storage log set before daily record data, need first to determine from all storage servers for storing depositing for the log set Store up server, the data storage server as the log set.Wherein, it is generally the case that only true for a log set A fixed data storage server, can be improved the efficiency of storage log set in this way.It such as needs for log set to be stored in more To avoid daily record data therein loss in a storage server, then can be stored on data storage server using duplication The mode of log set back up, the specific implementation of backup will be described in detail in embodiment below, no longer superfluous herein It states.
In addition, avoiding read-write task to make the disk I/O pressure energy of each storage server in system access balance It focuses on disk to execute, step 102 preferably determines the mode of data storage server, can be with are as follows: first in all storages The disk for being currently not carried out writing task is searched in server, then determines that residual memory space is maximum in the disk found One disk is determined as data storage service as the disk for storing log set, then by storage server where the disk Device.Wherein, it is not required for storing the disk of log set as the maximum disk of residual memory space in the disk that finds, It is also possible to any one disk for being currently not carried out writing task, as long as the residual memory space of the disk can satisfy storage and be somebody's turn to do The requirement of log set.
Step 103, the tables of data that the log set is generated on the data storage server, and by the log collection All log set in conjunction import the tables of data.
According to the explanation of above-mentioned steps 102, executed in order to avoid task focuses on a disk, storage log set is specific It can be to store the log set in the disk for being currently not carried out writing task into the data storage server, this is current The disk for being not carried out writing task is determined in step 102 for storing the disk of log set.
It should be noted that being stored as tables of data to log set, it can specifically include: by all days of log set Will data are sent to data storage server, and data storage server is that the log set creates tables of data, and by log set In all daily record datas import the tables of data storage.Wherein, a tables of data is usually created for a log set, in this way Be stored in actual storage in data storage server is exactly multiple tables of data, and the daily record data in each tables of data is to should belong to One period；Database where tables of data preferably uses mysql.Certainly, the daily record data quantity in log set is very big When, or a log set creates multiple tables of data.In addition, should importing all daily record datas in log set During tables of data stores, the index of tables of data is not created for daily record data, does not also specify a kind of major key (rope of tables of data Draw), but step 104 creation index is entered back into after daily record data imports tables of data completely.
After data storage server receives log set, it is generally completed after the storage completion of log set, can returns It restores storage and completes information, it is that storage server is completed all daily record datas in the log set which, which completes information, Import the tables of data and the information to return after tables of data creation index.If data storage server is to log collection There is mistake in the storage of conjunction, cannot complete the storing log set of the task, then data storage server would not be returned and be deposited Information is completed in storage.In order to which the daily record data guaranteed in log set can be stored in data storage server completely, in step Rapid 103 execution can use embodiment shown in Fig. 4 after completing, comprising:
Step 401 judges whether the data storage server returns to storage in preset storage time and complete information； The storage is completed information and is being completed for the storage server by all daily record datas importing institute in the log set State tables of data and the information to return after tables of data creation index: if not, entering step 402.
Step 402 is never determined for redefining in the storage server for storing the log set for depositing Store up the data storage server of the log set, and return step 103.
Step 403 periodically deletes unfinished storage in the data storage server according to preset erasing time Log set.
It should be noted that delete tables of data step 403 it is not necessary to step 401 or 402 complete after just into Capable, nor data storage server will be executed in storage log set every time.Since daily record data is continuous at any time Ground generates, so system constantly stores daily record data, so usually system needs to clear up week according to preset rubbish Phase, which recycles, executes step 403, is just avoided that in each storage server of system that there are excessive dirty data tables in this way, thus Expand available residual memory space.
By embodiment shown in Fig. 4, can occur log set caused by mistake to avoid data storage server Middle daily record data cannot be stored as completely tables of data.
It is then returned to the explanation to step 103.
In the present embodiment, if forming log set using embodiment 2 shown in Fig. 3, each log set The corresponding period is all same time interval.In this way, the daily record data item number just obtained in different time sections generally not phase Together.Therefore, the daily record data item number that the log set formed in this way may include is less, in turn results in and generates in step 103 Tables of data is also smaller.It, can many memory spaces of additional waste when the small data table in data storage server is excessive.For This problem is avoided, this implementation can be after step 103, into embodiment shown in fig. 5, comprising:
Step 501 judges on the data storage server according to preset merging time cycle property with the presence or absence of log Data bulk is less than the tables of data of preset storage quantity threshold value；If so, entering step 502.
Wherein, step 501 can carry out the execution of periodic cycle according to a preset merging time, in this way can be continuous Ground monitors on each data storage server with the presence or absence of compared with small data table.
Step 502, the tables of data that the daily record data quantity is less than to storage quantity threshold value merge.
By embodiment shown in fig. 5, reduce the quantity of tables of data, not only can much be stored to avoid additional waste Space, the tables of data quantity required to look up when can also save user query daily record data, to save query time.
It is then returned to Fig. 1 and enters step 104 after step 103 executes completion.
Step 104, after all daily record datas of the log set all import the tables of data, be the log The tables of data of set creates index.
For daily record data table create index, be log set all daily record datas all completely import tables of data and complete It is carried out after storage on data storage server.Wherein, index is created for log collective data table, that is, respectively Each tables of data has corresponding index in storage server.
In the embodiment of the present application, because of the side stored using will acquire daily record data formation log set by set Formula can first generate the tables of data of the log set and by daily record datas all in log set on the data storage server All imported on data storage server, all import after again for the log collective data table creation index, in this way it is avoided that One daily record data of every record requires the index of Update log storage equipment, so that the storage speed of log recording is accelerated, To prevent loss of data and shorten the delay time of user query.
Further, in the embodiment of the present application when storing log set, log set can also be stored to data and is deposited It is currently not carried out in the disk of writing task in storage server, in this way, disk does not execute other writing tasks when storing log set, And then storage speed when raising log data.
Further, storage log set also is completed to data storage server in the embodiment of the present application and is also provided with storage Time threshold will be redefined when data storage server is not completed to the store tasks of log set also by storage time Data storage server stores the log set, can occur log caused by mistake to avoid data storage server in this way Daily record data cannot be stored as completely tables of data in set.In addition, can also be set on the basis of storage time threshold value is arranged Erasing time has been set periodically to delete the log set for not completing storage, the rubbish in data storage server can be reduced in this way Rubbish data increase available space, improve utilization efficiency.
Further, storage quantity threshold is also provided with to the quantity of daily record data in log set in the embodiment of the present application Value, when in data storage server there are daily record data quantity be less than storage quantity threshold value tables of data when, just by these data Table merges, and can reduce the quantity of tables of data in this way, the tables of data number required to look up when saving user query daily record data Amount, to save query time.
Referring to Fig. 6, a kind of flow chart of logdata record embodiment of the method 2 of the application is shown, the present embodiment is also It may comprise steps of:
The daily record data got according to log acquisition condition is formed log set by step 601；The log acquisition item Part is to obtain log according to the time sequencing of daily record data, and the time of all daily record datas belongs to for the moment in the log set Between section；
Step 602 classifies the daily record data in the log set according to type of service, forms at least one a day measure Set；
When forming log subclass, can type be divided into different business models with by the log data set of same type of service At a log subclass, various businesses class can also be enclosed, there are one or more types of service under each scope of business, this The daily record data that type of service belongs to the same scope of business is formed a log subclass by sample.
Step 603 determines data storage server for storing the log set；
Step 604 stores the log set to the data storage server；
Step 605, according to the daily record data in log subclass each in the log set, it is corresponding to generate each log The subdatasheet of subclass；
The log set being sent on data storage server includes at least one log subclass, at this point, being each day Measure set creates a subdatasheet, and the daily record data in each log subclass is imported into corresponding subdatasheet In.
The subdatasheet of log subclass all in the log set is combined into the log set by step 606 Tables of data；
All log subclass of log set, which are completed, to be imported subdatasheet and then is combined into each subdatasheet The tables of data of corresponding log set.
Step 607, after all daily record datas of the log set all import the tables of data, be the log The tables of data of set creates index.
When creation index, subindex first can be created for each subdatasheet, then again by each subindex composition data table Index.
The present embodiment and the difference of embodiment 1 shown in FIG. 1 be, the present embodiment be actually according to daily record data when Between daily record data is divided into different log set and stores, then again by daily record data according to business in log set Type is divided into different log subclass.It is of course also possible to by the daily record data got in the same period directly according to industry Service type is divided into multiple types of service.In this way, quickly finding the log where target journaling when family inquiry log can be used The tables of data and subdatasheet of set, so that the tables of data quantity searched required for reducing, improves inquiry velocity.
Referring to Fig. 7, a kind of flow chart of logdata record embodiment of the method 3 of the application is shown, in the present embodiment In, after all steps for having executed previous embodiment 1, the present embodiment can with the following steps are included:
The tables of data storage of the log set is independent data list file set by step 701；
In the embodiment of the present application, tables of data is to be stored in storage server in the form of data list file.For Convenient for backing up to log set, the data list file that each log collective data table is formed is independent, that is, often Data in a data list file are all the same log set.It should be noted that the tables of data of each log set can To be one, it is also possible to multiple, also, each tables of data at least forms a data list file, so, each log set Tables of data generally all include multiple data list files, so, each log set it is corresponding on the whole be an independence Data list file set, a data list file set is a backup copies of the log set.Wherein, each data What table particular number was generally determined according to used database.In order to realize that each tables of data is stored as number independent According to list file, the present embodiment preferably uses mysql database, and under the MyISAM data engine in mysql database, each Tables of data is at least stored as three data list files, and three data list files distinguish storing daily record data (data file), rope Draw (MYI file) and table structure (FRM file), at this point, corresponding three data list files of a tables of data, include in log set It include three data list files in the data list file set of the log set when one tables of data.
The data list file of step 702, the corresponding data list file set of the duplication log set；
Since each log set is stored in independent data list file set, so when Backup Data, no longer It needs the first corresponding daily record data of lookup log set in storage server to replicate corresponding sentence again to go to be backed up, but Directly the data list file in the data list file set for being stored with log set can be replicated.
Step 703, determination is obtained for storing the duplication from the storage server of the not stored data list file Data list file backup storage server；
In the backup storage server for determining storing data list file, if the data list file collection of a log set All data list files in conjunction are stored in the same storage server, once then storage server failure log collection All data closed just can all lose.In order to avoid this problem, identified backup storage server can be currently without Store the storage server of the data list file of the log set.And when being backed up for the first time to a log set, the day The data list file of will set is only stored in its data storage server, so backup storage server determined by this time is only It needs different from data storage server.
Further, if all data list files in the data list file set of a log set are stored in together In storage server under one interchanger, once then all data of the interchanger failure log set also can all lose. In order to avoid this problem, interchanger belonging to identified backup storage server can be all storage services connected Device does not all store the log intersection of sets and changes planes.Log collective data is caused to lose problem, step in order to avoid above-mentioned as a result, 703 can specifically include: the data list file intersection of sets not stored in the storage server connected is changed planes determination For target switch；The institute obtained for storing the duplication is determined from the storage server that the target switch is connected State the backup storage server of data list file.
In addition, being avoided in task-set to make the disk I/O pressure energy of each storage server in system access balance It is executed to disk, step 703 preferably determines the mode of backup storage server, can be with are as follows: elder generation is from the not stored number According to the disk for being currently not carried out writing task is searched in the storage server of list file set, then determined in the disk found Disk of the maximum disk of residual memory space as the data list file set obtained for storage replication, then should Storage server where disk is determined as backup storage server.Wherein, it can be to look for for the disk of backup log set To disk in the maximum disk of residual memory space, the memory space of each disk can be made to be averaged use in this way, avoided Data are centrally stored to arrive the problem of loss of data risk caused by the disk of part increases.In addition, for backup log set Disk is not required to be also possible to any one and be currently not carried out for the maximum disk of residual memory space in the disk that finds The disk of read-write task, as long as the residual memory space of the disk can satisfy the requirement for storing the data list file.
It is understood that for a log set, the area of data storage server and backup storage server It is not only that, it is to store the log set in system first that data storage server, which needs to complete the record to the log set, Storage server.And after backup is completed, the log set is both stored in the form of data list file, herein In the backup again or inquiry executed afterwards, each backup storage server can be considered a data storage server.
Step 704 stores the obtained data list file that replicates correspondingly to the backup storage server Currently it is not carried out in the disk of writing task.
It, can be with by data list file storage into the disk for being currently not carried out writing task according to the explanation of above-mentioned steps 703 It avoids task from focusing on a disk to execute.In addition, in order to avoid data list file is stored in a backup storage server and The risk of the loss full backup of generation, the present embodiment store backup file in backup storage server correspondingly, It is stored in each part data list file of same log set in different backup storage servers, further, can also incite somebody to action Each part data list file set of same tables of data is stored in the backup storage server under different interchangers, in this way can be to prevent The problem of only backup caused by exchange fault is all lost or can not be read.
The present embodiment and the difference of embodiment 1 shown in FIG. 1 are that the present embodiment is on the basis of embodiment 1 to Recorded daily record data is backed up, and is stored in daily record data in multiple and different storage servers, in this way can be to avoid day Will loss of data guarantees the safety of daily record data being recorded.Further, since log set storage is independent number According to list file set, thus in backup can data list file directly in replicate data list file set store other In storage server, without saving standby by searching for the rear formation sql sentence complicated backup procedure that copy-statement executes again The time of part daily record data.
Referring to Fig. 8, a kind of flow chart of logdata record embodiment of the method 4 of the application is shown, in the present embodiment In, after all steps for having executed previous embodiment 2, the present embodiment can with the following steps are included:
The copy number of the data list file of step 801, the monitoring log set；
Wherein, the copy number of the data list file monitored is all tables of data text of the log set in system Part backed up after quantity.For a log set, to the data list file in its corresponding data list file set The monitoring of quantity can be after the log set backs up completion namely after step 704.
Size between step 802, the copy number for judging the data list file and preset copy number threshold values is closed System；
If the copy number of step 803, the data list file is greater than the copy number threshold value, by difference quantities Data list file delete；The difference quantities are between the copy number and the copy number threshold value of the data list file Difference；
If the copy number of step 804, the data list file is less than the copy number threshold value, return step 702。
It should be noted that when the copy number of data list file is less than the copy number threshold value, return step 702 The quantity to be replicated to obtain data list file can be equal to the difference quantities.
The present embodiment and the difference of embodiment of the method 3 shown in Fig. 7 are that the present embodiment is to back up to complete in log set The quantity of later monitoring data list file, and data list file set it is in varying numbers in copy number threshold value when carry out It adjusts, can guarantee that the backup quantity of log set in storage server maintains to stablize in this way, thus being not take up excessive storage Guarantee that all daily record datas will not be lost in the case where space.
Referring to Fig. 9, a kind of flow chart of inquiry log data method embodiment 1 of the application is shown, in the present embodiment In may include:
Step 901, by aforementioned either method storing daily record data, the time of all daily record datas in the log set Belong to the same period；
In the present embodiment, the acquisition needs of log set are obtained according to the time sequencing of log, log each in this way Set corresponds to a period, can determine the log set where log when to inquire according to logging time.
Step 902 passes through aforementioned either method backup log data；
It should be noted that step 902 is not have to carry out the step of, as long as that is, the target to be inquired Daily record data has been recorded, in storage to data storage server, so that it may be inquired, and be to be not carried out step at this time 902 complete data backup.But the log under normal conditions, to be inquired all is the daily record data having backed up.
Step 903, record between the log set and the data storage server, the log set with it is described standby Between part storage server and the log set and the corresponding relationship between the period；
It should be noted that step 901~903 are pre- the step of first carrying out, it is not that will be executed when inquiring every time.And And, it is generally the case that the target journaling inquired all has been recorded and has backed up, so, when inquiring target journaling every time, It is to be executed since step 904.
Step 904 receives daily record data inquiry request；
It include the information of the target journaling to be inquired, the packet in the present embodiment, in daily record data inquiry request Include time of log, type of service, User ID mark etc..In addition, inquiry proxy server can be passed through in the present embodiment Proxy provides the service of log query to be embodied as user.
Step 905, according to where the time of target journaling data in the inquiry request period and the log collection The corresponding relationship between the period is closed, determines the target journaling set where the target journaling data；
Period where target journaling can be one, be also possible to multiple, and corresponding log collection of each period Close usually be all multiple, so determine target journaling set have it is multiple.And usually target journaling belongs to one of target Set.
Step 906, according to the corresponding relationship and/or the log collection between the log set and data storage server The corresponding relationship between backup storage server is closed, is stored from the data for being stored with the target journaling collective data list file In server and backup storage server, the inquiry storage server for inquiring the target journaling is determined；
Due to determining target journaling set have it is multiple, so will determine one for each target journaling set Inquire storage server.Since each target journaling set has data list file in multiple storage servers, For a target journaling set, is executed in order to avoid task focuses on a disk, can first determine the target journaling The minimum disk of current loading condition in disk where collective data list file, then by the storage server where the disk It is determined as query service device, may be implemented to disperse disk task in each storage server in this way to improve the task execution of disk Efficiency shortens query time.It wherein, can in the load condition selection inquiry storage server of disk where according to tables of data With CPU is fastest or the maximum server of memory is chosen to be inquiry storage server, disk current task can also be gulped down The smallest server of the amount of spitting is chosen to be inquiry storage server.
In addition, if the target journaling is not backed up also when inquiring target journaling, then it can be by data storage service Device is determined as inquiring storage server.
Step 907 is searched the tables of data of the target journaling set in the inquiry storage server, and is being found Tables of data in search the target journaling data；
Inquiry proxy server proxy goes in inquiry storage server to search target journaling data.In the present embodiment, search It is preferred that using sql sentence.
The target journaling data found are merged output by step 908.
The target journaling of all information for meeting target journaling found from each inquiry storage server is merged It exports to user.
Explanation is additionally needed, if if the log set of each period is divided into according to type of service or the scope of business Log subclass, then the target journaling subset where can determining target journaling directly in target journaling set when searching It closes, can only be searched in the subdatasheet of the target journaling subclass in target journaling collective data table in this way.
Technical solution through this embodiment, since log set each in storage server correspondence generates different data Table thus can first determine the target collection where target journaling, then the target journaling set in inquiry storage server Tables of data on search the target journaling data, without inquiry storage server on search all tables of data, to mention The high speed of log query, reduces the delay time of inquiry.In addition, since target journaling is integrated into multiple storage servers On have the backup of tables of data, it is possible to the minimum storage server of disk present load where choosing the target journaling set It is determined as inquiring storage server, the speed for searching target journaling data in tables of data can be improved in this way, shortens inquiry institute The time needed.
It should be noted that above-mentioned data storage server, backup storage server and inquiry storage server are all For a log set, and for simply corresponding to the currently performed task of the log set.In practical day In will record, the data storage server of a log set either other log set data storage server, It can be the backup storage server or query service device of other log set.Moreover, data storage server is just for day The recording process of will data, just for the backup procedure of daily record data, inquire storage server is backup storage server For the query process of daily record data, and in fact, when system is not carried out task to the log set recorded, system can be with The storage server for being stored with the log set is only determined, without distinguishing data storage server, backup storage server and looking into Ask storage server.
Below with specific application scenarios, the embodiment party of the application logdata record method and querying method is described in detail Formula.
Referring to Figure 10, the flow chart of the scene embodiment of the application logdata record method is shown.In the present embodiment Scene in, the application logdata record method be applied to a log service system.The system includes log server Master and storage server worker a, wherein master corresponds to multiple worker.In the scene of the present embodiment, day Will server master will successively obtain daily record data from distributed file system sequentially in time and form a log collection Close, then by log set according to record storage to a storage server worker, daily record data is then backuped to other and is deposited It stores up on server worker.The present embodiment can specifically include:
Step 1001, generate daily record data: log buffer server receives the interactive information that operation system is sent, and generates Daily record data.
Step 1002, log cache data: generated daily record data is sent to distributed text by log buffer server Part system is cached；Distributed file system can be located at log buffer server local, can also be located at operation system, also It is local log service system can be located at.
Step 1003, log server obtain daily record data, and form log set: master is from distributed file system Generated daily record data is obtained, a log set is formed, and once obtained before the period of log set is determined as Moment at the time of this acquisition, records the log set and the corresponding relationship between the period.Daily record data is obtained by master After taking, it is removed from distributed file system.It, can also be by log collection if the daily record data quantity once obtained is excessive It closes and is divided into multiple log subclass according to type of service, according to log subclass classification storage when being stored to log set For different subdatasheets, when such inquiry log data, subdatasheet can be determined according to logging time and type of service, then It is searched in subdatasheet, and then reduces search range, save the inquiry used time.
Step 1004, the data storage server for determining the log set: master is selected from corresponding worker One is selected to be used to store the log set just obtained.In selection, first looks for all being not carried out each worker on and write times Then the disk of business determines that residual memory space maximum one is used to store the log set in the disk found, then Worker where the disk is determined as to the data storage server of the log set.If primary obtain forms multiple and different industry The log set of service type can also will determine a worker for each log set to complete subsequent step.
Step 1005 generates tables of data and imports log set: master sends step 1004 really for log set Fixed worker, the worker create a tables of data on the disk that step 1004 determines, and the log set is imported into In the tables of data.If log set forms multiple log subclass, need log subclass storing all respective subnumber A tables of data is formed according in table, then by subdatasheet.It should be noted that executing step while step 1005 carries out 1007。
Step 1006 all imports the tables of data creation index for the log set later: master is storing the data It is tables of data creation index on the worker of table, and the corresponding relationship between record log set and the worker.
Step 1007 judges whether data storage server returns to storage in default storage time and complete message, if It is no to enter step 1008, if it is entering step after log set is sent to the worker by 1009:master, wait to be received The record that worker is returned completes information.The worker complete by the log set all daily record datas storage after, Information is completed to master with regard to sending to record.If master does not receive record also within a preset time and completes message, then it is assumed that should Worker breaks down, and does not complete the logger task of the log set.
Step 1008, the data storage server for redefining the log set, return step 1005:master are not received When record completes information, a worker will be redefined in the worker for not also being confirmed as data storage server Store the log set.
Log set storage is independent data list file set by step 1009: master control stores the tables of data Worker the tables of data of the log set is stored in the form of independent data list file set, to form the log collection The backup copies closed.Under the MyISAM data engine of mysql, a tables of data is stored as three data list files.
Step 1010, the data list file set for replicating the tables of data: the data list file is integrated into its institute by master It is replicated on worker, to form new log set backup copies.If the tables of data had been backed up, master can It is replicated with the data list file set for selecting the log to be integrated on any one worker.
Step 1011 determines backup storage server: master is from the current not stored data list file set The worker of the data list file set obtained for storage replication is determined in worker.When determining backup storage server, It is necessary to meet following condition by the worker of storing data list file: having and is currently not carried out the disk of writing task, in these disks The remaining space maximum one data list file set obtained for storage replication；The number that the quantity of worker and duplication obtain Quantity according to list file set is identical；It is located under different interchangers between each worker.The table for storing data The worker of file includes the worker of the worker of the first record log set and backup copies that storage replication obtains, That is, including data storage server and backup storage server.
The data list file set that step 1012, storage replication obtain: step 1010 is replicated obtained data by master The designated disk in worker that the storage of list file set is determined to step 1011, the designated disk are currently to be not carried out writing task Disk in remaining space maximum one, a data list file set of each worker storage log set.? After master sends data list file set on corresponding worker, then etc. the worker to be received returns to storage backup The completion information of task.It, can be again by other if master is not received by the information in preset storage time Worker is determined as backup storage server, then is stored.After worker returns to completion information, master is recorded one by one Corresponding relationship between the log set and the worker of each data list file set for storing the log set.
Step 1013, monitor the log set data list file copy number.
Step 1014 judges whether the copy number of the data list file is equal to default copy number threshold value, if data The copy number of list file is greater than copy number threshold value and then enters step 1015, if the copy number of data list file is less than pair This number threshold value then return step 1010: default copy number threshold value is preferably 3.
Step 1015 deletes superfluous data list file.
It should be noted that 1015 or return step 1010 no matter are entered step after step 1015, it is deletion or multiple The quantity for the data list file being made is the difference between the copy number of current data list file and copy number threshold value.
Step 1016 judges with the presence or absence of small data table in each storage server, if yes then enter step 1017, if No, enter step 1018: the small data table refers to that wherein daily record data quantity is less than the tables of data for presetting storage quantity threshold value.By Need constantly to carry out the recording and storage of daily record data in system, so, this step 1016 can be according to a preset conjunction And time cycle property is executed to monitor whether store small data table in system, rather than backup must be completed in log set It executes again afterwards.
Small data table is merged into a tables of data by step 1017: merging is by the small data table on the same worker It merges.
Step 1018 deletes the log set that storage is not completed in each storage server: this step is for step 1006 Middle judging result be it is no in the case where the dirty data table that is generated on worker.Similarly, since system needs constantly to carry out day The recording and storage of will data, so, this step 1015 can also periodically be executed according to a preset erasing time to supervise It whether there is dirty data table in examining system, and it is not necessary to completing to execute again after backup in log set.
By this scene embodiment, the fast recording and quick backup to a large amount of daily record datas, balance system may be implemented The IO pressure of interior each storage server disk, and can prevent daily record data from losing, improve the utilization rate of system memory space, It additionally aids and realizes user to the quick search of daily record data.
Referring to Figure 11, the flow chart of the scene embodiment of the application inquiry log data method is shown.In the present embodiment Scene in, the application inquiry log data method can be applied to a log query system, which includes inquiry proxy Server proxy and log server master and storage server worker in embodiment illustrated in fig. 10.In this implementation In the scene of example, record of the daily record data in log service system and backup be all by embodiment shown in Fig. 10 come into Row.The present embodiment may include:
Step 1101 receives daily record data inquiry request: user is in inquiry log data, it is desirable that user must input mesh Mark the time of log.It, can also be if log set is divided into log subclass according to type of service in log recording User is required to input the type of service of target journaling when inquiry.
Step 1102 determines target journaling set: proxy according to time of target journaling in daily record data inquiry request, It determines the period where target journaling, then searches the corresponding relationship of the log set and period that record in master, really Set the goal log set.If user has input type of service, the log subclass recorded in master and business can also be searched The corresponding relationship of type determines target journaling subclass.Target journaling set and target journaling subclass generally are multiple, and Target journaling is pertaining only to one of them.
Step 1103 determines inquiry storage server: for each target journaling set, proxy first passes through master The log set of middle record and the corresponding relationship of worker search all storage target journaling collective data list files Then worker determines the disk for being stored with the target journaling collective data list file from the worker found again, then will A minimum disk of the load condition disk used as inquiry in determining disk, finally will be where the disk where inquiry Worker be determined as inquire storage server.
Step 1104 searches target journaling data in the tables of data of target journaling set: proxy is in each target journaling collection On the inquiry storage server worker of conjunction, the number of each target journaling set is searched according to the data table name of each target journaling set According to table, and is searched in the tables of data respectively found and meet the target journaling that log query request requires.Log query requests meeting Comprising information such as such as User ID marks, corresponding target journaling can be found in tables of data according to index according to these information. If target journaling subclass has been determined according to type of service, can only on the corresponding subdatasheet of target journaling subclass in It searches.
The target journaling found is merged and exports by step 1105: due to would generally be in multiple target journaling set searches To target journaling, therefore before exporting query result, proxy needs will be searched on each inquiry storage server worker Target journaling merge.
By this scene embodiment, the time needed for inquiry can be shortened, can also make to balance each storage service in system The IO pressure of disk on device improves IO performance.
Corresponding to above method embodiment, the embodiment of the present application also provides a kind of log services of logdata record Device.Referring to Figure 12, a kind of structural schematic diagram of the log server Installation practice 1 of logdata record is shown, can wrap It includes:
Log set forms module 1201, and the daily record data for will be got according to log acquisition condition forms log collection It closes；
Data storage server determining module 1202, for determining the data storage service for storing the log set Device；
Wherein, the data storage server determining module 1202, specifically for determining for storing the log set Data storage server in be currently not carried out the disk of writing task；
Tables of data generation module 1203, for generating the data of the log set on the data storage server Table；
Module 1204 is inserted into log, for all daily record datas in the log set to be imported the tables of data；
Index creation module 1205, for all daily record datas in the log set all import the tables of data it Afterwards, index is created for the tables of data of the log set.
With reference to Figure 13, the log set forms module 1201 and can specifically include in practical applications:
First daily record data acquisition submodule 1301, for once forming log before obtaining from distributed file system Collect the daily record data generated after the conjunction；
Quantity judging submodule 1302 is obtained, for judging whether the item number of the daily record data reaches preset acquisition number Data bulk threshold value；
Acquisition submodule 1303 is triggered, the feelings for being no in the judging result for obtaining quantity judging submodule 1302 Under condition, waited for the first waiting time triggered the daily record data later and obtain module；
Submodule 1304 is combined in first log, for being yes in the judging result for obtaining quantity judging submodule 1302 In the case where, the daily record data is formed into log set；The log set corresponding period is preceding primary composition log At the time of forming log set to this at the time of set.
With reference to Figure 14, the log set forms module 1201 and can also specifically include in practical applications:
Acquisition time judging submodule 1401 obtains whether the waiting time reaches preset waiting time threshold for judging Value, it is described obtain the waiting time be current time and it is preceding it is primary composition log set at the time of between time；
Triggered time judging submodule 1402 is no for the judging result in the acquisition time judging submodule 1401 In the case where, wait triggering acquisition time judging submodule 1401 after second waiting time；
Second log data acquisition submodule 1403, for the judging result in the acquisition time judging submodule 1401 It is primary before obtaining to form the user journal generated after the log collection conjunction from distributed file system in the case where being Data；
Submodule 1404 is combined in second log, for the daily record data to be formed a log set；The log collection At the time of the corresponding period is closed to form log set to this at the time of preceding primary composition log set.
Daily record data is recorded using the log server of the present embodiment, it can be to avoid because of one day of every record Will data require Update log storage equipment index caused by the slow phenomenon of log recording so that log recording Writing speed accelerate so that the network platform can record a large amount of daily record datas generated in the short time in time, prevent Loss of data, to and shorten delay times of user query.
Referring to Figure 15, a kind of structural schematic diagram of the log server Installation practice 2 of logdata record is shown, is removed Outside structure in log server Installation practice 1 shown in Figure 12, the log server can also include:
Tables of data completes judgment module 1501, for judging that the data storage server is in preset storage time Information is completed in no return storage；The storage is completed information and is completed for the storage server by the institute in the log set There is daily record data to import the tables of data and the information to return after tables of data creation index；
Data storage redefines module 1502, and the judging result for completing judgment module 1501 in the tables of data is In the case where no, never it is determined for redefining in the storage server for storing the log set described for storing The data storage server of log set；
Memory module 1503 is triggered, for triggering in the case where the data storage redefines module 1502 and starts Log set memory module 1203.
Referring to Figure 16, log server Installation practice 2 is other than all structures shown in figure 15, the log server Further include:
Collection modules 1601 are deleted, for periodically deleting in the data storage server according to preset erasing time The log set of storage is not completed.
Using the log server of the present embodiment come log data, mistake can occur to avoid data storage server Caused by log set daily record data cannot be stored as completely tables of data.Furthermore it is also possible to reduce data storage service Junk data in device increases available space, improves utilization efficiency.
Referring to Figure 17, a kind of structural schematic diagram of the log server Installation practice 3 of logdata record is shown, is removed Outside structure in log server Installation practice 1 shown in Figure 12, the log server can also include:
Data bulk judgment module 1701, for judging the data storage service according to preset merging time cycle property It is less than the tables of data of preset storage quantity threshold value on device with the presence or absence of daily record data quantity；
Gather merging module 1702, it, will be described in the case where the judging result of the data bulk 1701, which is, is The tables of data that daily record data quantity is less than storage quantity threshold value merges.
Using the log server of the present embodiment come log data, it is possible to reduce the quantity of tables of data saves user The tables of data quantity required to look up when inquiry log data, to save query time.
Referring to Figure 18, a kind of structural schematic diagram of the log server Installation practice 4 of logdata record is shown, is removed Outside structure in log server Installation practice 1 shown in Figure 12, the log server can also include:
Log subclass forms module 1801, for dividing the daily record data in the log set according to type of service Class forms at least one log subclass；
Correspondingly, the tables of data generation module 1204 includes:
Subdatasheet generates submodule 1802, for according to the log number in log subclass each in the log set According to the corresponding subdatasheet for generating each log subclass；
Subdatasheet combines submodule 1803, for by the subdata of the log subclass all in the log set Table pack at the log set tables of data.
Using the log server of the present embodiment come log data, when family inquiry log can be used, quickly search To the tables of data and subdatasheet of the log set where target journaling, so that the tables of data quantity searched required for reducing, mentions High inquiry velocity.
Referring to Figure 19, a kind of structural schematic diagram of the log server Installation practice 5 of logdata record is shown, is removed Outside structure in log server Installation practice 1 shown in Figure 12, the log server can also include:
Backup generation module 1901, for the tables of data storage of the log set to be independent data list file collection It closes；
Backup copy module 1902, for replicating the tables of data text of the corresponding data list file set of the log set Part；
Backup server determining module 1903 is used for determining from the storage server of the not stored data list file In the backup storage server for storing the data list file for replicating and obtaining；
Wherein, the backup server determining module 1903, can specifically include: first determines submodule, is used for institute The interchanger of the not stored data list file is determined as target switch in the storage server of connection；Second determines submodule Block, for determining the data obtained for storing the duplication from the storage server that the target switch is connected The backup storage server of list file.
Backup memory module 1904, for being stored the obtained data list file that replicates correspondingly to described standby Part storage server is currently not carried out in the disk of writing task.
Using the log server of the present embodiment come log data, the speed of backup file storage, section can be improved Save back up required for the time, and in balance system each disk IO pressure.
Referring to fig. 20, a kind of structural schematic diagram of the log server Installation practice 6 of logdata record is shown, is removed Outside structure in log server Installation practice 4 shown in Figure 19, the log server can also include:
Tables of data monitoring modular 2001, the copy number of the data list file for monitoring the log set；
Tables of data quantity judgment module 2002, for judging the copy number and preset copy number valve of data list file Size relation between value；
Tables of data removing module 2003 is the number for the judging result in the tables of data quantity judgment module 2002 In the case where being greater than the copy number threshold value according to the copy number of list file, the data list file of difference quantities is deleted；Institute State the difference between the copy number and preset copy number threshold value that difference quantities are the data list file；
Replication module 2004 is triggered, is the data for the judging result in the tables of data quantity judgment module 2002 In the case that the copy number of list file is less than the copy number threshold value, the tables of data replication module 1902 is triggered.
Using the log server of the present embodiment come log data, it is ensured that log set in storage server Data list file quantity maintains to stablize, to guarantee that daily record data will not be lost in the case where being not take up excessive memory space.
Corresponding to above method embodiment, the embodiment of the present application also provides a kind of system of logdata record.Referring to Figure 21 shows a kind of structural schematic diagram of the system embodiment 1 of logdata record, may include: log server 2101 With storage server 2102；The log server 2101 is any log server shown in 12~Figure 20 of earlier figures；It is described Storage server 2102 includes data storage server and/or backup storage server.
It, can be to avoid because one daily record data of every record requires using the system of the present embodiment come log data The slow phenomenon of log recording caused by the index of Update log storage equipment, so that the writing speed of log recording adds Fastly, so that the network platform can record a large amount of daily record datas generated in the short time in time, prevent loss of data and The delay time for shortening user query further can also improve the storage speed of daily record data in log set, avoid day The not storage or loss completely of will data, reduces the junk data in data storage server, increases available space, raising can Utilization rate saves query time, BACKUP TIME, the IO pressure etc. of each disk in balance system.
Corresponding to above method embodiment, the embodiment of the present application also provides a kind of system of inquiry log data.Referring to Figure 22 shows a kind of structural schematic diagram of the system embodiment 1 of inquiry log data, may include:
Log storage unit 2201, for passing through any log server storing daily record data shown in Figure 12~Figure 18, The time of all daily record datas belongs to the same period in the log set；
Log backup unit 2202, for passing through log server backup log data shown in Figure 19 or Figure 20；
Corresponding relationship recording unit 2203, for recording pair between the log set and the data storage server Between should being related to, between the log set and the backup storage server and the log set and between the period Corresponding relationship；
Inquiry request receiving unit 2204, for receiving daily record data inquiry request；
Target collection determination unit 2205, for according to the time where the time of target journaling in the inquiry request Section, determines the target journaling set where target journaling data；
Query service device determination unit 2206, for according to corresponding between the log set and data storage server Corresponding relationship between relationship and the log set and backup storage server, from being stored with the target journaling collective data In the data storage server and backup storage server of list file, determine that the inquiry for inquiring the target journaling stores clothes Business device；
Searching unit 2207, for searching the tables of data text of the target journaling set in the inquiry storage server Part；
Target journaling searching unit 2208, for the tables of data rope in the data list file according to the target journaling set Draw, the target journaling data are searched in the tables of data of the target journaling collective data list file；
Target journaling output module 2209, the target journaling data for will find merge output.
Using the system of the present embodiment come inquiry log data, the speed of log query can be improved, reduce inquiry Delay time makes the disk in storage server realize read and write abruption, disk I/O pressure etc. in balance system.
It should be noted that, in this document, relational terms such as first and second and the like are used merely to a reality Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation In any actual relationship or order or sequence.The terms "include", "comprise" or its any other variant are intended to non-row His property includes, so that the process, method, article or equipment for including a series of elements not only includes those elements, and And further include other elements that are not explicitly listed, or further include for this process, method, article or equipment institute it is intrinsic Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including institute State in the process, method, article or equipment of element that there is also other identical elements.
For system embodiments, since it corresponds essentially to embodiment of the method, so related place is referring to method reality Apply the part explanation of example.System embodiment described above is only schematical, wherein described be used as separation unit The unit of explanation may or may not be physically separated, and component shown as a unit can be or can also be with It is not physical unit, it can it is in one place, or may be distributed over multiple network units.It can be according to actual It needs that some or all of the modules therein is selected to achieve the purpose of the solution of this embodiment.Those of ordinary skill in the art are not In the case where making the creative labor, it can understand and implement.
The above is only the specific embodiment of the application, it is noted that for the ordinary skill people of the art For member, under the premise of not departing from the application principle, several improvements and modifications can also be made, these improvements and modifications are also answered It is considered as the protection scope of the application.

Claims (23)

1. a kind of method of logdata record characterized by comprising
The daily record data got according to log acquisition condition is formed into log set；
Determine the data storage server for storing the log set；
Generate the tables of data of the log set on the data storage server, and by all days in the log set Will data import the tables of data；
It is the tables of data of the log set after all daily record datas of the log set all import the tables of data Creation index；
Judge whether the data storage server returns to storage in preset storage time and complete information；The storage is completed Information is that the storage server is completed all daily record datas importing tables of data in the log set and is institute State the information returned after tables of data creation index；
If it is not, then being never determined for redefining in the storage server for storing the log set for storing The data storage server for stating log set generates the data of the log set on the data storage server redefined Table, and all daily record datas in the log set are imported into the tables of data.
2. the method according to claim 1, wherein the log number that will be got according to log acquisition condition According to composition log set, comprising:
It is primary before obtaining to form the daily record data generated after the log collection conjunction from distributed file system；
Judge whether the item number of the daily record data reaches preset acquisition data bulk threshold value；
If it is not, then once formed before returning to the judgement acquisition after waiting for the first waiting time the log collection conjunction it The step of daily record data generated afterwards；
If it is, the daily record data is formed log set；The log set corresponding period is preceding primary composition At the time of forming log set to this at the time of log set.
3. the method according to claim 1, wherein the log number that will be got according to log acquisition condition According to composition log set, comprising:
Judge to obtain whether the waiting time reaches preset waiting time threshold value, the acquisition waiting time is current time with before Time between at the time of primary composition log set；
Described judge the acquisition waiting time when whether reaching preset waiting if it is not, then returning after waiting for the second waiting time Between threshold value the step of；
If it is, from distributed file system, the daily record data generated after the primary composition log collection conjunction before obtaining, And the daily record data is formed into a log set；The log set corresponding period is preceding primary composition log set At the time of to this form log set at the time of.
4. the method according to claim 1, wherein the log acquisition condition is the time according to daily record data Sequence obtains log, and the time of all daily record datas belongs to the same period in the log set；
Correspondingly, by after the daily record data got according to log acquisition condition composition log set, further includes: by the day Daily record data in will set is classified according to type of service, forms at least one log subclass；
Correspondingly, the tables of data for generating the log set on the data storage server, comprising:
According to the daily record data in log subclass each in the log set, the corresponding subnumber for generating each log subclass According to table；
The subdatasheet of log subclass all in the log set is combined into the tables of data of the log set.
5. the method according to claim 1, wherein the data that the determination is used to store the log set are deposited Server is stored up, specifically: it determines in the data storage server for storing the log set and is currently not carried out writing task Disk.
6. the method according to claim 1, wherein the method also includes:
The log set that storage is not completed in the data storage server is periodically deleted according to preset erasing time.
7. the method according to claim 1, wherein described generate the day on the data storage server After the tables of data of will set, further includes:
Judge to be less than on the data storage server with the presence or absence of daily record data quantity according to preset merging time cycle property The tables of data of preset storage quantity threshold value；If it is, the daily record data quantity to be less than to the data of storage quantity threshold value Table merges.
8. the method according to claim 1, wherein the tables of data for the log set creates index Afterwards, further includes:
The tables of data storage of the log set is independent data list file set；
Replicate the data list file in the corresponding data list file set of the log set；
The tables of data obtained for storing the duplication is determined from the storage server of the not stored data list file The backup storage server of file；
The data list file that the duplication obtains is stored correspondingly to the backup storage server to be currently not carried out and is write In the disk of task.
9. according to the method described in claim 8, it is characterized in that, the storage service from the not stored data list file It is determined in device for storing the backup storage server for replicating the obtained data list file set, comprising:
The data list file intersection of sets not stored in the storage server connected is changed planes and is determined as target switch；
The tables of data obtained for storing the duplication is determined from the storage server that the target switch is connected The backup storage server of file.
10. according to the method described in claim 8, it is characterized in that, described store the obtained data list file that replicates After into the backup storage server, further includes:
Monitor the copy number of the data list file of the log set；
Judge the size relation between the copy number of the data list file and preset copy number threshold values；
If the copy number of the data list file is greater than the copy number threshold value, by the data list file of difference quantities It deletes；The difference quantities are the difference between the copy number and the copy number threshold value of the data list file；
If the copy number of the data list file is less than the copy number threshold value, the duplication log collection is returned The step of closing the data list file of corresponding data list file set.
11. a kind of method of inquiry log data, which is characterized in that pass through method described in claim 1~10 any one Storing daily record data, the time of all daily record datas belongs to the same period in the log set；Pass through claim 8,9 Or method backup log data described in 10；It records between the log set and the data storage server, the log Between set and the backup storage server and the log set and the corresponding relationship between the period；
The described method includes:
Receive daily record data inquiry request；
Period and the log set where the time of target journaling data in the inquiry request and period it Between corresponding relationship, determine the target journaling set where the target journaling data；
According to the corresponding relationship and/or the log set between the log set and data storage server and back up storage Corresponding relationship between server, from the data storage server and backup for being stored with the target journaling collective data list file In storage server, the inquiry storage server for inquiring the target journaling is determined；
The tables of data of the target journaling set is searched in the inquiry storage server, and is looked into the tables of data found Look for the target journaling data；
The target journaling data found are merged and are exported.
12. a kind of log server of log data characterized by comprising
Log set forms module, and the daily record data for will be got according to log acquisition condition forms log set；
Data storage server determining module, for determining the data storage server for storing the log set；
Tables of data generation module, for generating the tables of data of the log set on the data storage server；
Module is inserted into log, for all daily record datas in the log set to be imported the tables of data；
Index creation module, for after all daily record datas of the log set all import the tables of data, for institute State the tables of data creation index of log set；
Tables of data completes judgment module, deposits for judging whether the data storage server returns in preset storage time Information is completed in storage；It is that the storage server is completed all log numbers in the log set that information is completed in the storage Information according to the tables of data is imported and to be returned after tables of data creation index；
Data storage redefines module, in the case that the judging result for completing judgment module in the tables of data is no, Never it is determined for redefining in the storage server for storing the log set for storing the log set Data storage server；
Memory module is triggered, in the case where the data storage redefines module starting, triggering log set to be stored Module.
13. log server according to claim 12, which is characterized in that the log set forms module and includes:
First daily record data acquisition submodule, for once forming the log collection conjunction before obtaining from distributed file system The daily record data generated later；
Quantity judging submodule is obtained, for judging whether the item number of the daily record data reaches preset acquisition data bulk threshold Value；
Acquisition submodule is triggered, for waiting the in the case where the judging result for obtaining quantity judging submodule is no The daily record data is triggered after one waiting time obtains module；
Submodule is combined in first log, for inciting somebody to action in the case where the judging result for obtaining quantity judging submodule is to be The daily record data forms log set；At the time of the log set corresponding period is preceding primary composition log set extremely At the time of this composition log set.
14. log server according to claim 12, which is characterized in that the log set forms module and includes:
Acquisition time judging submodule obtains whether the waiting time reaches preset waiting time threshold value for judging, described to obtain Take the waiting time be current time and it is preceding it is primary composition log set at the time of between time；
Triggered time judging submodule, for the judging result of the acquisition time judging submodule be it is no in the case where, etc. To trigger the acquisition time judging submodule after the second waiting time；
Second log data acquisition submodule is the case where being for the judging result in the acquisition time judging submodule Under, it is primary before obtaining to form the daily record data generated after the log collection conjunction from distributed file system；
Submodule is combined in second log, for the daily record data to be formed a log set；The log set is corresponding At the time of forming log set to this at the time of period is preceding primary composition log set.
15. log server according to claim 12, which is characterized in that the log server further include:
Log subclass forms module, for the daily record data in the log set to be classified according to type of service, is formed extremely A few log subclass；
Correspondingly, the tables of data generation module includes:
Subdatasheet generates submodule, for corresponding to according to the daily record data in log subclass each in the log set Generate the subdatasheet of each log subclass；
Subdatasheet combines submodule, for the subdatasheet of the log subclass all in the log set to be combined into The tables of data of the log set.
16. log server according to claim 12, which is characterized in that the data storage server determining module, The disk of writing task is currently not carried out in the data storage server for storing the log set specifically for determining.
17. log server according to claim 16, which is characterized in that the log server further include:
Collection modules are deleted, are deposited for periodically deleting not completing in the data storage server according to preset erasing time The log set of storage.
18. log server according to claim 12, which is characterized in that the log server further include:
Data bulk judgment module, for according to preset merging time cycle property judge on the data storage server whether There are the tables of data that daily record data quantity is less than preset storage quantity threshold value；
Gather merging module, for the judging result of the data bulk be in the case where, by the daily record data quantity Tables of data less than storage quantity threshold value merges.
19. log server according to claim 12, which is characterized in that the log server further include:
Backup generation module, for the tables of data storage of the log set to be independent data list file set；
Backup copy module, for replicating the data list file of the corresponding data list file set of the log set；
Backup server determining module, for determining from the storage server of the not stored data list file for storing State the backup storage server for the data list file that duplication obtains；
Backup memory module, for being stored the obtained data list file that replicates correspondingly to backup storage clothes Business device is currently not carried out in the disk of writing task.
20. log server according to claim 19, which is characterized in that the backup server determining module, comprising:
First determines submodule, for the interchanger of the data list file not stored in the storage server connected is true It is set to target switch；
Second determines submodule, for determining described multiple for storing from the storage server that the target switch is connected The backup storage server for the data list file being made.
21. log server according to claim 19, which is characterized in that the log server further include:
Tables of data monitoring modular, the copy number of the data list file for monitoring the log set；
Tables of data quantity judgment module, for judging between the copy number of data list file and preset copy number threshold values Size relation；
Tables of data removing module is the pair of the data list file for the judging result in the tables of data quantity judgment module In the case that this number is greater than the copy number threshold value, the data list file of difference quantities is deleted；The difference quantities are Difference between the copy number of the data list file and preset copy number threshold value；
Replication module is triggered, is the copy of the data list file for the judging result in the tables of data quantity judgment module In the case that number is less than the copy number threshold value, the backup copy module is triggered.
22. a kind of system of log data, which is characterized in that the system comprises log servers and storage server； The log server is log server described in claim 12~21 any one；The storage server includes data Storage server and/or backup storage server.
23. a kind of system of inquiry log data, which is characterized in that the system comprises:
Log storage unit, for passing through log server storing daily record data described in claim 12~21 any one, The time of all daily record datas belongs to the same period in the log set；
Log backup unit, for passing through log server backup log data described in claim 19,20 or 21；
Corresponding relationship recording unit, for recording between the log set and the data storage server, the log collection It closes between the backup storage server and the log set and the corresponding relationship between the period；
Inquiry request receiving unit, for receiving daily record data inquiry request；
Target collection determination unit, for determining mesh according to the period where the time of target journaling in the inquiry request Mark the target journaling set where daily record data；
Query service device determination unit, for according between the log set and data storage server corresponding relationship and/ Or the corresponding relationship between the log set and backup storage server, it is literary from the target journaling collective data table is stored with In the data storage server and backup storage server of part, the inquiry storage service for inquiring the target journaling is determined Device；
Searching unit, for searching the tables of data of the target journaling set in the inquiry storage server；
Target journaling searching unit, for searching the target journaling data in the tables of data found；
Target journaling output module, the target journaling data for will find merge output.
CN201310306476.3A 2013-07-19 2013-07-19 A kind of method of logdata record, log server and system Active CN104301360B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201310306476.3A CN104301360B (en) 2013-07-19 2013-07-19 A kind of method of logdata record, log server and system
HK15102899.1A HK1202722A1 (en) 2013-07-19 2015-03-23 Method for log data recording, logging server and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310306476.3A CN104301360B (en) 2013-07-19 2013-07-19 A kind of method of logdata record, log server and system

Publications (2)

Publication Number Publication Date
CN104301360A CN104301360A (en) 2015-01-21
CN104301360B true CN104301360B (en) 2019-03-12

Family

ID=52320927

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310306476.3A Active CN104301360B (en) 2013-07-19 2013-07-19 A kind of method of logdata record, log server and system

Country Status (2)

Country Link
CN (1) CN104301360B (en)
HK (1) HK1202722A1 (en)

Families Citing this family (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105989059B (en) * 2015-02-09 2019-12-24 阿里巴巴集团控股有限公司 Data record checking method and device
CN105260399A (en) * 2015-09-17 2016-01-20 西安未来国际信息股份有限公司 Method for acquiring and retrieving distributed log
CN105391772B (en) * 2015-10-16 2019-02-22 百度在线网络技术（北京）有限公司 Service request processing method, log processing method and device
CN106815274B (en) * 2015-12-02 2022-02-18 中兴通讯股份有限公司 Hadoop-based log data mining method and system
CN106897281B (en) * 2015-12-17 2020-08-14 阿里巴巴集团控股有限公司 Log fragmentation method and device
CN105528454A (en) * 2015-12-25 2016-04-27 北京奇虎科技有限公司 Log treatment method and distributed cluster computing device
CN107239377B (en) * 2016-03-29 2021-02-26 阿里巴巴集团控股有限公司 Method and device for acquiring running state of Java virtual machine
CN107306416A (en) * 2016-04-25 2017-10-31 中国移动通信集团云南有限公司 The recording method of business diary and device
CN106055706B (en) * 2016-06-23 2019-08-06 杭州迪普科技股份有限公司 A kind of cache resources storage method and device
CN107590162A (en) * 2016-07-08 2018-01-16 中兴通讯股份有限公司 The storage method and device of journal file
CN107644017A (en) * 2016-07-20 2018-01-30 平安科技（深圳）有限公司 The querying method and device of journal file
CN106777371B (en) * 2017-01-23 2019-12-06 北京齐尔布莱特科技有限公司 Log collection system and method
CN108509455B (en) * 2017-02-28 2021-12-24 百度在线网络技术（北京）有限公司 Data table processing method and device
CN107609895B (en) * 2017-08-10 2021-08-27 腾讯科技（深圳）有限公司 Processing method and equipment for merging service areas
CN110019069A (en) * 2017-10-30 2019-07-16 阿里巴巴集团控股有限公司 A kind of log processing method, device and equipment
CN107835095A (en) * 2017-11-20 2018-03-23 杭州迪普科技股份有限公司 A kind of processing method and processing device of daily record
CN109828862B (en) * 2017-11-23 2023-08-22 成都华为技术有限公司 Method and device for replaying log
CN109117312B (en) * 2018-08-23 2022-03-01 北京小米智能科技有限公司 Data recovery method and device
US11120908B2 (en) * 2018-09-20 2021-09-14 Abiomed, Inc. Data storage and retrieval system for non-contiguous medical device operational data
CN109471851B (en) * 2018-10-17 2021-05-28 上海达梦数据库有限公司 Data processing method, device, server and storage medium
CN109460318B (en) * 2018-10-26 2021-01-01 珠海市时杰信息科技有限公司 Import method of rollback archive collected data, computer device and computer readable storage medium
CN111198853B (en) * 2018-11-16 2023-08-22 北京微播视界科技有限公司 Data processing method, device, electronic equipment and computer readable storage medium
CN109766218A (en) * 2018-12-21 2019-05-17 中信百信银行股份有限公司 Data back up method based on distributed storage
CN109922490B (en) * 2019-01-28 2022-01-28 广东中视信息科技有限公司 Data interruption-proof continuous transmission management system
CN110232048B (en) * 2019-06-12 2023-07-07 腾讯科技(成都)有限公司 Log file acquisition method, device and storage medium
CN110502545B (en) * 2019-08-15 2023-12-05 中国平安财产保险股份有限公司 Data storage method, device, equipment and computer readable storage medium
CN111414343B (en) * 2020-02-24 2023-08-04 北京百度云途腾科技有限责任公司 Log writing method, device, electronic equipment and medium
CN111400368B (en) * 2020-03-06 2023-12-12 网宿科技股份有限公司 Log searching method and device of distributed server system
CN111830039B (en) * 2020-07-22 2021-07-27 南京认知物联网研究院有限公司 Intelligent product quality detection method and device
CN112395360B (en) * 2020-12-01 2023-06-23 中国联合网络通信集团有限公司 Data synchronization method, device, apparatus and medium based on non-relational database

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1932812A (en) * 2005-09-16 2007-03-21 腾讯科技（深圳）有限公司 Method and apparatus for holding journal
CN101605028A (en) * 2009-02-17 2009-12-16 北京安天电子设备有限公司 A kind of combining log records method and system
CN101763593A (en) * 2009-12-17 2010-06-30 中国电力科学研究院 Method and device for realizing audit log of system
CN101799894A (en) * 2010-01-11 2010-08-11 中冶东方工程技术有限公司 EPC engineering data processing system and method in complex network environment
CN102750326A (en) * 2012-05-30 2012-10-24 浪潮电子信息产业股份有限公司 Log management optimization method of cluster system based on downsizing strategy
WO2012154838A2 (en) * 2011-05-09 2012-11-15 Google Inc. Generating application recommendations based on user installed applications
WO2013033123A2 (en) * 2011-08-31 2013-03-07 Google Inc. Method and system for collecting and managing tv viewership data
CN102981944A (en) * 2012-12-12 2013-03-20 青岛海信宽带多媒体技术有限公司 Log storage method based on file system

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1932812A (en) * 2005-09-16 2007-03-21 腾讯科技（深圳）有限公司 Method and apparatus for holding journal
CN101605028A (en) * 2009-02-17 2009-12-16 北京安天电子设备有限公司 A kind of combining log records method and system
CN101763593A (en) * 2009-12-17 2010-06-30 中国电力科学研究院 Method and device for realizing audit log of system
CN101799894A (en) * 2010-01-11 2010-08-11 中冶东方工程技术有限公司 EPC engineering data processing system and method in complex network environment
WO2012154838A2 (en) * 2011-05-09 2012-11-15 Google Inc. Generating application recommendations based on user installed applications
WO2013033123A2 (en) * 2011-08-31 2013-03-07 Google Inc. Method and system for collecting and managing tv viewership data
CN102750326A (en) * 2012-05-30 2012-10-24 浪潮电子信息产业股份有限公司 Log management optimization method of cluster system based on downsizing strategy
CN102981944A (en) * 2012-12-12 2013-03-20 青岛海信宽带多媒体技术有限公司 Log storage method based on file system

Also Published As

Publication number Publication date
CN104301360A (en) 2015-01-21
HK1202722A1 (en) 2015-10-02

Similar Documents

Publication Publication Date Title
CN104301360B (en) A kind of method of logdata record, log server and system
Santos et al. Real-time data warehouse loading methodology
US7689602B1 (en) Method of creating hierarchical indices for a distributed object system
CN102521406B (en) Distributed query method and system for complex task of querying massive structured data
CN105930228B (en) Data back up method and system
CN101103355B (en) Methods and apparatus for managing deletion of data
US11138227B2 (en) Consistent query execution in hybrid DBMS
CN102012933B (en) Distributed file system and method for storing data and providing services by utilizing same
US10417265B2 (en) High performance parallel indexing for forensics and electronic discovery
CN107835983A (en) Backup-and-restore is carried out in distributed data base using consistent database snapshot
CN106662981A (en) Storage device, program, and information processing method
CN103646111A (en) System and method for realizing real-time data association in big data environment
CN101673192B (en) Method for time-sequence data processing, device and system therefor
CN107003935A (en) Optimize database duplicate removal
CN107798130A (en) A kind of Snapshot Method of distributed storage
KR102038529B1 (en) System for processing real-time data modification of in-memory database
JP7030831B2 (en) Manage large association sets with optimized bitmap representations
EP2380090B1 (en) Data integrity in a database environment through background synchronization
CN104317957B (en) A kind of open platform of report form processing, system and report processing method
CN102332004B (en) Data processing method and system for managing mass data
CN110019469A (en) Distributed data base data processing method, device, storage medium and electronic device
CN109144416A (en) The method and apparatus for inquiring data
CN109408536A (en) A kind of cloud platform data managing method
CN102779138A (en) Hard disk access method of real time data
Santos et al. Optimizing data warehouse loading procedures for enabling useful-time data warehousing

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 1202722

Country of ref document: HK

GR01 Patent grant
GR01 Patent grant