CN104301360A - Method, log server and system for recording log data - Google Patents

Method, log server and system for recording log data Download PDF

Info

Publication number
CN104301360A
CN104301360A CN201310306476.3A CN201310306476A CN104301360A CN 104301360 A CN104301360 A CN 104301360A CN 201310306476 A CN201310306476 A CN 201310306476A CN 104301360 A CN104301360 A CN 104301360A
Authority
CN
China
Prior art keywords
daily record
data
storage server
record set
log
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201310306476.3A
Other languages
Chinese (zh)
Other versions
CN104301360B (en
Inventor
刘宾
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201310306476.3A priority Critical patent/CN104301360B/en
Publication of CN104301360A publication Critical patent/CN104301360A/en
Priority to HK15102899.1A priority patent/HK1202722A1/en
Application granted granted Critical
Publication of CN104301360B publication Critical patent/CN104301360B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a method, log server and system for recording log data. The method comprises that log data obtained according to the log obtaining condition forms a log set; a data storage server for storing the log set is determined; the log set is stored in the data storage server, and a data sheet of the log set is generated in the data storage server; and after all log data of the log set is introduced into the data sheet, an index is created for the data sheet of the log set. According to the method, log server and system, the storage speed of log data can be improved, a network platform can timely record a lot of log data that is generated in a short time, data loss is prevented, and delayed query time can be shortened.

Description

A kind of method of logdata record, log server and system
Technical field
The application relates to network data processing field, particularly relates to a kind of method of logdata record, log server and system.
Background technology
Along with internet communication is more and more universal, and the number of users by network interaction between the network platform gets more and more, and number of times mutual between user and platform also gets more and more.For the ease of user can inquire completed alternately for information about, the mode of usual employing is the daily record data that interactive information is generated user by the network platform, then each daily record data produced alternately of each user is recorded, like this, when user needs to inquire about completed interactive information, the daily record data that can directly record in the network platform always searches this mutual corresponding daily record data, and then the daily record data found is returned to user, such inquiry just can obtain required interactive information.
In existing logdata record method, whenever generation user journal data, the insertion of this daily record data is just stored on a certain memory device in system by log service system, like this, in order to when user inquires about, log service system can find the memory location of daily record data in daily record memory device, all to the index in daily record memory device once be upgraded while every bar daily record data is inserted.
Carry out log data by prior art, all need once to upgrade the index of daily record memory device owing to often inserting a daily record, so the speed of logdata record is comparatively slow, cause the efficiency of logdata record lower.And further, the current increase due to network platform traffic carrying capacity, the daily record data quantity that the network platform produces gets more and more, log data is carried out by prior art, be difficult to a large amount of daily record datas recording generation in time, like this, not only can cause the loss of daily record data, also can cause and inquire about less than interactive information in the long period of user after completing alternately, also namely the time of delay of user's inquiry is longer.
Summary of the invention
Technical problems to be solved in this application are, a kind of method of logdata record, Apparatus and system are provided, with the technical problem that the efficiency solving the logdata record that the mode of logdata record causes is conventionally lower, further, can also solve in the short time produce the network platform that daily record data reaches hundred million number of levels be difficult to record in time a large amount of daily record data of generation and time of delay that daily record data is lost and user inquires about of causing longer technical problem.
For solving the problems of the technologies described above, the embodiment of the present application provides a kind of method of logdata record, and the method comprises:
By the daily record data composition daily record set got according to log acquisition condition;
Determine the data storage server storing described daily record set;
Described data storage server generates the tables of data of described daily record set, and all daily record datas in described daily record set are imported described tables of data;
After all daily record datas of described daily record set all import described tables of data, for the tables of data of described daily record set creates index.
Preferably, the described daily record data composition daily record set will got according to log acquisition condition, comprising:
From distributed file system, the daily record data generated after once forming the daily record set moment before acquisition;
Judge the acquisition number data bulk the threshold value whether number of described daily record data reaches default;
If not, then waited for for the first stand-by period after return the described acquisition of described judgement before once form the daily record set moment after the step of daily record data that generates;
If so, then by described daily record data composition daily record set; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
Preferably, the described daily record data composition daily record set will got according to log acquisition condition, comprising:
Judge whether reach default stand-by period threshold value, the described acquisition stand-by period is the time between current time and the front moment once forming daily record set if obtaining the stand-by period;
If not, then wait for described second stand-by period after return and describedly judge whether obtain the stand-by period reaches the step of default stand-by period threshold value;
If so, then from distributed file system, the daily record data generated after once forming the daily record set moment before acquisition, and described daily record data is formed a daily record set; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
Preferably, described log acquisition condition is obtain daily record according to the time sequencing of daily record data, and in described daily record set, the time of all daily record datas belongs to the same time period;
Accordingly, after the daily record data got according to log acquisition condition is formed daily record set, also comprise: the daily record data in described daily record set is classified according to type of service, form at least one daily record subclass;
Accordingly, the described tables of data generating described daily record set on described data storage server, comprising:
According to the daily record data in daily record subclass each in described daily record set, the corresponding subdatasheet generating each daily record subclass;
The subdatasheet of daily record subclass all in described daily record set is combined into the tables of data of described daily record set.
Preferably, described in determine the data storage server storing described daily record set, be specially: determine storing the current disk not performing writing task in the data storage server of described daily record set.
Preferably, described on described data storage server, generate the tables of data of described daily record set after, also comprise:
Judge whether described data storage server returns storage and complete information within the memory time of presetting; Described storage completes information, and to be described storage server import described tables of data by all daily record datas in described daily record set and the information returned after creating index for described tables of data completing;
If not, then never determined storing in the storage server of described daily record set the data storage server redefined for storing described daily record set, and returned the described step described daily record set being stored into described data storage server.
Preferably, described method also comprises:
Periodically the daily record set not completing storage in described data storage server is deleted according to the erasing time preset.
Preferably, described on described data storage server, generate the tables of data of described daily record set after, also comprise:
Judge whether described data storage server exists according to the merging time cycle property preset the tables of data that daily record data quantity is less than default stored number threshold value; If so, then the tables of data described daily record data quantity being less than stored number threshold value merges.
Preferably, described is that the tables of data of described daily record set creates after index, also comprises:
The tables of data of described daily record set is stored as independently data list file set;
Copy the data list file in data list file set corresponding to described daily record set;
Never the back-up storage server determining copying the described data list file obtained described in storage is stored in the storage server of described data list file;
The described data list file obtained that copies is stored into that described back-up storage server is current not to be performed in the disk of writing task one to one.
Preferably, described never store in the storage server of described data list file determine storing described in copy the back-up storage server of the described data list file set obtained, comprising:
Change planes be defined as target switch by all not storing described data list file intersection of sets in connected storage server;
The back-up storage server copying the described data list file obtained described in storing is determined from the storage server that described target switch connects.11, method according to claim 9, is characterized in that, describedly copies after the data list file obtained is stored in described back-up storage server by described, also comprises:
Monitor the copy number of the data list file of described daily record set;
Judge the magnitude relationship between the copy number of described data list file and default copy number threshold values;
If the copy number of described data list file is greater than described copy number threshold value, then the data list file of difference quantities is deleted; Described difference quantities is the difference between the copy number of described data list file and described copy number threshold value;
If the copy number of described data list file is less than described copy number threshold value, then copy the step of the data list file of data list file set corresponding to described daily record set described in returning.
Disclosed herein as well is a kind of method of inquiry log data, by the method storing daily record data of aforementioned any one, in described daily record set, the time of all daily record datas all belongs to the same time period; By aforesaid method backup log data; Record between described daily record set and described data storage server, between described daily record set and described back-up storage server and described daily record set and the corresponding relation between the time period;
Described method comprises:
Receive daily record data inquiry request;
According to time period at the time place of target journaling data in described inquiry request and described daily record set and the corresponding relation between the time period, determine the target journaling set at described target journaling data place;
According to the corresponding relation between described daily record set and data storage server and/or the corresponding relation between described daily record set and back-up storage server, from the data storage server storing described target journaling collective data list file and back-up storage server, determine the inquiry storage server inquiring about described target journaling;
Described inquiry storage server is searched the tables of data of described target journaling set, and in the tables of data found, searches described target journaling data;
The target journaling data found are merged and exports.
Disclosed herein as well is a kind of log server of log data, comprising:
Daily record set forms module, for the daily record data composition daily record set will got according to log acquisition condition;
Data storage server determination module, for determining the data storage server storing described daily record set;
Tables of data generation module, for generating the tables of data of described daily record set on described data storage server;
Daily record insert module, for importing described tables of data by all daily record datas in described daily record set;
Index creation module, after all importing described tables of data at all daily record datas of described daily record set, for the tables of data of described daily record set creates index.
Preferably, described daily record set formation module comprises:
First daily record data obtains submodule, for from distributed file system, and the daily record data generated after once forming the daily record set moment before acquisition;
Obtain quantity and judge submodule, for judging the acquisition number data bulk the threshold value whether number of described daily record data reaches default;
Trigger and obtain submodule, for when described acquisition quantity judges that the judged result of submodule is no, after waiting for for the first stand-by period, trigger described daily record data acquisition module;
First daily record combination submodule, for when described acquisition quantity judges the judged result of submodule for being, by described daily record data composition daily record set; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
Preferably, described daily record set formation module comprises:
Acquisition time judges submodule, for judging whether reach default stand-by period threshold value, the described acquisition stand-by period is the time between current time and the front moment once forming daily record set if obtaining the stand-by period;
Triggered time judges submodule, for judging that the judged result of submodule is no at described acquisition time, triggering described acquisition time and judging submodule after waiting for described second stand-by period;
Second daily record data acquisition submodule, for when described acquisition time judges the judged result of submodule for being, from distributed file system, the daily record data generated after once forming the daily record set moment before acquisition;
Second daily record combination submodule, for forming a daily record set by described daily record data; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
Preferably, described log server also comprises:
Daily record subclass forms module, for being classified according to type of service by the daily record data in described daily record set, forms at least one daily record subclass;
Accordingly, described tables of data generation module comprises:
Subdatasheet generates submodule, for according to the daily record data in daily record subclass each in described daily record set, and the corresponding subdatasheet generating each daily record subclass;
Subdatasheet combination submodule, for being combined into the tables of data of described daily record set by the subdatasheet of described daily record subclass all in described daily record set.
Preferably, described data storage server determination module, specifically for determine storing described daily record set data storage server in the current disk not performing writing task.
Preferably, described log server also comprises:
Tables of data completes judge module, for judging whether described data storage server returns storage and complete information within the memory time of presetting; Described storage completes information, and to be described storage server import described tables of data by all daily record datas in described daily record set and the information returned after creating index for described tables of data completing;
Data storage redefines module, when being no for completing the judged result of judge module in described tables of data, never determined storing in the storage server of described daily record set the data storage server redefined for storing described daily record set;
Trigger memory module, for redefining module startup in described data storage, trigger daily record set memory module.
Preferably, described log server also comprises:
Delete collection modules, for periodically deleting the daily record set not completing storage in described data storage server according to the erasing time preset.
Preferably, described log server also comprises:
Data bulk judge module, for judging whether described data storage server exists the tables of data that daily record data quantity is less than default stored number threshold value according to the merging time cycle property preset;
Set merges module, and for when the judged result of described data bulk is for being, the tables of data described daily record data quantity being less than stored number threshold value merges.
Preferably, described log server also comprises:
Backup generation module, for being stored as independently data list file set by the tables of data of described daily record set;
Backup copy module, for copying the data list file of data list file set corresponding to described daily record set;
Backup server determination module, for never store described data list file storage server in determine storing described in copy the back-up storage server of the data list file obtained;
By the described data list file obtained that copies, back-up storage module, for being stored into that described back-up storage server is current not to be performed in the disk of writing task one to one.
Preferably, described backup server determination module, comprising:
First determines submodule, for the switch all not storing described data list file in connected storage server is defined as target switch;
Second determines submodule, for copying the back-up storage server of the described data list file obtained described in determining in the storage server that connects from described target switch storing.
Preferably, described log server also comprises:
Tables of data monitoring modular, for monitoring the copy number of the data list file of described daily record set;
Tables of data quantity judge module, for judging the magnitude relationship between the copy number of data list file and default copy number threshold values;
Tables of data removing module, when being greater than described copy number threshold value for the copy number that is described data list file in the judged result of described tables of data quantity judge module, deletes the data list file of difference quantities; Described difference quantities is the difference between the copy number of described data list file and default copy number threshold value;
Trigger replication module, when being less than described copy number threshold value for the copy number that is described data list file in the judged result of described tables of data quantity judge module, trigger described backup copy module.
Disclosed herein as well is a kind of system of log data, it is characterized in that, described system comprises log server and storage server; Described log server is the log server of aforementioned any one; Described storage server comprises data storage server and/or back-up storage server.
Disclosed herein as well is a kind of system of inquiry log data, described system comprises:
Log storage unit, for passing through aforesaid log server storing daily record data, in described daily record set, the time of all daily record datas all belongs to the same time period;
Log backup unit, for by aforesaid log server backup log data;
Corresponding relation record cell, for recording between described daily record set and described data storage server, between described daily record set and described back-up storage server and described daily record set and the corresponding relation between the time period;
Inquiry request receiving element, for receiving daily record data inquiry request;
Goal set determining unit, for the time period at the time place according to target journaling in described inquiry request, determines the target journaling set at target journaling data place;
Querying server determining unit, for according to the corresponding relation between described daily record set and data storage server and/or the corresponding relation between described daily record set and back-up storage server, from the data storage server storing described target journaling collective data list file and back-up storage server, determine the inquiry storage server inquiring about described target journaling;
Search unit, for searching the tables of data of described target journaling set on described inquiry storage server;
Target journaling searches unit, for searching described target journaling data in the tables of data found;
Target journaling output module, merges output for the target journaling data that will find.
As can be seen from above-mentioned technical scheme, the method of logdata record disclosed in the embodiment of the present application, daily record set is formed by gathering the mode carrying out storing by obtaining daily record data because adopt, the daily record data composition daily record set first will obtained, then the data storage server of storing daily record set is determined, so just first can generate the tables of data of this daily record set and all daily record datas in this daily record set are all imported on data storage server on this data storage server, be this daily record collective data table establishment index again after all daily record datas of this daily record set all import described tables of data, so just can avoid because often record daily record data all need the index of Update log memory device and the slow phenomenon of log recording that causes, the writing speed of log recording is accelerated, thus make the network platform can record a large amount of daily record datas produced in the short time in time, prevent loss of data and shorten user inquiry time of delay, make user can inquire completed interactive information more in time.
Further, in the embodiment of the present application when determining the data storage server of storing daily record set, can determine currently in data storage server not perform in the disk of writing task, like this, the disk of storing daily record set does not perform other writing tasks when storing, and can improve the storage speed of daily record data in daily record set.
Further, in the embodiment of the present application, daily record set storage is completed to data storage server and be also provided with threshold value memory time, when data storage server has not also returned storing message through memory time, just will redefine data storage server to store this daily record set, can avoid like this data storage server occur wrong and in the daily record set that causes daily record data can not be stored as tables of data completely.In addition, on the basis that threshold value memory time is set, erasing time can also be provided with periodically to delete the daily record set not completing storage, the junk data in data storage server can be reduced like this, increase free space, improve utilization ratio.
Further, in the embodiment of the present application, stored number threshold value is also provided with to the quantity of daily record data in tables of data, when there is daily record data quantity in data storage server and being less than the tables of data of stored number threshold value, just these tables of data are merged, the quantity of tables of data can be reduced like this, need the tables of data quantity of searching when saving user's inquiry log data, thus save query time.
Further, in the embodiment of the present application when the daily record set of back-up storage, tables of data due to each daily record set is independent data list file set of preserving, therefore can data list file in the data list file set that directly replication log set is corresponding be stored in back-up storage server, the time required for backup can be saved.In addition, when data table memory file, data list file can be stored into and currently on back-up storage server not perform in the disk of writing task, the speed that data list file stores can be improved like this, save the time required for backup further.
Further, each back-up storage server of each data list file set of storage same daily record set can be made in the embodiment of the present application to be positioned at different switch when determining backup server under, each data list file set of same daily record set can be avoided like this to be stored in the storage server under same switch and this switch of causing damage after the problem that cannot inquire about of daily record data in this daily record set.
Further, in the embodiment of the present application, daily record set is provided with to the copy number threshold value of data list file, when the copy number of the data list file of daily record set is greater than default copy number threshold value, with regard to the data list file of Delete superfluous, when the copy number of data list file is less than copy number threshold value, just copy data list file again, can ensure that the copy number of the data list file of daily record set in storage server remains stable like this, thus ensure that when not taking too much memory space daily record data can not be lost.
Further, in the embodiment of the present application when inquiry has completed the target journaling data of record, because daily record set correspondence each on storage server generates different tables of data, so just first can determine the goal set at target journaling place, this target journaling data are searched again in the tables of data of inquiring about this target journaling set on storage server, and do not need to search all tables of data on inquiry storage server, thus improve the speed of log query, reduce the time of delay of inquiry.
Certainly, the arbitrary product implementing the application might not need to reach above-described all advantages simultaneously.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present application or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, the accompanying drawing that the following describes is only some embodiments recorded in the application, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the flow chart of the embodiment of the method 1 of the application's logdata record;
The flow chart of the execution mode 1 of step 101 in Fig. 2 the application embodiment of the method 1;
The flow chart of the execution mode 2 of step 101 in Fig. 3 the application embodiment of the method 1;
Fig. 4 be the application's embodiment of the method 1 can the flow chart of Additional embodiments 1;
Fig. 5 be the application's embodiment of the method 1 can the flow chart of Additional embodiments 2;
Fig. 6 is the flow chart of the embodiment of the method 2 of the application's logdata record;
Fig. 7 is the flow chart of the embodiment of the method 3 of the application's logdata record;
Fig. 8 is the flow chart of the embodiment of the method 4 of the application's logdata record;
Fig. 9 is the flow chart of the embodiment of the method 1 of the application's inquiry log data;
Figure 10 is the flow chart of the method scene embodiment of the application's logdata record;
Figure 11 is the flow chart of the method scene embodiment of the application's inquiry log data;
Figure 12 is the structure chart of the log services apparatus embodiment 1 of the application's logdata record;
Figure 13 is the structure chart that in the application's log services apparatus embodiment 1, daily record set forms the embodiment 1 of module 1201;
Figure 14 is the structure chart that in the application's log services apparatus embodiment 1, daily record set forms the embodiment 2 of module 1201;
Figure 15 is the structure chart of the log services apparatus embodiment 2 of the application's logdata record;
Figure 16 is the another structure chart of the log services apparatus embodiment 2 of the application's logdata record;
Figure 17 is the structure chart of the log services apparatus embodiment 3 of the application's logdata record;
Figure 18 is the structure chart of the log services apparatus embodiment 4 of the application's logdata record;
Figure 19 is the structure chart of the log services apparatus embodiment 5 of the application's logdata record;
Figure 20 is the structure chart of the log services apparatus embodiment 6 of the application's logdata record;
Figure 21 is the structure chart of the system embodiment 1 of the application's logdata record;
Figure 22 is the structure chart of the system embodiment 1 of the application's inquiry log data.
Embodiment
The application's scheme is understood better in order to make those skilled in the art person, below in conjunction with the accompanying drawing in the embodiment of the present application, technical scheme in the embodiment of the present application is clearly and completely described, obviously, described embodiment is only some embodiments of the present application, instead of whole embodiments.Based on the embodiment in the application, those of ordinary skill in the art are not making the every other embodiment obtained under creative work prerequisite, all belong to the scope of the application's protection.
The application can be used in numerous general or special purpose computing system environment or configuration.Such as: personal computer, server computer, handheld device or portable set, laptop device, multicomputer system, system, set top box, programmable consumer-elcetronics devices, network PC, minicom, mainframe computer, the distributed computing environment (DCE) comprising above any system or equipment etc. based on microprocessor.
The application can describe in the general context of computer executable instructions, such as program module.Usually, program module comprises the routine, program, object, assembly, data structure etc. that perform particular task or realize particular abstract data type.Also can put into practice the application in distributed meter environment, in these distributed computing environment (DCE), be executed the task by the remote processing devices be connected by communication network.In a distributed computing environment, program module can be arranged in the local and remote computer-readable storage medium comprising memory device.
One of main thought of the application can comprise, adopt the mode comprising the daily record set of multiple daily record data, first by multiple daily record data composition daily record set needing record, and this daily record set is stored into completely data storage server generate tables of data, the more all daily record datas in daily record set all import tables of data after for this daily record set tables of data create index.All upgrade index owing to no longer needing often to record a daily record data, the method for this log can improve the speed of daily record data storage, and then saves the time stored.
See Fig. 1, show the flow chart of the embodiment of the method 1 of a kind of logdata record of the application, the present embodiment can comprise:
Step 101, the daily record data composition daily record set will got according to log acquisition condition.
The embodiment of the present application is when implementing, and the daily record data obtained is the daily record data generated.The source of daily record data is the operation system of network platform application cluster.A large number of users and operation system are carried out a large amount of mutual, operation system is just corresponding creates a large amount of interactive information, these interactive information are sent to log buffer server by operation system immediately, then daily record data corresponding to each interactive information is generated by log buffer server, and this daily record data of buffer memory, with etc. this daily record data to be recorded.In order to make the daily record data of buffer memory not easily damaged or loss, daily record data after log buffer server generates daily record data, can be cached in distributed file system by the present embodiment.Wherein, it is local that distributed file system can be placed in log buffer server, also can be placed in operation system.In addition, after daily record data is in the buffer acquired, usually need to delete from buffer memory, to keep having enough memory spaces to carry out the newly-generated daily record data of buffer memory in buffer memory.
It should be noted that, because daily record data produced by operation system and user interactions, and operation system is constantly carried out a large amount of mutual in time with a large number of users, so usual daily record data is all producing all the time.Therefore, when obtaining the daily record data of composition daily record set, log acquisition condition can for obtain according to time sequencing successively.Like this, in the daily record set once obtaining daily record data formation, daily record data in this daily record set is all generate and buffer memory upper once getting in this time period obtained, and there is in daily record data the time of daily record generation usually, so, in this daily record set, the time of all daily record datas all belongs in this time period, as can be seen here, obtain according to time sequencing the daily record set that daily record data formed successively, in same daily record set, the time of all daily record datas all belongs to section at the same time.
For the above-mentioned log acquisition condition obtaining daily record data according to time sequencing successively, present embodiments provide the embodiment of two kinds of steps 101.
See Fig. 2, show the flow chart of the execution mode 1 of step 101, step 101 can specifically comprise:
Step 201, from distributed file system, the daily record data generated after once forming the daily record set moment before acquisition;
Step 202 judges the acquisition number data bulk the threshold value whether number of described daily record data reaches default; If not, enter step 203, if so, enter step 204;
Step 203, waited for for the first stand-by period after return step 201;
Wherein, the first stand-by period was a default time value.To the setting of the first stand-by period, need the daily record data number that can get in guarantee first stand-by period can not exceed acquisition number data bulk threshold value, so it is all less that usual first stand-by period is arranged.
Step 204, by described daily record data composition daily record set; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
It should be noted that, after step 204 is complete, just complete the composition of a daily record set, define a daily record set.After step 204 completes, for this daily record set, step 102 can be entered and complete record to daily record data in daily record set; Meanwhile, the execution mode 1 that step 201 carries out Fig. 2 again can be returned, to carry out the acquisition of daily record data next time and to form next daily record set.
Be understandable that, in the execution mode 1 of the step 101 shown in Fig. 2, in the daily record set of each composition, the number of daily record data is identical, is all default acquisition number data bulk threshold value.
See Fig. 3, show the flow chart of the execution mode 2 of step 101, step 101 can specifically comprise:
Step 301, judge obtain the stand-by period whether reach default stand-by period threshold value, the described acquisition stand-by period is the time between current time and the front moment once forming daily record set; If not, enter step 302, if so, enter step 303;
Step 302, wait for described second stand-by period after return step 301;
Wherein, the second stand-by period was default time.When arranging the second stand-by period, the second stand-by period can directly be set to stand-by period threshold value, also can be set to be less than stand-by period threshold value.It is all very little that usual second stand-by period is arranged, and like this, is equivalent to step 301 and uninterruptedly carries out the judgement obtaining the stand-by period, once judged result is yes, then enter step 303.
Step 303, from distributed file system, the daily record data generated after once forming the daily record set moment before acquisition, and described daily record data is formed a daily record set; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
It should be noted that, after step 303 is complete, just complete the composition of a daily record set, define a daily record set.After step 303 completes, for this daily record set, step 102 can be entered and complete record to daily record data in daily record set; Meanwhile, the execution mode 2 that step 301 carries out Fig. 3 again can be returned, to carry out the acquisition of daily record data next time and to form next daily record set.
Be understandable that, in the execution mode 2 of the step 101 shown in Fig. 3, in the daily record set of each composition, the time of place time period time of daily record data is all equal, is all default stand-by period threshold value.
Then return Fig. 1, after step 101 is finished, enter step 102.
Step 102, determine the data storage server storing described daily record set.
Owing to needing to store a large amount of user journal data, so be generally used for, in the log service system of log, all there is multiple storage server, and each storage server have multiple disk daily record data is stored.Therefore, in storing daily record set before daily record data, need first from all storage servers, to determine the storage server storing this daily record set, as the data storage server of this daily record set.Wherein, under normal circumstances, a data storage server is only determined for a daily record set, the efficiency of storing daily record set can be improved like this.As needed, daily record set is stored on multiple storage server to avoid daily record data wherein to lose, the mode copying the daily record set be stored on data storage server then can be adopted to back up, the specific implementation of backup describes in detail in embodiment below, does not repeat them here.
In addition, balance is accessed in order to make the disk I/O pressure energy of each storage server in system, avoid read-write task to focus on a disk to perform, step 102 preferably determines the mode of data storage server, can be: first in all storage servers, search the current disk not performing writing task, then in the disk found, determine that the maximum disk of residual memory space is as the disk for storing daily record set, then this disk place storage server is defined as data storage server.Wherein, not be necessary for the disk that in the disk that finds, residual memory space is maximum for the disk of storing daily record set, also can be any one current disk not performing writing task, as long as the residual memory space of this disk can meet the requirement storing this daily record set.
Step 103, on described data storage server, generate the tables of data of described daily record set, and all daily record set in described daily record set are imported described tables of data.
According to the explanation of above-mentioned steps 102, perform in order to avoid task focuses on a disk, storing daily record set is specifically as follows described daily record set to be stored into and currently in described data storage server does not perform in the disk of writing task, and this current disk not performing writing task is the disk for storing daily record set determined in step 102.
It should be noted that, tables of data is stored as to daily record set, specifically can comprise: all daily record datas of daily record set are sent to data storage server, data storage server is that this daily record set creates tables of data, and all daily record datas in daily record set is imported the storage of this tables of data.Wherein, usually create a tables of data for a daily record set, what be stored in actual storage in data storage server like this is exactly multiple tables of data, and the daily record data correspondence in each tables of data belongs to a time period; Tables of data place database preferably adopts mysql.Certainly, when the daily record data quantity in daily record set is very large, also can be that a daily record set creates multiple tables of data.In addition, in the process all daily record datas in daily record set being imported the storage of this tables of data, be not that daily record data creates the index of tables of data, do not specify major key (a kind of index of tables of data) yet, but after daily record data imports tables of data completely, enter step 104 again create index.
After data storage server receives daily record set, usually complete after daily record set storage completes, can return storage and complete information, this storage completes information, and to be storage server import described tables of data by all daily record datas in described daily record set and the information returned after creating index for described tables of data completing.If mistake has appearred in the storage of data storage server to daily record set, can not complete daily record set being stored of task, then data storage server would not return storage and complete information.In order to ensure that the daily record data in daily record set can be stored in data storage server completely, the execution mode shown in Fig. 4 can be adopted after step 103 is complete, comprises:
Step 401, judge whether described data storage server returns storage and complete information within the memory time of presetting; Described storage completes information, and to be described storage server import described tables of data by all daily record datas in described daily record set and the information returned after creating index for described tables of data completing: if not, enter step 402.
Step 402, never determined storing in the storage server of described daily record set the data storage server redefined for storing described daily record set, and returned step 103.
Step 403, periodically delete the daily record set not completing storage in described data storage server according to the erasing time preset.
It should be noted that, the step 403 of deleting tables of data is not just must carry out after step 401 or 402 complete, and neither all will perform when each storing daily record set by data storage server.Because daily record data constantly produces in time, so system constantly stores daily record data, so system needs to circulate according to the garbage-cleaning cycle of presetting to perform step 403 usually, so just can avoid that each storage server of system exists too much dirty data table, thus expand available residual memory space.
By the execution mode shown in Fig. 4, data storage server can be avoided to occur mistake and in the daily record set that causes daily record data can not be stored as tables of data completely.
Then the explanation to step 103 is returned.
In the present embodiment, if adopt the execution mode 2 shown in Fig. 3 to form daily record set, then the time period that each daily record set is corresponding is all same time interval.Like this, the daily record data number just obtained in different time sections is general not identical.Therefore, the daily record data number that the daily record set formed like this may comprise is less, and then causes the tables of data generated in step 103 also less.When the small data table in data storage server is too much time, can a lot of memory space of additional waste.In order to avoid this problem, this enforcement after step 103, can enter the execution mode shown in Fig. 5, comprising:
Step 501, judge whether described data storage server exists the tables of data that daily record data quantity is less than default stored number threshold value according to the merging time cycle property preset; If so, step 502 is entered.
Wherein, step 501 according to the execution of a default merging time periodic cycle, can constantly can monitor on each data storage server whether there is comparatively small data table like this.
Step 502, described daily record data quantity is less than stored number threshold value tables of data merge.
By the execution mode shown in Fig. 5, decrease the quantity of tables of data, not only can avoid a lot of memory space of additional waste, need the tables of data quantity of searching when can also save user's inquiry log data, thus save query time.
Then return Fig. 1, after step 103 is complete, enter step 104.
Step 104, after all daily record datas of described daily record set all import described tables of data, be described daily record set tables of data create index.
For daily record data table creates index, be carry out after all daily record datas of daily record set import tables of data all completely and complete the storage on data storage server.Wherein, index is daily record collective data table and creating, and also, on each storage server, each tables of data has corresponding index.
In the embodiment of the present application, daily record set is formed by gathering the mode carrying out storing by obtaining daily record data because adopt, first can generate the tables of data of this daily record set and daily record datas all in daily record set are all imported on data storage server on this data storage server, be that this daily record collective data table creates index again after whole importing, doing so avoids the index often recording a daily record data and all need Update log memory device, the storage speed of log recording is accelerated, thus prevent loss of data and shorten user inquiry time of delay.
Further, in the embodiment of the present application when storing daily record set, daily record set can also be stored into and currently in data storage server not perform in the disk of writing task, like this, disk does not perform other writing tasks when storing daily record set, and then storage speed during raising log data.
Further, also storing daily record set is completed to data storage server in the embodiment of the present application and be also provided with threshold value memory time, when data storage server does not also complete the store tasks to daily record set through memory time, data storage server will be redefined to store this daily record set, data storage server can be avoided like this to occur mistake and in the daily record set that causes daily record data can not be stored as tables of data completely.In addition, on the basis that threshold value memory time is set, erasing time can also be provided with periodically to delete the daily record set not completing storage, the junk data in data storage server can be reduced like this, increase free space, improve utilization ratio.
Further, in the embodiment of the present application, stored number threshold value is also provided with to the quantity of daily record data in daily record set, when there is daily record data quantity in data storage server and being less than the tables of data of stored number threshold value, just these tables of data are merged, the quantity of tables of data can be reduced like this, need the tables of data quantity of searching when saving user's inquiry log data, thus save query time.
See Fig. 6, show the flow chart of a kind of logdata record embodiment of the method 2 of the application, the present embodiment can also comprise the following steps:
Step 601, the daily record data composition daily record set will got according to log acquisition condition; Described log acquisition condition is obtain daily record according to the time sequencing of daily record data, and in described daily record set, the time of all daily record datas belongs to the same time period;
Step 602, the daily record data in described daily record set to be classified according to type of service, form at least one daily record subclass;
When forming daily record subclass, different business models can be divided into so that the daily record data of same type of service is formed a daily record subclass by type, also miscellaneous service class can be enclosed, under each scope of business, there is one or more type of service, like this, daily record data type of service being belonged to the same scope of business forms a daily record subclass.
Step 603, determine the data storage server storing described daily record set;
Step 604, described data storage server is stored in described daily record set;
Step 605, according to the daily record data in daily record subclass each in described daily record set, the corresponding subdatasheet generating each daily record subclass;
The daily record set be sent on data storage server comprises at least one daily record subclass, now, for each daily record subclass creates a subdatasheet, and is imported to by the daily record data in each daily record subclass in corresponding subdatasheet and goes.
Step 606, the subdatasheet of daily record subclass all in described daily record set is combined into the tables of data of described daily record set;
After all daily record subclass of daily record set are completed and import subdatasheet, more each subdatasheet is combined into the tables of data of corresponding daily record set.
Step 607, after all daily record datas of described daily record set all import described tables of data, be described daily record set tables of data create index.
When creating index, can be first that each subdatasheet creates subindex, and then by the index of each subindex composition tables of data.
The difference of the embodiment 1 shown in the present embodiment and Fig. 1 is, daily record data is divided in different daily record set and stores by the present embodiment time be actually according to daily record data, then in daily record set, again daily record data is divided into different daily record subclass according to type of service.Certainly, also the daily record data got in the same time period directly can be divided into multiple type of service according to type of service.Like this, when can make user's inquiry log, fast finding is to the tables of data of the daily record set at target journaling place and subdatasheet, thus the tables of data quantity of searching required for reducing, improve inquiry velocity.
See Fig. 7, show the flow chart of a kind of logdata record embodiment of the method 3 of the application, in the present embodiment, execute previous embodiment 1 institute in steps after, the present embodiment can also comprise the following steps:
Step 701, the tables of data of described daily record set is stored as independently data list file set;
In the embodiment of the present application, tables of data is carry out storing with the form of data list file on storage server.For the ease of backing up daily record set, the data list file that each daily record collective data table is formed is independently, and also, the data in each data list file are same daily record set.It should be noted that, the tables of data of each daily record set can be one, also can be multiple, further, each tables of data at least forms a data list file, so, the tables of data of each daily record set generally comprises multiple data list file, so corresponding to each daily record is integrated on the whole is an independently data list file set, and a data list file set is a backup copies of this daily record set.Wherein, the concrete quantity of each tables of data is generally determined according to used database.Independently data list file is stored as separately in order to realize each tables of data, the present embodiment preferably adopts mysql database, and under MyISAM data engine in mysql database, each tables of data is at least stored as three data list files, three data list files storing daily record data (data file), index (MYI file) and list structure (FRM file) respectively, now, corresponding three data list files of tables of data, when daily record set comprises a tables of data, the data list file set of this daily record set comprises three data list files.
Step 702, copy the data list file of data list file set corresponding to described daily record set;
Because each daily record set is stored in independently data list file set, so when Backup Data, no longer need first to search on storage server daily record data corresponding to daily record set to copy corresponding statement again and go to back up, but can directly the data list file stored in the data list file set of daily record set be copied.
Step 703, never store described data list file storage server in determine storing described in copy the back-up storage server of the data list file obtained;
When determining the back-up storage server of data table memory file, if all data list files in the data list file set of a daily record set are all stored on same storage server, then this storage server just all can be lost once all data of this daily record set of fault.In order to avoid this problem, determined back-up storage server can be the current storage server not storing the data list file of this daily record set.And when backing up first a daily record set, the data list file of this daily record set is only stored in its data storage server, so now determined back-up storage server only needs different from data storage server.
Further, if all data list files in the data list file set of a daily record set are all stored on the storage server under same switch, then this switch is once all data of this daily record set of fault also all can be lost.In order to avoid this problem, the switch belonging to determined back-up storage server can be that connected all storage servers all do not store this daily record intersection of sets and change planes.Thus, in order to avoid the above-mentioned daily record collective data that causes loses problem, step 703 can specifically comprise: change planes be defined as target switch by all not storing described data list file intersection of sets in connected storage server; The back-up storage server copying the described data list file obtained described in storing is determined from the storage server that described target switch connects.
In addition, balance is accessed in order to make the disk I/O pressure energy of each storage server in system, avoid task to focus on a disk to perform, step 703 preferably determines the mode of back-up storage server, can be: first never store in the storage server of described data list file set and search the current disk not performing writing task, then in the disk found, determine that the maximum disk of residual memory space is as the disk of the data list file set obtained for storage replication, is then defined as back-up storage server by this disk place storage server.Wherein, disk for backup log set can be the disk that in the disk found, residual memory space is maximum, the memory space of each disk can be made so on average to be used, avoid data centralization to be stored into the problem of the loss of data risk increase that part disk causes.In addition, not be necessary for the disk that in the disk that finds, residual memory space is maximum for the disk of backup log set, also can be any one current disk not performing read-write task, as long as the residual memory space of this disk can meet the requirement storing this data list file.
Be understandable that, for a daily record set, the difference of data storage server and back-up storage server is only, data storage server has needed the record to this daily record set, is the storage server first storing this daily record set in system.And after backup completes, both store this daily record set with the form of data list file, in the backup again after this performed or inquiry, each back-up storage server can be considered a data storage server.
Step 704, the described data list file obtained that copies is stored into that described back-up storage server is current not to be performed in the disk of writing task one to one.
According to the explanation of above-mentioned steps 703, data list file is stored into and does not currently perform in the disk of writing task, task can be avoided to focus on a disk and perform.In addition, in order to avoid data list file is stored in the risk of the loss full backup that a back-up storage server produces, backup file is stored in back-up storage server by the present embodiment one to one, each part data list file of same daily record set is made to be stored on different back-up storage servers, further, can also be stored on the back-up storage server under different switch by each part data list file set of same tables of data, the problem that maybe cannot read all is lost in the backup that can prevent exchange fault from causing like this.
The difference of the embodiment 1 shown in the present embodiment and Fig. 1 is, the present embodiment backs up the daily record data recorded on the basis of embodiment 1, daily record data is made to be stored in multiple different storage server, daily record data can be avoided like this to lose, ensure the fail safe of the daily record data be recorded.Further, due to daily record set is stored as independently data list file set, so can be stored on other storage servers by the data list file directly in the set of copy data list file when backing up, and by searching the rear formation sql statement complicated backup procedure that performs of copy-statement again, the time of backup log data need not be saved.
See Fig. 8, show the flow chart of a kind of logdata record embodiment of the method 4 of the application, in the present embodiment, execute previous embodiment 2 institute in steps after, the present embodiment can also comprise the following steps:
Step 801, monitor the copy number of the data list file of described daily record set;
Wherein, the copy number of the data list file of monitoring is the quantity after all data list files of this daily record set in system back up.For a daily record set, after can having backed up from this daily record set the monitoring of the data list file quantity in the data list file set of its correspondence, also namely step 704 starts afterwards.
Step 802, the magnitude relationship judging between the copy number of described data list file and default copy number threshold values;
If the copy number of the described data list file of step 803 is greater than described copy number threshold value, then the data list file of difference quantities is deleted; Described difference quantities is the difference between the copy number of described data list file and described copy number threshold value;
If the copy number of the described data list file of step 804 is less than described copy number threshold value, then return step 702.
It should be noted that, when the copy number of data list file is less than described copy number threshold value, returns step 702 and will copy the quantity obtaining data list file and can equal described difference quantities.
The difference of the embodiment of the method 3 shown in the present embodiment and Fig. 7 is, the present embodiment is the quantity of Monitoring Data list file after daily record set has been backed up, and regulate when the quantity of data list file set is not equal to copy number threshold value, can ensure that the backup quantity of daily record set in storage server remains stable like this, thus ensure that when not taking too much memory space all daily record datas can not be lost.
See Fig. 9, show the flow chart of a kind of inquiry log data method embodiment 1 of the application, can comprise in the present embodiment:
Step 901, by aforementioned either method storing daily record data, in described daily record set, the time of all daily record datas all belongs to the same time period;
In the present embodiment, the acquisition of daily record set needs to obtain according to the time sequencing of daily record, and each like this daily record set corresponds to a time period, so that can according to the daily record set at logging time determination daily record place when inquiring about.
Step 902, by aforementioned either method backup log data;
It should be noted that, step 902 is not a step that must perform, and is yet, as long as the target journaling data of inquiring about are recorded, be stored on data storage server, just can inquire about, and be now do not perform step 902 to have carried out data backup.But the daily record under normal circumstances, inquired about is all the daily record data backed up.
Step 903, record between described daily record set and described data storage server, between described daily record set and described back-up storage server and described daily record set and the corresponding relation between the time period;
It should be noted that, step 901 ~ 903 are the steps performed in advance, all will perform when being not and inquiring about at every turn.Further, under normal circumstances, the target journaling inquired about all has been recorded and has backed up, so, when each query aim daily record, perform from step 904.
Step 904, reception daily record data inquiry request;
In the present embodiment, comprise the information of the target journaling will inquired about to some extent in daily record data inquiry request, this information comprises the time, type of service, user ID mark etc. of daily record.In addition, in the present embodiment, can be embodied as by inquiry proxy server proxy the service that user provides log query.
Step 905, according to time period at the time place of target journaling data in described inquiry request and described daily record set and the corresponding relation between the time period, determine the target journaling set at described target journaling data place;
The time period at target journaling place can be one, also can be multiple, and daily record set corresponding to each time period is multiple usually, so the target journaling set determined has multiple.And target journaling belongs to one of them goal set usually.
Step 906, according to the corresponding relation between described daily record set and data storage server and/or the corresponding relation between described daily record set and back-up storage server, from the data storage server storing described target journaling collective data list file and back-up storage server, determine the inquiry storage server inquiring about described target journaling;
Because the target journaling set determined has multiple, so for each target journaling set, an inquiry storage server all to be determined.Because each target journaling set has data list file on multiple storage server, therefore, for a target journaling set, perform in order to avoid task focuses on a disk, first can determine the disk that in the disk at this target journaling collective data list file place, current loading condition is minimum, then the storage server at this disk place is defined as querying server, can realize like this disperseing disk task on each storage server thus the tasks carrying efficiency of raising disk, shorten query time.Wherein, when selecting inquiry storage server according to the load condition of tables of data place disk, inquiry storage server can be chosen to be by CPU is fastest or that internal memory is maximum server, also server minimum for disk current task throughput can be chosen to be inquiry storage server.
In addition, if when query aim daily record, this target journaling also backs up, then data storage server can be defined as inquiring about storage server.
Step 907, on described inquiry storage server, search the tables of data of described target journaling set, and in the tables of data found, search described target journaling data;
Inquiry proxy server proxy goes target journaling data searched by inquiry storage server.In the present embodiment, search and preferably adopt sql statement.
Step 908, the target journaling data merging output that will find.
The all target journalings meeting the information of target journaling found from each inquiry storage server are merged and exports to user.
Need in addition illustrate, if if the daily record set of each time period divide into daily record subclass according to type of service or the scope of business, then directly can determine the target journaling subclass at target journaling place when searching in target journaling set, can search in the subdatasheet of target journaling subclass only in target journaling collective data table like this.
By the technical scheme of the present embodiment, because daily record set correspondence each on storage server generates different tables of data, so just first can determine the goal set at target journaling place, this target journaling data are searched again in the tables of data of inquiring about this target journaling set on storage server, and do not need to search all tables of data on inquiry storage server, thus improve the speed of log query, reduce the time of delay of inquiry.In addition, because target journaling is integrated into the backup multiple storage server having tables of data, be defined as inquiring about storage server so the minimum storage server of this target journaling set place disk present load can be chosen, the speed of searching target journaling data in tables of data can be improved like this, shorten the time needed for inquiry.
It should be noted that, above-mentioned data storage server, back-up storage server and inquiry storage server all correspond to a daily record set, and just correspond to the current performed task of this daily record set.In actual log record, the data storage server of a daily record set both can be the data storage server of other daily record set, also can be back-up storage server or the querying server of other daily record set.And, data storage server is just for the recording process of daily record data, back-up storage server is just for the backup procedure of daily record data, inquiry storage server is just for the query script of daily record data, and in fact, when system is not executed the task to the daily record set of having recorded, system only can determine the storage server storing this daily record set, and does not distinguish data storage server, back-up storage server and inquiry storage server.
Below with concrete application scenarios, describe the execution mode of the application's logdata record method and querying method in detail.
See Figure 10, show the flow chart of the scene embodiment of the application's logdata record method.In the scene of the present embodiment, the application's logdata record method is applied to a log service system.This system comprises log server master and storage server worker, wherein, and a corresponding multiple worker of master.In the scene of the present embodiment, log server master will obtain daily record data successively according to time sequencing and form a daily record set from distributed file system, again daily record set is stored into a storage server worker according to record, then daily record data is backuped on other storage servers worker.The present embodiment specifically can comprise:
Step 1001, generates daily record data: log buffer server receives the interactive information that operation system sends, and generates daily record data.
Step 1002, log cache data: the daily record data generated sends to distributed file system to carry out buffer memory by log buffer server; It is local that distributed file system can be positioned at log buffer server, also can be positioned at operation system, can also be positioned at log service system this locality.
Step 1003, log server obtain daily record data, composition daily record set: master obtains the daily record data generated from distributed file system, form a daily record set, and the moment that the moment once obtained before being defined as the time period of daily record set obtains to this, record this daily record set and the corresponding relation between the time period.It removes after being obtained by master by daily record data from distributed file system.If the daily record data quantity once obtained is excessive, daily record set can also be divided into multiple daily record subclass according to type of service, different subdatasheets is stored as according to the classification of daily record subclass during to store daily record set, during such inquiry log data, can according to logging time and type of service determination subdatasheet, search in subdatasheet again, and then reduce hunting zone, save the inquiry used time.
Step 1004, determine the data storage server of this daily record set: master selects one to be used for storing the daily record set just obtained from the worker corresponding with it.When selecting, first the disks not performing writing task all on each worker is searched, then in the disk found, determining that maximum one of residual memory space is for storing this daily record set, then this disk place worker being defined as the data storage server of this daily record set.If once obtain the daily record set defining multiple different service types, also can will determine that for each daily record set a worker is to complete subsequent step.
Step 1005, generate tables of data daily record set being imported: daily record set is sent to the worker that step 1004 is determined by master, an establishment tables of data on the disk that this worker determines in step 1004, and this daily record set is imported in this tables of data.If daily record set defines multiple daily record subclass, then need daily record subclass to store in all respective subdatasheet, then subdatasheet is formed a tables of data.It should be noted that, while step 1005 is carried out, perform step 1007.
Step 1006, all import after be that to create index: master be this tables of data establishment index on the worker storing this tables of data for the tables of data of this daily record set, and the corresponding relation between log set and this worker.
Step 1007, judge whether data storage server returns storage and complete message within default memory time, enter step 1008 if not, if enter after daily record set sends to this worker by step 1009:master, the record that wait-receiving mode worker returns completes information.This worker, after completing all daily record datas storages in this daily record set, just sends record and completes information to master.If master does not also receive record and completes message in Preset Time, then think that this worker breaks down, do not complete the logger task of this daily record set.
Step 1008, redefine the data storage server of this daily record set, return step 1005:master and do not receive record when completing information, a worker will be redefined to store this daily record set in the worker not also being confirmed as data storage server.
Step 1009, daily record set is stored as independently data list file set: the tables of data of this daily record set stores with the form of independently data list file set by the worker of this tables of data of master control store, thus forms a backup copies of this daily record set.Under the MyISAM data engine of mysql, a tables of data is stored as three data list files.
Step 1010, copy the data list file set of this tables of data: this data list file is integrated on the worker of its place and copies by master, thus forms new daily record set backup copies.If this tables of data was backed up, then the data list file set that master can select this daily record to be integrated on any one worker copied.
Step 1011, determine back-up storage server: master does not store the worker of this data list file set from current the worker determining the data list file set that storage replication obtains.When determining back-up storage server, the following condition of worker demand fulfillment of data table memory file: there is the current disk not performing writing task, the data list file set obtained for storage replication that in these disks, remaining space is maximum; The quantity of worker is identical with the quantity copying the data list file set obtained; Be positioned under different switches between each worker.The described worker for data table memory file comprises the worker of the backup copies that the worker of this daily record set of first record and storage replication obtain, and also, comprises data storage server and back-up storage server.
Step 1010 is copied the data list file set obtained and is stored into designated disk in the worker that step 1011 determines by the data list file set that step 1012, storage replication obtain: master, this designated disk is currently do not perform maximum one of remaining space in the disk of writing task, a data list file set of each worker storing daily record set.After data list file set to be sent on corresponding worker by master, then what this worker of wait-receiving mode returned storage backup task completes information.If master does not receive this information within the memory time of presetting, then again other worker can be defined as back-up storage server, then store.After worker has returned information, master has recorded the corresponding relation between the worker of the data list file set of this daily record set and this daily record set of each storage one by one.
Step 1013, monitor the copy number of the data list file of this daily record set.
Step 1014, judge whether the copy number of this data list file equals default copy number threshold value, if the copy number of data list file is greater than copy number threshold value, enter step 1015, if the copy number of data list file is less than copy number threshold value, return step 1010: preset copy number threshold value and be preferably 3.
The data list file of step 1015, deletion surplus.
It should be noted that no matter enter step 1015 or return step 1010 after step 1015, deletion or the quantity that copies the data list file obtained be difference between the copy number of current data list file and copy number threshold value.
Step 1016, judge whether each storage server exists small data table, if yes then enter step 1017, if not, enter step 1018: this small data table refers to that wherein daily record data quantity is less than the tables of data of default stored number threshold value.Because system needs record and the storage of constantly carrying out daily record data, so, this step 1016 can perform according to a default merging time cycle property whether to store small data table in monitoring system, instead of must perform after daily record set completes backup again.
Step 1017, small data table is merged into a tables of data: merging is merged by the small data table on same worker.
Step 1018, delete the daily record set not completing storage in each storage server: this step is the dirty data table produced on worker when being no for judged result in step 1006.Equally, because system needs record and the storage of constantly carrying out daily record data, so this step 1015 also periodically can perform whether there is dirty data table in monitoring system according to a default erasing time, and is not must perform after daily record set completes backup again.
By this scene embodiment, the fast recording to a large amount of daily record data and quick backup can be realized, the IO pressure of each storage server disk in balance sysmte, and can prevent daily record data from losing, improve the utilance of system memory space, also contribute to realizing the quick search of user to daily record data.
See Figure 11, show the flow chart of the scene embodiment of the application's inquiry log data method.In the scene of the present embodiment, the application's inquiry log data method can be applied to a log query system, this system comprise inquiry proxy server proxy and embodiment illustrated in fig. 10 in log server master and storage server worker.In the scene of the present embodiment, the record of daily record data in log service system and backup are all undertaken by the embodiment shown in Figure 10.The present embodiment can comprise:
Step 1101, reception daily record data inquiry request: user, when inquiry log data, requires that user must input the time of target journaling.If daily record set be divide into daily record subclass according to type of service when log recording, then can also require that when inquiring about user inputs the type of service of target journaling.
Step 1102, determine target journaling set: proxy is according to the time of target journaling in daily record data inquiry request, determine the time period at target journaling place, then search the daily record set and the corresponding relation of time period of recording in master, determine target journaling set.If user have input type of service, the corresponding relation of daily record subclass and the type of service recorded in master can also be searched, determine target journaling subclass.Target journaling set and target journaling subclass generally are multiple, and target journaling only belongs to one of them.
Step 1103, determine inquire about storage server: for each target journaling set, proxy is first by the corresponding relation of the daily record set of recording in master and worker, search the worker of this target journaling collective data list file of all storages, and then from the worker found, determine the disk storing this target journaling collective data list file, again using minimum for load condition in the disk a determined disk as inquiry disk used, finally the worker at the disk place at inquiry place is defined as inquiring about storage server.
Step 1104, in the tables of data of target journaling set, search target journaling data: proxy is on the inquiry storage server worker of each target journaling set, search the tables of data of each target journaling set according to the data table name of each target journaling set, and in the tables of data respectively found, search the target journaling meeting log query request and require.Log query request can comprise as information such as user ID marks, can find corresponding target journaling according to these information according to index in tables of data.If determine target journaling subclass according to type of service, then can only search the subdatasheet that target journaling subclass is corresponding is upper.
Step 1105, the target journaling that finds is merged and exports: due to usually can in multiple target journaling set search to target journaling; therefore, before output Query Result, proxy needs the target journaling searched on each inquiry storage server worker to merge.
By this scene embodiment, the time needed for inquiry can be shortened, the IO pressure balancing disk on each storage server in system can also be made, improve IO performance.
Corresponding to embodiment of the method above, the embodiment of the present application also provides a kind of log server of logdata record.See Figure 12, show a kind of structural representation of log services apparatus embodiment 1 of logdata record, can comprise:
Daily record set forms module 1201, for the daily record data composition daily record set will got according to log acquisition condition;
Data storage server determination module 1202, for determining the data storage server storing described daily record set;
Wherein, described data storage server determination module 1202, specifically for determine storing described daily record set data storage server in the current disk not performing writing task;
Tables of data generation module 1203, for generating the tables of data of described daily record set on described data storage server;
Daily record insert module 1204, for importing described tables of data by all daily record datas in described daily record set;
Index creation module 1205, after all importing described tables of data at all daily record datas of described daily record set, for the tables of data of described daily record set creates index.
With reference to Figure 13, described daily record set formation module 1201 can specifically comprise in actual applications:
First daily record data obtains submodule 1301, for from distributed file system, and the daily record data generated after once forming the daily record set moment before acquisition;
Obtain quantity and judge submodule 1302, for judging the acquisition number data bulk the threshold value whether number of described daily record data reaches default;
Trigger and obtain submodule 1303, for when described acquisition quantity judges that the judged result of submodule 1302 is no, after waiting for for the first stand-by period, trigger described daily record data acquisition module;
First daily record combination submodule 1304, for when described acquisition quantity judges the judged result of submodule 1302 for being, by described daily record data composition daily record set; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
With reference to Figure 14, described daily record set formation module 1201 also can specifically comprise in actual applications:
Acquisition time judges submodule 1401, for judging whether reach default stand-by period threshold value, the described acquisition stand-by period is the time between current time and the front moment once forming daily record set if obtaining the stand-by period;
Triggered time judges submodule 1402, for judging that the judged result of submodule 1401 is no at described acquisition time, triggering acquisition time and judging submodule 1401 after waiting for described second stand-by period;
Second daily record data acquisition submodule 1403, for when described acquisition time judges the judged result of submodule 1401 for being, from distributed file system, the user journal data generated after once forming the daily record set moment before acquisition;
Second daily record combination submodule 1404, for forming a daily record set by described daily record data; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
The log server of the present embodiment is adopted to carry out record to daily record data, can avoid because often record daily record data all need the index of Update log memory device and the slow phenomenon of log recording that causes, the writing speed of log recording is accelerated, thus make the network platform can record a large amount of daily record datas produced in the short time in time, prevent loss of data, thus and shorten user inquiry time of delay.
See Figure 15, show a kind of structural representation of log services apparatus embodiment 2 of logdata record, except the structure in the log services apparatus embodiment 1 shown in Figure 12, described log server can also comprise:
Tables of data completes judge module 1501, for judging whether described data storage server returns storage and complete information within the memory time of presetting; Described storage completes information, and to be described storage server import described tables of data by all daily record datas in described daily record set and the information returned after creating index for described tables of data completing;
Data storage redefines module 1502, when being no for completing the judged result of judge module 1501 in described tables of data, never determined storing in the storage server of described daily record set the data storage server redefined for storing described daily record set;
Trigger memory module 1503, for redefining module 1502 startup in described data storage, trigger daily record set memory module 1203.
See Figure 16, log services apparatus embodiment 2 is except all structures shown in Figure 15, and described log server also comprises:
Delete collection modules 1601, for periodically deleting the daily record set not completing storage in described data storage server according to the erasing time preset.
Adopt the log server of the present embodiment to carry out log data, data storage server can be avoided to occur mistake and in the daily record set that causes daily record data can not be stored as tables of data completely.In addition, the junk data in data storage server can also be reduced, increase free space, improve utilization ratio.
See Figure 17, show a kind of structural representation of log services apparatus embodiment 3 of logdata record, except the structure in the log services apparatus embodiment 1 shown in Figure 12, described log server can also comprise:
Data bulk judge module 1701, for judging whether described data storage server exists the tables of data that daily record data quantity is less than default stored number threshold value according to the merging time cycle property preset;
Set merges module 1702, and for when the judged result of described data bulk 1701 is for being, the tables of data described daily record data quantity being less than stored number threshold value merges.
Adopt the log server of the present embodiment to carry out log data, the quantity of tables of data can be reduced, need the tables of data quantity of searching when saving user's inquiry log data, thus save query time.
See Figure 18, show a kind of structural representation of log services apparatus embodiment 4 of logdata record, except the structure in the log services apparatus embodiment 1 shown in Figure 12, described log server can also comprise:
Daily record subclass forms module 1801, for being classified according to type of service by the daily record data in described daily record set, forms at least one daily record subclass;
Accordingly, described tables of data generation module 1204 comprises:
Subdatasheet generates submodule 1802, for according to the daily record data in daily record subclass each in described daily record set, and the corresponding subdatasheet generating each daily record subclass;
Subdatasheet combination submodule 1803, for being combined into the tables of data of described daily record set by the subdatasheet of described daily record subclass all in described daily record set.
Adopt the log server of the present embodiment to carry out log data, when can make user's inquiry log, fast finding is to the tables of data of the daily record set at target journaling place and subdatasheet, thus the tables of data quantity of searching required for reducing, raising inquiry velocity.
See Figure 19, show a kind of structural representation of log services apparatus embodiment 5 of logdata record, except the structure in the log services apparatus embodiment 1 shown in Figure 12, described log server can also comprise:
Backup generation module 1901, for being stored as independently data list file set by the tables of data of described daily record set;
Backup copy module 1902, for copying the data list file of data list file set corresponding to described daily record set;
Backup server determination module 1903, for never store described data list file storage server in determine storing described in copy the back-up storage server of the data list file obtained;
Wherein, described backup server determination module 1903, specifically can comprise: first determines submodule, for the switch all not storing described data list file in connected storage server is defined as target switch; Second determines submodule, for copying the back-up storage server of the described data list file obtained described in determining in the storage server that connects from described target switch storing.
By the described data list file obtained that copies, back-up storage module 1904, for being stored into that described back-up storage server is current not to be performed in the disk of writing task one to one.
Adopt the log server of the present embodiment to carry out log data, the speed that backup file stores can be improved, save the time required for backup, and the IO pressure of each disk in balance sysmte.
See Figure 20, show a kind of structural representation of log services apparatus embodiment 6 of logdata record, except the structure in the log services apparatus embodiment 4 shown in Figure 19, described log server can also comprise:
Tables of data monitoring modular 2001, for monitoring the copy number of the data list file of described daily record set;
Tables of data quantity judge module 2002, for judging the magnitude relationship between the copy number of data list file and default copy number threshold values;
Tables of data removing module 2003, when being greater than described copy number threshold value for the copy number that is described data list file in the judged result of described tables of data quantity judge module 2002, deletes the data list file of difference quantities; Described difference quantities is the difference between the copy number of described data list file and default copy number threshold value;
Trigger replication module 2004, when being less than described copy number threshold value for the copy number that is described data list file in the judged result of described tables of data quantity judge module 2002, trigger described tables of data replication module 1902.
Adopt the log server of the present embodiment to carry out log data, can ensure that the data list file quantity of daily record set in storage server remains stable, thus ensure that when not taking too much memory space daily record data can not be lost.
Corresponding to embodiment of the method above, the embodiment of the present application also provides a kind of system of logdata record.See Figure 21, show a kind of structural representation of system embodiment 1 of logdata record, can comprise: log server 2101 and storage server 2102; Described log server 2101 is the arbitrary log server shown in aforementioned Figure 12 ~ Figure 20; Described storage server 2102 comprises data storage server and/or back-up storage server.
The system of the present embodiment is adopted to carry out log data, can avoid because often record daily record data all need the index of Update log memory device and the slow phenomenon of log recording that causes, the writing speed of log recording is accelerated, thus make the network platform can record a large amount of daily record datas produced in the short time in time, prevent loss of data and shorten user inquiry time of delay, further, the storage speed of daily record data in daily record set can also be improved, avoid not storing completely or losing of daily record data, reduce the junk data in data storage server, increase free space, improve availability, save query time, BACKUP TIME, the IO pressure etc. of each disk in balance sysmte.
Corresponding to embodiment of the method above, the embodiment of the present application also provides a kind of system of inquiry log data.See Figure 22, show a kind of structural representation of system embodiment 1 of inquiry log data, can comprise:
Log storage unit 2201, for passing through the arbitrary log server storing daily record data shown in Figure 12 ~ Figure 18, in described daily record set, the time of all daily record datas all belongs to the same time period;
Log backup unit 2202, for passing through the log server backup log data shown in Figure 19 or Figure 20;
Corresponding relation record cell 2203, for recording between the corresponding relation between described daily record set and described data storage server, between described daily record set and described back-up storage server and described daily record set and the corresponding relation between the time period;
Inquiry request receiving element 2204, for receiving daily record data inquiry request;
Goal set determining unit 2205, for the time period at the time place according to target journaling in described inquiry request, determines the target journaling set at target journaling data place;
Querying server determining unit 2206, for according to the corresponding relation between described daily record set and data storage server and the corresponding relation between described daily record set and back-up storage server, from the data storage server storing described target journaling collective data list file and back-up storage server, determine the inquiry storage server inquiring about described target journaling;
Search unit 2207, for searching the data list file of described target journaling set on described inquiry storage server;
Target journaling searches unit 2208, for according to the tables of data index in the data list file of described target journaling set, in the tables of data of described target journaling collective data list file, searches described target journaling data;
Target journaling output module 2209, merges output for the target journaling data that will find.
Adopt the system of the present embodiment to carry out inquiry log data, the speed of log query can be improved, reduce the time of delay of inquiry, make the disk on storage server realize read and write abruption, disk I/O pressure etc. in balance sysmte.
It should be noted that, in this article, the such as relational terms of first and second grades and so on is only used for an entity or operation to separate with another entity or operating space, and not necessarily requires or imply the relation that there is any this reality between these entities or operation or sequentially.Term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thus make to comprise the process of a series of key element, method, article or equipment and not only comprise those key elements, but also comprise other key elements clearly do not listed, or also comprise by the intrinsic key element of this process, method, article or equipment.When not more restrictions, the key element limited by statement " comprising ... ", and be not precluded within process, method, article or the equipment comprising described key element and also there is other identical element.
For system embodiment, because it corresponds essentially to embodiment of the method, so relevant part illustrates see the part of embodiment of the method.System embodiment described above is only schematic, the wherein said unit illustrated as separating component or can may not be and physically separates, parts as unit display can be or may not be physical location, namely can be positioned at a place, or also can be distributed in multiple network element.Some or all of module wherein can be selected according to the actual needs to realize the object of the present embodiment scheme.Those of ordinary skill in the art, when not paying creative work, are namely appreciated that and implement.
The above is only the embodiment of the application; it should be pointed out that for those skilled in the art, under the prerequisite not departing from the application's principle; can also make some improvements and modifications, these improvements and modifications also should be considered as the protection range of the application.

Claims (25)

1. a method for logdata record, is characterized in that, comprising:
By the daily record data composition daily record set got according to log acquisition condition;
Determine the data storage server storing described daily record set;
Described data storage server generates the tables of data of described daily record set, and all daily record datas in described daily record set are imported described tables of data;
After all daily record datas of described daily record set all import described tables of data, for the tables of data of described daily record set creates index.
2. method according to claim 1, is characterized in that, the described daily record data composition daily record set will got according to log acquisition condition, comprising:
From distributed file system, the daily record data generated after once forming the daily record set moment before acquisition;
Judge the acquisition number data bulk the threshold value whether number of described daily record data reaches default;
If not, then waited for for the first stand-by period after return the described acquisition of described judgement before once form the daily record set moment after the step of daily record data that generates;
If so, then by described daily record data composition daily record set; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
3. method according to claim 1, is characterized in that, the described daily record data composition daily record set will got according to log acquisition condition, comprising:
Judge whether reach default stand-by period threshold value, the described acquisition stand-by period is the time between current time and the front moment once forming daily record set if obtaining the stand-by period;
If not, then wait for described second stand-by period after return and describedly judge whether obtain the stand-by period reaches the step of default stand-by period threshold value;
If so, then from distributed file system, the daily record data generated after once forming the daily record set moment before acquisition, and described daily record data is formed a daily record set; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
4. method according to claim 1, is characterized in that, described log acquisition condition is obtain daily record according to the time sequencing of daily record data, and in described daily record set, the time of all daily record datas belongs to the same time period;
Accordingly, after the daily record data got according to log acquisition condition is formed daily record set, also comprise: the daily record data in described daily record set is classified according to type of service, form at least one daily record subclass;
Accordingly, the described tables of data generating described daily record set on described data storage server, comprising:
According to the daily record data in daily record subclass each in described daily record set, the corresponding subdatasheet generating each daily record subclass;
The subdatasheet of daily record subclass all in described daily record set is combined into the tables of data of described daily record set.
5. method according to claim 1, is characterized in that, described in determine the data storage server storing described daily record set, be specially: determine storing the current disk not performing writing task in the data storage server of described daily record set.
6. method according to claim 1, is characterized in that, described on described data storage server, generate the tables of data of described daily record set after, also comprise:
Judge whether described data storage server returns storage and complete information within the memory time of presetting; Described storage completes information, and to be described storage server import described tables of data by all daily record datas in described daily record set and the information returned after creating index for described tables of data completing;
If not, then never determined storing in the storage server of described daily record set the data storage server redefined for storing described daily record set, and returned the described step described daily record set being stored into described data storage server.
7. method according to claim 6, is characterized in that, described method also comprises:
Periodically the daily record set not completing storage in described data storage server is deleted according to the erasing time preset.
8. method according to claim 1, is characterized in that, described on described data storage server, generate the tables of data of described daily record set after, also comprise:
Judge whether described data storage server exists according to the merging time cycle property preset the tables of data that daily record data quantity is less than default stored number threshold value; If so, then the tables of data described daily record data quantity being less than stored number threshold value merges.
9. method according to claim 1, is characterized in that, described is that the tables of data of described daily record set creates after index, also comprises:
The tables of data of described daily record set is stored as independently data list file set;
Copy the data list file in data list file set corresponding to described daily record set;
Never the back-up storage server determining copying the described data list file obtained described in storage is stored in the storage server of described data list file;
The described data list file obtained that copies is stored into that described back-up storage server is current not to be performed in the disk of writing task one to one.
10. method according to claim 9, is characterized in that, described never store in the storage server of described data list file determine storing described in copy the back-up storage server of the described data list file set obtained, comprising:
Change planes be defined as target switch by all not storing described data list file intersection of sets in connected storage server;
The back-up storage server copying the described data list file obtained described in storing is determined from the storage server that described target switch connects.
11. methods according to claim 9, is characterized in that, describedly copy after the data list file obtained is stored in described back-up storage server by described, also comprise:
Monitor the copy number of the data list file of described daily record set;
Judge the magnitude relationship between the copy number of described data list file and default copy number threshold values;
If the copy number of described data list file is greater than described copy number threshold value, then the data list file of difference quantities is deleted; Described difference quantities is the difference between the copy number of described data list file and described copy number threshold value;
If the copy number of described data list file is less than described copy number threshold value, then copy the step of the data list file of data list file set corresponding to described daily record set described in returning.
The method of 12. 1 kinds of inquiry log data, is characterized in that, by the method storing daily record data described in claim 1 ~ 11 any one, in described daily record set, the time of all daily record datas all belongs to the same time period; By the method backup log data described in claim 9,10 or 11; Record between described daily record set and described data storage server, between described daily record set and described back-up storage server and described daily record set and the corresponding relation between the time period;
Described method comprises:
Receive daily record data inquiry request;
According to time period at the time place of target journaling data in described inquiry request and described daily record set and the corresponding relation between the time period, determine the target journaling set at described target journaling data place;
According to the corresponding relation between described daily record set and data storage server and/or the corresponding relation between described daily record set and back-up storage server, from the data storage server storing described target journaling collective data list file and back-up storage server, determine the inquiry storage server inquiring about described target journaling;
Described inquiry storage server is searched the tables of data of described target journaling set, and in the tables of data found, searches described target journaling data;
The target journaling data found are merged and exports.
The log server of 13. 1 kinds of log data, is characterized in that, comprising:
Daily record set forms module, for the daily record data composition daily record set will got according to log acquisition condition;
Data storage server determination module, for determining the data storage server storing described daily record set;
Tables of data generation module, for generating the tables of data of described daily record set on described data storage server;
Daily record insert module, for importing described tables of data by all daily record datas in described daily record set;
Index creation module, after all importing described tables of data at all daily record datas of described daily record set, for the tables of data of described daily record set creates index.
14. log servers according to claim 13, is characterized in that, described daily record set forms module and comprises:
First daily record data obtains submodule, for from distributed file system, and the daily record data generated after once forming the daily record set moment before acquisition;
Obtain quantity and judge submodule, for judging the acquisition number data bulk the threshold value whether number of described daily record data reaches default;
Trigger and obtain submodule, for when described acquisition quantity judges that the judged result of submodule is no, after waiting for for the first stand-by period, trigger described daily record data acquisition module;
First daily record combination submodule, for when described acquisition quantity judges the judged result of submodule for being, by described daily record data composition daily record set; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
15. log servers according to claim 13, is characterized in that, described daily record set forms module and comprises:
Acquisition time judges submodule, for judging whether reach default stand-by period threshold value, the described acquisition stand-by period is the time between current time and the front moment once forming daily record set if obtaining the stand-by period;
Triggered time judges submodule, for judging that the judged result of submodule is no at described acquisition time, triggering described acquisition time and judging submodule after waiting for described second stand-by period;
Second daily record data acquisition submodule, for when described acquisition time judges the judged result of submodule for being, from distributed file system, the daily record data generated after once forming the daily record set moment before acquisition;
Second daily record combination submodule, for forming a daily record set by described daily record data; Time period corresponding to described daily record set once forms the moment of moment to this composition daily record set of daily record set before being.
16. log servers according to claim 13, is characterized in that, described log server also comprises:
Daily record subclass forms module, for being classified according to type of service by the daily record data in described daily record set, forms at least one daily record subclass;
Accordingly, described tables of data generation module comprises:
Subdatasheet generates submodule, for according to the daily record data in daily record subclass each in described daily record set, and the corresponding subdatasheet generating each daily record subclass;
Subdatasheet combination submodule, for being combined into the tables of data of described daily record set by the subdatasheet of described daily record subclass all in described daily record set.
17. log servers according to claim 13, is characterized in that, described data storage server determination module, specifically for determine storing described daily record set data storage server in the current disk not performing writing task.
18. log servers according to claim 13, is characterized in that, described log server also comprises:
Tables of data completes judge module, for judging whether described data storage server returns storage and complete information within the memory time of presetting; Described storage completes information, and to be described storage server import described tables of data by all daily record datas in described daily record set and the information returned after creating index for described tables of data completing;
Data storage redefines module, when being no for completing the judged result of judge module in described tables of data, never determined storing in the storage server of described daily record set the data storage server redefined for storing described daily record set;
Trigger memory module, for redefining module startup in described data storage, trigger daily record set memory module.
19. log servers according to claim 18, is characterized in that, described log server also comprises:
Delete collection modules, for periodically deleting the daily record set not completing storage in described data storage server according to the erasing time preset.
20. log servers according to claim 13, is characterized in that, described log server also comprises:
Data bulk judge module, for judging whether described data storage server exists the tables of data that daily record data quantity is less than default stored number threshold value according to the merging time cycle property preset;
Set merges module, and for when the judged result of described data bulk is for being, the tables of data described daily record data quantity being less than stored number threshold value merges.
21. log servers according to claim 13, is characterized in that, described log server also comprises:
Backup generation module, for being stored as independently data list file set by the tables of data of described daily record set;
Backup copy module, for copying the data list file of data list file set corresponding to described daily record set;
Backup server determination module, for never store described data list file storage server in determine storing described in copy the back-up storage server of the data list file obtained;
By the described data list file obtained that copies, back-up storage module, for being stored into that described back-up storage server is current not to be performed in the disk of writing task one to one.
22. log servers according to claim 21, is characterized in that, described backup server determination module, comprising:
First determines submodule, for the switch all not storing described data list file in connected storage server is defined as target switch;
Second determines submodule, for copying the back-up storage server of the described data list file obtained described in determining in the storage server that connects from described target switch storing.
23. log servers according to claim 21, is characterized in that, described log server also comprises:
Tables of data monitoring modular, for monitoring the copy number of the data list file of described daily record set;
Tables of data quantity judge module, for judging the magnitude relationship between the copy number of data list file and default copy number threshold values;
Tables of data removing module, when being greater than described copy number threshold value for the copy number that is described data list file in the judged result of described tables of data quantity judge module, deletes the data list file of difference quantities; Described difference quantities is the difference between the copy number of described data list file and default copy number threshold value;
Trigger replication module, when being less than described copy number threshold value for the copy number that is described data list file in the judged result of described tables of data quantity judge module, trigger described backup copy module.
The system of 24. 1 kinds of log data, is characterized in that, described system comprises log server and storage server; Described log server is the log server described in claim 13 ~ 23 any one; Described storage server comprises data storage server and/or back-up storage server.
The system of 25. 1 kinds of inquiry log data, is characterized in that, described system comprises:
Log storage unit, for passing through the log server storing daily record data described in claim 13 ~ 23 any one, in described daily record set, the time of all daily record datas all belongs to the same time period;
Log backup unit, for passing through the log server backup log data described in claim 21,22 or 23;
Corresponding relation record cell, for recording between described daily record set and described data storage server, between described daily record set and described back-up storage server and described daily record set and the corresponding relation between the time period;
Inquiry request receiving element, for receiving daily record data inquiry request;
Goal set determining unit, for the time period at the time place according to target journaling in described inquiry request, determines the target journaling set at target journaling data place;
Querying server determining unit, for according to the corresponding relation between described daily record set and data storage server and/or the corresponding relation between described daily record set and back-up storage server, from the data storage server storing described target journaling collective data list file and back-up storage server, determine the inquiry storage server inquiring about described target journaling;
Search unit, for searching the tables of data of described target journaling set on described inquiry storage server;
Target journaling searches unit, for searching described target journaling data in the tables of data found;
Target journaling output module, merges output for the target journaling data that will find.
CN201310306476.3A 2013-07-19 2013-07-19 A kind of method of logdata record, log server and system Active CN104301360B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201310306476.3A CN104301360B (en) 2013-07-19 2013-07-19 A kind of method of logdata record, log server and system
HK15102899.1A HK1202722A1 (en) 2013-07-19 2015-03-23 Method for log data recording, logging server and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310306476.3A CN104301360B (en) 2013-07-19 2013-07-19 A kind of method of logdata record, log server and system

Publications (2)

Publication Number Publication Date
CN104301360A true CN104301360A (en) 2015-01-21
CN104301360B CN104301360B (en) 2019-03-12

Family

ID=52320927

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310306476.3A Active CN104301360B (en) 2013-07-19 2013-07-19 A kind of method of logdata record, log server and system

Country Status (2)

Country Link
CN (1) CN104301360B (en)
HK (1) HK1202722A1 (en)

Cited By (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105260399A (en) * 2015-09-17 2016-01-20 西安未来国际信息股份有限公司 Method for acquiring and retrieving distributed log
CN105391772A (en) * 2015-10-16 2016-03-09 百度在线网络技术（北京）有限公司 Service request processing method, log processing method and service request processing device
CN105528454A (en) * 2015-12-25 2016-04-27 北京奇虎科技有限公司 Log treatment method and distributed cluster computing device
CN105989059A (en) * 2015-02-09 2016-10-05 阿里巴巴集团控股有限公司 Data record checking method and device
CN106055706A (en) * 2016-06-23 2016-10-26 杭州迪普科技有限公司 Cache resource storage method and device
CN106777371A (en) * 2017-01-23 2017-05-31 北京齐尔布莱特科技有限公司 Result collection system and method
CN106815274A (en) * 2015-12-02 2017-06-09 中兴通讯股份有限公司 Daily record data method for digging and system based on Hadoop
CN106897281A (en) * 2015-12-17 2017-06-27 阿里巴巴集团控股有限公司 A kind of daily record sharding method and device
CN107239377A (en) * 2016-03-29 2017-10-10 阿里巴巴集团控股有限公司 The method and apparatus for obtaining Java Virtual Machine running status
CN107306416A (en) * 2016-04-25 2017-10-31 中国移动通信集团云南有限公司 The recording method of business diary and device
CN107590162A (en) * 2016-07-08 2018-01-16 中兴通讯股份有限公司 The storage method and device of journal file
CN107609895A (en) * 2017-08-10 2018-01-19 腾讯科技（深圳）有限公司 A kind of processing method and its equipment for merging service area
CN107644017A (en) * 2016-07-20 2018-01-30 平安科技（深圳）有限公司 The querying method and device of journal file
CN107835095A (en) * 2017-11-20 2018-03-23 杭州迪普科技股份有限公司 A kind of processing method and processing device of daily record
CN108509455A (en) * 2017-02-28 2018-09-07 百度在线网络技术（北京）有限公司 Tables of data treating method and apparatus
CN109117312A (en) * 2018-08-23 2019-01-01 北京小米智能科技有限公司 Data reconstruction method and device
CN109460318A (en) * 2018-10-26 2019-03-12 珠海市时杰信息科技有限公司 It is a kind of can rollback archives acquisition data introduction method, computer installation and computer readable storage medium
CN109471851A (en) * 2018-10-17 2019-03-15 上海达梦数据库有限公司 Data processing method, device, server and storage medium
CN109766218A (en) * 2018-12-21 2019-05-17 中信百信银行股份有限公司 Data back up method based on distributed storage
CN109828862A (en) * 2017-11-23 2019-05-31 成都华为技术有限公司 A kind of method and apparatus playing back log
CN109922490A (en) * 2019-01-28 2019-06-21 行目数据科技有限公司 A kind of anti-interruption continuous transmission management system of data
CN110019069A (en) * 2017-10-30 2019-07-16 阿里巴巴集团控股有限公司 A kind of log processing method, device and equipment
CN110232048A (en) * 2019-06-12 2019-09-13 腾讯科技(成都)有限公司 Acquisition methods, device and the storage medium of journal file
CN110502545A (en) * 2019-08-15 2019-11-26 中国平安财产保险股份有限公司 Date storage method, device, equipment and computer readable storage medium
CN111198853A (en) * 2018-11-16 2020-05-26 北京微播视界科技有限公司 Data processing method and device, electronic equipment and computer readable storage medium
CN111400368A (en) * 2020-03-06 2020-07-10 网宿科技股份有限公司 Log searching method and device of distributed server system
CN111414343A (en) * 2020-02-24 2020-07-14 北京云途腾科技有限责任公司 Log writing method and device, electronic equipment and medium
CN111830039A (en) * 2020-07-22 2020-10-27 南京认知物联网研究院有限公司 Intelligent product quality detection method and device
CN112395360A (en) * 2020-12-01 2021-02-23 中国联合网络通信集团有限公司 Data synchronization method, device, apparatus and medium based on non-relational database
CN112771624A (en) * 2018-09-20 2021-05-07 阿比奥梅德公司 Data storage and retrieval system for non-contiguous medical device operational data

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1932812A (en) * 2005-09-16 2007-03-21 腾讯科技（深圳）有限公司 Method and apparatus for holding journal
CN101605028A (en) * 2009-02-17 2009-12-16 北京安天电子设备有限公司 A kind of combining log records method and system
CN101763593A (en) * 2009-12-17 2010-06-30 中国电力科学研究院 Method and device for realizing audit log of system
CN101799894A (en) * 2010-01-11 2010-08-11 中冶东方工程技术有限公司 EPC engineering data processing system and method in complex network environment
CN102750326A (en) * 2012-05-30 2012-10-24 浪潮电子信息产业股份有限公司 Log management optimization method of cluster system based on downsizing strategy
WO2012154838A2 (en) * 2011-05-09 2012-11-15 Google Inc. Generating application recommendations based on user installed applications
WO2013033123A2 (en) * 2011-08-31 2013-03-07 Google Inc. Method and system for collecting and managing tv viewership data
CN102981944A (en) * 2012-12-12 2013-03-20 青岛海信宽带多媒体技术有限公司 Log storage method based on file system

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1932812A (en) * 2005-09-16 2007-03-21 腾讯科技（深圳）有限公司 Method and apparatus for holding journal
CN101605028A (en) * 2009-02-17 2009-12-16 北京安天电子设备有限公司 A kind of combining log records method and system
CN101763593A (en) * 2009-12-17 2010-06-30 中国电力科学研究院 Method and device for realizing audit log of system
CN101799894A (en) * 2010-01-11 2010-08-11 中冶东方工程技术有限公司 EPC engineering data processing system and method in complex network environment
WO2012154838A2 (en) * 2011-05-09 2012-11-15 Google Inc. Generating application recommendations based on user installed applications
WO2013033123A2 (en) * 2011-08-31 2013-03-07 Google Inc. Method and system for collecting and managing tv viewership data
CN102750326A (en) * 2012-05-30 2012-10-24 浪潮电子信息产业股份有限公司 Log management optimization method of cluster system based on downsizing strategy
CN102981944A (en) * 2012-12-12 2013-03-20 青岛海信宽带多媒体技术有限公司 Log storage method based on file system

Cited By (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105989059A (en) * 2015-02-09 2016-10-05 阿里巴巴集团控股有限公司 Data record checking method and device
CN105989059B (en) * 2015-02-09 2019-12-24 阿里巴巴集团控股有限公司 Data record checking method and device
CN105260399A (en) * 2015-09-17 2016-01-20 西安未来国际信息股份有限公司 Method for acquiring and retrieving distributed log
CN105391772A (en) * 2015-10-16 2016-03-09 百度在线网络技术（北京）有限公司 Service request processing method, log processing method and service request processing device
CN105391772B (en) * 2015-10-16 2019-02-22 百度在线网络技术（北京）有限公司 Service request processing method, log processing method and device
CN106815274A (en) * 2015-12-02 2017-06-09 中兴通讯股份有限公司 Daily record data method for digging and system based on Hadoop
CN106897281A (en) * 2015-12-17 2017-06-27 阿里巴巴集团控股有限公司 A kind of daily record sharding method and device
CN105528454A (en) * 2015-12-25 2016-04-27 北京奇虎科技有限公司 Log treatment method and distributed cluster computing device
CN107239377B (en) * 2016-03-29 2021-02-26 阿里巴巴集团控股有限公司 Method and device for acquiring running state of Java virtual machine
CN107239377A (en) * 2016-03-29 2017-10-10 阿里巴巴集团控股有限公司 The method and apparatus for obtaining Java Virtual Machine running status
CN107306416A (en) * 2016-04-25 2017-10-31 中国移动通信集团云南有限公司 The recording method of business diary and device
CN106055706A (en) * 2016-06-23 2016-10-26 杭州迪普科技有限公司 Cache resource storage method and device
CN106055706B (en) * 2016-06-23 2019-08-06 杭州迪普科技股份有限公司 A kind of cache resources storage method and device
CN107590162A (en) * 2016-07-08 2018-01-16 中兴通讯股份有限公司 The storage method and device of journal file
CN107644017A (en) * 2016-07-20 2018-01-30 平安科技（深圳）有限公司 The querying method and device of journal file
CN106777371A (en) * 2017-01-23 2017-05-31 北京齐尔布莱特科技有限公司 Result collection system and method
CN106777371B (en) * 2017-01-23 2019-12-06 北京齐尔布莱特科技有限公司 Log collection system and method
CN108509455A (en) * 2017-02-28 2018-09-07 百度在线网络技术（北京）有限公司 Tables of data treating method and apparatus
CN107609895A (en) * 2017-08-10 2018-01-19 腾讯科技（深圳）有限公司 A kind of processing method and its equipment for merging service area
CN110019069A (en) * 2017-10-30 2019-07-16 阿里巴巴集团控股有限公司 A kind of log processing method, device and equipment
CN107835095A (en) * 2017-11-20 2018-03-23 杭州迪普科技股份有限公司 A kind of processing method and processing device of daily record
CN109828862B (en) * 2017-11-23 2023-08-22 成都华为技术有限公司 Method and device for replaying log
CN109828862A (en) * 2017-11-23 2019-05-31 成都华为技术有限公司 A kind of method and apparatus playing back log
CN109117312B (en) * 2018-08-23 2022-03-01 北京小米智能科技有限公司 Data recovery method and device
CN109117312A (en) * 2018-08-23 2019-01-01 北京小米智能科技有限公司 Data reconstruction method and device
CN112771624A (en) * 2018-09-20 2021-05-07 阿比奥梅德公司 Data storage and retrieval system for non-contiguous medical device operational data
CN109471851A (en) * 2018-10-17 2019-03-15 上海达梦数据库有限公司 Data processing method, device, server and storage medium
CN109471851B (en) * 2018-10-17 2021-05-28 上海达梦数据库有限公司 Data processing method, device, server and storage medium
CN109460318A (en) * 2018-10-26 2019-03-12 珠海市时杰信息科技有限公司 It is a kind of can rollback archives acquisition data introduction method, computer installation and computer readable storage medium
CN109460318B (en) * 2018-10-26 2021-01-01 珠海市时杰信息科技有限公司 Import method of rollback archive collected data, computer device and computer readable storage medium
CN111198853B (en) * 2018-11-16 2023-08-22 北京微播视界科技有限公司 Data processing method, device, electronic equipment and computer readable storage medium
CN111198853A (en) * 2018-11-16 2020-05-26 北京微播视界科技有限公司 Data processing method and device, electronic equipment and computer readable storage medium
CN109766218A (en) * 2018-12-21 2019-05-17 中信百信银行股份有限公司 Data back up method based on distributed storage
CN109922490A (en) * 2019-01-28 2019-06-21 行目数据科技有限公司 A kind of anti-interruption continuous transmission management system of data
CN109922490B (en) * 2019-01-28 2022-01-28 广东中视信息科技有限公司 Data interruption-proof continuous transmission management system
CN110232048B (en) * 2019-06-12 2023-07-07 腾讯科技(成都)有限公司 Log file acquisition method, device and storage medium
CN110232048A (en) * 2019-06-12 2019-09-13 腾讯科技(成都)有限公司 Acquisition methods, device and the storage medium of journal file
CN110502545B (en) * 2019-08-15 2023-12-05 中国平安财产保险股份有限公司 Data storage method, device, equipment and computer readable storage medium
CN110502545A (en) * 2019-08-15 2019-11-26 中国平安财产保险股份有限公司 Date storage method, device, equipment and computer readable storage medium
CN111414343B (en) * 2020-02-24 2023-08-04 北京百度云途腾科技有限责任公司 Log writing method, device, electronic equipment and medium
CN111414343A (en) * 2020-02-24 2020-07-14 北京云途腾科技有限责任公司 Log writing method and device, electronic equipment and medium
CN111400368A (en) * 2020-03-06 2020-07-10 网宿科技股份有限公司 Log searching method and device of distributed server system
CN111400368B (en) * 2020-03-06 2023-12-12 网宿科技股份有限公司 Log searching method and device of distributed server system
CN111830039A (en) * 2020-07-22 2020-10-27 南京认知物联网研究院有限公司 Intelligent product quality detection method and device
CN112395360B (en) * 2020-12-01 2023-06-23 中国联合网络通信集团有限公司 Data synchronization method, device, apparatus and medium based on non-relational database
CN112395360A (en) * 2020-12-01 2021-02-23 中国联合网络通信集团有限公司 Data synchronization method, device, apparatus and medium based on non-relational database

Also Published As

Publication number Publication date
HK1202722A1 (en) 2015-10-02
CN104301360B (en) 2019-03-12

Similar Documents

Publication Publication Date Title
CN104301360A (en) Method, log server and system for recording log data
CN105447075B (en) The computer implemented method divided for dynamic
US8250033B1 (en) Replication of a data set using differential snapshots
CN113626525B (en) System and method for implementing scalable data storage services
US7480643B2 (en) System and method for migrating databases
US8799601B1 (en) Techniques for managing deduplication based on recently written extents
Santos et al. Real-time data warehouse loading methodology
CN102110121B (en) A kind of data processing method and system thereof
US20130110873A1 (en) Method and system for data storage and management
CN102169507A (en) Distributed real-time search engine
CN102136003A (en) Large-scale distributed storage system
CN106934048A (en) Online data moving method, agent node
EP2380090B1 (en) Data integrity in a database environment through background synchronization
CN103336849A (en) Method and device for increasing retrieval speed in database retrieval system
CN103902623A (en) Method and system for accessing files on a storage system
CN103455526A (en) ETL (extract-transform-load) data processing method, device and system
CN108694188B (en) Index data updating method and related device
CN104462389A (en) Method for implementing distributed file systems on basis of hierarchical storage
CN105573859A (en) Data recovery method and device of database
CN110737682A (en) cache operation method, device, storage medium and electronic equipment
CN105787037A (en) Repeated data deleting method and device
CN110008197A (en) A kind of data processing method, system and electronic equipment and storage medium
CN106874343B (en) Data deletion method and system for time sequence database
CN104035822A (en) Low-cost efficient internal storage redundancy removing method and system
CN105426119A (en) Storage apparatus and data processing method

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 1202722

Country of ref document: HK

GR01 Patent grant
GR01 Patent grant