CN102447633A - Log transmission method and system - Google Patents
Log transmission method and system Download PDFInfo
- Publication number
- CN102447633A CN102447633A CN2011104526934A CN201110452693A CN102447633A CN 102447633 A CN102447633 A CN 102447633A CN 2011104526934 A CN2011104526934 A CN 2011104526934A CN 201110452693 A CN201110452693 A CN 201110452693A CN 102447633 A CN102447633 A CN 102447633A
- Authority
- CN
- China
- Prior art keywords
- log information
- log
- buffer queue
- data center
- preset buffer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention provides a log transmission method and system. The method comprises the following steps: judging whether log information written in a first preset buffer queue of a game point exceeds a first preset length threshold, and if not, locally reading the log information from the game point and writing in the first preset buffer queue in the current game point; if so, stopping locally reading the log information from the game point; and transmitting the log information in the first preset buffer queue to a data center. The invention can ensure the log transmission stability and reliability, and enhance the transmission efficiency under various possible abnormal conditions and extremely complex network states.
Description
Technical field
The application relates to the computer data transmission technology field, particularly relates to a kind of method of log transmission, and, a kind of system of log transmission.
Background technology
Now, the Internet technology high speed development that makes rapid progress, it is more and more wider, trans-regional that the business of a lot of Internet firms covers the region, transprovincially, so that transnational.In the face of so large-scale region, a lot of companies all are chosen in the each province and city deployment server, and general headquarters dispose the mode of central server, to guarantee favorable user experience.Therefore, the daily record of the server in each province and city generation will be sent to central server through a kind of safe, real-time, complete load mode.
Application number is that 201010536426 one Chinese patent application discloses a kind of log collection device and log collection method on February 9th, 2011, and with reference to Fig. 1, this system comprises:
At least one client that is used for log collection is used for the server of log record with at least one; Carrying out real-time log information through network between the said client and server sends.Client generates log information and log information is write the disk file of client; Read the log information in the client disk file, and send connection request to server; After server is received connection request, set up network with client and be connected and receive log information; The log information that receives is sent to the sever log writing module respectively to server and server first day of the lunar year will writes DBM; The sever log writing module writes the server disk file with said log information; Sever log writes DBM said log information is write database.
There is following problem in above-mentioned patent application:
1, the client log read module is out of contior to the speed that the client log sending module sends daily record; The client log read module sends speed from daily record to the client log sending module can occur and send the speed of daily record to server, make process of transmitting unstable greater than the client log sending module.
If the network between the 2 client and server ends breaks down when breaking off, continue transmission when continuation was broken off from last time when can not guarantee network recovery.
3, the sever log receiver module speed of accepting daily record can not be controlled, and can cause speed that the sever log receiver module accepts daily record to handle the speed of daily record greater than the sever log processing module, sever log is received and storage unstable.
So those skilled in the art press for the problem of solution: guarantee when various possible abnormal conditions and extremely complicated network state, the stability of log transmission and reliability, and improve the efficient of transmission.
Summary of the invention
The application's technical problem to be solved provides a kind of method of log transmission, in order to guarantee when various possible abnormal conditions and the extremely complicated network state stability of log transmission and reliability; Improve the efficient of transmission.
Accordingly, the application also provides a kind of system of log transmission, in order to ensure the application of said method in reality.
In order to address the above problem, the application discloses a kind of method of log transmission, specifically can comprise:
Judge whether the log information that writes in the first preset buffer queue of office point exceeds the first preset length threshold value, if not, then reads log information from the office point, and write the preset buffer queue in first in the current office point; If then suspend and read log information from the office point;
Send the log information in the said first preset buffer queue to data center.
Preferably, described method can also comprise:
Judge that whether the log information that writes in the second preset buffer queue of said data center exceeds the second preset length threshold value, if not, then writes the second preset buffer queue with the log information that receives; If then suspend the reception log information;
Store the log information in the said second preset buffer queue into target location.
Preferably, described method can also comprise:
If the log information in the said first preset buffer queue transmits failure in sending the process of data center to, then the log information with current transmission writes in the preset buffer memory of current office point local hard drive;
Send the log information in the said preset buffer memory to data center.
Preferably, said log information is for forming the log record of journal file, and said journal file has filename, and first log record in the said journal file has the filename and the owner record sign of said journal file;
The said step that stores the log information in the second preset buffer queue into target location can comprise:
Have the filename and the owner record sign of said journal file when receiving log information, then according to the newly-built journal file of the filename of said journal file;
In said newly-built journal file, begin to write the log information of reception.
Preferably; Identical configuration and application program can be adopted with data center in said office point; Be provided with first configuration file in the said office point; Be provided with second configuration file in the said data center, said office point and data center are distinguished according to said first configuration file and second configuration file.
Preferably, said target location can comprise the local hard drive of said data center, and/or, in distributed system that said data center links to each other.
Preferably, the log information in the said preset buffer memory can have precedence over the log information transmission in the said first preset buffer queue.
The application also discloses a kind of system of log transmission simultaneously, specifically can comprise:
Judge module is read in daily record, is used for judging whether the log information that the first preset buffer queue of office point writes exceeds the first preset length threshold value, if not, then calls the daily record read module, if then call daily record and read the time-out module;
Daily record writes the first preset buffer queue module, is used for reading log information from the office point, and writes the preset buffer queue in first in the current office point;
The time-out module is read in daily record, is used for suspending reading log information from the office point;
The daily record sending module is used for sending the log information of the said first preset buffer queue to data center.
Preferably, described system can also comprise:
Daily record receives judge module, is used for judging whether the log information that the second preset buffer queue of said data center writes exceeds the second preset length threshold value, if not, then calls the daily record receiver module, if then call daily record and receive and suspend module;
Daily record writes the second preset buffer queue module, is used for the log information that receives is write the second preset buffer queue;
Daily record receives and suspends module, is used for then suspending the reception log information;
The log store module is used for storing the log information of the said second preset buffer queue into target location.
Preferably, described system can also comprise:
Judge module is sent in daily record; Be used for sending the log information of the said first preset buffer queue to data center; And judge whether the log information in the said first preset buffer queue transmits failure in sending the process of data center to, if then call the log buffer module; If not, then call daily record and receive judge module;
The log buffer module is used for the log information of current transmission is write the preset buffer memory of current office point local hard drive;
The daily record delivery module is used for sending the log information of said preset buffer memory to data center.
Preferably, said log information is for forming the log record of journal file, and said journal file has filename, and first log record in the said journal file has the filename and the owner record sign of said journal file;
Said log store module can comprise:
Journal file classification submodule; Be used for having the filename and the owner record sign of said journal file when receiving log information; Then according to the newly-built journal file of the filename of said journal file, and in said newly-built journal file, begin to write the log information of reception.
Preferably; Identical configuration and application program can be adopted with data center in said office point; Be provided with first configuration file in the said office point; Be provided with second configuration file in the said data center, said office point and data center are distinguished according to said first configuration file and second configuration file.
Compared with prior art, the application has the following advantages:
The application provides a kind of distributed method and system that can carry out the real-time Transmission daily record.The office point is set handles the work of daily record transmission and the work that data center handles the daily record reception.Through the first preset buffer queue is set; At the daily record sending link local daily record reading speed and transmission speed are regulated and control; Be specially and read log information from the office point and write the preset buffer queue in first the current office point; Log information in the said first preset buffer queue, writing exceeds the first preset length threshold value, suspends and reads log information from the office point, thereby avoided causing the unsettled situation of log transmission in local daily record reading speed greater than transmission speed; Strengthened the stability of log transmission.
The application also through local cache is set, receives link to sending storing and continuous biography of failure in daily record.Be specially when daily record sending link network failure, the log information that is sending is stored.In the time of network recovery, preferentially the log information in the local cache is started and continue to send.Realize the breakpoint transmission in the daily record process of transmitting, thereby avoided when network failure, causing the insecure problem of log transmission, improved the reliability of log transmission.
The application is through being provided with the second preset buffer queue; Receiving link in daily record regulates and control local daily record inbound pacing and storage speed; Be specially the second preset buffer queue that the log information of data center's reception is write data center; Log information in the said second preset buffer queue, writing exceeds the second preset length threshold value, suspends from the office point local reception log information, thereby has avoided causing the stable situation of log transmission in the daily record inbound pacing greater than storage speed; Strengthened the stability of log transmission.
Description of drawings
Fig. 1 is a kind of log collection apparatus structure block diagram of prior art;
Fig. 2 is the flow chart of the method embodiment 1 of a kind of log transmission of the application;
Fig. 3 is the flow chart of the method embodiment 2 of a kind of log transmission of the application;
Fig. 4 is the flow chart of the method embodiment 3 of a kind of log transmission of the application;
Fig. 5 is the structured flowchart of the system embodiment of a kind of log transmission of the application.
Embodiment
For above-mentioned purpose, the feature and advantage that make the application can be more obviously understandable, the application is done further detailed explanation below in conjunction with accompanying drawing and embodiment.
One of core idea of the application embodiment is, the office point is set handles the work of daily record transmission and the work that data center handles the daily record reception.Through the first preset buffer queue is set; At the daily record sending link local daily record reading speed and transmission speed are regulated and control; Be specially and read log information from the office point and write the preset buffer queue in first the current office point; Log information in the said first preset buffer queue, writing exceeds the first preset length threshold value; Time-out reads log information from the office point, thereby has avoided causing the unsettled situation of log transmission in local daily record reading speed greater than transmission speed, thereby has improved the stability and the reliability of log transmission.
With reference to Fig. 2, show the flow chart of the method embodiment 1 of a kind of log transmission of the application, specifically can comprise:
In concrete the realization, the office point in the log transmission system is the server that is deployed in each subordinate branch, in order to read local daily record, is transferred to data center.
The office point produces daily record; Owing to the configuration of office point machine reason such as high-performance is not bad; Analyzing and processing can not be carried out to daily record in the office point, so the scheme that the daily record of each office point is mentioned through this patent is transferred to data center, data center is a high performance server cluster; Have powerful computing ability, data center to data analyze, follow-up work such as processing.In reality; Can branch of different provinces and cities; Local log acquisition link is disposed a plurality of office points; In general headquarters the link of same processing of daily record and storage is disposed a data center, a cover code can be shared by said office point and data center, through configuration file office point, each province and city server and data center server is configured.
Particularly, the office point can be provided with daily record transmission thread and read journal file from this locality, passes to the first preset buffer queue the inside of log processing after the compression.When authorities' point reads the speed of local daily record and sends thread and send daily record fireballing than the daily record of office point; The daily record meeting is in the first preset buffer queue accumulation; Reach the threshold value (maximum length can dispose from first configuration file) of maximum length when the first preset buffer queue; Stop to read the buffer queue that journal file passes to log processing module, drop to the certain proportion of maximum length up to the buffer queue size.
The first preset buffer queue can be provided with according to the size of internal memory, and it is exactly the block space in the internal memory in fact, as long as be no more than the size of internal memory, the operation that does not influence other programs is just passable, if internal memory 2G, buffer queue can be set to 1G.For stopping to read the buffer queue that journal file passes to log processing module, drop to the certain proportion of maximum length up to the buffer queue size, generally can be configured to 10%～30% of the first preset buffer queue size.Certainly, above-mentioned setting is only as example, those skilled in the art according to actual conditions be provided with arbitrarily all can, the application does not limit this.
In concrete the realization, the log information that thread writes the first preset buffer queue is sent in the daily record of office point, through after the processed compressed, is transferred to data center.
For example: local daily record is read in the office point, is sent to the first preset buffer queue after the compression.At this moment the office point is read the speed of local daily record and is sent thread than the daily record of office point to send the speed of daily record fast, and stored log has reached the maximum length of buffer queue in the buffer queue.At this moment suspend and read local daily record, the daily record of waiting for the office point is sent thread and is sent the daily record at the first preset buffer queue.When stored log in the first preset buffer queue drop to the buffer queue maximum length 30% the time, restart to read local log store to buffer queue.
The application is provided with the office point and handles the work of daily record transmission and the work that data center handles the daily record reception.Through the first preset buffer queue is set; At the daily record sending link local daily record reading speed and transmission speed are regulated and control; Be specially and read log information from the office point and write the preset buffer queue in first the current office point; Log information in the said first preset buffer queue, writing exceeds the first preset length threshold value, suspends and reads log information from the office point, thereby avoided causing the unsettled situation of log transmission in local daily record reading speed greater than transmission speed; Strengthened the stability of log transmission.
With reference to Fig. 3, show the flow chart of the method embodiment 2 of a kind of log transmission of the application, specifically can comprise:
In concrete the realization; The speed that receives log information when data center is than data center's storing daily record information fireballing the time; The daily record meeting is preset the buffer queue accumulation in second of data center, and when the log information in the second preset buffer queue reaches the threshold value of maximum length, data center will slow down; Or suspend reception log information to the second and preset buffer queue, the log information amount drops to the certain proportion of maximum length in the second preset buffer queue.
The threshold value of the threshold value of the second preset buffer queue maximum length and the first preset buffer queue maximum length all is to lean on the first preset configuration file and the second preset configuration file configuration, can be identical, and also can be different.
The said second preset buffer queue can be provided with according to the size of internal memory, if internal memory 2G, buffer queue can be set to 1G.For stopping to read the buffer queue that journal file passes to log processing module, drop to the certain proportion of maximum length up to the buffer queue size, generally can be configured to 10%～30% of the second preset buffer queue size.Certainly, above-mentioned setting is only as example, those skilled in the art according to actual conditions be provided with arbitrarily all can, the application does not limit this.
In a kind of preferred embodiment of the application, said log information is for forming the log record of journal file, and said journal file has filename, and first log record in the said journal file has the filename and the owner record sign of said journal file.
In a kind of preferred embodiment of the application, said step 208 specifically can comprise following substep:
Substep S11, has filename and the owner record sign of said journal file when receiving log information, then according to the newly-built journal file of the filename of said journal file;
Substep S12, in said newly-built journal file, begin to write the log information of reception.
In concrete the realization, said target location can comprise the local hard drive of said data center, and/or, in distributed system that said data center links to each other.Distributed system can adopt HDFS (Hadoop Distributed File System, HDFS) group system.The daily record of said office point is sent and the daily record reception of data center can parallel processing.
Because in fact whole log transmission process be similar to a stream, daily record flows to data center continually from the office point, but in the whole stream only is the content of daily record, i.e. daily record information.For example, if the office point journal file has two: a and b have some daily records among file a and the file b.Be transferred to the content that is merely file a and file b (being some daily records) of data center so, and data center can not know that which daily record belongs to journal file a, which daily record belongs to journal file b.
Because being order when sending, log content sends; Also be that order receives when receiving; So the application embodiment tells data center through the owner record sign when sending article one log information of each journal file; This is a new journal file, and the filename of subsidiary this journal file.Such as, when sending article one daily record of each journal file in the office point, add an owner record sign in this daily record front: " magic string ", the back adds the filename of this file then.When log information is received by data center; Detect earlier and whether contain owner record sign " magic string "; If have; Learn that then it is article one log information of a new journal file,, create the journal file opening entry log information of corresponding document name in data center then according to filename subsidiary behind the magic string.
The application is through being provided with the second preset buffer queue; Receiving link in daily record regulates and control local daily record inbound pacing and storage speed; Be specially the second preset buffer queue that the log information of data center's reception is write data center; Log information in the said second preset buffer queue, writing exceeds the second preset length threshold value, suspends from the office point local reception log information, thereby has avoided causing the stable situation of log transmission in the daily record inbound pacing greater than storage speed; Further strengthened the stability of log transmission.
With reference to Fig. 4, show the flow chart of the method embodiment 3 of a kind of log transmission of the application, specifically can comprise:
In concrete the realization; When in the log information in the first preset buffer queue is sending the process of data center to, transmitting failure; Log processing module is buffered in local hard drive earlier with daily record; Wait until the data that read the local hard drive of buffer memory when network is unimpeded more then, be transferred to second of data center's log processing module and preset buffer queue.
In a kind of preferred embodiment of the present invention, the log information that the log information in the said preset buffer memory has precedence in the said first preset buffer queue transmits.
In a kind of preferred embodiment of the present invention; Identical configuration and application program are adopted with data center in said office point; Be provided with first configuration file in the said office point; Be provided with second configuration file in the said data center, said office point and data center are distinguished according to said first configuration file and second configuration file.
The application also through local cache is set, receives link to sending storing and continuous biography of failure in daily record.Be specially when daily record sending link network failure, the log information that is sending is stored.In the time of network recovery, preferentially the log information in the local cache is started and continue to send.Realize the breakpoint transmission in the daily record process of transmitting, thereby avoided when network failure, causing the insecure problem of log transmission, improved the reliability of log transmission.
For making those skilled in the art understand the present invention better, below better set forth the application's implementation procedure with a complete embodiment:
1, at office point and data center's difference deployment server, two cover servers are the same, and just configuration file is different.The office point disposes with first configuration file, and first configuration file is called judian.conf, after application program is resolved the filename of first configuration file, starts and reads local daily record, sends daily record to network; Data center disposes with second configuration file, and second configuration file is called zhongxin.conf, after application program is resolved the filename of second configuration file, starts and reads network data, to HDFS storage data.
2, the daily record of office point is sent thread and is read the local log information of office point; Be marked with magic string for newly-built one of article one log information that reads; Using mark and suffix is former journal file name; New file as the local daily record of reading writes daily record, presets the buffer queue the inside with passing to first after the daily record compression.
3, at this moment, local daily record reading speed is greater than transmission speed, and log information presets the buffer queue accumulation first, reaches the threshold value of maximum length, stops to read journal file and passes to first and preset buffer queue;
4, first log information that preset in the buffer queue is sent to data center; When first log information that preset in the buffer queue drops to 30% of maximum length, daily record is sent the thread center and is restarted to read local log information and write first and preset buffer queue.
5, log information is sent to data center second and presets buffer memory and read, when receiving the completion signal that data center returns, continue the transmission of next bar daily record.
6, the log information in the first preset buffer queue transmits failure in sending the process of data center to; The log processing thread is buffered in local hard drive earlier with daily record; When network is unimpeded by the time; From the local hard drive of buffer memory, read log information again, preset the buffer queue transmission to second of data center again.
7, the daily record receiving thread of data center writes the second preset buffer queue with the log information that receives, and the log store thread reads log information from the second preset buffer queue.
8, the speed of daily record this moment receiving thread reception log information is faster than the speed of log store threads store log information; Log information is in data center's second preset buffer queue accumulation; Reach the second preset buffer queue maximum length and put into threshold value, the daily record receiving thread suspends the reception log information.
9, the log store thread continues the log information in the second preset buffer queue is stored among the distributed system HDFS; Log information in the second preset buffer queue drops to 30% o'clock of the second preset buffer queue maximum length, from newly beginning to receive log information.
10, data center's log information of detecting current reception contains " magic string ", behind the decompress(ion), according to new journal file of second configuration file configuration, log information is write the new journal file of people and is deposited the hdfs distributed file system.
Need to prove; For aforesaid each method embodiment, for simple description, so it all is expressed as a series of combination of actions; But those skilled in the art should know; The present invention does not receive the restriction of described sequence of movement, because according to the present invention, some step can adopt other orders or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in the specification all belongs to preferred embodiment, and related action and module might not be that the present invention is necessary.Moreover above-mentioned each method embodiment all adopts the mode of going forward one by one to describe, and what each embodiment stressed all is and the difference of other embodiment that identical similar part is mutually referring to getting final product between each embodiment.
With reference to Fig. 5, show the structured flowchart of the system embodiment of a kind of log transmission of the application, specifically can comprise:
Daily record writes the first preset buffer queue module 402, is used for reading log information from the office point, and writes the preset buffer queue in first in the current office point;
Daily record is read and is suspended module 403, is used for suspending reading log information from the office point;
Log buffer module 405 is used for the log information of current transmission is write the preset buffer memory of current office point local hard drive;
Daily record delivery module 406 is used for sending the log information of said preset buffer memory to data center.
Daily record receives judge module 407; Be used for judging whether the log information that the second preset buffer queue of said data center writes exceeds the second preset length threshold value, if not, then calls daily record and writes the second preset buffer queue module 408; If then call daily record and receive time-out module 409;
Daily record writes the second preset buffer queue module 408, is used for the log information that receives is write the second preset buffer queue;
Daily record receives and suspends module 409, is used for then suspending the reception log information;
In a kind of preferred embodiment of the present invention, said log information is for forming the log record of journal file, and said journal file has filename, and first log record in the said journal file has the filename and the owner record sign of said journal file.
Said log store module 410 specifically can comprise following submodule:
Journal file classification submodule, be used for having the filename and the owner record sign of said journal file, then according to the newly-built journal file of the filename of said journal file when receiving log information;
The newly-built submodule of journal file, be used for beginning to write the log information of reception at said newly-built journal file.
In a kind of preferred embodiment of the present invention; Identical configuration and application program can be adopted with data center in said office point; Be provided with first configuration file in the said office point; Be provided with second configuration file in the said data center, said office point and data center are distinguished according to said first configuration file and second configuration file.
In a kind of preferred embodiment of the present invention, said target location specifically can comprise the local hard drive of said data center, and/or, in distributed system that said data center links to each other.
In a kind of preferred embodiment of the present invention, the log information that the log information in the said preset buffer memory has precedence in the said first preset buffer queue transmits.
Because said system embodiment is basically corresponding to preceding method embodiment, so not detailed part in the description of present embodiment can just not given unnecessary details at this referring to the related description in the previous embodiment.
The application can be used in numerous general or special purpose computingasystem environment or the configuration.For example: personal computer, server computer, handheld device or portable set, plate equipment, multicomputer system, the system based on microprocessor, set top box, programmable consumer-elcetronics devices, network PC, minicom, mainframe computer, comprise DCE of above any system or equipment or the like.
The application can describe in the general context of the computer executable instructions of being carried out by computer, for example program module.Usually, program module comprises the routine carrying out particular task or realize particular abstract, program, object, assembly, data structure or the like.Also can in DCE, put into practice the application, in these DCEs, by through communication network connected teleprocessing equipment execute the task.In DCE, program module can be arranged in this locality and the remote computer storage medium that comprises memory device.
At last; Also need to prove; In this article; Relational terms such as first and second grades only is used for an entity or operation are made a distinction with another entity or operation, and not necessarily requires or hint relation or the order that has any this reality between these entities or the operation.And; Term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability; Thereby make and comprise that process, method, article or the equipment of a series of key elements not only comprise those key elements; But also comprise other key elements of clearly not listing, or also be included as this process, method, article or equipment intrinsic key element.Under the situation that do not having much more more restrictions, the key element that limits by statement " comprising ... ", and be not precluded within process, method, article or the equipment that comprises said key element and also have other identical element.
More than the method for a kind of log transmission that the application provided and a kind of system of log transmission have been carried out detailed introduction; Used concrete example among this paper the application's principle and execution mode are set forth, the explanation of above embodiment just is used to help to understand the application's method and core concept thereof; Simultaneously, for one of ordinary skill in the art, according to the application's thought, the part that on embodiment and range of application, all can change, in sum, this description should not be construed as the restriction to the application.
Claims (12)
1. the method for a log transmission is characterized in that, said log transmission relates to the process that daily record is transmitted to data center from the office point, and said method comprises:
Judge whether the log information that writes in the first preset buffer queue of office point exceeds the first preset length threshold value, if not, then reads log information from the office point, and write the preset buffer queue in first in the current office point; If then suspend and read log information from the office point;
Send the log information in the said first preset buffer queue to data center.
2. the method for claim 1 is characterized in that, also comprises:
Judge that whether the log information that writes in the second preset buffer queue of said data center exceeds the second preset length threshold value, if not, then writes the second preset buffer queue with the log information that receives; If then suspend the reception log information;
Store the log information in the said second preset buffer queue into target location.
3. method as claimed in claim 2 is characterized in that, also comprises:
If the log information in the said first preset buffer queue transmits failure in sending the process of data center to, then the log information with current transmission writes in the preset buffer memory of current office point local hard drive;
Send the log information in the said preset buffer memory to data center.
4. like claim 1 or 2 or 3 described methods; It is characterized in that; Said log information is for forming the log record of journal file, and said journal file has filename, and first log record in the said journal file has the filename and the owner record sign of said journal file;
The said step that stores the log information in the second preset buffer queue into target location comprises:
Have the filename and the owner record sign of said journal file when receiving log information, then according to the newly-built journal file of the filename of said journal file;
In said newly-built journal file, begin to write the log information of reception.
5. like claim 1 or 2 or 3 described methods; It is characterized in that; Identical configuration and application program are adopted with data center in said office point; Be provided with first configuration file in the said office point, be provided with second configuration file in the said data center, said office point and data center are distinguished according to said first configuration file and second configuration file.
6. method as claimed in claim 2 is characterized in that said target location comprises the local hard drive of said data center, and/or, in distributed system that said data center links to each other.
7. method as claimed in claim 3 is characterized in that, the log information that the log information in the said preset buffer memory has precedence in the said first preset buffer queue transmits.
8. the system of a log transmission is characterized in that, comprising:
Judge module is read in daily record, is used for judging whether the log information that the first preset buffer queue of office point writes exceeds the first preset length threshold value, if not, then calls the daily record read module, if then call daily record and read the time-out module;
Daily record writes the first preset buffer queue module, is used for reading log information from the office point, and writes the preset buffer queue in first in the current office point;
The time-out module is read in daily record, is used for suspending reading log information from the office point;
The daily record sending module is used for sending the log information of the said first preset buffer queue to data center.
9. system as claimed in claim 8 is characterized in that, also comprises:
Daily record receives judge module, is used for judging whether the log information that the second preset buffer queue of said data center writes exceeds the second preset length threshold value, if not, then calls the daily record receiver module, if then call daily record and receive and suspend module;
Daily record writes the second preset buffer queue module, is used for the log information that receives is write the second preset buffer queue;
Daily record receives and suspends module, is used for then suspending the reception log information;
The log store module is used for storing the log information of the said second preset buffer queue into target location.
10. system as claimed in claim 9 is characterized in that, also comprises:
Judge module is sent in daily record; Be used for sending the log information of the said first preset buffer queue to data center; And judge whether the log information in the said first preset buffer queue transmits failure in sending the process of data center to, if then call the log buffer module; If not, then call daily record and receive judge module;
The log buffer module is used for the log information of current transmission is write the preset buffer memory of current office point local hard drive;
The daily record delivery module is used for sending the log information of said preset buffer memory to data center.
11. like claim 8 or 9 or 10 described systems; It is characterized in that; Said log information is for forming the log record of journal file, and said journal file has filename, and first log record in the said journal file has the filename and the owner record sign of said journal file;
Said log store module comprises:
Journal file classification submodule; Be used for having the filename and the owner record sign of said journal file when receiving log information; Then according to the newly-built journal file of the filename of said journal file, and in said newly-built journal file, begin to write the log information of reception.
12. like claim 8 or 9 or 10 described systems; It is characterized in that; Identical configuration and application program are adopted with data center in said office point; Be provided with first configuration file in the said office point, be provided with second configuration file in the said data center, said office point and data center are distinguished according to said first configuration file and second configuration file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011104526934A CN102447633A (en) | 2011-12-29 | 2011-12-29 | Log transmission method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011104526934A CN102447633A (en) | 2011-12-29 | 2011-12-29 | Log transmission method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102447633A true CN102447633A (en) | 2012-05-09 |
Family
ID=46009734
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2011104526934A Pending CN102447633A (en) | 2011-12-29 | 2011-12-29 | Log transmission method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102447633A (en) |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064779A (en) * | 2012-12-26 | 2013-04-24 | 北京思特奇信息技术股份有限公司 | Method for processing log information rapidly |
| CN103532943A (en) * | 2013-10-08 | 2014-01-22 | 北京神州绿盟信息安全科技股份有限公司 | Web application firewall device and asynchronous security protection log processing method |
| CN103617287A (en) * | 2013-12-12 | 2014-03-05 | 用友软件股份有限公司 | Log management method and device in distributed environment |
| CN104468399A (en) * | 2014-12-08 | 2015-03-25 | 北京奇虎科技有限公司 | Data transmission method and device and server |
| CN105391772A (en) * | 2015-10-16 | 2016-03-09 | 百度在线网络技术（北京）有限公司 | Service request processing method, log processing method and service request processing device |
| CN105429783A (en) * | 2015-10-30 | 2016-03-23 | 上海帝联信息科技股份有限公司 | Edge node server, log file processing method and log file processing system |
| CN105429784A (en) * | 2015-10-30 | 2016-03-23 | 上海帝联信息科技股份有限公司 | Central collection server, and method and system for log file processing |
| CN106095959A (en) * | 2016-06-16 | 2016-11-09 | 北京中电普华信息技术有限公司 | A kind of collecting method, Apparatus and system |
| CN106504402A (en) * | 2016-11-08 | 2017-03-15 | 深圳怡化电脑股份有限公司 | Log information recording method and device |
| CN106649834A (en) * | 2016-12-29 | 2017-05-10 | 北京奇虎科技有限公司 | Asynchronous transmission method, device and system of log data |
| CN106708578A (en) * | 2016-12-23 | 2017-05-24 | 北京五八信息技术有限公司 | Dual-thread-based journal output method and device |
| CN106815124A (en) * | 2015-12-01 | 2017-06-09 | 北京国双科技有限公司 | Journal file treating method and apparatus |
| CN106899643A (en) * | 2015-12-21 | 2017-06-27 | 阿里巴巴集团控股有限公司 | A kind of user journal storage method and equipment |
| CN108234595A (en) * | 2017-11-29 | 2018-06-29 | 山东鲁能软件技术有限公司 | Log transmission method and system |
| CN108399186A (en) * | 2018-01-12 | 2018-08-14 | 联动优势科技有限公司 | A kind of collecting method and device |
| CN108804295A (en) * | 2017-04-28 | 2018-11-13 | 北京京东尚科信息技术有限公司 | log information recording method and device |
| CN109491862A (en) * | 2018-10-24 | 2019-03-19 | 聚好看科技股份有限公司 | A kind of service requesting information collection method and device based on cloud platform |
| WO2020000745A1 (en) * | 2018-06-29 | 2020-01-02 | 平安科技（深圳）有限公司 | Log management method and apparatus, computer device, and storage medium |
| CN112612760A (en) * | 2020-12-30 | 2021-04-06 | 中国农业银行股份有限公司 | Log message output method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1794646A (en) * | 2005-04-27 | 2006-06-28 | 华为技术有限公司 | Method for obtaining of daily information from network element equipment by network management server |
| CN101909075A (en) * | 2010-06-21 | 2010-12-08 | 深圳市五巨科技有限公司 | System core data storage method and device thereof |
| CN101969386A (en) * | 2010-11-09 | 2011-02-09 | 道有道(北京)科技有限公司 | Log acquisition device and log acquisition method |
-
2011
- 2011-12-29 CN CN2011104526934A patent/CN102447633A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1794646A (en) * | 2005-04-27 | 2006-06-28 | 华为技术有限公司 | Method for obtaining of daily information from network element equipment by network management server |
| CN101909075A (en) * | 2010-06-21 | 2010-12-08 | 深圳市五巨科技有限公司 | System core data storage method and device thereof |
| CN101969386A (en) * | 2010-11-09 | 2011-02-09 | 道有道(北京)科技有限公司 | Log acquisition device and log acquisition method |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064779A (en) * | 2012-12-26 | 2013-04-24 | 北京思特奇信息技术股份有限公司 | Method for processing log information rapidly |
| CN103064779B (en) * | 2012-12-26 | 2016-08-03 | 北京思特奇信息技术股份有限公司 | A kind of method of quick process log information |
| CN103532943A (en) * | 2013-10-08 | 2014-01-22 | 北京神州绿盟信息安全科技股份有限公司 | Web application firewall device and asynchronous security protection log processing method |
| CN103617287A (en) * | 2013-12-12 | 2014-03-05 | 用友软件股份有限公司 | Log management method and device in distributed environment |
| CN104468399A (en) * | 2014-12-08 | 2015-03-25 | 北京奇虎科技有限公司 | Data transmission method and device and server |
| CN104468399B (en) * | 2014-12-08 | 2020-01-17 | 北京奇虎科技有限公司 | Data transmission method and device and server |
| WO2016091085A1 (en) * | 2014-12-08 | 2016-06-16 | 北京奇虎科技有限公司 | Data transmission method, device, and server |
| CN105391772A (en) * | 2015-10-16 | 2016-03-09 | 百度在线网络技术（北京）有限公司 | Service request processing method, log processing method and service request processing device |
| CN105391772B (en) * | 2015-10-16 | 2019-02-22 | 百度在线网络技术（北京）有限公司 | Service request processing method, log processing method and device |
| CN105429784A (en) * | 2015-10-30 | 2016-03-23 | 上海帝联信息科技股份有限公司 | Central collection server, and method and system for log file processing |
| CN105429783A (en) * | 2015-10-30 | 2016-03-23 | 上海帝联信息科技股份有限公司 | Edge node server, log file processing method and log file processing system |
| CN106815124A (en) * | 2015-12-01 | 2017-06-09 | 北京国双科技有限公司 | Journal file treating method and apparatus |
| CN106899643A (en) * | 2015-12-21 | 2017-06-27 | 阿里巴巴集团控股有限公司 | A kind of user journal storage method and equipment |
| CN106095959A (en) * | 2016-06-16 | 2016-11-09 | 北京中电普华信息技术有限公司 | A kind of collecting method, Apparatus and system |
| CN106504402A (en) * | 2016-11-08 | 2017-03-15 | 深圳怡化电脑股份有限公司 | Log information recording method and device |
| CN106708578A (en) * | 2016-12-23 | 2017-05-24 | 北京五八信息技术有限公司 | Dual-thread-based journal output method and device |
| CN106649834A (en) * | 2016-12-29 | 2017-05-10 | 北京奇虎科技有限公司 | Asynchronous transmission method, device and system of log data |
| CN108804295A (en) * | 2017-04-28 | 2018-11-13 | 北京京东尚科信息技术有限公司 | log information recording method and device |
| CN108234595A (en) * | 2017-11-29 | 2018-06-29 | 山东鲁能软件技术有限公司 | Log transmission method and system |
| CN108399186A (en) * | 2018-01-12 | 2018-08-14 | 联动优势科技有限公司 | A kind of collecting method and device |
| WO2020000745A1 (en) * | 2018-06-29 | 2020-01-02 | 平安科技（深圳）有限公司 | Log management method and apparatus, computer device, and storage medium |
| CN109491862A (en) * | 2018-10-24 | 2019-03-19 | 聚好看科技股份有限公司 | A kind of service requesting information collection method and device based on cloud platform |
| CN112612760A (en) * | 2020-12-30 | 2021-04-06 | 中国农业银行股份有限公司 | Log message output method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102447633A (en) | Log transmission method and system | |
| CN105373448B (en) | The restoration methods and system of fault data in database | |
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| CN101997823B (en) | Distributed file system and data access method thereof | |
| JP7230083B2 (en) | Method and apparatus for map service testing | |
| WO2020233212A1 (en) | Log record processing method, server, and storage medium | |
| CN103051692B (en) | Support the mobile operating system works platform of limit network environment | |
| CN109309631A (en) | A kind of method and device based on universal network file system write-in data | |
| CN105630810B (en) | A method of mass small documents are uploaded in distributed memory system | |
| US9501377B2 (en) | Generating and implementing data integration job execution design recommendations | |
| CN109241414A (en) | System message method for pushing and terminal device based on message template | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| EP2696297B1 (en) | System and method for generating information file based on parallel processing | |
| US20230114100A1 (en) | Small file restore performance in a deduplication file system | |
| CN106095957A (en) | Cross-domain many wave files synchronous method of distributed file system and device | |
| CN102722405A (en) | Counting method in high concurrent and multithreaded application and system | |
| CN114968953A (en) | Log storage and retrieval method, system, terminal equipment and medium | |
| US8539453B2 (en) | Autonomic I/O tracing and performance tuning | |
| CN113656673A (en) | Master-slave distributed content crawling robot for advertisement delivery | |
| CN109992469B (en) | Method and device for merging logs | |
| CN102916987A (en) | Data acquisition and management system based on cloud computing framework | |
| CN107357526A (en) | For the method and apparatus of network data, server and storage medium | |
| CN113343312A (en) | Page tamper-proofing method and system based on front-end point burying technology | |
| CN202268912U (en) | Data management system based on cloud computation structure | |
| CN112650755A (en) | Data storage method, method for querying data, database and readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20120509 |