CN106776249B - Method and system for processing concurrently generated service logs - Google Patents
Method and system for processing concurrently generated service logs Download PDFInfo
- Publication number
- CN106776249B CN106776249B CN201611071496.7A CN201611071496A CN106776249B CN 106776249 B CN106776249 B CN 106776249B CN 201611071496 A CN201611071496 A CN 201611071496A CN 106776249 B CN106776249 B CN 106776249B
- Authority
- CN
- China
- Prior art keywords
- log
- log area
- service
- query
- condition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3404—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for parallel or distributed programming
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention reduces the system pressure by processing the concurrently generated service logs and separating the buffer log area from the physical storage, thereby reducing the problem of slow response of the system and improving the insertion and storage efficiency of the service logs. The method comprises the following steps: writing the service logs generated concurrently into a buffer log area; judging whether the state of the buffering log area reaches a preset condition or not; when the state of the buffering log area reaches a preset condition, transferring the service log in the buffering log area to a long-term storage medium; and updating the query result set of the awakening log area according to the existing query conditions in the awakening log area so as to realize query for the service log.
Description
Technical Field
The present invention relates to the field of Java Web, and more particularly, to a method and system for processing a concurrently generated service log.
Background
In conventional web projects, various log outputs are often required to help management personnel locate problems, monitor hazardous operations, and the like. General logs fall into two categories:
(1) system logging: system level and code level debugging information used for system debugging and program debugging;
(2) service log: the method refers to the relevant operation records of the service category and the security authority change, and is used for monitoring key operations of users and the like.
At present, the common practice is to add extra log codes into service processing codes to insert log records into a service log table of a user database, and although the method has the advantages of simple and understandable codes, high real-time performance and the like in a low-concurrency scene, the method is not satisfactory in a high-concurrency environment. In a daily use scene, a service log can be called at a high frequency, after the user amount reaches a certain amount, database reading and writing and data source connection request can be frequently carried out, great burden is caused to an application program and a database, the system response is slow and even blocked, the service function response is blocked, and the user experience is seriously reduced.
Disclosure of Invention
The invention aims to provide a method and a system for processing a concurrently generated service log, which solve the problems of slow system response and the like caused by system pressure when the service log is inserted and inquired at high concurrency.
According to an aspect of the present invention, there is provided a method for processing a concurrently generated traffic log, the method comprising,
writing the service logs generated concurrently into a buffer log area;
judging whether the state of the buffering log area reaches a preset condition or not;
when the state of the buffering log area reaches a preset condition, transferring the service log in the buffering log area to a long-term storage medium; and
and updating the query result set of the awakening log area according to the existing query conditions in the awakening log area so as to realize query for the service log.
Preferably, the preset conditions are: and whether the data size of the service log cached in the log buffer area reaches a critical value or not.
Preferably, the preset conditions are: whether the interval time satisfies a predetermined period of time for performing the synchronization operation.
Preferably, when the interval time meets a predetermined time period for performing the synchronization operation, if no data exists in the buffer log area, the synchronization is interrupted and the timing is restarted.
Preferably, the service log is written to the buffered log area by calling the insertion interface.
Preferably, the data stored in the wake-up log area is a corresponding relation between a query condition and a query result set, where the query condition does not include a paging condition and a time dimension.
Preferably, the query for the service log is:
obtaining a set R1 of service logs meeting the query condition in the buffer log area;
inquiring whether an inquiry result set meeting the inquiry condition exists in the awakening log area or not;
if the inquiry result set meeting the inquiry condition exists in the awakening log area, acquiring a set R2 of the service log according to the content of the inquiry result set;
if the inquiry result set meeting the inquiry condition does not exist in the awakening log area, inquiring in the long-term storage medium, acquiring a set R2 of the service log meeting the inquiry condition, and updating the inquiry result set of the awakening log area;
calculate the union R of R1 and R2-And to R-And filtering the time dimension and the paging condition to obtain a set R of all the service logs meeting the query condition.
Preferably, the filtering method is an iterative traversal method.
Preferably, if the query condition does not exist in the wakeup log area, the query result set of the wakeup log area is not updated.
According to another aspect of the present invention, there is provided a processing system for concurrently generated traffic logs, comprising:
the log writing unit is used for writing the service logs generated concurrently into the buffer log area;
the state judgment unit is used for judging whether the state of the buffering log area reaches a preset condition or not;
the unloading unit is used for unloading the service logs in the buffer log area into a long-term storage medium for storing the service logs for a long term;
the log query unit is used for acquiring the service logs in the buffer log area, the awakening log area and the long-term storage medium according to query conditions and obtaining query results;
and the awakening log area is used for storing the corresponding relation between the query condition and the query result set.
Preferably, the buffer log area is cached in list form.
Preferably, the wake-up log area is stored in a map form.
Preferably, the unloading unit unloads when the state judging unit judges that the state of the buffer log area reaches a preset condition.
Preferably, the preset conditions are: and whether the data size of the service log cached in the log buffer area reaches a critical value or not.
Preferably, the preset conditions are: whether the interval time satisfies a predetermined period of time for performing the synchronization operation.
The invention separates the insertion and query service logs from the physical storage of the logs through the buffer log area, the speed of insertion and query in the buffer log area is increased compared with the speed of storage and query in a file system or a database system, and the insertion and query service logs are centralized and multi-threaded when being synchronized to a long-term storage medium, thereby bringing less pressure to the system and reducing the problem of slow response of the system.
Drawings
A more complete understanding of exemplary embodiments of the present invention may be had by reference to the following drawings in which:
FIG. 1 is a flow chart of a processing method according to a preferred embodiment of the present invention;
FIG. 2 is a flow chart of a method for querying a service log according to a preferred embodiment of the present invention;
fig. 3 is a block diagram of a processing system in accordance with a preferred embodiment of the present invention.
Detailed Description
The exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, however, the present invention may be embodied in many different forms and is not limited to the embodiments described herein, which are provided for complete and complete disclosure of the present invention and to fully convey the scope of the present invention to those skilled in the art. The terminology used in the exemplary embodiments illustrated in the accompanying drawings is not intended to be limiting of the invention. In the drawings, the same units/elements are denoted by the same reference numerals.
Unless otherwise defined, terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Further, it will be understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense.
FIG. 1 is a flow chart of a processing method according to a preferred embodiment of the present invention. As shown in fig. 1, a method 100 for processing concurrently generated traffic logs begins at step 101. In step 101, writing the concurrently generated service log into the buffer log area for buffering by calling the insertion interface. Preferably, the cache log area may be a local memory or an independent cache server, because the speed of reading and writing by the memory or the independent cache server is high, when there are many service logs generated at high concurrency, the service logs can be cached quickly, and the problems of slow system response and the like caused by frequent writing of the service logs are reduced.
In step 102, determining whether the state of the cache log area reaches a preset condition, and if the state of the cache log area reaches the preset condition, performing step 103; and if the state of the cache log area does not reach the preset condition, not operating. Preferably, the preset condition is whether the data size of the service log cached in the log buffer area reaches a critical value; or, whether the interval time meets the predetermined time period for performing the synchronization operation may be determined, and when the interval time meets the predetermined time period for performing the synchronization operation, if there is no data in the buffer log area, the synchronization is interrupted and the timer is restarted.
In step 103, the service log in the buffer log area is transferred to a long-term storage medium for long-term storage. Preferably, the long-term storage medium may be a database or other storage medium that can achieve the same effect, and the read-write speed of the long-term storage medium is slower than that of a local memory or an independent cache server, but data storage can be performed safely for a long time.
In step 104, the query result set in the wakeup log area is updated according to the query conditions existing in the wakeup log area, so as to implement the query for the service log. Preferably, the data stored in the wakeup log area is a corresponding relation between the query condition and the query result set, that is, a key-value relation pair, where the key is the query condition, the query condition does not include a paging condition and a time dimension, and the value is the query result set. When the awakening log area is updated, the number of the service log which is newly transferred to the long-term storage medium and meets the query condition is added into the corresponding value according to the existing key of the awakening log area.
Fig. 2 is a flowchart of a method for querying a service log according to a preferred embodiment of the present invention. As shown in fig. 2, a method 200 for querying a traffic log begins at step 201. In step 201, according to the query condition, a set R1 of service logs meeting the query condition in the buffer log area is obtained first. Preferably, a part of the service logs which are not transferred to the long-term storage medium are cached in the buffer log area, and when the service logs need to be queried, because all the generated service logs are queried, and a part of the logs are stored in the buffer log area and the long-term storage medium, the logs meeting the query conditions in the buffer log area are obtained first.
In step 202, inquiring whether an inquiry result set meeting the inquiry condition exists in the awakening log area, and if so, performing step 203; if not, proceed to step 204. Preferably, the query result set meeting the query condition here means that there is a value in the wakeup log area, and the key corresponding to the value is completely consistent with the query condition of this time, for example, the query condition of this time is "high-risk operation with little human operator", and if there is a key in the wakeup log area as "high-risk operation with little human operator", the service log corresponding to the value in the value corresponding to the key is returned as the result R2 of this query.
When the query result set meeting the query condition exists in the wakeup log area, step 203 is performed. In step 203, the query result set meeting the query conditions in the wake log area is returned as R2. Preferably, when the service logs are transferred from the buffered log area to the long-term storage medium, the update operation of the wakeup log area is performed synchronously, and if a key in the wakeup log area is consistent with the query condition of this time, the set of the service logs corresponding to the value of the value thereof must represent the set of all the service logs meeting the query condition in the long-term storage medium, that is, the set R2.
When there is no query result set meeting the query condition in the wakeup log area, go to step 204. In step 204, if there is no query result set meeting the query condition in the wakeup log area, querying the service log meeting the condition in the long-term storage medium, and obtaining a set R2 of query results. Preferably, the long-term storage medium queries the service log stored in the long-term storage medium only when there is no query result set meeting the query condition in the wakeup log area due to the reasons of large storage capacity, relatively slow read-write speed, and the like.
When step 204 is completed, step 205 is performed. And updating the data of the awakening log area according to the set of the service logs meeting the query conditions in the long-term storage medium, adding a new entry, and storing the corresponding relation of taking the query conditions as keys and the query result in the long-term storage medium as values so as to be directly applied in the next query.
When step 203 or 205 is completed, step 206 is performed. In step 206, the union R of R1 and R2 is calculated-And to R-And filtering the time dimension and the paging condition to obtain a set R of all the service logs meeting the query condition. Preferably, the filtering method is an iterative traversal method, and the filtering is performed in the order of performing the paging condition after the time dimension.
Fig. 3 is a block diagram of a processing system in accordance with a preferred embodiment of the present invention. As shown in fig. 3, the processing system 300 is composed of a log writing unit 301, a state judgment unit 302, a dump unit 303, a log query unit 304, a buffer log area 305, a wakeup log area 306, and a long-term storage medium 307.
Preferably, the log writing unit 301 is configured to write the concurrently generated service log into the buffered log area 305, so the log writing unit may be an interface inserted in the Java Web or another interface capable of achieving a writing effect.
Preferably, the state determination unit 302 is configured to determine whether the state of the buffer log area 305 reaches a preset condition, and when the state of the buffer log area 305 reaches the preset condition, the unloading unit 303 unloads all data in the buffer log area 305 to the long-term storage medium 307. Preferably, the preset condition is whether the data size of the service log cached in the buffer log area 305 reaches a critical value; it may be determined whether the interval time satisfies a predetermined time period for performing the synchronization operation, and if there is no data in the buffer log area 305 when the interval time satisfies the predetermined time period for performing the synchronization operation, the unloading operation may be interrupted and the time may be counted again.
Preferably, the log query unit 304 is mainly used for obtaining the service logs in the buffer log area 305, the wakeup log area 306 and the long-term storage medium 307 during query. In the obtaining process, firstly, a set R1 of the service logs meeting the query condition in the buffer log area 305 is obtained, secondly, it is determined whether a query result set meeting the query condition exists in the wakeup log area 306, if so, a service log set R2 corresponding to the query result set is obtained, and if not, a service log set R2 meeting the query condition in the long-term storage medium 307 is obtained. And finally, calculating a union set of R1 and R2, and filtering the time dimension and the paging condition to obtain a set R of all service logs meeting the query condition.
Preferably, the buffering log area 305 caches the service log in a list form, and the wakeup log area stores the key-value relationship pair in a map form.
The invention has been described with reference to a few embodiments. However, other embodiments of the invention than the one disclosed above are equally possible within the scope of the invention, as would be apparent to a person skilled in the art from the appended patent claims.
Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a/an/the [ device, component, etc ]" are to be interpreted openly as referring to at least one instance of said device, component, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.
Claims (13)
1. A method for processing concurrently generated traffic logs, the method comprising,
writing the service logs generated concurrently into a buffer log area;
judging whether the state of the buffering log area reaches a preset condition or not;
when the state of the buffering log area reaches a preset condition, transferring the service log in the buffering log area to a long-term storage medium; when the service log is transferred from the buffering log area to the long-term storage medium, the updating operation of the awakening log area is synchronously performed; and
updating the query result set of the awakening log area according to the existing query conditions in the awakening log area so as to realize query for the service log;
the data stored in the wake-up log area is the corresponding relation between a query condition and a query result set, wherein the query condition does not include a paging condition and a time dimension;
the query for the service log is:
obtaining a set R1 of service logs meeting the query condition in the buffer log area;
inquiring whether an inquiry result set meeting the inquiry condition exists in the awakening log area or not;
if the inquiry result set meeting the inquiry condition exists in the awakening log area, acquiring a set R2 of the service log according to the content of the inquiry result set;
if the inquiry result set meeting the inquiry condition does not exist in the awakening log area, inquiring in the long-term storage medium, acquiring a set R2 of the service log meeting the inquiry condition, and updating the inquiry result set of the awakening log area;
and calculating the union R-of R1 and R2, and filtering the R-by time dimension and paging conditions to obtain a set R of all service logs meeting the query conditions.
2. The method according to claim 1, wherein the preset condition is: and whether the data size of the service log cached in the log buffer area reaches a critical value or not.
3. The method according to claim 1, wherein the preset condition is: whether the interval time satisfies a predetermined period of time for performing the synchronization operation.
4. The method of claim 3, wherein when the interval time satisfies a predetermined time period for performing the synchronization operation, if there is no data in the buffered log area, the synchronization is interrupted and the timing is restarted.
5. The method of claim 1, wherein the service log is written to the buffered log area by calling an insertion interface.
6. The method of claim 1, wherein the filtering method is an iterative traversal method.
7. The method of claim 1, wherein the query result set of the wake log area is not updated if the query condition does not exist in the wake log area.
8. A processing system for concurrently generated traffic logs, comprising:
the log writing unit is used for writing the service logs generated concurrently into the buffer log area;
the state judgment unit is used for judging whether the state of the buffering log area reaches a preset condition or not;
the unloading unit is used for unloading the service logs in the buffer log area into a long-term storage medium for storing the service logs for a long term; when the service log is transferred from the buffering log area to the long-term storage medium, the updating operation of the awakening log area is synchronously performed;
the log query unit is used for acquiring the service logs in the buffer log area, the awakening log area and the long-term storage medium according to query conditions and obtaining query results;
the wake-up log area is used for storing the corresponding relation between the query condition and the query result set; the query condition does not include a paging condition and a time dimension;
the query for the service log is:
obtaining a set R1 of service logs meeting the query condition in the buffer log area;
inquiring whether an inquiry result set meeting the inquiry condition exists in the awakening log area or not;
if the inquiry result set meeting the inquiry condition exists in the awakening log area, acquiring a set R2 of the service log according to the content of the inquiry result set;
if the inquiry result set meeting the inquiry condition does not exist in the awakening log area, inquiring in the long-term storage medium, acquiring a set R2 of the service log meeting the inquiry condition, and updating the inquiry result set of the awakening log area;
and calculating the union R-of R1 and R2, and filtering the R-by time dimension and paging conditions to obtain a set R of all service logs meeting the query conditions.
9. The system of claim 8, wherein the buffered log area is cached in list form.
10. The system of claim 8, wherein the wake log area is stored in a map.
11. The system according to claim 8, wherein the unloading unit unloads when the status determining unit determines that the status of the buffered log area reaches a preset condition.
12. The system according to claim 8, wherein the preset condition is: and whether the data size of the service log cached in the log buffer area reaches a critical value or not.
13. The system according to claim 8, wherein the preset condition is: whether the interval time satisfies a predetermined period of time for performing the synchronization operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611071496.7A CN106776249B (en) | 2016-11-28 | 2016-11-28 | Method and system for processing concurrently generated service logs |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611071496.7A CN106776249B (en) | 2016-11-28 | 2016-11-28 | Method and system for processing concurrently generated service logs |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106776249A CN106776249A (en) | 2017-05-31 |
| CN106776249B true CN106776249B (en) | 2020-07-28 |
Family
ID=58905360
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611071496.7A Active CN106776249B (en) | 2016-11-28 | 2016-11-28 | Method and system for processing concurrently generated service logs |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106776249B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108197233A (en) * | 2017-12-29 | 2018-06-22 | 飞狐信息技术（天津）有限公司 | A kind of data managing method, middleware and data management system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102200972A (en) * | 2010-03-24 | 2011-09-28 | 腾讯科技（北京）有限公司 | Database maintenance method and database maintenance equipment |
| CN103164511A (en) * | 2013-02-21 | 2013-06-19 | 烽火通信科技股份有限公司 | Method for storage event log to automatically filter repeated jitter data |
| CN105488201A (en) * | 2015-12-08 | 2016-04-13 | 北京皮尔布莱尼软件有限公司 | Log inquiry method and system |
-
2016
- 2016-11-28 CN CN201611071496.7A patent/CN106776249B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102200972A (en) * | 2010-03-24 | 2011-09-28 | 腾讯科技（北京）有限公司 | Database maintenance method and database maintenance equipment |
| CN103164511A (en) * | 2013-02-21 | 2013-06-19 | 烽火通信科技股份有限公司 | Method for storage event log to automatically filter repeated jitter data |
| CN105488201A (en) * | 2015-12-08 | 2016-04-13 | 北京皮尔布莱尼软件有限公司 | Log inquiry method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106776249A (en) | 2017-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10133679B2 (en) | Read cache management method and apparatus based on solid state drive | |
| WO2019085471A1 (en) | Database synchronization method, application server, and computer readable storage medium | |
| CN106462592B (en) | System and method for optimizing multi-version support for indexes | |
| CA2491731A1 (en) | System and method for caching data for a mobile application | |
| US20060248125A1 (en) | Method and system for data processing with data backup | |
| CN112948398B (en) | Hierarchical storage system and method for cold and hot data | |
| JP7101566B2 (en) | Multiversion Concurrency Control (MVCC) in non-volatile memory | |
| US10621104B2 (en) | Variable cache for non-volatile memory | |
| EP2797014A1 (en) | Database update execution according to power management schemes | |
| EP2541423A1 (en) | Replacement policy for resource container | |
| US20110145186A1 (en) | Online access to database snapshots | |
| CN111221828A (en) | Method and terminal for improving consistency of database data and cache data | |
| CN114706836B (en) | Data life cycle management method based on airborne embedded database | |
| CN106776249B (en) | Method and system for processing concurrently generated service logs | |
| CN110554914B (en) | Resource lock management method, device, server and storage medium | |
| US7240065B2 (en) | Providing mappings between logical time values and real time values | |
| US9317432B2 (en) | Methods and systems for consistently replicating data | |
| US11860840B2 (en) | Update of deduplication fingerprint index in a cache memory | |
| CN115129618A (en) | Method and apparatus for optimizing data caching | |
| US7251660B2 (en) | Providing mappings between logical time values and real time values in a multinode system | |
| CN107506147B (en) | Method and system for keeping consistency of metadata | |
| CN108038121B (en) | Caching method and terminal | |
| US10372699B2 (en) | Patch-up operations on invalidity data | |
| KR101932967B1 (en) | Method for flushing metadata in journaling file system and apparatus for writing data using the method | |
| CN116719851A (en) | Redis cache elimination method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |