CN112306827A - Log collection device, method and computer readable storage medium - Google Patents
Log collection device, method and computer readable storage medium Download PDFInfo
- Publication number
- CN112306827A CN112306827A CN202010218588.3A CN202010218588A CN112306827A CN 112306827 A CN112306827 A CN 112306827A CN 202010218588 A CN202010218588 A CN 202010218588A CN 112306827 A CN112306827 A CN 112306827A
- Authority
- CN
- China
- Prior art keywords
- log
- queue
- pointer
- circular queue
- log collection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000004044 response Effects 0.000 claims abstract description 8
- 230000008569 process Effects 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 11
- 238000012545 processing Methods 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 22
- 238000007726 management method Methods 0.000 description 7
- 230000000903 blocking effect Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3442—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for planning or managing the needed capacity
Abstract
The invention discloses a log collection device, a log collection method and a computer readable storage medium, and relates to the technical field of computers. The log collection method comprises the following steps: in response to obtaining the log, writing the log into the circular queue through a plurality of enqueue threads, circularly trying to obtain an enqueue pointer corresponding to the circular queue and writing the log into a position next to the position pointed by the enqueue pointer when the log is written through each enqueue thread until the log is successfully written, and successfully writing the log in the condition that the position next to the position pointed by the enqueue pointer is empty; reading logs from the circular queue according to the dequeue pointers corresponding to the circular queue; and sending the read log to a server. The invention can improve the efficiency and the processing performance of log acquisition and meet the requirements of high throughput and low time delay.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a log collection device, a log collection method, and a computer-readable storage medium.
Background
In many internet client (or C-side for short) service systems, it is often necessary to record a service data log, which can be used to record behavior data of the service system, so as to trace problems and retain traces. For example, the log may record core state changes, external system calls. Besides, the logs can be used for calculating and analyzing trends in real time.
Such logging behavior has several characteristics or requirements:
(1) the method does not belong to the core business process, and the influence on the core business process is reduced as much as possible;
(2) the calling amount is large, and the occupied space of log storage is large;
(3) the requirement of part of service logs on timeliness is high, and the service logs need to be recorded as soon as possible so as to conduct data analysis such as order quantity real-time prediction and state change tracks as soon as possible.
In the related art, the following log collection scheme may be used. For example, an independent service log service may be created, and the service system writes the log into the log service synchronously through a Remote Procedure Call (RPC) interface. As another example, log collection is accomplished using a third party tool. Printing the log to a local disk through a log frame, such as log frame log4j for JAVA, and then extracting a log file using a log collection agent (agent) and forwarding the extracted log file to a background log service; or, the log frame is utilized and matched with a log collecting tool such as kafka, the log is printed and input into the log collecting tool and is forwarded to the kafka server side for message consumption.
Disclosure of Invention
After the analysis, the inventor finds that, in the related art, the call performance of the core service is interfered by the RPC synchronous call mode, which causes instability of the service system. The method of using the third-party tool is not flexible enough in deployment, and the timeliness of the logging cannot be satisfied.
The embodiment of the invention aims to solve the technical problem that: how to improve the system stability and timeliness in the log collection process.
According to a first aspect of some embodiments of the present invention, there is provided a log collecting method, including: in response to obtaining the log, writing the log into the circular queue through a plurality of enqueue threads, circularly trying to obtain an enqueue pointer corresponding to the circular queue and writing the log into a position next to the position pointed by the enqueue pointer when the log is written through each enqueue thread until the log is successfully written, and successfully writing the log in the condition that the position next to the position pointed by the enqueue pointer is empty; reading logs from the circular queue according to the dequeue pointers corresponding to the circular queue; and sending the read log to a server.
In some embodiments, each circular queue corresponds to one dequeue thread; reading the log from the circular queue according to the dequeue pointer corresponding to the circular queue comprises: and acquiring logs in the corresponding annular queue in batch by using the dequeuing thread and atomic operation, and emptying the read logs in the corresponding annular queue.
In some embodiments, there are a plurality of circular queues; the log collection method further comprises the following steps: determining a hash value of the time of acquiring the log; and distributing the log to a ring queue corresponding to the hash value for storage.
In some embodiments, the log collection method further comprises: and in the case that the position next to the position pointed by the enqueue pointer is not empty, allocating other ring queues for the corresponding logs to store.
In some embodiments, the log collection method further comprises: initializing a preset number of circular queues; in response to the current queues all being full, a new circular queue is created.
In some embodiments, the log collection method further comprises: and if the number of the current ring queues is higher than the number of the initial queues and the time length of the last log written time of the ring queues from the current time exceeds a threshold value, destroying the corresponding ring queues.
In some embodiments, the log collection method further comprises: after an instruction for finishing a process corresponding to the log acquisition service is acquired, locally storing the logs in the ring queue; and when the log collection service is started next time, sending the locally stored message to the server.
In some embodiments, the circular queue is implemented by an atomic array structure.
According to a second aspect of some embodiments of the present invention, there is provided a log collecting apparatus, including: a log collection interface configured to obtain a log; the ring queue is configured to store logs, and an enqueue pointer corresponding to the ring queue points to the last written position in the ring queue and a dequeue pointer points to the last read position in the ring queue; an enqueuing module comprising a plurality of enqueuing threads, each enqueuing thread configured to cyclically attempt to acquire an enqueuing pointer and write a log to a location next to the location pointed by the enqueuing pointer until the write is successful, the write being successful if the location next to the location pointed by the enqueuing pointer is empty; the dequeue module is configured to read the log from the circular queue according to the dequeue pointer and send the read log to the log sending component; and the log sending component is configured to send the acquired log to the server.
According to a third aspect of some embodiments of the present invention, there is provided a log collecting apparatus, including: a memory; and a processor coupled to the memory, the processor configured to perform any of the aforementioned log collection methods based on instructions stored in the memory.
According to a fourth aspect of some embodiments of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any one of the aforementioned log collection methods.
Some embodiments of the above invention have the following advantages or benefits: the invention can realize asynchronous log collection. In addition, in the process of temporarily storing the log to the local, the stability and the timeliness of the system in the log acquisition process are improved by utilizing the annular queue and the lock-free and non-blocking multithread operation, and the requirements of high throughput and low time delay are met.
Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 illustrates a flow diagram of a log collection method according to some embodiments of the inventions.
FIG. 2 illustrates a flow diagram of a circular queue allocation method according to some embodiments of the invention.
FIG. 3 illustrates a flow diagram of a resiliency management method according to some embodiments of the invention.
FIG. 4 illustrates a flow diagram of a service exit handling method according to some embodiments of the invention.
FIG. 5 illustrates a schematic diagram of a log collection apparatus according to some embodiments of the invention.
Fig. 6 is a schematic structural diagram of a log collection apparatus according to other embodiments of the present invention.
FIG. 7 illustrates a schematic diagram of a log collection apparatus according to further embodiments of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
FIG. 1 illustrates a flow diagram of a log collection method according to some embodiments of the inventions. As shown in fig. 1, the log collection method of this embodiment includes steps S102 to S106.
In step S102, in response to acquiring the log, writing the log into the circular queue by the plurality of enqueue threads, and when the log is written by each enqueue thread, circularly attempting to acquire an enqueue pointer corresponding to the circular queue and writing the log into a position next to a position pointed by the enqueue pointer until the writing is successful, wherein the writing is successful in a case where the position next to the position pointed by the enqueue pointer is empty. After the write is successful, the enqueue thread updates the write pointer so that the write pointer points to the location of the data written by the enqueue thread.
The circular queue is used for storing logs. Each circular queue corresponds to an enqueue pointer and a dequeue pointer. For example, the enqueue pointer and dequeue pointer are represented by pushIndex and pullIndex, respectively. The circular queue may be implemented by an array. The pushIndex represents the subscript value of the latest log enqueued in the array. Each log is enqueued, the pushIndex increments by 1, and returns to 0 when the value of the pushIndex grows to the array capacity boundary, thereby forming a ring-type data structure. The pullIndex represents the subscript value in the array of the latest log of the dequeue of the message. Each dequeue of a log, pullIndex increments by 1. The value of pullIndex will also return to 0 when it grows to the array capacity boundary. Thus, data may be stored to or read from the queue sequentially at all times. The structure is suitable for the log temporary storage scene.
In some embodiments of the present invention, the,
in some embodiments, one or more circular queues are provided for staging logs as needed. Each queue corresponds to a plurality of enqueue threads for writing logs.
Thread pools in the related art (e.g., JAVA thread pools) typically use a blocking queue. When the amount of enqueue messages is large, large overhead is caused by inter-thread switching and lock contention caused by blocking, and therefore the processing speed of the messages is reduced. And the manner adopted by step S102 avoids lock contention and thread switching.
Ideally, the next position where the enqueue pointer points to a position should be empty. And if the next position is not empty, the log is written into the next position, which indicates that other threads have preempted the log. At this time, the enqueue pointer does not need to be reloaded, and only the process of obtaining the enqueue pointer and judging whether the next position of the position pointed by the enqueue pointer is empty is needed to be repeated until the writing condition is met and the writing is successful. This approach does not require locking and blocking and also maintains log write accuracy.
In some embodiments, the circular queue is implemented by an atomic array structure, that is, operations on the circular queue are all atomic, which ensures the accuracy of the operations. In some embodiments, the write process may be implemented using a Compare And Set (CAS) mechanism.
In step S104, the log is read from the circular queue according to the dequeue pointer corresponding to the circular queue.
In some embodiments, each circular queue corresponds to a dequeue thread. The dequeue thread reads messages by polling without stalling. In this case, there is no contention between threads, and therefore, the locking process is not required.
In some embodiments, the logs in the corresponding ring queue are obtained in batch by using the atomic operation through the dequeue thread, and the read logs in the corresponding ring queue are emptied. Thus, the timeliness of log reading can be improved. And the atomic operation ensures the accuracy of log reading.
Pointers of the circular queue need to be recycled. In order to increase the calculation speed, an exponent having a capacity of 2 may be set to calculate the pointer by way of bit operations, thereby increasing the calculation efficiency. In some embodiments, the enqueue and dequeue pointers are represented in binary form. Taking the enqueue pointer as an example, after the log is written and the value of the enqueue pointer is added with 1, the enqueue pointer and the binary representation of the capacity of the circular queue minus 1 are subjected to and operation. When the capacity of the circular queue is exceeded after the enqueue pointer is added by 1, the enqueue pointer can return to the initial value through the operation, and the circular effect is achieved.
In step S106, the read log is transmitted to the server. For example, the log may be sent to the server via Hypertext Transfer Protocol (HTTP).
By the method of the embodiment, the asynchronous log collection can be realized. In addition, in the process of temporarily storing the log to the local, the stability and the timeliness of the system in the log acquisition process are improved by utilizing the annular queue and the lock-free and non-blocking multithread operation, and the requirements of high throughput and low time delay are met.
When the number of the ring queues is multiple, the ring queues can be allocated for the logs to be written to be stored. An embodiment of a circular queue allocation method is described below with reference to fig. 2.
FIG. 2 illustrates a flow diagram of a circular queue allocation method according to some embodiments of the invention. As shown in fig. 2, the circular queue allocation method of this embodiment includes steps S202 to S204.
In step S202, a hash value of the time at which the log is acquired is determined. For example, the time to acquire the log is divided by the total number of the ring queues, and the remainder is used as the hash value.
In step S204, the log is assigned to the ring queue corresponding to the hash value for storage.
If the assigned ring queue is full, the log may be reassigned to another ring queue for storage, e.g., to the next ring queue. In some embodiments, a determination is made as to whether the location next to the location pointed to by the enqueue pointer is empty. And if the log is not empty, allocating other ring queues for the corresponding logs to be stored.
By the method of the embodiment, the logs can be uniformly distributed among the annular queues as much as possible.
In order to adapt to the condition of increasing the number of logs, the invention can also realize the elastic capacity expansion of the ring queue. An embodiment of the resiliency management method of the invention is described below with reference to fig. 3.
FIG. 3 illustrates a flow diagram of a resiliency management method according to some embodiments of the invention. As shown in fig. 3, the elasticity management method of this embodiment includes steps S302 to S304.
In step S302, a preset number of circular queues are initialized. The preset number may be configured in advance.
In step S304, a new ring queue is created in response to the current queues all being full.
In some embodiments, an upper limit on the number of circular queues may be set. When the created ring queue reaches the upper limit, but new logs are generated, the logs which cannot be stored in the ring queue can be directly sent to the server. Therefore, under the condition of higher log operation pressure, the synchronous and asynchronous modes of log recording are combined to reduce the influence on the system performance.
The invention can also realize the elastic capacity shrinkage of the circular queue. For example, the elasticity management method may further include step S306.
In step S306, in a case where the number of the current ring queues is higher than the initial queue number, and the time length of the last log written of the ring queue from the current time exceeds a threshold, the corresponding ring queue is destroyed.
Therefore, when the log operation pressure is reduced, the circular queues can be gradually destroyed, and finally the number of the circular queues is kept at the initial level, so that the performance of the system is improved.
When the log collection service exits, there may be unread messages in the ring queue. The present invention can appropriately cope with such a situation. An embodiment of the service logout processing method of the present invention is described below with reference to fig. 4.
FIG. 4 illustrates a flow diagram of a service exit handling method according to some embodiments of the invention. As shown in fig. 4, the service logout processing method of the embodiment includes steps S402 to S404.
In step S402, after acquiring an instruction to end a process corresponding to the log collection service, the logs in the ring queue are locally stored.
In some embodiments, after the instruction for ending the process corresponding to the log collection service is obtained, the log in the ring queue may be tried to be dequeued first, and the main process is instructed to wait; and after the waiting time exceeds the preset waiting time, if the data still exist in the circular queue, locally storing the logs.
In step S404, when the log collection service is started next time, a locally stored message is sent to the server.
In some embodiments, when reading from or writing to locally stored logs, a random access file (RandomAccessFile) class may be used for processing. In the processing process, the read-write pointer of the file can be recorded, and the end-point continuous transmission of the file content is supported, so that the log uploading stability is improved. In some embodiments, when a read operation is performed, the read data is copied to avoid the read and write threads interfering with each other.
By the method of the embodiment, safe and reliable log collection service exit can be realized.
An embodiment of the log collection apparatus of the present invention is described below with reference to fig. 5.
FIG. 5 illustrates a schematic diagram of a log collection apparatus according to some embodiments of the invention. As shown in fig. 5, the log collection apparatus 50 of this embodiment includes a log collection interface 510, a circular queue 520, an enqueue module 530, a dequeue module 540, and a log sending component 550.
The log collection interface 510 is configured to obtain logs.
In some embodiments, the log collection device 50 may be packaged as a module, such as in the form of a Jar package, library file, package, or the like. The business process initiates a call to the module through the log collection interface 510 to transfer in the log to be recorded. Therefore, the log processing logic does not need to be integrated in the service logic, and only the log collection device 50 needs to be introduced into the service system and the service identifier needs to be configured, so that the server can distinguish from which service system the received log is obtained.
The circular queue 520 is configured to store a log, wherein an enqueue pointer corresponding to the circular queue points to a last written location in the circular queue and a dequeue pointer points to a last read location in the circular queue.
The enqueuing module 530 includes a plurality of enqueuing threads, each enqueuing thread configured to cycle through attempts to obtain an enqueuing pointer and write a log to a location next to the location pointed to by the enqueuing pointer until the write is successful, wherein the write is successful if the location next to the location pointed to by the enqueuing pointer is empty.
The dequeue module 540 is configured to read the log from the circular queue according to the dequeue pointer and send the read log to the log sending component.
The log sending component 550 is configured to send the obtained log to a server.
In some embodiments, the dequeue module 540 includes dequeue threads, one for each circular queue; the dequeue thread is configured to obtain the logs in the corresponding ring queue in batches through atomic operation, and empty the read logs in the corresponding ring queue.
In some embodiments, the log collection device 50 further comprises: and a queue allocation module 560 configured to allocate the ring queue for storage for the logs obtained by the log collection interface 510.
In some embodiments, the queue allocation module 560 is further configured to determine a hash value of a time when the log collection interface 510 obtains the log, and allocate the log to a circular queue corresponding to the hash value for storage.
In some embodiments, the enqueue thread is further configured to feed back a write failure to the queue assignment module 560 in the event that the next location where the enqueue pointer points to a location is not empty, such that the queue assignment module 560 assigns other circular queues for storage for the corresponding log.
In some embodiments, the log collection device 50 further comprises: a queue management module 570 configured to initialize the initial queue number of ring queues and create a new ring queue in response to the current queues all being full.
In some embodiments, the queue management module 570 is further configured to destroy the respective ring queue if the number of current ring queues is higher than the initial number of queues and the length of time that the last log of the ring queue was written to is greater than a threshold from the current time.
In some embodiments, the log collection device 50 further comprises: and the log caching module 580 is configured to, when an instruction to end the process corresponding to the log collecting apparatus 50 is obtained, locally store the logs in the ring queue, so that when the log collecting apparatus is started next time, the locally stored message is sent to the server through the log sending component.
In some embodiments, the circular queue is implemented by an atomic array structure.
Fig. 6 is a schematic structural diagram of a log collection apparatus according to other embodiments of the present invention. As shown in fig. 6, the log collection apparatus 60 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610, the processor 620 being configured to perform the log collection method of any of the foregoing embodiments based on instructions stored in the memory 610.
FIG. 7 illustrates a schematic diagram of a log collection apparatus according to further embodiments of the present invention. As shown in fig. 7, the log collection device 70 of this embodiment includes: the memory 710 and the processor 720 may further include an input/output interface 730, a network interface 740, a storage interface 750, and the like. These interfaces 730, 740, 750, as well as the memory 710 and the processor 720, may be connected, for example, by a bus 760. The input/output interface 730 provides a connection interface for input/output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 740 provides a connection interface for various networking devices. The storage interface 750 provides a connection interface for external storage devices such as an SD card and a usb disk.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement any one of the aforementioned log collection methods.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (11)
1. A log collection method, comprising:
in response to obtaining the log, writing the log into a circular queue through a plurality of enqueue threads, and when the log is written through each enqueue thread, circularly attempting to obtain an enqueue pointer corresponding to the circular queue and writing the log into a position next to a position pointed by the enqueue pointer until the writing is successful, wherein the writing is successful in the case that the position next to the position pointed by the enqueue pointer is empty;
reading logs from the circular queue according to the dequeue pointer corresponding to the circular queue;
and sending the read log to a server.
2. A log collection method as defined in claim 1 wherein each circular queue corresponds to a dequeue thread;
the reading the log from the circular queue according to the dequeue pointer corresponding to the circular queue includes:
and acquiring logs in the corresponding annular queue in batch by using the dequeue thread and atomic operation, and emptying the read logs in the corresponding annular queue.
3. A log collecting method according to claim 1, wherein there are a plurality of said circular queues;
the log collection method further comprises the following steps:
determining a hash value of the time of acquiring the log;
and distributing the log to a ring queue corresponding to the hash value for storage.
4. The log collection method of claim 1, further comprising:
and under the condition that the next position of the position pointed by the enqueue pointer is not empty, allocating other ring queues for the corresponding logs for storage.
5. The log collection method of claim 1, further comprising:
initializing a preset number of circular queues;
in response to the current queues all being full, a new circular queue is created.
6. The log collection method of claim 1, further comprising:
and if the number of the current ring queues is higher than the initial queue number and the time length of the last log written time of the ring queues from the current time exceeds a threshold value, destroying the corresponding ring queues.
7. The log collection method of claim 1, further comprising:
after an instruction for finishing the process corresponding to the log acquisition service is acquired, locally storing the logs in the annular queue;
and when the log collection service is started next time, sending the locally stored message to a server.
8. A log collection method as claimed in any one of claims 1 to 7, wherein said circular queue is implemented by an atomic array structure.
9. A log collection apparatus, comprising:
a log collection interface configured to obtain a log;
a circular queue configured to store logs, wherein an enqueue pointer corresponding to the circular queue points to a last written position in the circular queue and a dequeue pointer points to a last read position in the circular queue;
an enqueuing module comprising a plurality of enqueuing threads, each enqueuing thread configured to circularly attempt to acquire the enqueuing pointer and write the log to a location next to a location pointed by the enqueuing pointer until the write is successful, wherein the write is successful in a case where the location next to the location pointed by the enqueuing pointer is empty;
the dequeue module is configured to read the log from the circular queue according to the dequeue pointer and send the read log to the log sending component;
and the log sending component is configured to send the acquired log to the server.
10. A log collection apparatus, comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform the log collection method of any of claims 1-8 based on instructions stored in the memory.
11. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the log collection method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010218588.3A CN112306827A (en) | 2020-03-25 | 2020-03-25 | Log collection device, method and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010218588.3A CN112306827A (en) | 2020-03-25 | 2020-03-25 | Log collection device, method and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112306827A true CN112306827A (en) | 2021-02-02 |
Family
ID=74336745
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010218588.3A Pending CN112306827A (en) | 2020-03-25 | 2020-03-25 | Log collection device, method and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112306827A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114500403A (en) * | 2022-01-24 | 2022-05-13 | 中国联合网络通信集团有限公司 | Data processing method and device and computer readable storage medium |
| CN117407249A (en) * | 2023-12-12 | 2024-01-16 | 苏州元脑智能科技有限公司 | Drive log management method, device and system, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140372486A1 (en) * | 2013-06-12 | 2014-12-18 | Oracle International Corporation | In-Database Sharded Queue |
| CN106648461A (en) * | 2016-11-15 | 2017-05-10 | 努比亚技术有限公司 | Memory management device and method |
| US20180144015A1 (en) * | 2016-11-18 | 2018-05-24 | Microsoft Technology Licensing, Llc | Redoing transaction log records in parallel |
| CN108399186A (en) * | 2018-01-12 | 2018-08-14 | 联动优势科技有限公司 | A kind of collecting method and device |
| CN108509327A (en) * | 2018-04-20 | 2018-09-07 | 深圳市文鼎创数据科技有限公司 | A kind of log-output method, device, terminal device and storage medium |
| CN108710531A (en) * | 2018-04-20 | 2018-10-26 | 深圳市文鼎创数据科技有限公司 | Method for writing data, device, terminal device and the storage medium of round-robin queue |
| CN110019498A (en) * | 2017-08-14 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Log synchronous method and device, storage medium, electronic equipment |
-
2020
- 2020-03-25 CN CN202010218588.3A patent/CN112306827A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140372486A1 (en) * | 2013-06-12 | 2014-12-18 | Oracle International Corporation | In-Database Sharded Queue |
| CN106648461A (en) * | 2016-11-15 | 2017-05-10 | 努比亚技术有限公司 | Memory management device and method |
| US20180144015A1 (en) * | 2016-11-18 | 2018-05-24 | Microsoft Technology Licensing, Llc | Redoing transaction log records in parallel |
| CN110019498A (en) * | 2017-08-14 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Log synchronous method and device, storage medium, electronic equipment |
| CN108399186A (en) * | 2018-01-12 | 2018-08-14 | 联动优势科技有限公司 | A kind of collecting method and device |
| CN108509327A (en) * | 2018-04-20 | 2018-09-07 | 深圳市文鼎创数据科技有限公司 | A kind of log-output method, device, terminal device and storage medium |
| CN108710531A (en) * | 2018-04-20 | 2018-10-26 | 深圳市文鼎创数据科技有限公司 | Method for writing data, device, terminal device and the storage medium of round-robin queue |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114500403A (en) * | 2022-01-24 | 2022-05-13 | 中国联合网络通信集团有限公司 | Data processing method and device and computer readable storage medium |
| CN117407249A (en) * | 2023-12-12 | 2024-01-16 | 苏州元脑智能科技有限公司 | Drive log management method, device and system, electronic equipment and storage medium |
| CN117407249B (en) * | 2023-12-12 | 2024-03-01 | 苏州元脑智能科技有限公司 | Drive log management method, device and system, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7698602B2 (en) | Systems, methods and computer products for trace capability per work unit | |
| US8904067B2 (en) | Adaptive multi-threaded buffer | |
| CN110231995B (en) | Task scheduling method, device and storage medium based on Actor model | |
| US9069566B1 (en) | Implementation of a multiple writer single reader queue in a lock free and a contention free manner | |
| US7681196B2 (en) | Providing optimal number of threads to applications performing multi-tasking using threads | |
| CN112306827A (en) | Log collection device, method and computer readable storage medium | |
| CN112579263A (en) | Task execution method and device, storage medium and electronic equipment | |
| CN109842621A (en) | A kind of method and terminal reducing token storage quantity | |
| CN114168302A (en) | Task scheduling method, device, equipment and storage medium | |
| CN112231069A (en) | Transaction identifier processing method and device and electronic equipment | |
| CN108304272B (en) | Data IO request processing method and device | |
| CN107977275B (en) | Task processing method based on message queue and related equipment | |
| CN112698959A (en) | Multi-core communication method and device | |
| CA2697994C (en) | Method and apparatus for data processing using queuing | |
| CN115827506A (en) | Data writing method, data reading method, device, processing core and processor | |
| CN109298888B (en) | Queue data access method and device | |
| CN111282263A (en) | Event message processing method and device, electronic equipment and readable storage medium | |
| CN111597056B (en) | Distributed scheduling method, system, storage medium and equipment | |
| CN111190541B (en) | Flow control method of storage system and computer readable storage medium | |
| WO2018094509A9 (en) | System for inter-process communication | |
| CN110515749B (en) | Method, device, server and storage medium for queue scheduling of information transmission | |
| EP2318924B1 (en) | Synchronization of multiple processor cores | |
| CN108830724B (en) | Resource data packet processing method and terminal equipment | |
| CN113826081A (en) | Method for transmitting message in computing system and computing system | |
| CN104769553A (en) | System and method for supporting work sharing muxing in a cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210302 Address after: 100176 room 701, 7 / F, building 1, yard 18, Kechuang 11th Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing Applicant after: Beijing Jingdong tuoxian Technology Co.,Ltd. Address before: Room A402, 4th floor, building 2, No.18, Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant before: BEIJING WODONG TIANJUN INFORMATION TECHNOLOGY Co.,Ltd. Applicant before: BEIJING JINGDONG CENTURY TRADING Co.,Ltd. |