CN113656645A - Log consumption method and device - Google Patents
Log consumption method and device Download PDFInfo
- Publication number
- CN113656645A CN113656645A CN202010396856.0A CN202010396856A CN113656645A CN 113656645 A CN113656645 A CN 113656645A CN 202010396856 A CN202010396856 A CN 202010396856A CN 113656645 A CN113656645 A CN 113656645A
- Authority
- CN
- China
- Prior art keywords
- log
- target
- logs
- target log
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 230000004044 response Effects 0.000 claims abstract description 27
- 230000008859 change Effects 0.000 claims abstract description 18
- 238000003780 insertion Methods 0.000 claims description 39
- 230000037431 insertion Effects 0.000 claims description 39
- 238000012545 processing Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 9
- 230000000875 corresponding effect Effects 0.000 description 65
- 238000010586 diagram Methods 0.000 description 14
- 230000006870 function Effects 0.000 description 9
- 230000008569 process Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000007596 consolidation process Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the disclosure discloses a log consumption method and a log consumption device. One embodiment of the method comprises: acquiring a log pointed by a head pointer of a target log queue, wherein the log in the target log queue is used for recording information for performing row change operation on a target database; in response to that the obtained logs meet a preset merging condition, merging a first number of logs in a target log list corresponding to a target log queue to obtain a second number of merged logs, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number; and consuming the merged log of the second quantity of bars. The implementation mode reduces the number of logs required to be consumed, improves the consumption speed of the logs and reduces the pressure of log consumption of the electronic equipment.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to a log consumption method and device.
Background
Typically, databases have a log that records all transactions (events) and the modifications that each transaction makes to the database.
For example, binlog is a log used to record database changes. In practice, consuming binlog logs through MQ (message queuing) is a relatively common means in production. Among them, binlog has three modes, which are raw, state and mixed. Generally, the raw mode is most commonly used. In the raw mode, the specific contents of which table and which row in the database are changed can be known. When a large number of update operations are performed on data in a database, many change events are often generated.
Disclosure of Invention
The disclosure provides a log consumption method and device.
In a first aspect, an embodiment of the present disclosure provides a log consumption method, including: acquiring a log pointed by a head pointer of a target log queue, wherein the log in the target log queue is used for recording information for performing row change operation on a target database; in response to that the obtained logs meet a preset merging condition, merging a first number of logs in a target log list corresponding to a target log queue to obtain a second number of merged logs, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number; and consuming the merged log of the second quantity of bars.
In some embodiments, the log includes operation information and table information, the operation information indicates operations performed on data in the target database, the table information indicates a table including the operated data, the operation information included in each log included in the target log list indicates the same operation, and the table information included in each log included in the target log list indicates the same table.
In some embodiments, the preset merge condition comprises: a target log list corresponding to the target log queue is not empty, and one of: the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is greater than or equal to the preset time interval; the time difference between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval; and the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the acquired log indicate different operations or tables.
In some embodiments, merging a first number of logs in a target log list corresponding to a target log queue includes: and merging the logs included in the target log list into a group of logs.
In some embodiments, before merging a first number of logs in a target log list corresponding to a target log queue in response to the obtained log meeting a preset merging condition to obtain a second number of merged logs, the method further includes: in response to the target log list corresponding to the target log queue being empty, inserting the acquired log into the target log list corresponding to the target log queue, and recording an insertion time of the acquired log.
In some embodiments, before merging a first number of logs in a target log list corresponding to a target log queue in response to the obtained log meeting a preset merging condition to obtain a second number of merged logs, the method further includes: in response to the preset insertion condition being met, inserting the acquired logs into a target log list corresponding to the target log queue; wherein the preset insertion condition comprises: a target log list corresponding to the target log queue is not empty; the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval; the operation information included in the log pointed by the column tail pointer included in the target log list and the operation information included in the acquired log indicate the same operation; the log pointed by the column tail pointer included in the target log list includes table information indicating the same table as the table information included in the acquired log.
In some embodiments, after obtaining the log pointed to by the head-of-line pointer of the target log queue, the method further comprises: and deleting the acquired log pointed by the head of queue pointer of the target log queue from the target log queue.
In some embodiments, after consuming the second number of merged logs, the method further comprises: in response to completing consumption of the second number of merged logs, consumed logs included in the target log list are deleted from the target log list.
In a second aspect, an embodiment of the present disclosure provides a log consumption apparatus, including: the acquisition unit is configured to acquire a log pointed by a head pointer of a target log queue, wherein the log in the target log queue is used for recording information for performing row change operation on a target database; the merging unit is configured to merge a first number of logs in a target log list corresponding to a target log queue to obtain a second number of merged logs in response to the acquired logs meeting a preset merging condition, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number; and the consumption unit is configured to consume the merged log of the second quantity of bars.
In some embodiments, the log includes operation information and table information, the operation information indicates operations performed on data in the target database, the table information indicates a table including the operated data, the operation information included in each log included in the target log list indicates the same operation, and the table information included in each log included in the target log list indicates the same table.
In some embodiments, the preset merge condition comprises: a target log list corresponding to the target log queue is not empty, and one of: the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is greater than or equal to the preset time interval; the time difference between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval; and the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the acquired log indicate different operations or tables.
In some embodiments, the merging unit comprises: and the merging subunit is configured to merge the logs included in the target log list into a group of logs.
In some embodiments, the apparatus further comprises: a first inserting unit configured to insert the acquired log into a target log list corresponding to a target log queue in response to the target log list corresponding to the target log queue being empty, and to record an insertion time of the acquired log.
In some embodiments, the apparatus further comprises: a second insertion unit configured to insert the acquired log into a target log list corresponding to a target log queue in response to satisfaction of a preset insertion condition; wherein the preset insertion condition comprises: a target log list corresponding to the target log queue is not empty; the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval; the operation information included in the log pointed by the column tail pointer included in the target log list and the operation information included in the acquired log indicate the same operation; the log pointed by the column tail pointer included in the target log list includes table information indicating the same table as the table information included in the acquired log.
In some embodiments, the apparatus further comprises: and the first deleting unit is configured to delete the acquired log pointed by the head of line pointer of the target log queue from the target log queue.
In some embodiments, the apparatus further comprises: a second deleting unit configured to delete consumed logs included in the target log list from the target log list in response to completion of consumption of the second number of merged logs.
In a third aspect, an embodiment of the present disclosure provides an electronic device, including: one or more processors; a storage device, on which one or more programs are stored, which, when executed by the one or more processors, cause the one or more processors to implement the method of any of the embodiments of the log consumption method described above.
In a fourth aspect, embodiments of the present disclosure provide a computer-readable medium, on which a computer program is stored, which when executed by a processor, implements the method of any of the embodiments of the log consumption method described above.
According to the log consumption method and device provided by the embodiment of the disclosure, logs pointed by a head of line pointer of a target log queue are obtained, wherein the logs in the target log queue are used for recording information for performing a row change operation on a target database, then, in response to that the obtained logs meet a preset merging condition, a first number of logs in a target log list corresponding to the target log queue are merged, a second number of merged logs are obtained, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number, and finally, the second number of merged logs are consumed, so that the number of logs required to be consumed is reduced, the consumption speed of the logs is improved, and the pressure of electronic equipment for consuming the logs is reduced.
Drawings
Other features, objects and advantages of the disclosure will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present disclosure may be applied;
FIG. 2 is a flow diagram of one embodiment of a log consumption method according to the present disclosure;
FIG. 3 is a schematic diagram of one application scenario of a log consumption method according to the present disclosure;
FIG. 4 is a flow diagram of yet another embodiment of a log consumption method according to the present disclosure;
FIG. 5 is a flow diagram for one application scenario of FIG. 4;
FIG. 6 is a schematic block diagram of one embodiment of a log consumption device according to the present disclosure;
FIG. 7 is a schematic block diagram of a computer system suitable for use with an electronic device implementing embodiments of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that, in the present disclosure, the embodiments and features of the embodiments may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates an exemplary system architecture 100 to which embodiments of a log consumption method or log consumption apparatus of embodiments of the present disclosure may be applied.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 101, 102, 103 to interact with the server 105 over the network 104 to receive or transmit data or the like. The terminal devices 101, 102, 103 may have various client applications installed thereon, such as shopping applications, video playing software, news applications, image processing applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, and the like.
The terminal apparatuses 101, 102, and 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices with data transmission function, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like. When the terminal apparatuses 101, 102, 103 are software, they can be installed in the electronic apparatuses listed above. It may be implemented as multiple pieces of software or software modules (e.g., software or software modules used to provide distributed services) or as a single piece of software or software module. And is not particularly limited herein.
The server 105 may be a server providing various services, such as a background server operating a database (e.g. a database for storing user data) based on data sent by the terminal devices 101, 102, 103. The background server can update, add, delete, query, etc. the database based on the data sent by the terminal devices 101, 102, 103. As an example, the server 105 may be a cloud server.
The server may be hardware or software. When the server is hardware, it may be implemented as a distributed server cluster formed by multiple servers, or may be implemented as a single server. When the server is software, it may be implemented as multiple pieces of software or software modules (e.g., software or software modules used to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.
It should be further noted that the log consumption method provided by the embodiments of the present disclosure may be executed by a server, may also be executed by a terminal device, and may also be executed by the server and the terminal device in cooperation with each other. Accordingly, each part (for example, each unit and sub-unit) included in the log consuming apparatus may be entirely disposed in the server, may be entirely disposed in the terminal device, and may be disposed in the server and the terminal device, respectively.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation. When the electronic device on which the log consumption method is executed does not need to perform data transmission with other electronic devices, the system architecture may include only the electronic device (e.g., a server or a terminal device) on which the log consumption method is executed.
With continued reference to FIG. 2, a flow 200 of one embodiment of a log consumption method according to the present disclosure is shown. The log consumption method comprises the following steps:
In this embodiment, an execution main body (for example, a server or a terminal device shown in fig. 1) of the log consumption method may obtain, from the target log queue, the log pointed by the head-of-line pointer of the target log queue, from another electronic device or locally through a wired connection manner or a wireless connection manner. The logs in the target log queue are used for recording and recording information for performing row change operation on the target database.
The target log queue may be a log queue stored in an MQ (message queue). Typically, for each MQ, a coroutine may be initiated.
As an example, the logs in the target log queue may be binlog logs. The binlog log is a binary log of a MySQL (relational database management system) database and is used for recording information of row change operation on a target database. Here, the user can perform a row change operation on the database through the SQL statement. In practice, the binlog log may be consumed by the MQ. Among them, binlog has three modes, which are raw, state and mixed. Generally, the raw mode is most commonly used. In the raw mode, the specific contents of which table and which row in the database are changed can be known. When a large number of update operations are performed on data in a database, many change events are often generated.
In some optional implementations of this embodiment, the log includes operation information and table information. Wherein the operation information indicates an operation performed on the data in the target database. The table information indicates a table including operated data. The operation information included in each log in the target log list indicates the same operation. The table information included in each log in the target log list indicates the same table.
In this embodiment, in the case that the logs acquired in step 201 meet the preset merging condition, the execution main body may merge a first number of logs in the target log list corresponding to the target log queue to obtain a second number of merged logs. Wherein the first number is greater than or equal to 2 and is less than or equal to the number of logs included in the current (e.g., when performing step 202) target log list. The second number is less than the first number and is greater than or equal to 1. In general, the first number of logs may be adjacent to each other, that is, there is no log that needs to be merged between the first number of logs for merging in the target log list. Each target log queue may correspond to a target log list. The correspondence between the target log queue and the target log list may be established in advance.
It will be appreciated that consuming the second number of consolidated logs will produce the same result as consuming the first number of logs. As an example, a first number of logs may call 1000 times a method (function) for decrementing the current value by 1. After back-end processing, the second quantity bar merged log may be used to call 1 method for decrementing the current value by 1000. Illustratively, the back-end may generate 1 piece of processing information from the second number of merged logs. The process information may call 1 method for decrementing the current value by 1000.
The preset merging condition may be a predetermined condition for indicating merging of logs in the target log list. As an example, the preset merge condition may include: the storage space occupied by the acquired log is larger than or equal to a preset storage space threshold value.
As an example, in a case that the obtained log meets a preset merging condition, the execution main body may merge all logs in the target log list into 1 log, so as to obtain 1 merged log. Optionally, when the obtained logs meet the preset merging condition, the executing body may merge 10 logs, which are inserted into the target log list and have a farthest time from the current time (for example, the time of executing step 202), into 9 logs, so as to obtain 9 merged logs.
And step 203, consuming the merged log of the second quantity of bars.
In this embodiment, the executing entity may consume the second number of merged logs obtained in step 202.
In practice, the second number of merged logs may be consumed based on the binlog consumption component. Optionally, the log consumption logic may also be determined according to actual requirements, so that the log after the second number of pieces are merged is consumed.
Here, the log consumption method of the present disclosure may be applied in a raw, state, or mixed mode. Because in the raw mode, when a large number of update operations are performed on data in a database, many change events (i.e., logs) are often generated. Therefore, when the log consumption method disclosed by the disclosure is applied to the raw mode, the number of logs required to be consumed can be further reduced, the consumption speed of the logs is improved, and the pressure of log consumption of the electronic equipment is reduced compared with other modes.
With continued reference to fig. 3, fig. 3 is a schematic diagram of an application scenario of the log consumption method according to the present embodiment. In the application scenario of fig. 3, the server 301 first obtains a log (e.g. log 1) pointed by a head-of-line pointer of the target log queue 302, where the log in the target log queue is used to record information for performing a row change operation on a target database, and then, in a case that the obtained log (i.e. log 1) meets a preset merging condition, the server 301 merges a first number of logs (e.g. log a and log b, two logs) in the target log list 303 corresponding to the target log queue 302 to obtain a second number of merged logs 304 (in the illustration, log x, 1 merged log). Finally, server 301 consumes the second number of merged logs 304.
The prior art generally carries out binlog log consumption one by one, and one binlog log ack (a confirmation operation after consumption) is consumed once every time, and only after one binlog log is consumed, the next binlog log is consumed.
In the method provided by the foregoing embodiment of the present disclosure, logs pointed by a head-of-line pointer of a target log queue are obtained, where the logs in the target log queue are used to record information for performing a row change operation on a target database, then, when the obtained logs meet a preset merge condition, a first number of logs in a target log list corresponding to the target log queue are merged, so as to obtain a second number of merged logs, where the first number is greater than or equal to 2, and the second number is smaller than the first number, and finally, the second number of merged logs are consumed, so that the number of logs to be consumed is reduced, the consumption speed of the logs is increased, and the pressure of electronic equipment for consuming the logs is reduced. And the back end can directly consume the merged log, thereby reducing a large amount of network request time delay.
In some optional implementations of this embodiment, before performing step 202, in a case that the target log list corresponding to the target log queue is empty, the executing main body may further insert the acquired log into the target log list corresponding to the target log queue, and record an insertion time of the acquired log.
In some optional implementations of this embodiment, the preset combining condition includes: a target log list corresponding to the target log queue is not empty, and any of:
in the first item, the time interval between the acquisition time of the acquired log and the insertion time of the first log inserted into the target log list included in the target log list is greater than or equal to the preset time interval.
The preset time interval may be a preset fixed value or a preset and changeable value.
It is understood that when the preset merge condition includes the first item described above, the throughput of log consumption can be controlled by setting a preset time interval.
A second item, wherein the time difference between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than a preset time interval; and the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the acquired log indicate different operations or tables.
Wherein, the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the obtained log indicate different operations or tables, that is: the operation information included in the log pointed by the column tail pointer included in the target log list and the operation information included in the acquired log indicate different operations; or the table information included in the log pointed by the column tail pointer included in the target log list and the table information included in the acquired log indicate different tables.
It is to be understood that when the preset merging condition includes the second item described above, it may be ensured that the included operation information of each unconsumed log in the target log list corresponding to the target log queue indicates the same operation, and the included table information of each unconsumed log in the target log list corresponding to the target log queue indicates the same table, so that log merging and log consumption are performed in order.
In some optional implementation manners of this embodiment, before performing step 202, in a case that a preset insertion condition is satisfied, the executing main body may further insert the obtained log into a target log list corresponding to the target log queue.
Wherein the preset insertion condition comprises:
a target log list corresponding to the target log queue is not empty;
the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval;
the operation information included in the log pointed by the column tail pointer included in the target log list and the operation information included in the acquired log indicate the same operation; and
the log pointed by the column tail pointer included in the target log list includes table information indicating the same table as the table information included in the acquired log.
It will be appreciated that in the above alternative implementation, it may be ensured that the included operation information of each unconsumed log in the target log list corresponding to the target log queue indicates the same operation, and the included table information of each unconsumed log in the target log list corresponding to the target log queue indicates the same table, so as to facilitate the ordered progression of log consolidation and log consumption.
In some optional implementation manners of this embodiment, after the step 201 is executed, the execution main body may further delete, from the target log queue, the obtained log pointed by the head-of-line pointer of the target log queue.
It is understood that after the above step 201 is executed, the log obtained in step 201 may be processed, for example, the log obtained in step 201 is inserted into the target log list. Therefore, the execution main body can delete the acquired logs pointed by the head of queue pointer of the target log queue from the target log queue, so that the head of queue pointer of the target log queue points to other logs, other logs pointed by the head of queue pointer can be acquired subsequently, other logs are processed, and repeated processing of the logs is avoided.
In some optional implementation manners of this embodiment, after the step 203 is executed, in a case that the consumption of the second number of merged logs is completed, the executing main body may further delete consumed logs included in the target log list from the target log list.
It will be appreciated that after performing step 203 above, consumption of the second number of merged logs obtained in step 203 may generally be completed. Therefore, the execution main body can delete consumed logs included in the target log list from the target log list, so that merged logs which are not consumed in the target log list are consumed subsequently, repeated consumption of the logs is avoided, consumed logs included in the target log list are deleted from the target log list under the condition that consumption of the second number of merged logs is completed, and the logs can be guaranteed not to be lost.
With further reference to FIG. 4, a flow 400 of yet another embodiment of a log consumption method is shown. The process 400 of the log consumption method includes the following steps:
In this embodiment, step 401 is substantially the same as step 201 in the corresponding embodiment of fig. 2, and is not described here again.
In this embodiment, in a case where the target log list corresponding to the target log queue is empty, an execution subject of the log consumption method (for example, the server or the terminal device shown in fig. 1) may insert the acquired log into the target log list corresponding to the target log queue and record an insertion time of the acquired log.
In this embodiment, step 403 is substantially the same as step 202 in the corresponding embodiment of fig. 2, and is not described herein again.
In this embodiment, step 404 is substantially the same as step 203 in the corresponding embodiment of fig. 2, and is not described herein again.
It should be noted that, besides the above-mentioned contents, the embodiment of the present disclosure may also include the same or similar features and effects as the embodiment corresponding to fig. 2, and no further description is provided herein.
As can be seen from fig. 4, in the case that the target log list corresponding to the target log queue is empty, the process 400 of the log consumption method in this embodiment may insert the obtained log into the target log list corresponding to the target log queue, and record the insertion time of the obtained log, so as to obtain the time when the target log list is inserted into the first log, which is beneficial to implement more timely log consumption according to the insertion time.
By way of example, referring to fig. 5, fig. 5 is a flow diagram for one application scenario of fig. 4. The process 500 of the application scenario includes the following steps:
In this embodiment, in a case that a target log list corresponding to a target log queue is empty, an execution subject of the log consumption method (for example, a server or a terminal device shown in fig. 1) may obtain a log pointed by a head-of-queue pointer of the target log queue.
Step 502 determines whether the target log list corresponding to the target log queue is empty. Then, if the target log list corresponding to the target log queue is empty, execute step 504; if the target log list corresponding to the target log queue is not empty, step 503 is executed.
In this embodiment, the execution body may determine whether the target log list corresponding to the target log queue is empty.
In step 503, it is determined whether a time interval between the obtaining time of the obtained log and the inserting time of the first inserted log included in the target log list is greater than or equal to a preset time interval. Then, if the time interval between the obtaining time of the obtained log and the inserting time of the first inserted log included in the target log list is greater than or equal to the preset time interval, executing step 507; if the time interval between the obtaining time of the obtained log and the inserting time of the first inserted log included in the target log list is less than the preset time interval, step 505 is executed.
In this embodiment, in the case where it is determined in step 502 that the target log list corresponding to the target log queue is not empty, the execution main body may determine whether a time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is greater than or equal to a preset time interval.
In this embodiment, in the case where it is determined in step 502 that the target log list corresponding to the target log queue is empty, the execution main body may insert the acquired log into the target log list corresponding to the target log queue and record the insertion time of the acquired log.
In this embodiment, in the case where it is determined in step 502 that the target log list corresponding to the target log queue is not empty, and the time interval between the acquisition time of the log acquired in step 503 and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval, the execution main body may determine whether the operation information and table information included in the log pointed by the column end pointer included in the target log list and the operation information and table information included in the acquired log indicate the same operation and table.
In the present embodiment, it is determined in step 502 that the target log list corresponding to the target log queue is not empty; in the case that the time interval between the acquisition time of the log acquired in step 503 and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval, and the operation information and the table information included in the log pointed by the column end pointer included in the target log list in step 505 and the operation information and the table information included in the acquired log indicate the same operation and table, respectively, the execution main body may insert the acquired log into the target log list corresponding to the target log queue.
In this embodiment, the execution main body may merge the logs included in the target log list into a group of logs, so as to obtain a merged log. Wherein, the consumption of the group of logs obtained after merging and the consumption of the first number of logs can generate the same result. As an example, a first number of logs may call 1000 times a method (function) for decrementing the current value by 1. After the back-end processing, the combined set of logs can be used to call 1 method for subtracting 1000 from the current value. For example, the back-end may generate 1 piece of processing information according to a group of logs obtained after merging. The process information may call 1 method for decrementing the current value by 1000.
And step 508, consuming the merged logs, and deleting each log included in the target log list from the target log list.
In this embodiment, the execution main body may consume the merged log, and delete each log included in the target log list from the target log list.
In some optional implementations of this embodiment, after the step 505 is executed, the execution main body may further execute the step 507, the step 508 and the step 504 at a time.
It should be noted that, besides the above-mentioned contents, the embodiment of the present disclosure may also include the same or similar features and effects as the embodiment corresponding to fig. 2, and no further description is provided herein.
It is understood that the flow 500 of the log consumption method in this embodiment may accelerate log consumption by merging logs. Moreover, the log consumption can be guaranteed to be orderly and not lost. By adjusting the preset time interval, the throughput of log consumption is controlled.
With further reference to fig. 6, as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a log consumption device, which corresponds to the embodiment of the method shown in fig. 2, and which may include the same or corresponding features as the embodiment of the method shown in fig. 2 and produce the same or corresponding effects as the embodiment of the method shown in fig. 2, in addition to the features described below. The device can be applied to various electronic equipment.
As shown in fig. 6, the log consumption apparatus 600 of the present embodiment includes: an obtaining unit 601 configured to obtain a log pointed by a head of queue pointer of a target log queue, where the log in the target log queue is information for recording a row change operation performed on a target database; a merging unit 602, configured to merge, in response to that the obtained logs meet a preset merging condition, a first number of logs in a target log list corresponding to a target log queue to obtain a second number of merged logs, where the first number is greater than or equal to 2, and the second number is smaller than the first number; a consuming unit 603 configured to consume the second number of merged logs.
In this embodiment, the acquisition unit 601 of the log consumption device 600 may acquire the log pointed to by the head-of-line pointer of the target log queue. The logs in the target log queue are used for recording information for performing row change operation on the target database.
In this embodiment, in a case that the log acquired by the acquiring unit 601 meets a preset merging condition, the merging unit 602 may merge a first number of logs in the target log list corresponding to the target log queue to obtain a second number of merged logs. Wherein the first number is greater than or equal to 2 and the second number is less than the first number.
In this embodiment, the consuming unit 603 may consume the second number of combined logs obtained by the combining unit 602.
In some optional implementations of this embodiment, the log includes operation information and table information, the operation information indicates an operation performed on data in the target database, the table information indicates a table including the operated data, the operation information included in each log included in the target log list indicates the same operation, and the table information included in each log included in the target log list indicates the same table.
In some optional implementations of this embodiment, the preset combining condition includes: a target log list corresponding to the target log queue is not empty, and one of: the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is greater than or equal to the preset time interval; the time difference between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval; and the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the acquired log indicate different operations or tables.
In some optional implementations of this embodiment, the merging unit 603 includes: and a merging subunit (not shown in the figure) configured to merge the respective logs included in the target log list into a set of logs.
In some optional implementations of this embodiment, the apparatus 600 further includes: a first inserting unit (not shown in the figure) configured to insert the acquired log into a target log list corresponding to a target log queue in response to the target log list corresponding to the target log queue being empty, and to record an insertion time of the acquired log.
In some optional implementations of this embodiment, the apparatus 600 further includes: and a second inserting unit (not shown in the figure) configured to insert the acquired log into a target log list corresponding to the target log queue in response to satisfaction of a preset inserting condition. Wherein the preset insertion condition comprises: a target log list corresponding to the target log queue is not empty; the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval; the operation information included in the log pointed by the column tail pointer included in the target log list and the operation information included in the acquired log indicate the same operation; the log pointed by the column tail pointer included in the target log list includes table information indicating the same table as the table information included in the acquired log.
In some optional implementations of this embodiment, the apparatus 600 further includes: and a first deleting unit (not shown in the figure) configured to delete the acquired log pointed by the head-of-line pointer of the target log queue from the target log queue.
In some optional implementations of this embodiment, the apparatus 600 further includes: a second deleting unit (not shown in the figure) configured to delete the consumed logs included in the target log list from the target log list in response to completion of consumption of the second number of merged logs.
In the apparatus provided by the foregoing embodiment of the present disclosure, the obtaining unit 601 obtains logs pointed by a head pointer of a target log queue, where the logs in the target log queue are used to record information for performing a row change operation on a target database, then, when the obtained logs meet a preset merging condition, the merging unit 602 merges a first number of logs in a target log list corresponding to the target log queue to obtain a second number of merged logs, where the first number is greater than or equal to 2, and the second number is smaller than the first number, and finally, the consuming unit 603 consumes the second number of merged logs, so as to reduce the number of logs to be consumed, improve the consumption speed of the logs, and reduce the pressure of log consumption by the electronic device. And the back end can directly consume the merged log, thereby reducing a large amount of network request time delay.
Referring now to fig. 7, a schematic diagram of an electronic device (e.g., the server or terminal device of fig. 1) 700 suitable for use in implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. The terminal device/server shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 7, electronic device 700 may include a processing means (e.g., central processing unit, graphics processor, etc.) 701 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from storage 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for the operation of the electronic apparatus 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Generally, the following devices may be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 707 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 708 including, for example, magnetic tape, hard disk, etc.; and a communication device 709. The communication means 709 may allow the electronic device 700 to communicate wirelessly or by wire with other devices to exchange data. While fig. 7 illustrates an electronic device 700 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 7 may represent one device or may represent multiple devices as desired.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via the communication means 709, or may be installed from the storage means 708, or may be installed from the ROM 702. The computer program, when executed by the processing device 701, performs the above-described functions defined in the methods of embodiments of the present disclosure.
It should be noted that the computer readable medium described in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In embodiments of the present disclosure, however, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a log pointed by a head pointer of a target log queue, wherein the log in the target log queue is used for recording information for performing row change operation on a target database; in response to that the obtained logs meet a preset merging condition, merging a first number of logs in a target log list corresponding to a target log queue to obtain a second number of merged logs, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number; and consuming the merged log of the second quantity of bars.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes an acquisition unit, a merging unit, and a consumption unit. The names of these units do not in some cases constitute a limitation on the unit itself, and for example, the acquisition unit may also be described as "a unit that acquires a log pointed to by the head-of-line pointer of the target log queue".
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.
Claims (11)
1. A log consumption method, comprising:
acquiring a log pointed by a head pointer of a target log queue, wherein the log in the target log queue is used for recording information for performing row change operation on a target database;
in response to that the obtained logs meet a preset merging condition, merging a first number of logs in a target log list corresponding to the target log queue to obtain a second number of merged logs, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number;
and consuming the merged log of the second quantity of bars.
2. The method of claim 1, wherein the log comprises operation information and table information, the operation information indicating operations performed on the data in the target database, the table information indicating a table including the operated data, the operation information included in each log included in the target log list indicating the same operations, and the table information included in each log included in the target log list indicating the same table.
3. The method of claim 2, wherein the preset merge condition comprises: a target log list corresponding to the target log queue is not empty, and one of:
the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is greater than or equal to the preset time interval;
the time difference between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval; and the operation information or table information included in the log pointed by the column tail pointer included in the target log list and the operation information or table information included in the acquired log indicate different operations or tables.
4. The method of claim 3, wherein the merging the first number of logs in the list of target logs corresponding to the queue of target logs comprises:
and merging the logs included in the target log list into a group of logs.
5. The method according to one of claims 1 to 4, wherein before merging a first number of logs in a target log list corresponding to the target log queue to obtain a second number of merged logs in response to the obtained logs meeting a preset merging condition, the method further comprises:
in response to the target log list corresponding to the target log queue being empty, inserting the acquired log into the target log list corresponding to the target log queue, and recording the insertion time of the acquired log.
6. The method according to one of claims 1 to 4, wherein before merging a first number of logs in a target log list corresponding to the target log queue to obtain a second number of merged logs in response to the obtained logs meeting a preset merging condition, the method further comprises:
in response to the preset insertion condition being met, inserting the acquired logs into a target log list corresponding to the target log queue;
wherein the preset insertion condition comprises:
a target log list corresponding to the target log queue is not empty;
the time interval between the acquisition time of the acquired log and the insertion time of the first inserted log included in the target log list is smaller than the preset time interval;
the operation information included in the log pointed by the column tail pointer included in the target log list and the operation information included in the acquired log indicate the same operation;
the log pointed by the column tail pointer included in the target log list includes table information indicating the same table as the table information included in the acquired log.
7. The method of any of claims 1-4, wherein after obtaining the log pointed to by the head-of-line pointer of the target log queue, the method further comprises:
and deleting the acquired logs pointed by the head pointer of the target log queue from the target log queue.
8. The method of any of claims 1-4, wherein after the consuming the second number of merged logs, the method further comprises:
in response to completing consumption of the second number of merged logs, deleting consumed logs included in the target log list from the target log list.
9. A log consumption device, comprising:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is configured to acquire a log pointed by a head pointer of a target log queue, and the log in the target log queue is used for recording information for performing row change operation on a target database;
the merging unit is configured to merge a first number of logs in a target log list corresponding to the target log queue to obtain a second number of merged logs in response to that the obtained logs meet a preset merging condition, wherein the first number is greater than or equal to 2, and the second number is smaller than the first number;
a consumption unit configured to consume the merged log of the second number of bars.
10. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.
11. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010396856.0A CN113656645A (en) | 2020-05-12 | 2020-05-12 | Log consumption method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010396856.0A CN113656645A (en) | 2020-05-12 | 2020-05-12 | Log consumption method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113656645A true CN113656645A (en) | 2021-11-16 |
Family
ID=78476861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010396856.0A Pending CN113656645A (en) | 2020-05-12 | 2020-05-12 | Log consumption method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113656645A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101605028A (en) * | 2009-02-17 | 2009-12-16 | 北京安天电子设备有限公司 | A kind of combining log records method and system |
| CN101625703A (en) * | 2009-08-21 | 2010-01-13 | 华中科技大学 | Method and system for merging logs of memory database |
| CN108984686A (en) * | 2018-07-02 | 2018-12-11 | 中国电子科技集团公司第五十二研究所 | A kind of distributed file system indexing means and device merged based on log |
| CN109992469A (en) * | 2017-12-29 | 2019-07-09 | 北京奇虎科技有限公司 | A kind of method and device merging log |
-
2020
- 2020-05-12 CN CN202010396856.0A patent/CN113656645A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101605028A (en) * | 2009-02-17 | 2009-12-16 | 北京安天电子设备有限公司 | A kind of combining log records method and system |
| CN101625703A (en) * | 2009-08-21 | 2010-01-13 | 华中科技大学 | Method and system for merging logs of memory database |
| CN109992469A (en) * | 2017-12-29 | 2019-07-09 | 北京奇虎科技有限公司 | A kind of method and device merging log |
| CN108984686A (en) * | 2018-07-02 | 2018-12-11 | 中国电子科技集团公司第五十二研究所 | A kind of distributed file system indexing means and device merged based on log |
Non-Patent Citations (1)
| Title |
|---|
| 许小明: "多源异构日志的数据归并和预处理技术", 《中国优秀硕士学位论文全文数据库信息科技辑（月刊）》, no. 06, 15 June 2009 (2009-06-15) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110781373B (en) | List updating method and device, readable medium and electronic equipment | |
| CN111309747A (en) | Data synchronization method, system and device | |
| CN111163336B (en) | Video resource pushing method and device, electronic equipment and computer readable medium | |
| CN110856004B (en) | Message processing method and device, readable storage medium and electronic equipment | |
| CN111881216A (en) | Data acquisition method and device based on shared template | |
| CN112148744A (en) | Page display method and device, electronic equipment and computer readable medium | |
| CN112581043A (en) | Device control method, device, electronic device and computer readable medium | |
| CN112307393A (en) | Information issuing method and device and electronic equipment | |
| CN113988992B (en) | Order information sending method, order information sending device, electronic equipment and computer readable medium | |
| CN115658655A (en) | Information processing method, device, electronic equipment and storage medium | |
| CN110619079B (en) | Method and device for pushing information | |
| CN112200643B (en) | Article information pushing method and device, electronic equipment and computer readable medium | |
| CN113656645A (en) | Log consumption method and device | |
| CN114116247A (en) | Redis-based message processing method, device, system, server and medium | |
| CN110061907B (en) | Method and equipment for drawing resources and distributing resources | |
| CN112685075A (en) | Gray scale distribution method and device, electronic equipment and computer readable medium | |
| CN112115154A (en) | Data processing and data query method, device, equipment and computer readable medium | |
| CN111291254A (en) | Information processing method and device | |
| CN112100211A (en) | Data storage method and device, electronic equipment and computer readable medium | |
| CN112311840A (en) | Multi-terminal data synchronization method, device, equipment and medium | |
| CN111294657A (en) | Information processing method and device | |
| CN111787043A (en) | Data request method and device | |
| CN111294321B (en) | Information processing method and device | |
| CN111294611B (en) | Video insertion method and device, electronic equipment and computer readable storage medium | |
| CN111367592A (en) | Information processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |