CN116170273A

CN116170273A - Log asynchronous output processing method and device

Info

Publication number: CN116170273A
Application number: CN202310003004.4A
Authority: CN
Inventors: 杨阳; 杨崇现
Original assignee: Everbright Technology Co ltd
Current assignee: Everbright Technology Co ltd
Priority date: 2023-01-03
Filing date: 2023-01-03
Publication date: 2023-05-26

Abstract

The invention provides a log asynchronous output processing method and device, wherein the method comprises the following steps: acquiring the length of an original log; transcoding and splicing the original log according to the length of the original log to obtain a spliced log; performing binary reorganization on the spliced log according to the length of the spliced log to obtain a reorganized log; the method has the advantages that the problem of inter-service cross-network access can be solved on the premise of avoiding invasion of a service system in the related art by carrying out asynchronous output processing on the reorganized log according to the non-lock ring-shaped queue, and the method can be recycled and has higher performance based on the non-lock ring-shaped queue; the logs are processed through splicing and reorganization, so that reorganized logs are obtained, efficiency is improved, and safety is improved.

Description

Log asynchronous output processing method and device

Technical Field

The invention relates to the field of data processing, in particular to a log asynchronous output processing method and device.

Background

When the multiple systems interact through online transactions, if the transactions are frequent or the message length is large, resources and time are consumed when the transaction logs are synchronously output, and the throughput of the systems is affected. For some special transactions involving sensitive information, if these data are output directly to the log, mismanagement may result in disclosure of the private data. The async appliance asynchronous log output component is adopted in the existing asynchronous log of the log back, and essentially, the log is stored by using a blocking queue, so that the performance is lower; and the log back does not process log sensitive information.

At present, an asynchronous log async applicator of the log back stores log events through a default ArrayB l ock i ngQueue array, and the performance is lower due to the adoption of a blocking queue. The log output of the log back is plaintext, and when the log is larger, the output efficiency is low, and the plaintext is unsafe.

Aiming at the problem of how to access between services across network domains on the premise of avoiding intrusion into a service system in the related art, no solution has been proposed yet.

Disclosure of Invention

The embodiment of the invention provides a log asynchronous output processing method and device, which at least solve the problem of how to access between services across network domains on the premise of avoiding intrusion into a service system in the related art.

According to an embodiment of the present invention, there is provided a log asynchronous output processing method, including:

acquiring the length of an original log;

transcoding and splicing the original log according to the length of the original log to obtain a spliced log;

performing binary reassembly on the spliced log according to the length of the spliced log to obtain a reassembly log;

and carrying out asynchronous output processing on the reorganized log according to the lock-ring-free queue.

Optionally, transcoding and splicing the original log according to the length of the original log, and obtaining the spliced log includes:

judging whether the length of the original log is larger than a preset threshold value or not;

if the judgment result is yes, carrying out compression processing on the original log, transcoding the compressed log to obtain a transcoded log, and splicing a first preset letter in front of the transcoded log to obtain the spliced log;

and under the condition that the judgment result is negative, transcoding the original log to obtain a transcoded log, and splicing a second preset letter in front of the transcoded log to obtain the spliced log.

Optionally, before acquiring the length of the original log, the method further comprises:

judging whether the original log needs to be subjected to desensitization treatment or not;

and if the judgment result is yes, desensitizing the original log.

Optionally, performing binary reorganization on the spliced log according to the length of the spliced log, and obtaining the reorganized log includes:

dividing the spliced log into a first log section and a second log section according to the length of the spliced log;

and sequentially resetting the first log segment and the second log segment, and then combining to obtain the reorganized log.

Optionally, performing asynchronous output processing on the reorganized log according to the lock-free ring queue includes:

establishing the lock-free ring queue for storing log events;

adding the reorganized log to the lock-free ring queue;

invoking the reassembly log from the lock-free ring queue via a single read thread;

and writing the reorganized log into a corresponding log file through an appender list.

Optionally, adding the reassembly log to the lock-free ring queue includes:

acquiring the next position in the lock-free ring-shaped queue;

setting the reorganization log;

the reassembly log is added to a next location in the non-circular queue.

Optionally, before asynchronous output processing is performed on the reassembly log according to a ring queue, the method further includes:

defining the list of appenders in a configuration file;

inserting the configuration file into a database;

loading a configuration file in the database at the time of starting; and/or

And newly creating a timing task, and reloading configuration files in the database every preset time period.

According to another embodiment of the present invention, there is also provided a log asynchronous output processing apparatus including:

the acquisition module is used for acquiring the length of the original log;

the splicing module is used for transcoding and splicing the original log according to the length of the original log to obtain a spliced log;

the reorganization module is used for carrying out binary reorganization on the spliced log according to the length of the spliced log to obtain a reorganized log;

and the asynchronous output module is used for carrying out asynchronous output processing on the recombined log according to the lock-ring-free queue.

Optionally, the splicing module includes:

the judging submodule is used for judging whether the length of the original log is larger than a preset threshold value or not;

the first splicing sub-module is used for carrying out compression processing on the original log under the condition that the judgment result is yes, transcoding the compressed log to obtain a transcoded log, and splicing a first preset letter in front of the transcoded log to obtain the spliced log;

and the second splicing sub-module is used for transcoding the original log to obtain a transcoded log under the condition that the judgment result is negative, and splicing a second preset letter in front of the transcoded log to obtain the spliced log.

Optionally, the apparatus further comprises:

the judging module is used for judging whether the original log needs to be subjected to desensitization treatment or not;

and the desensitization module is used for carrying out desensitization processing on the original log under the condition that the judgment result is yes.

Optionally, the reorganization module is further configured to divide the spliced log into a first log segment and a second log segment according to the length of the spliced log; and sequentially resetting the first log segment and the second log segment, and then combining to obtain the reorganized log.

Optionally, the asynchronous output module includes:

a building sub-module for building the lock-free ring queue for storing log events;

an adding submodule for adding the reorganized log to the lock-free ring queue;

a calling sub-module for calling the reorganized log from the non-lock ring queue through a single read thread;

and the log writing module is used for writing the reorganized log into a corresponding log file through the appender list.

Optionally, the adding submodule is further configured to obtain a next position in the lock-free ring queue; setting the reorganization log; the reassembly log is added to a next location in the non-circular queue.

Optionally, the apparatus further comprises:

a definition module for defining the list of appenders in a configuration file;

the inserting module is used for inserting the configuration file into a database;

the loading module is used for loading the configuration files in the database when the loading module is started; and/or

And the reloading module is used for newly creating a timing task and reloading the configuration files in the database at intervals of a preset time period.

According to a further embodiment of the invention, there is also provided a computer-readable storage medium having stored therein a computer program, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.

According to a further embodiment of the invention, there is also provided an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.

According to the method and the device, the length of the original log is obtained; transcoding and splicing the original log according to the length of the original log to obtain a spliced log; performing binary reassembly on the spliced log according to the length of the spliced log to obtain a reassembly log; the method has the advantages that the problem of inter-service cross-network domain access can be solved on the premise of avoiding invasion of a service system in the related art by carrying out asynchronous output processing on the reorganized logs according to the non-lock ring-shaped queue, and the method is recyclable and has higher performance based on the non-lock ring-shaped queue; the logs are processed through splicing and reorganization, so that reorganized logs are obtained, efficiency is improved, and safety is improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:

FIG. 1 is a block diagram of a hardware configuration of a mobile terminal of a log asynchronous output processing method according to an embodiment of the present invention;

FIG. 2 is a flow chart of a log asynchronous output processing method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of log processing according to an embodiment of the invention;

FIG. 4 is a flow chart of log obfuscation according to an embodiment of the invention;

FIG. 5 is a schematic diagram of log asynchronous output according to an embodiment of the invention;

fig. 6 is a block diagram of a log asynchronous output processing apparatus according to an embodiment of the present invention.

Detailed Description

The invention will be described in detail hereinafter with reference to the drawings in conjunction with embodiments. It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.

The method embodiment provided in the first embodiment of the present application may be executed in a mobile terminal, a computer terminal or a similar computing device. Taking a mobile terminal as an example, fig. 1 is a block diagram of a hardware structure of a mobile terminal according to an embodiment of the present invention, where, as shown in fig. 1, the mobile terminal may include one or more (only one is shown in fig. 1) processors 102 (the processors 102 may include, but are not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data, and optionally, the mobile terminal may further include a transmission device 106 for a communication function and an input/output device 108. It will be appreciated by those skilled in the art that the structure shown in fig. 1 is merely illustrative and not limiting of the structure of the mobile terminal described above. For example, the mobile terminal may also include more or fewer components than shown in fig. 1, or have a different configuration than shown in fig. 1.

The memory 104 may be used to store a computer program, for example, a software program of application software and a module, such as a computer program corresponding to a log asynchronous output processing method in an embodiment of the present invention, and the processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, that is, implement the above-mentioned method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, which may be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission means 106 is arranged to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network I nterface Contro l l er, abbreviated NIC) that can communicate with other network equipment via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (Rad i o Frequency, abbreviated as RF) module for communicating with the internet wirelessly.

In this embodiment, a log asynchronous output processing method running on the mobile terminal or the network architecture is provided and applied to a master node, and fig. 2 is a flowchart of the log asynchronous output processing method according to an embodiment of the present invention, as shown in fig. 2, where the flowchart includes the following steps:

step S202, obtaining the length of an original log;

step S204, transcoding and splicing the original log according to the length of the original log to obtain a spliced log;

step S206, performing binary reorganization on the spliced log according to the length of the spliced log to obtain a reorganized log;

and step S208, carrying out asynchronous output processing on the reorganized log according to the lock-ring-free queue.

Through the steps S202 to S208, the problem of how to access between services across network domains on the premise of avoiding invasion to a service system in the related art can be solved, and the service system can be recycled based on a ring-free queue, and has higher performance; the logs are processed through splicing and reorganization, so that reorganized logs are obtained, efficiency is improved, and safety is improved.

In the embodiment of the present invention, step S204 may specifically include: judging whether the length of the original log is larger than a preset threshold value or not; if the judgment result is yes, carrying out compression processing on the original log, transcoding the compressed log to obtain a transcoded log, and splicing a first preset letter in front of the transcoded log to obtain the spliced log; and under the condition that the judgment result is negative, transcoding the original log to obtain a transcoded log, and splicing a second preset letter in front of the transcoded log to obtain the spliced log.

In an alternative embodiment, before the step S202, the method further includes: judging whether the original log needs to be subjected to desensitization treatment or not; and if the judgment result is yes, desensitizing the original log.

In the embodiment of the present invention, step S206 may specifically include:

In the embodiment of the present invention, step S208 may specifically include:

establishing the lock-free ring queue for storing log events; adding the reorganized log into the non-lock ring-shaped queue, and further, acquiring the next position in the non-lock ring-shaped queue; setting the reorganization log; adding the reorganized log to the next position in the non-ring queue, specifically, calling the next position in the seq= r i ngBuffer. Next () acquisition queue, calling Va l ueEvent= r i ngBuffer. Get (seq) to acquire Va l ueEvent objects, setting log events through Va l ueEvent. SetEvent (eventObject), and calling r i ngBuffer. Pub ish (seq) to add the log events to r i ngBuffer;

invoking the reassembly log from the lock-free ring queue via a single read thread; and writing the reorganized log into a corresponding log file through an appender list.

In another alternative embodiment, prior to step S208, the method further includes: defining the list of appenders in a configuration file; inserting the configuration file into a database; loading a configuration file in the database at the time of starting; and/or newly creating a timing task, and reloading configuration files in the database every preset time period.

FIG. 3 is a schematic diagram of log processing, as shown in FIG. 3, according to an embodiment of the invention, including:

1. and loading an l ogback.xml configuration file when the business system is started.

2. Log confusion, fig. 4 is a flowchart of log confusion according to an embodiment of the invention, as shown in fig. 4, comprising:

step S401, obtaining an original log, creating a message converter class Sens it I veDataConverter, inheriting the class C l ass I cConverter, and realizing a overt method, and obtaining original log data through an I log I ngEvent.

Step S402, judging whether desensitization processing is needed, if yes, executing step S403, otherwise executing step S404;

step S403, desensitizing the log sensitive information, for the system with higher security level requirement, firstly desensitizing the original log data, otherwise, skipping the step. The specific desensitization rules are as follows:

(1) Define key values that require desensitization. Because the key naming rules of the same meaning are inconsistent for each system, unified collection is needed. Such as a handset number key value definition: mobile_key=mobi i l e|phone|l ephone|mobi l ephone;

(2) A regular expression is defined that identifies va l ue. Such as regularization for cell phone number: mobile_regex= "'" (;

(3) Different desensitization rules are defined for different keys. For example, for the mobile phone number, a replacement rule is adopted, and other characters except the first three characters and the last four characters are replaced by 'x';

(4) Detecting the original log according to the rule, and performing data desensitization.

Step S404, acquiring log length, and adopting different strategies for logs with different lengths.

Step S405, judging whether the log length is more than 600, if yes, executing step S406, otherwise executing step S409;

in step S406-408, if the log length is greater than or equal to 600, the log data is compressed by GZ I POutputStream, and then the compressed data stream is transcoded by BASE 64. To facilitate log restoration, the letter "B" is finally spelled in front of the transcoded log.

In step S409-410, if the log length is less than 600, the log is directly transcoded by BASE 64. Likewise, the letter "S" is spelled before the transcoded log.

And S411-S412, performing binary recombination to obtain a log after confusion. Specifically, the log length after the processing is obtained and divided by 2, the log length is divided into two sections according to the length, and the two sections of log are recombined into new data after the sequence of the two sections of log replacement. Such as: if the log is BCDTB6 r=, the new data after the replacement is: b6r=bcdt; if the log is SCS6R, the new data after replacement is: s6RSC.

3. And (5) asynchronously outputting the log.

FIG. 5 is a schematic diagram of asynchronous output of a log according to an embodiment of the present invention, as shown in FIG. 5, the so-called asynchronous output log is conventionally defined as a queue for storing the log, then a log is added to the queue by calling the l ogger.i nfo () method, and finally the log in the queue is written to a file by using a new thread. The current storage log queue adopted by the log back is a blocking queue ArrayB l ock i ngQueue, the performance is lower, and the optimization space is replaced by a non-blocking queue Di sruper.

For the development level, log back can achieve asynchronous log printing by adding the configuration shown in fig. 3 to log back.

As can be seen from the configuration, l ogback is printed by async appliance to implement asynchronous journaling. The principle layer is disassembled from the following.

Calling l ogger.i nfo () adds a log to the queue. When calling the l ogger.i nfo () method, l ogback defines an object of AppenderAttachab l e Imp l < I log I ngEvent > type in log, which is used to process an app defined in a configuration file. AppenderAttachab l e Imp l maintains a list of appers and provides methods for addition, deletion, etc. of appers. The object calls the application () method in class Unsynchron i zedAppenderBase, adding a log event. application () is an abstract method that facilitates subclass rewriting. The asynchronous log processing subclass async appenderbase carried by log back implements the abstract method of application () and places log events into a defined one b l ock i ngQueue blocking queue. The async appliance inherits async appliance base, indirectly realizes application (). The task of l ogger.i nfo () is completed.

The newly created thread obtains the information from B l ock i ngQueue and writes to the file. An internal class called a workbench is defined in the async appenderbase, inherits Thread, belongs to a newly built Thread processing class, is responsible for taking out information from B l ock i ngQueue and processing, and is not deeply analyzed because the di srupptor is already packaged in the concrete processing flow.

The start () method of async appenderbase rewrites Unsynchron i zedAppenderBase, starts the work thread, rewrites stop () method, and ends the work thread.

Through the above analysis, various hook methods have been defined in Unsynchron i zedAppenderBase, and only the hook method in the class needs to be rewritten, so that a set of self asynchronous log mechanism can be realized. The specific implementation is as follows:

firstly, defining a plurality of objects, and facilitating subsequent use.

(1.1) defining a Start () method Start thread, stop () method end thread, and processing event hand l eEventsWith () method processing log event for an Interferon d i sruper object to call the object

(1.2) defining a circular buffer rI ngBuffer queue object

(1.3) define a Log event handler l ogEventHand l er object, add the Log to the list of appender appenders at AppenderAttachab l e Imp l for the Di sruper to handle the object

D i sruptorAppender class is newly built, inheriting Unsynchron i zedAppenderBase. The method for rewriting application () adds a log event to the r i ngBuffer, and mainly realizes the following steps:

(2.1) call seq= r i ngbuffer.next () next position in get queue

(2.2) calling Va l ueEvent= r i ngBuffer. Get (seq) to get Va l ueEvent object, and setting log event by Va l ueEvent. SetEvent (eventObject)

(2.3) call r i ngBuffer. Publ ish (seq) adds the log event to r i ngBuffer

The start () method is rewritten and d i srupptor.hand l eEventsWith (l ogEventHand l er) processes the log in the application. Call d i sruppor. Start () starts the thread.

The stop () method is rewritten, and d i srupptor.shutdown () end thread is called.

The replacement of ArrayB l ock i ngQueue by the non-blocking queue di srupptor is accomplished by integrating di srupptor and Unsynchron i zedAppenderBase.

4. Dynamic configuration is realized by the l ogback xml, the l ogback configuration file is inserted into the database, and the database l ogback configuration file is loaded when the project is started; the timing task is newly built, and the database i log back configuration file is reloaded every half hour (time is configurable).

According to another embodiment of the present invention, there is also provided a log asynchronous output processing apparatus, fig. 6 is a block diagram of the log asynchronous output processing apparatus according to an embodiment of the present invention, as shown in fig. 6, the apparatus including:

an obtaining module 62, configured to obtain a length of an original log;

the splicing module 64 is configured to transcode and splice the original log according to the length of the original log to obtain a spliced log;

the reorganization module 66 is configured to reorganize the spliced log in two halves according to the length of the spliced log, so as to obtain a reorganized log;

and the asynchronous output module 68 is used for performing asynchronous output processing on the reorganized log according to the lock-free ring-shaped queue.

Optionally, the splicing module 64 includes:

Optionally, the apparatus further comprises:

Optionally, the reorganizing module 66 is further configured to divide the spliced log into a first log segment and a second log segment according to a length of the spliced log; and sequentially resetting the first log segment and the second log segment, and then combining to obtain the reorganized log.

Optionally, the asynchronous output module 68 includes:

an adding submodule for adding the reorganized log to the lock-free ring queue;

Optionally, the apparatus further comprises:

a definition module for defining the list of appenders in a configuration file;

It should be noted that each of the above modules may be implemented by software or hardware, and for the latter, it may be implemented by, but not limited to: the modules are all located in the same processor; alternatively, the above modules may be located in different processors in any combination.

Embodiments of the present invention also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.

Alternatively, in the present embodiment, the above-described storage medium may be configured to store a computer program for performing the steps of:

s1, acquiring the length of an original log;

s2, transcoding and splicing the original log according to the length of the original log to obtain a spliced log;

s3, performing binary reorganization on the spliced log according to the length of the spliced log to obtain a reorganized log;

s4, carrying out asynchronous output processing on the recombined log according to the lock-ring-free queue.

Alternatively, in the present embodiment, the storage medium may include, but is not limited to: a usb disk, a Read-On-y Memory (ROM), a random access Memory (RandomAccess Memory, RAM), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing a computer program.

An embodiment of the invention also provides an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.

Optionally, the electronic apparatus may further include a transmission device and an input/output device, where the transmission device is connected to the processor, and the input/output device is connected to the processor.

Alternatively, in the present embodiment, the above-described processor may be configured to execute the following steps by a computer program:

s1, acquiring the length of an original log;

Alternatively, specific examples in this embodiment may refer to examples described in the foregoing embodiments and optional implementations, and this embodiment is not described herein.

It will be appreciated by those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may alternatively be implemented in program code executable by computing devices, so that they may be stored in a memory device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps within them may be fabricated into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log asynchronous output processing method, the method comprising:

acquiring the length of an original log;

2. The method of claim 1, wherein transcoding and splicing the original log according to the length of the original log to obtain a spliced log comprises:

3. The method of claim 2, wherein prior to obtaining the length of the original log, the method further comprises:

and if the judgment result is yes, desensitizing the original log.

4. The method of claim 1, wherein performing binary reassembly of the spliced log according to a length of the spliced log, obtaining a reassembly log comprises:

5. The method of claim 1, wherein asynchronously outputting the reorganized log according to a lock-free queue comprises:

establishing the lock-free ring queue for storing log events;

adding the reorganized log to the lock-free ring queue;

6. The method of claim 5, wherein adding the reassembly log to the non-circular queue comprises:

acquiring the next position in the lock-free ring-shaped queue;

setting the reorganization log;

the reassembly log is added to a next location in the non-circular queue.

7. The method of claim 5, wherein prior to asynchronously outputting the reassembly log in accordance with a ring queue, the method further comprises:

defining the list of appenders in a configuration file;

inserting the configuration file into a database;

loading a configuration file in the database at the time of starting; and/or

8. A log asynchronous output processing apparatus, the apparatus comprising:

the acquisition module is used for acquiring the length of the original log;

9. A computer-readable storage medium, characterized in that the storage medium has stored therein a computer program, wherein the computer program is arranged to execute the method of any of the claims 1 to 7 when run.

10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to run the computer program to perform the method of any of the claims 1 to 7.