KR102639786B1 - System and method for non-disruptive transmitting log using message queue - Google Patents

Abstract

A non-stop log transmission system and method using a message queue is provided. The system includes a first server that issues logs logged by the log generation process in different ways depending on whether there is a network failure, and an offset in a preset order for storing the logs received from the first server. , and a second server that distributes the log to at least one analysis system based on the offset.

Description

Non-stop log transmission system and method using message queue {SYSTEM AND METHOD FOR NON-DISRUPTIVE TRANSMITTING LOG USING MESSAGE QUEUE}

The present invention relates to a system and method for uninterrupted log transmission using a message queue.

A log is a record of computer processing or usage status over time. When an accident occurs, it is helpful in restoring data or identifying the cause of the accident, and prevents and utilizes illegal use of the network or data destruction. It is used as the basis for calculating fees.

In this way, log data transmitted over time may be related to each other. If log transmission is interrupted due to a network failure, even if the failure is recovered and log transmission is restarted, the previously stored log data and the newly transmitted log data Because the connection with is broken, the relationship between the two data cannot be clearly identified, and there is a problem that accurate results cannot be derived when analyzing log data later.

Therefore, a method is needed to transmit log data without interruption even when a network failure occurs.

Public Patent Publication No. 10-2013-0107276, 2013.10.01.

The problem to be solved by the present invention is to provide a system and method for uninterrupted log transmission using a message queue.

However, the problem to be solved by the present invention is not limited to the problems described above, and other problems may exist.

An uninterrupted log transmission system using a message queue according to one aspect of the present invention to solve the above-described problem includes a first server that issues logs logged by a log creation process in different ways depending on whether or not there is a network failure, and the first server. A second server stores the logs received from the server with an offset in a preset order, and distributes the logs to at least one analysis system based on the offsets. The first server includes: If a failure does not occur in the network, the log is issued in the form of a single data and transmitted to the second server. If a failure occurs in the network, a plurality of logs logged during the time the failure persists are stored in the form of a file. , When the failure is recovered, the plurality of logs stored in the file format are issued and transmitted to the second server.

Additionally, the second server may check the log transmission status from the first server at preset intervals, and determine that a failure has occurred in the network if the log is not transmitted for a preset time before and after the current point of time.

In addition, if one log is logged by the log generation process at the time the failure is recovered, the first server first issues the plurality of logs stored sequentially during the time the failure persists, and then issues the one log. can be issued.

In addition, the offset is an integer value sequentially assigned to the logs according to the order of log publication, and the second server stores the logs last published before the occurrence of the failure for the plurality of logs received after the failure is recovered. Offsets can be given sequentially, starting with the value corresponding to the next value of the given offset.

In addition, whenever the failure occurs, the first server calculates the strength, period, and constancy of the signal for data transmission at each point in time when the failure occurred, and calculates a plurality of intensity values calculated for each time point, A first reference value for the intensity, a second reference value for the period, and a third reference value for the constancy are set based on a plurality of period values and a plurality of constancy values, and the first reference value, the It is possible to determine in advance whether the network has a failure using at least one of the second reference value and the third reference value.

In addition, each of the first reference value, the second reference value, and the third reference value is set to an average value, minimum value, or maximum value of each of the plurality of intensity values, the plurality of period values, and the plurality of constancy values. It can be.

Additionally, the first server determines that the failure will occur if the intensity of the signal at the current time is less than the first reference value, and if the period of the signal at the current time is greater than the second reference value. It may be determined that the failure will occur, and if the constancy of the signal at the current time is greater than the third reference value, it may be determined that the failure will occur.

Additionally, when transmitting the plurality of logs after the failure is recovered, the first server may transmit them based on the time the failure lasted and the log transmission speed before the failure occurred.

The uninterrupted log transmission method using a message queue according to another aspect of the present invention to solve the above-described problem is a method performed by a system including a first server for log collection and a second server for loading logs, issuing logs logged by the log generation process in different ways depending on whether there is a network failure; and the second server storing the logs received from the first server by assigning an offset to them in a preset order, Distributing the log to at least one analysis system based on the offset, wherein the first server, when no failure occurs in the network, issues the log in a single data form to the second server. When a failure occurs in the network, a plurality of logs logged during the time the failure persisted are stored in the form of a file, and when the failure is recovered, the plurality of logs stored in the file form are issued and transmitted to the second server. do.

In addition to this, another method for implementing the present invention, another system, and a computer-readable recording medium recording a computer program for executing the method may be further provided.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention described above, logs generated during a time when the network is disconnected are separately stored and then transferred to the big data system after normalization, so that the logs can be loaded in order despite network failure. Accordingly, analysis systems can analyze loaded logs more accurately.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figure 1 is a diagram for explaining an uninterrupted log transmission system using a message queue according to the present invention.
Figure 2 is a flowchart of a non-stop log transmission method using a message queue according to the present invention.
Figures 3a and 3b are diagrams for explaining a log transmission method depending on whether there is a network failure according to the present invention.
Figure 4 is a diagram for explaining offset granting according to the log issuance order according to the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide a general understanding of the technical field to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and every combination of one or more of the referenced elements. Although “first”, “second”, etc. are used to describe various components, these components are of course not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may also be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

Prior to explanation, the meaning of terms used in this specification will be briefly explained. However, since the explanation of terms is intended to aid understanding of the present specification, it should be noted that if it is not explicitly described as limiting the present invention, it is not used in the sense of limiting the technical idea of the present invention.

In this specification, 'device' includes all various devices that can perform computational processing and provide results to the user. For example, devices can be in the form of computers and mobile terminals. The computer may be in the form of a server that receives requests from clients and performs information processing. Additionally, a computer may include a sequencing device that performs sequencing. The mobile terminal includes mobile phones, smart phones, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, laptop PCs, slate PCs, tablet PCs, and ultrabooks. ), wearable devices (for example, a watch-type terminal (smartwatch), a glass-type terminal (smart glass), a head mounted display (HMD)), etc. may be included.

Figure 1 is a diagram for explaining an uninterrupted log transmission system using a message queue according to the present invention.

Figure 2 is a flowchart of a non-stop log transmission method using a message queue according to the present invention.

Figures 3a and 3b are diagrams for explaining a log transmission method depending on whether there is a network failure according to the present invention.

Figure 4 is a diagram for explaining offset granting according to the log issuance order according to the present invention.

System 1 of the present invention may include a first server 10, a second server 20, and an analysis system 30. However, in some embodiments, system 1 may include fewer or more components than those shown in FIG. 1 .

The first server 10 may be a device that collects logged logs through a log generation process, and may be, for example, a log collector. Specifically, the first server 10 may collect recorded logs by logging system status information and operation information received from an external device over time using the log generation process.

The first server 10 may be a producer that issues collected logs. The first server 10 may issue logs in different ways depending on whether there is a network failure. A detailed description of this will be provided later.

As shown in FIG. 1, the first server 10 may include a communication unit 11, a memory 12, and a processor 13. However, in some embodiments, the first server 10 may include fewer or more components than those shown in FIG. 1 .

The communication unit 11 can receive various status and operation information related to system operation from an external device through a communication network. The communication unit 11 may transmit the log to the second server 20 through a communication network.

Here, various types of communication networks may be used, such as wireless LAN (WLAN), Wi-Fi, Wibro, Wimax, and High Speed Downlink Packet Access (HSDPA). Communication methods or wired communication methods such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to The Curb), and FTTH (Fiber To The Home) may be used.

Meanwhile, the communication network is not limited to the communication methods presented above, and may include all types of communication methods that are widely known or will be developed in the future in addition to the communication methods described above.

At least one process for transmitting logs without interruption using a message queue may be stored in the memory 12. The memory 12 may store various information related to log collection.

The processor 13 can perform overall functions for controlling the first server 10 and various operations for transmitting logs without interruption using a message queue. For example, the processor 13 performs the overall function of controlling the first server 10 by executing a program or process stored in the memory 12 and various operations to transmit logs without interruption using a message queue. can do. The processor 13 is implemented as a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), NPU (Neural Processing Unit), or AP (Application Processor) provided in the first server 10. It may be, but is not limited to this.

The second server 20 may be a device that loads logs received from the first server 10 into the log storage 24.

The second server 20 may be a broker that stores published logs with offsets in a preset order. The second server 20 can sequentially store logs issued intermittently due to network failures by sequentially assigning offsets. The second server 20 may distribute the loaded log to at least one analysis system 30 based on the offset. A detailed description of this will be provided later.

As shown in FIG. 1, the second server 10 may include a communication unit 21, a memory 22, and a processor 23. However, in some embodiments, the second server 20 may include fewer or more components than those shown in FIG. 1 .

The communication unit 21 may receive logs from the first server 10 through a communication network. The communication unit 21 may distribute and transmit the loaded logs to at least one analysis system 30 through a communication network.

Here, various types of communication networks may be used, such as wireless LAN (WLAN), Wi-Fi, Wibro, Wimax, and High Speed Downlink Packet Access (HSDPA). Communication methods or wired communication methods such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to The Curb), and FTTH (Fiber To The Home) may be used.

Meanwhile, the communication network is not limited to the communication methods presented above, and may include all other types of communication methods that are widely known or will be developed in the future in addition to the communication methods described above.

At least one process for transmitting logs without interruption using a message queue may be stored in the memory 22. Various information related to log loading may be stored in the memory 12.

The processor 23 can perform various operations to transmit logs without interruption using the overall function for controlling the second server 20 and the message queue. For example, the processor 23 performs the overall function of controlling the second server 20 by executing a program or process stored in the memory 22 and various operations to transmit logs without interruption using a message queue. can do. The processor 23 is implemented as a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), NPU (Neural Processing Unit), or AP (Application Processor) provided in the second server 20. It may be, but is not limited to this.

The external device is a device that provides various status and operation information related to system operation to the first server 10. For example, it may be, but is not limited to, a user terminal.

The analysis system 30 may be a consumer that distributes and processes logs loaded in large quantities. Each of the analysis systems 30 subscribes to a specific topic, and the analysis systems 30 that subscribe to the same topic can distribute and process the logs of the topic.

Hereinafter, with reference to FIGS. 2 and 4, the uninterrupted log transmission method using a message queue performed by the system 1 of the present invention will be described in detail. Each operation of the method according to the present invention may be performed by the first server 10 or the second server 20, and more specifically, the processor 13 of the first server 10 or the second server ( 20) Can be performed by the processor 23.

Referring to FIG. 2, the first server 10 may generate a log by logging system status information and operation information received from an external device through a log creation process (S110).

The first server 10 may determine whether there is a network failure (S120). Since the first server 10 issues logs in different ways depending on whether a network failure occurs, it must first determine whether there is a failure.

Depending on the embodiment, the second server 20 may also determine whether there is a network failure. In more detail, the second server 20 checks the log transmission status from the first server 10 at a preset period, and if the log is not transmitted for a preset time before and after the current point of time, a failure has occurred in the network. It can be judged that For example, the period may be one hour, and the time may be 10 seconds, but the period and time may be set to suit the system situation without being limited thereto.

In step S120, if it is determined that no failure has occurred in the network, the first server 10 can immediately issue a log (S140). More specifically, the first server 10 may issue the log in the form of single data and transmit it to the second server 20. Referring to FIG. 3A, when the network is normally connected, the first server 10 can sequentially issue each log logged by the log generation process one by one and transmit them to the second server 20 in real time.

In step S120, if it is determined that a failure has occurred in the network, the first server 10 may issue a log after the failure is recovered (S140). In more detail, the first server 10 temporarily stores a plurality of logs logged during the time the failure persists in the form of a file (S130), and when the failure is recovered, the first server 10 issues the plurality of logs stored in the file form to It can be transmitted to the second server 20. Referring to FIG. 3b, the first server 10 sequentially and temporarily stores logs that are continuously logged by the log creation process on the internal disk when the network is disconnected, and then stores the accumulated logs when the network is normally restored. It can be published and transmitted to the second server 20 at once.

In step S140, if one log is logged by the log generation process at the time the failure is recovered, the first server 10 first issues the plurality of logs sequentially stored during the time the failure persists and then generates the one log. logs can be published. In this way, by prioritizing the publication of logs logged during a failure over logs logged at the time of recovery, it is possible to maintain the order of logs published regardless of a network failure.

Referring again to FIG. 2, the second server 20 may load the log received from the first server 10 into the log storage 24 (S150). In more detail, the second server 20 may store the received logs with offsets in a preset order.

Here, the offset may be an integer value sequentially given to the log according to the log issuing order.

In a situation where the network is normally connected, the second server 20 may assign an offset value to the log that increases from 0 to 1 in the order in which the log is issued. This offset value can be an ID that identifies the log being loaded.

In a situation where the network is disconnected and then restored normally, the second server 20 determines the plurality of logs (logs temporarily stored in the first server 10 during the failure) received after the failure has been recovered, in response to the failure occurring. Offsets can be given sequentially, starting with the value corresponding to the next value of the offset given to the last published log. Referring to FIG. 4, when a network failure occurs while offset values of 1, 2, and 3 are assigned to log 1, log 2, and log 3 issued before the failure occurs, log 4, log 5, and log 4 generated during the failure occur. Log 6 is temporarily stored within the first server 10. After the failure is recovered, the temporarily stored log 4, log 5, and log 6 are issued, and the second server 20 can sequentially assign 4, 5, and 6 to the published log 4, log 5, and log 6, respectively. .

Depending on the embodiment, the second server 20 may classify and load logs by topic. That is, the logs that are the first topic can be grouped into a first group, offsets can be sequentially given to the logs grouped in the first group and loaded, and the logs that are the second topic can be grouped into a second group. By sequentially assigning offsets to the logs grouped into the second group, each group can be loaded separately.

Referring again to FIG. 2, the second server 20 may transmit the loaded log based on the offset to at least one analysis system 30 (S160). When delivering to the analysis system 30, first-in-first-out is used as a rule, and the logs may be distributed and delivered to at least one analysis system 30 in order, starting from the log with an offset value of 0.

Depending on the embodiment, when logs are classified and loaded by topic, the log of the topic may be delivered to the analysis system 30 that has subscribed to the specific topic. For example, when the analysis systems 30 a, b, and c subscribe to the first topic, the logs of the first topic may be distributed and delivered to each of the analysis systems 30 a, b, and c.

According to the embodiment, whenever the failure occurs, the first server 10 calculates the strength, period, and constancy of the signal for data transmission at each point in time when the failure occurs, and calculates the signal strength, period, and constancy for data transmission at each point in time when the failure occurs, and calculates the signal strength, period, and constancy for data transmission at each point in time when the failure occurs. Setting a first reference value for the intensity, a second reference value for the period, and a third reference value for the constancy based on a plurality of intensity values, a plurality of period values, and a plurality of constancy values, and It is possible to determine in advance whether the network has a failure using at least one of the 1 reference value, the second reference value, and the third reference value.

In other words, since the strength, period, and constancy of the signal when a network failure occurs are different from the strength, period, and constancy of the signal under normal circumstances, the corresponding values are calculated each time a failure occurs, and a reference value based on the corresponding values is used. Afterwards, the presence of a failure can be determined in advance by determining whether there is an abnormality in the strength, period, and constancy of the signal.

Each of the first reference value, the second reference value, and the third reference value may be set to an average value, minimum value, or maximum value of each of the plurality of intensity values, the plurality of period values, and the plurality of constancy values. there is.

For example, when a third failure occurs, the intensity, period, and constancy at the time of the failure are calculated, and the intensity, period, and constancy values for the third failure are calculated along with the previously stored intensities for the first and second failures, Using the period and constancy values, each reference value can be set. More specifically, the average value, minimum value, or maximum value of the three values for intensity is calculated and set as the first reference value, and the average value, minimum value, or maximum value of the three values for the period are calculated and set as the second reference value. And, the average value, minimum value, or maximum value of the three values for constancy can be calculated and set as the third reference value.

Then, the first server 10 continues to monitor the signal strength, period, and constancy, and if at least one of the signal strength, period, and constancy values at the current point is different from each reference value. It can be predicted that a network failure will occur.

Specifically, the first server 10 may determine that the failure will occur if the signal strength at the current time is less than the first reference value.

The first server 10 may determine that the failure will occur if the signal period at the current time is greater than the second reference value.

The first server 10 may determine that the failure will occur if the constancy of the signal at the current time is greater than the third reference value.

Depending on the embodiment, when transmitting the plurality of logs after the failure is recovered, the first server 10 may transmit them based on the time the failure lasted and the log transmission speed before the failure occurred.

In more detail, the log transmission speed (v ₂ ) after failure recovery can be calculated based on Equation 1 below.

Here, v ₁ is the original transmission speed, v ₂ is the log transmission speed after failure recovery, t ₁ is the failure duration, and t _r is the reference time.

According to Equation 1 above, as the failure duration increases, data can be transmitted faster than the original transmission speed. Here, t _r may be set in advance.

Figure 2 depicts steps S110 to S160 as being sequentially executed, but this is merely an illustrative explanation of the technical idea of this embodiment, and those skilled in the art will understand the steps of this embodiment. Various modifications and modifications can be made by executing by changing the order shown in FIG. 2 or executing steps S110 to S160 in parallel without departing from the essential characteristics, so FIG. 2 is not limited to a time-series order. .

Meanwhile, in the above description, steps S110 to S160 may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present invention. Additionally, some steps may be omitted or the order between steps may be changed as needed.

The uninterrupted log transmission method using a message queue according to the present invention described above may be implemented as a program (or application) to be executed in conjunction with a computer as hardware and stored in a computer-readable recording medium.

The above-mentioned program is C, C++, JAVA, Ruby, and It may include code encoded in a computer language such as machine language. These codes may include functional codes related to functions that define the necessary functions for executing the methods, and include control codes related to execution procedures necessary for the computer's processor to execute the functions according to predetermined procedures. can do. In addition, these codes may further include memory reference-related codes that indicate at which location (address address) in the computer's internal or external memory additional information or media required for the computer's processor to execute the above functions should be referenced. there is. In addition, if the computer's processor needs to communicate with any other remote computer or server in order to execute the above functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes regarding whether communication should be performed and what information or media should be transmitted and received during communication.

The storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers that the computer can access or on various recording media on the user's computer. Additionally, the medium may be distributed to computer systems connected to a network, and computer-readable code may be stored in a distributed manner.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as single may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

10: First server
11: Department of Communications
12: memory
13: processor
20: Second server
21: Department of Communications
22: memory
23: processor
24: Log storage
30: Analysis system

Claims (10)

a first server that issues logs logged by the log generation process in different ways depending on whether there is a network failure; and
A second server that stores the logs received from the first server with offsets in a preset order and distributes the logs to at least one analysis system based on the offsets,
The first server is,
If no failure occurs in the network, the log is issued in a single data form and transmitted to the second server,
When a failure occurs in the network, a plurality of logs logged during the time the failure persisted are stored in the form of a file, and when the failure is recovered, the plurality of logs stored in the file form are issued and transmitted to the second server,
The first server is,
Whenever the failure occurs, calculate the strength, period, and constancy of the signal for data transmission at each point in time when the failure occurs,
A first reference value for the intensity, a second reference value for the period, and a third reference for the constancy based on the plurality of intensity values, plurality of period values, and plurality of constancy values calculated for each time point. An uninterrupted log transmission system using a message queue that sets a value and determines in advance whether there is a problem in the network using at least one of the first reference value, the second reference value, and the third reference value. According to claim 1,
The second server is,
An uninterrupted log transmission system using a message queue that checks the log transmission status from the first server at a preset period and determines that a failure has occurred in the network if the log is not transmitted for a preset time before and after the current point of time. According to claim 1,
The first server is,
If one log is logged by the log creation process at the time the failure is recovered, the plurality of logs sequentially stored during the time the failure persists are first issued and then the single log is issued. Non-stop using a message queue. Log transmission system. According to claim 1,
The offset is an integer value sequentially given to the log according to the log issuing order,
The second server sequentially assigns offsets to the plurality of logs received after recovering from the failure, starting with a value corresponding to the next value of the offset given to the log last published before the failure occurred. Non-stop log transmission system using a queue. delete According to claim 1,
Each of the first reference value, the second reference value, and the third reference value is,
An uninterrupted log transmission system using a message queue, wherein each of the plurality of intensity values, the plurality of period values, and the plurality of constancy values is set to an average value, minimum value, or maximum value. According to claim 1,
The first server is,
If the intensity of the signal at the current time is less than the first reference value, it is determined that the failure will occur,
If the period of the signal at the current time is greater than the second reference value, it is determined that the failure will occur,
An uninterrupted log transmission system using a message queue that determines that the failure will occur if the constancy of the signal at the current time is greater than the third reference value. According to claim 1,
The first server is,
An uninterrupted log transmission system using a message queue that transmits the plurality of logs after the failure is recovered, based on the time the failure lasted and the log transmission speed before the failure occurred. In a method performed by a system including a first server for log collection and a second server for log loading,
issuing logs logged by the log generation process in different ways depending on whether there is a network failure; and
The second server stores the logs received from the first server with offsets in a preset order, and distributes the logs to at least one analysis system based on the offsets; ,
The first server is,
If no failure occurs in the network, the log is issued in a single data form and transmitted to the second server,
When a failure occurs in the network, a plurality of logs logged during the time the failure persisted are stored in the form of a file, and when the failure is recovered, the plurality of logs stored in the file form are issued and transmitted to the second server,
The first server is,
Whenever the failure occurs, calculate the strength, period, and constancy of the signal for data transmission at each point in time when the failure occurs,
A first reference value for the intensity, a second reference value for the period, and a third reference for the constancy based on the plurality of intensity values, plurality of period values, and plurality of constancy values calculated for each time point. A method of non-stop log transmission using a message queue, wherein a value is set and at least one of the first reference value, the second reference value, and the third reference value is used to determine whether the network has a failure in advance. A program coupled to a computer and stored on a computer-readable recording medium for executing the method of claim 9.

Images