CN117614999A - Service log transmission and ordered storage method and system - Google Patents

Service log transmission and ordered storage method and system Download PDF

Info

Publication number
CN117614999A
CN117614999A CN202311392446.9A CN202311392446A CN117614999A CN 117614999 A CN117614999 A CN 117614999A CN 202311392446 A CN202311392446 A CN 202311392446A CN 117614999 A CN117614999 A CN 117614999A
Authority
CN
China
Prior art keywords
node
target
information
log
storage system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311392446.9A
Other languages
Chinese (zh)
Inventor
覃开游
金彪
杨东
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang Green Huilian Co ltd
Original Assignee
Zhejiang Green Huilian Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang Green Huilian Co ltd filed Critical Zhejiang Green Huilian Co ltd
Priority to CN202311392446.9A priority Critical patent/CN117614999A/en
Publication of CN117614999A publication Critical patent/CN117614999A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/54—Presence management, e.g. monitoring or registration for receipt of user log-on information, or the connection status of the users
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0803—Configuration setting
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a method and a system for transmitting and orderly storing service logs, which relate to the technical field of data processing, and the method comprises the following steps: the application node sends a registration request to a registration control center, wherein the registration request comprises node configuration information; the registration control center obtains target middleware information corresponding to the application node according to the node configuration information, returns the target middleware information to the application node, and establishes a target log database in a storage system according to the target middleware information; the target middleware corresponding to the target middleware information sends the log data monitored by the application node to a storage system; the storage system stores the log data in a target log database. The invention can realize the registration of the application node and the transmission and storage of the log data, ensures the sequence, consistency and reliability of the log data, and is convenient for the subsequent analysis and processing of the log data.

Description

Service log transmission and ordered storage method and system
Technical Field
The invention relates to the technical field of data processing, in particular to a method and a system for transmitting and orderly storing service logs.
Background
With the continuous development of internet technology, the architecture of an online service system is also evolving continuously. In a high-traffic and high-concurrency scenario, a business system typically needs to employ clustered deployment to deploy applications on multiple nodes. However, in such a distributed environment, application logs generated by individual nodes need to be collected and uniformly stored for subsequent analysis and querying.
The current mainstream log service architecture design typically includes five levels: the system comprises a collection layer, a transmission layer, a processing layer, a storage layer and an access layer, wherein the collection layer is responsible for reading various service logs and is usually realized through various agents. The agents are distributed on each node of the cluster along with the deployment of the service application, read the service log according to the rule and send to the transmission layer; the transport layer is typically implemented by various message middleware, which functions asynchronously and peak clipping and valley filling. The transmission layer receives the log data from the collection layer and transmits the log data to the processing layer; the processing layer receives the log data of the transmission layer, performs necessary processing and conversion, and stores the log data in various storage devices, however, in the process, since the logs of different nodes are written into different log files and sent out by different agents, the order of the logs cannot be ensured, and the disorder brings great inconvenience to the context retrieval of the logs.
Therefore, how to ensure accurate and orderly storage of logs in a cluster is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a method and a system for transmitting and orderly storing business logs, which are used for solving the problem that the logs in the prior art cannot be accurately and orderly stored.
In one aspect, the present invention provides a method for transmitting and orderly storing service logs, including:
the application node sends a registration request to a registration control center, wherein the registration request comprises node configuration information;
the registration control center obtains target middleware information corresponding to the application node according to the node configuration information, returns the target middleware information to the application node, and establishes a target log database in a storage system according to the target middleware information;
the target middleware corresponding to the target middleware information sends the log data monitored by the application node to a storage system;
the storage system stores the log data in a target log database.
Preferably, the node configuration information includes node IP, node name, environment to which the node belongs, application name, and log file path.
Preferably, the target middleware information comprises a transmission intermediary and a physical partition corresponding to the transmission intermediary, and the target log database comprises a library table; one application node corresponds to one physical partition and one physical partition corresponds to one library table.
Preferably, the registration control center obtains target middleware information corresponding to the application node according to the node configuration information, including:
carrying out hash calculation on the node configuration information to obtain a node hash value;
and determining the target middleware information from the registration node mapping table according to the node hash value.
Preferably, the registration node mapping table is used for storing a mapping relation between the node hash value and the middleware information.
Preferably, determining the target middleware information from the registered node mapping table according to the node hash value includes:
judging whether the node hash value exists in a registered node mapping table;
if yes, the middleware information corresponding to the node hash value is read from the registered node mapping table, and the read middleware information is used as target middleware information;
if not, creating a transmission intermediary according to the node name in the node configuration information, creating a physical partition corresponding to the transmission intermediary according to the hash value, and taking the created transmission intermediary and the physical partition as target middleware information.
Preferably, the method further comprises: and acquiring a query request input by a user through a user interface, and acquiring target log data from a target log database according to the query request.
In another aspect, the present invention provides a service log transmission and ordered storage system, including:
the application node is used for sending a registration request to the registration control center, wherein the registration request comprises node configuration information and is also used for monitoring log data;
the registration control center is used for acquiring target middleware information corresponding to the application node according to the node configuration information, returning the target middleware information to the application node, and establishing a target log database in a storage system according to the target middleware information;
the target middleware corresponding to the middleware information is used for sending the monitored log data to a corresponding storage system by the application node;
and the storage system is used for storing the monitored log data to the corresponding target log database.
The beneficial effects of the invention are as follows: a business log transmission and orderly storage method and system can realize the registration of application nodes and the transmission and storage of log data, ensure the sequence, consistency and reliability of the log data and facilitate the subsequent analysis and processing of the log data.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. Like elements or portions are generally identified by like reference numerals throughout the several figures. In the drawings, elements or portions thereof are not necessarily drawn to scale.
FIG. 1 is a flowchart of a method for transmitting and orderly storing service logs according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a service log transmission and ordered storage system according to an embodiment of the present invention;
fig. 3 is a working flow chart of each module in a service log transmission and ordered storage system according to an embodiment of the present invention.
Detailed Description
Embodiments of the technical scheme of the present invention will be described in detail below with reference to the accompanying drawings. The following examples are only for more clearly illustrating the technical aspects of the present invention, and thus are merely examples, and are not intended to limit the scope of the present invention.
It is noted that unless otherwise indicated, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.
Example 1
As shown in fig. 1, the present invention provides a method for transmitting and orderly storing service logs, which includes:
step 1, an application node sends a registration request to a registration control center, wherein the registration request comprises node configuration information;
the service system needs clustered deployment, applications in the whole cluster are deployed in a plurality of application nodes, log data of each application node needs to be collected and uniformly stored so as to be convenient for analysis and processing, namely one application at least corresponds to one application node, each application node is deployed with an Agent, and registration, monitoring and the like are realized through the Agent. Specifically, the node configuration information includes node IP, node name, environment to which the node belongs, application name, and log file path.
Step 2, the registration control center obtains target middleware information corresponding to the application node according to the node configuration information, returns the target middleware information to the application node, and establishes a target log database in a storage system according to the target middleware information;
in the embodiment of the invention, the registration control center obtains the target middleware information corresponding to the application node according to the node configuration information, and the method comprises the following steps: carrying out hash calculation on the node configuration information to obtain a node hash value; and determining the target middleware information from the registration node mapping table according to the node hash value.
Specifically, the target middleware information includes a transmission intermediary (topic) and a physical partition (partition) corresponding to the transmission intermediary, and the target log database includes a library table; one application node corresponds to one physical partition and one physical partition corresponds to one library table.
The registration control center obtains target middleware information corresponding to the application node according to the node configuration information, ensures the connection between the application node and the storage system and the preparation work of data transmission, establishes a target log database according to the node configuration information, provides a uniform storage position, and facilitates subsequent data storage and management.
By using the ordering characteristic of the transmission medium, the ordered writing of the log data into the table of the storage system can be ensured, the log of the same node is written into the same physical partition, the log ordering of the same node is ensured, and meanwhile, the node and the physical partition form a one-to-one relationship by using a hash algorithm, so that the conflict or disorder phenomenon of log writing is avoided, and the consistency of the log is improved.
Step 3, the target middleware corresponding to the target middleware information sends the log data monitored by the application node to a storage system;
and step 4, the storage system stores the log data into a target log database.
The storage system stores the log data into the target log database, so that unified storage of the log data is realized, and subsequent data analysis and processing are facilitated.
In an embodiment of the present invention, the method further includes: and acquiring a query request input by a user through a user interface, and acquiring target log data from a target log database according to the query request.
The user interface (WebUI) provides a user-friendly graphical interface in which a user can enter query conditions, select a particular log type or time range, etc. These query requests will be sent to the storage system, which will then retrieve eligible target log data from the target log database upon request.
For example, a user may fill in a query form on the WebUI interface, specify the start and end time ranges, and provide other filtering conditions, after the user submits a query, the WebUI will send a query request to the backend system, the storage system will parse the query request and perform the corresponding database query operation, retrieve the log data satisfying the conditions from the target log database, and these data may then be displayed to the user on the WebUI for viewing and analysis.
In summary, the embodiment of the invention provides a service log transmission and orderly storage method, which can realize the registration of application nodes and the transmission and storage of log data, ensure the sequence, consistency and reliability of the log data and facilitate the subsequent analysis and processing of the log data.
Example 2
As shown in the figure 2 of the drawings,
the invention provides a business log transmission and ordered storage system, comprising: the application node is used for sending a registration request to the registration control center, wherein the registration request comprises node configuration information and is also used for monitoring log data; the registration control center is used for acquiring target middleware information corresponding to the application node according to the node configuration information, returning the target middleware information to the application node, and establishing a target log database in a storage system according to the target middleware information; the target middleware corresponding to the middleware information is used for sending the monitored log data to a corresponding storage system by the application node; and the storage system is used for storing the monitored log data to the corresponding target log database.
It should be understood that, for the same inventive concept, the service log transmission and ordered storage system provided by the embodiment of the present invention and the service log transmission and ordered storage method provided by the foregoing embodiment, reference may be made to the foregoing embodiment for more specific working principles of each module in the embodiment of the present invention, which is not repeated in the embodiment of the present invention.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention, and are intended to be included within the scope of the appended claims and description.

Claims (8)

1. A method for transmitting and orderly storing service logs, comprising the steps of:
the application node sends a registration request to a registration control center, wherein the registration request comprises node configuration information;
the registration control center obtains target middleware information corresponding to the application node according to the node configuration information, returns the target middleware information to the application node, and establishes a target log database in a storage system according to the target middleware information;
the target middleware corresponding to the target middleware information sends the log data monitored by the application node to a storage system;
the storage system stores the log data in a target log database.
2. The method for transmitting and orderly storing service logs according to claim 1, wherein the node configuration information includes node IP, node name, environment to which the node belongs, application name, and log file path.
3. The method for transmitting and orderly storing service logs according to claim 1, wherein the target middleware information comprises a transmission intermediary and a physical partition corresponding to the transmission intermediary, and the target log database comprises a library table; one application node corresponds to one physical partition and one physical partition corresponds to one library table.
4. The method for transmitting and orderly storing service logs according to claim 2, wherein the registration control center obtains target middleware information corresponding to the application node according to the node configuration information, and the method comprises the following steps:
carrying out hash calculation on the node configuration information to obtain a node hash value;
and determining the target middleware information from the registration node mapping table according to the node hash value.
5. The method for transmitting and orderly storing service logs according to claim 4, wherein the registration node mapping table is used for storing the mapping relation between the node hash value and the middleware information.
6. The method for transmitting and orderly storing service logs according to claim 5, wherein determining the target middleware information from the registered node mapping table according to the node hash value comprises:
judging whether the node hash value exists in a registered node mapping table;
if yes, the middleware information corresponding to the node hash value is read from the registered node mapping table, and the read middleware information is used as target middleware information;
if not, creating a transmission intermediary according to the node name in the node configuration information, creating a physical partition corresponding to the transmission intermediary according to the hash value, and taking the created transmission intermediary and the physical partition as target middleware information.
7. The method for transmitting and orderly storing service logs according to claim 1, further comprising: and acquiring a query request input by a user through a user interface, and acquiring target log data from a target log database according to the query request.
8. A traffic log transmission and ordered storage system, comprising:
the application node is used for sending a registration request to the registration control center, wherein the registration request comprises node configuration information and is also used for monitoring log data;
the registration control center is used for acquiring target middleware information corresponding to the application node according to the node configuration information, returning the target middleware information to the application node, and establishing a target log database in a storage system according to the target middleware information;
the target middleware corresponding to the middleware information is used for sending the monitored log data to a corresponding storage system by the application node;
and the storage system is used for storing the monitored log data to the corresponding target log database.
CN202311392446.9A 2023-10-24 2023-10-24 Service log transmission and ordered storage method and system Pending CN117614999A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311392446.9A CN117614999A (en) 2023-10-24 2023-10-24 Service log transmission and ordered storage method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311392446.9A CN117614999A (en) 2023-10-24 2023-10-24 Service log transmission and ordered storage method and system

Publications (1)

Publication Number Publication Date
CN117614999A true CN117614999A (en) 2024-02-27

Family

ID=89952322

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311392446.9A Pending CN117614999A (en) 2023-10-24 2023-10-24 Service log transmission and ordered storage method and system

Country Status (1)

Country Link
CN (1) CN117614999A (en)

Similar Documents

Publication Publication Date Title
CN106326381B (en) HBase data retrieval method based on MapDB building
CN109117275B (en) Account checking method and device based on data slicing, computer equipment and storage medium
US8775489B2 (en) Database-based logs exposed via LDAP
CN111597148B (en) Distributed metadata management method for distributed file system
CN112445626B (en) Data processing method and device based on message middleware
CN108228322B (en) Distributed link tracking and analyzing method, server and global scheduler
CN110413845B (en) Resource storage method and device based on Internet of things operating system
US11755531B1 (en) System and method for storage of data utilizing a persistent queue
US20180248977A1 (en) Selective distribution of messages in a publish-subscribe system
US11775480B2 (en) Method and system for deleting obsolete files from a file system
CN111831618A (en) Data writing method, data reading method, device, equipment and storage medium
CN111752945B (en) Time sequence database data interaction method and system based on container and hierarchical model
US20060112083A1 (en) Object relation information management program, method, and apparatus
US20120297010A1 (en) Distributed Caching and Cache Analysis
CN107493309B (en) File writing method and device in distributed system
CN111666344A (en) Heterogeneous data synchronization method and device
CN111556123A (en) Self-adaptive network rapid configuration and load balancing system based on edge calculation
CN113094166B (en) Link tracking method, device, medium and computing equipment
CN111274004B (en) Process instance management method and device and computer storage medium
CN107180034A (en) The group system of MySQL database
CN113301558A (en) Message transmission method, device, system and storage medium
CN117614999A (en) Service log transmission and ordered storage method and system
US10812390B2 (en) Intelligent load shedding of traffic based on current load state of target capacity
CN112579552A (en) Log storage and calling method, device and system
US11947490B2 (en) Index generation and use with indeterminate ingestion patterns

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination