CN111143286B - Cloud platform log management method and system - Google Patents
Cloud platform log management method and system Download PDFInfo
- Publication number
- CN111143286B CN111143286B CN201911280687.8A CN201911280687A CN111143286B CN 111143286 B CN111143286 B CN 111143286B CN 201911280687 A CN201911280687 A CN 201911280687A CN 111143286 B CN111143286 B CN 111143286B
- Authority
- CN
- China
- Prior art keywords
- log
- lightweight
- search server
- request
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007726 management method Methods 0.000 title claims abstract description 56
- 230000000007 visual effect Effects 0.000 claims abstract description 19
- 238000000034 method Methods 0.000 claims abstract description 16
- 230000008859 change Effects 0.000 claims abstract description 6
- 238000004148 unit process Methods 0.000 claims abstract description 5
- 238000012544 monitoring process Methods 0.000 claims abstract description 3
- 238000012545 processing Methods 0.000 claims description 42
- 230000004044 response Effects 0.000 claims description 15
- 238000006243 chemical reaction Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000012800 visualization Methods 0.000 abstract description 16
- 230000006872 improvement Effects 0.000 description 12
- 238000004458 analytical method Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000012550 audit Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 229920001971 elastomer Polymers 0.000 description 2
- 239000000806 elastomer Substances 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- WGZDBVOTUVNQFP-UHFFFAOYSA-N N-(1-phthalazinylamino)carbamic acid ethyl ester Chemical compound C1=CC=C2C(NNC(=O)OCC)=NN=CC2=C1 WGZDBVOTUVNQFP-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 239000010979 ruby Substances 0.000 description 1
- 229910001750 ruby Inorganic materials 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides a cloud platform log management method and a cloud platform log management system, which are applied to electronic equipment, wherein the method comprises the following steps: the service micro-service unit processes the service Request input from the visual interface and binds the Request ID for the service Request; collecting call information and user information associated with a service Request, and forming a log corresponding to the service Request together with a Request ID to write a log file; monitoring the change condition of a log file through a lightweight log collector, converting the collected log into a lightweight text according to a data template corresponding to a configuration file, and sending the lightweight text to a search server; the log micro-service unit queries the lightweight text from a query interface configured by the search server and displays the lightweight text in the visual interface. The invention realizes the visualization and integrated management of the log file generated by the service request, and improves the query and statistics efficiency of the log file.
Description
Technical Field
The invention relates to the technical field of log file management, in particular to a cloud platform log management method and system.
Background
With the increasing size and complexity of software systems and cloud platforms, a large amount of log information is generated, and these logs need to be stored for query and analysis, while conventional relational databases (e.g., mySQL) have a certain limitation on the ability of log storage, query, and analysis. For example, log information is usually stored in time sequence, a plurality of pieces of log information are formed in a response process of one request, log information of different requests is mixed and stored in a database, and when the log information amount is very large, it is difficult to separate log information generated by a specific request when analyzing a log file.
Currently, a software architecture of a micro service is generally adopted in a cloud platform. Micro-services are supporting micro-services that split a large single application and service into tens of. Massive operations and logs generated by request events can be generated in a cloud platform based on a micro-service software architecture. Thereby causing log files in the cloud platform to assume discrete and unordered logical states. Therefore, it is particularly important to manage logs in a cloud platform based on a micro-service software architecture.
Disclosure of Invention
The invention aims to disclose a cloud platform log management method and a cloud platform log management system based on the cloud platform log management method, which are used for realizing the visual management of cloud platform logs so as to adapt to the increasingly-increased analysis and management capacity of log files and provide references for reasonable resource allocation of service requests.
In order to achieve the first object, the present invention provides a cloud platform log management method, which is applied to an electronic device, and includes the following steps:
s1, a service micro-service unit processes a service Request input from a visual interface and binds a Request ID for the service Request;
s2, collecting call information and user information associated with the service Request, and forming a log corresponding to the service Request together with the Request ID so as to write a log file;
s3, monitoring the change condition of the log file through a lightweight log collector, converting the collected log into a lightweight text according to a data template corresponding to the configuration file, and sending the lightweight text to a search server;
s4, the log micro-service unit queries the lightweight text from a query interface configured by the search server and displays the lightweight text in the visual interface.
As a further improvement of the present invention, the lightweight text includes JSON file, XML file, TXT file, or Binlog file.
As a further improvement of the present invention, the service request in the step S1 is processed by a plurality of service micro service units respectively to form a plurality of log files, and the plurality of log files are associated to form log details.
As a further improvement of the present invention, the log file includes a service Request time stamp, service Request call information, service Request processing result, and Request ID.
As a further improvement of the present invention, the step S2 further includes: and taking the Request ID and the time stamp as the processing time length of the input calculation service Request, and writing the processing time length into a log file.
As a further improvement of the present invention, the step S3 further includes: detecting the request pressure of a search server, wherein the lightweight log collector adjusts the receiving and transmitting speed between the lightweight text and the search server according to the request pressure of the search server;
the method specifically detects the request pressure of the search server, and comprises the following steps:
judging whether the event number of the lightweight text sent to the search server exceeds the maximum batch processing event number set by the search server or not through the lightweight log collector;
if yes, reducing the receiving and transmitting speed between the lightweight text and the search server;
if not, the transceiving speed between the lightweight text and the search server is increased, and the mass processing maximum event number set by the search server is increased.
As a further improvement of the present invention, after the lightweight text is sent to the search server in step S3, the method further includes:
and writing the lightweight text into a search server response body.
Based on the same thought, the application also discloses a cloud platform log management system which is applied to the electronic equipment and comprises the following steps:
the service micro-service unit processes the service Request input from the visual interface, binds a Request ID for the service Request, collects call information and user information associated with the service Request, and forms a log corresponding to the service Request together with the Request ID so as to write a log file;
the conversion module is used for converting the collected logs into lightweight texts;
the lightweight log collector monitors the change condition of the log file, converts the collected log into a lightweight text according to a data template corresponding to the configuration file and sends the lightweight text to the search server;
the log micro-service unit inquires a lightweight text from an inquiry interface configured by the search server and displays the lightweight text in a visual interface;
and the search server receives and stores the log converted into the lightweight text and responds to the log micro-service unit.
As a further improvement of the present invention, the lightweight text includes JSON file, XML file, TXT file, or Binlog file; the log file comprises a service Request time stamp, service Request calling information, a service Request processing result and a Request ID.
As a further improvement of the present invention, the search server sets a batch processing maximum event number;
the lightweight log collector judges whether the event number of the lightweight text sent to the search server exceeds the maximum batch processing event number set by the search server;
if yes, reducing the receiving and transmitting speed between the lightweight text and the search server;
if not, the transceiving speed between the lightweight text and the search server is increased, and the mass processing maximum event number set by the search server is increased.
As a further improvement of the present invention, the lightweight log collector is selected from Filebeat, flume, fluentd, logagent, rsyslog or Syslog-ng; the search server is selected from the group consisting of an elastic search.
As a further improvement of the invention, the lightweight log collector and the search server are deployed in an elastic search cluster;
as a further improvement of the invention, the service micro-service unit takes the Request ID and the time stamp as the input to calculate the processing time length of the service Request, and writes the processing time length into the log file.
Compared with the prior art, the invention has the beneficial effects that:
according to the cloud platform log management method and system disclosed by the invention, the efficient management of the log files generated based on the service requests initiated by users and/or administrators can be efficiently integrated, and the visual management of the cloud platform log is realized, so that the method and system are better suitable for executing statistics, audit, positioning, analysis and management capabilities on increasingly-increased log files, and can provide references for reasonable resource configuration of the service requests.
Drawings
FIG. 1 is a general flow chart of a cloud platform log management method of the present invention;
FIG. 2 is a topology diagram of a cloud platform log management system running the cloud platform log management method;
FIG. 3 is a topology diagram of a cloud platform log management system running the cloud platform log management method in a variation;
FIG. 4 is a diagram showing an interface for performing a visualization operation on a log file embedded in a visualization interface in a cloud platform log management method;
fig. 5 is a topology diagram of an electronic device running the cloud platform log management method.
Detailed Description
The present invention will be described in detail below with reference to the embodiments shown in the drawings, but it should be understood that the embodiments are not limited to the present invention, and functional, method, or structural equivalents and alternatives according to the embodiments are within the scope of protection of the present invention by those skilled in the art.
Embodiment one:
the embodiment firstly discloses a specific implementation manner of a cloud platform log management method.
The cloud platform log management method is run and applied in the electronic device 100. The electronic device 100 may be configured not only as a data center, a cloud computing platform (or simply "cloud platform 10") or a server based on a super fusion architecture, but also as other devices having independent computing storage communication functions, such as a personal computer, tablet computer, or wearable smart device. In particular, the "server" disclosed in the present embodiment includes, but is not limited to, a file server, a proxy server, or an application server. The log file related in the cloud platform log management method refers to: the methods and systems disclosed in the various embodiments disclosed herein are particularly well suited in distributed scenarios based on the electronic device 100 receiving user and/or administrator initiated access requests, responses, and log files or collections thereof of events generated during execution of transactions.
Meanwhile, each application server (e.g., tomcat, jboss) of the electronic device 100 configures the lightweight log collector 13, and the application server generates a unique Request ID based on one or more service micro-service units 14 configured in the cloud platform 10 according to the access Request, and outputs call information, user name, authentication information generated based on the access Request to the log according to the access Request (e.g., create cloud host, cancel cloud host, migrate cloud host, purchase application software, pay commodity price, modify commodity order, etc.) initiated by the user. The Request ID may form the individual logs into a log call chain. The log carrying the Request ID is convenient for later fault detection, inquiring the content or statistics of the user Request and other requirements.
In the cloud platform log management method and system disclosed in the application, through the lightweight log collection 13 and the search server 12, a log call chain with a Request ID generated by a plurality of business micro-service units 14 is configured in the search server 12 to be used for inquiring a query interface of a lightweight text, so that the query interface is finally displayed outwards through a visual interface 17 configured by the log micro-service unit 11, and a user or an administrator can inquire, count, analyze, locate and audit the log through the visual interface 17, so that the cloud platform log management method and system has great technical application value in application scenes of electronic commerce and electronic payment. The visual interface 17 may be loaded in a medium such as a display, browser, etc.; wherein, the meaning of "show to the outside" means: receiving instructions input by a user and/or an administrator in a UI manner or presenting visual results presented based on the instructions input by the user and/or the administrator, and the presented visual results are in forms including, but not limited to, graphics, tables, interactive files, audio files, flash files, and the like.
Hereinafter, the cloud platform log management method is further described in detail.
Referring to fig. 1 and fig. 2, this embodiment discloses a specific embodiment of a cloud platform log management method.
In the present embodiment, the cloud platform log management method (hereinafter referred to as "method") is applied to the electronic device 100. The electronic device 100 is described in detail using the cloud platform 10 as an example in this embodiment. The method comprises the following steps:
first, step S1 is performed, and the service micro service unit 14 processes the service Request input from the visualization interface 17 (see arrow 111 in fig. 2), and binds the Request ID for the service Request. The service request in this step S1 is processed by the plurality of service micro service units 14, respectively, to form a plurality of log files 15, and the plurality of log files 15 are associated to form log details.
The user/administrator initiates an access request to cloud platform 10 through visualization interface 17 and presents the visualization results to the user/administrator through visualization interface 17, with logic shown by double-headed arrow 110. The cloud platform 10 passes the service request in the direction indicated by arrow 101 to the service micro service unit 14 for processing according to the service type, respectively, and writes one or more log files 15 in the direction indicated by arrow 102 to the storage medium 200 for saving the log files 15. The storage medium 200 includes, but is not limited to, a SAN storage device, NAS storage device, CEPH storage device, NAND storage device, NVDIMM storage device, or cache queue.
Then, step S2 is performed to collect call information and user information associated with the service Request, and form a log corresponding to the service Request together with the Request ID, so as to write the log file 15.
The log file 15 includes a service Request time stamp, service Request call information, service Request processing result, and Request ID. Preferably, in this embodiment, the step S2 further includes: the Request ID and the time stamp are taken as the processing time length of the input calculation service Request, and the processing time length is written into the log file 15. By calculating the processing time length of the service request, the user/administrator can conveniently know the processing progress and the processing result of the specific access request by the service micro-service unit 14 in real time in a transparent mode; when the service processing capability of the service micro service unit 14 is insufficient to support the access request of the user/administrator in response to the access request, reporting to the cloud platform 10 and expanding the capacity of the service micro service unit 14; of course, in order to improve the resource utilization efficiency of the service micro service unit 14, the service micro service unit 14 may be scaled when the service processing capability of the service micro service unit 14 exceeds the access request of the user/administrator. Through the capacity expansion and contraction operation, the service micro service unit 14 running in the cloud platform obviously improves the use efficiency of virtual resources in the cloud platform 10, avoids the waste of the virtual resources, and can provide references for reasonable resource configuration of service requests.
Then, step S3 is executed, and the change condition of the log file 15 is monitored by the lightweight log collector 13, and the collected log is converted into a lightweight text according to the data template corresponding to the configuration file and then sent to the search server 12 (shown by reference arrow 104). The log file 15 is called by the conversion module 16 in the direction indicated by arrow 105 and converted to lightweight text, such as JSON file, XML file, TXT file or Binlog file, and sent to the search server 12 in the direction indicated by arrow 106.
Specifically, the lightweight log collector 13 is selected from Filebeat, flume, fluentd, logagent, rsyslog or Syslog-ng; the search server 12 is selected from the group consisting of elastiscearch. In this embodiment, the lightweight log collector 13 does not use logstack, but uses Filebeat, flume, fluentd, logagent, rsyslog or Syslog-ng to have more excellent technical advantages. The lightweight log collector 13 in the form of logstack is adopted to have a severe performance requirement on a physical server in the cloud platform 10, and does not support caching, so that the lightweight log collector is not applicable to the cloud platform based on the micro-service architecture. Thus, in this embodiment, the lightweight log collector 13 is selected from Filebeat, flume, fluentd, logagent, rsyslog or Syslog-ng, and is most preferably Filebeat. The filebean may send the collected and converted lightweight text to the search server 12 in the direction indicated by arrow 106 and have the search server 12 perform parsing and/or storage without the use of a cache. It should be noted that the conversion operation performed by the conversion module 16 in fig. 2 may also be performed by a unit similar to the conversion module 16 built in the search server 12. The filecoat can rapidly collect incremental content of the log file 15, and can continuously collect the log file 15 of the incremental part at the position of the node where the program is interrupted or the service is crashed, so that the integrity of the log file 15 is ensured, and the loss of data and processes is prevented.
Various modules (auditd, apache, NGINX, system, mySQL and the like) are arranged in the filecoat, so that the collection, analysis and visualization processes of the logs in a common format can be greatly simplified, and the collection, analysis and visualization of the logs can be realized by only one command. The plurality of filebeans also includes a preconfigured Machine Learning task.
Meanwhile, fluent d is developed by using Ruby language, and has the advantage of being very easy to write and maintain, and almost all source and target stores have plug-ins, so that fluent d can be used for connecting all lightweight files in series. In addition, loggent occupies minimal machine CPU, memory resources, thereby enabling an improvement in quality of service (QoS) of the cloud platform 10 for access requests.
In this embodiment, after the lightweight text is sent to the search server 12, the method further includes: and writing the lightweight text into a search server response body.
For example, the index (_index) is website, the type (_type) is blog, the_id is 123, the version number (_version) is 1, and the state is true, and the data format of the search server response body is as follows:
by the technical means of writing the lightweight text into the search server response body, the difficulty in subsequent work such as searching, statistics, analysis and the like of the lightweight text through the search server response body is obviously reduced.
The lightweight log collector 13 retrieves the log file 15 from the storage medium 200 in the direction indicated by arrow 103. The lightweight log collector 13 creates a search server response body from the time axis to save log information to the search server response body.
In particular, in the present embodiment, the step S3 further includes: detecting the request pressure of the search server 12, wherein the lightweight log collector 13 adjusts the transceiving speed between the lightweight text and the search server 12 according to the request pressure of the search server 12;
the detection of the request pressure of the search server 12 specifically includes:
judging whether the event number of the lightweight text sent to the search server 12 exceeds the batch processing maximum event number (bulk_max_size) set by the search server 12 by the lightweight log collector 13;
if yes, the transceiving speed between the lightweight text and the search server 12 is reduced;
if not, the speed of sending and receiving the lightweight text to and from the search server 12 is increased to the maximum number of batch processing events set by the search server 12. In the present embodiment, the batch processing maximum event number (bulk_max_size) is set to 50. By the technical scheme, the reliability and the usability of the cloud platform log management method are further improved, so that the efficiency of the lightweight date collector 13 writing the log file 15 converted into the lightweight text into the search server 12 can be improved within the set range of the maximum batch processing event number of the search server 12, congestion of the search server 12 when receiving the lightweight text is prevented, and the reliability of a user/manager for inquiring and positioning the log file 15 is improved.
Finally, step S4 is executed, and the log micro service unit 11 queries the lightweight text from the query interface configured by the search server 12, and displays the lightweight text to the outside in the visualization interface 17. Specifically, the log micro service unit 11 inquires the lightweight text from the inquiry interface configured by the search server 12 in the direction shown by an arrow 107, and sends the inquired lightweight text into the visualization interface 17 as shown by an arrow 109 to be learned by the user/administrator. When a user/administrator searches the visualization interface 17 for a specific log file 15, the cloud platform 10 initiates a call Request to the log micro service unit 11 through an arrow 108, and according to the Request ID, the log call chains formed by the logs can be visualized in the visualization interface 17 in the direction indicated by an arrow 109. Logically, as shown in connection with fig. 2, the lightweight date collector 13 and the search server 12 are both deployed in the elastomer search cluster 20 at the same time, and the elastomer search cluster 20 may be built by a Docker.
As shown in fig. 4, the visual interface 17 is embedded with a domain name input field 201, a search field 202, a log list 203, log details 204, and a log call chain 205 corresponding to the log details 204 based on the query; the log list 203 includes fields such as operation time, operation action, operation object, operation duration, operation details, etc., and the fields in the log list 203 may be arbitrarily set according to the actual needs of the user/administrator. Clicking on the "view" button control 206 in the log list 203 turns around the dialog box displaying log details 204, which log details 204 contain the log call chain 205. Meanwhile, in this embodiment, the log call chain 205 displays the log call chain formed based on the operation corresponding to a certain access request in the form of "waterfall stream", and displays the state (for example, success or failure) of each log call chain, and can visually display the prompt information of the log file call error. The search bar 202 can manually key in operation objects, operation actions and the like to search log files so as to improve the efficiency of searching, positioning, auditing and counting the log files.
Embodiment two:
referring to fig. 2 and fig. 3, the present embodiment further discloses a cloud platform log management system based on the inventive concept of the cloud platform log management method disclosed in the first embodiment.
A cloud platform log management system, applied to an electronic device 100, comprising: the system comprises a business micro-service unit 14, a conversion module 16, a lightweight log collector 13, a search server 12, a log micro-service unit 11 and a visualization interface 17.
Specifically, the service microservice unit 14 processes the service Request input from the visualization interface 17, binds the Request ID for the service Request, collects call information and user information associated with the service Request, and forms a log corresponding to the service Request together with the Request ID to write the log file 15. Preferably, the service micro service unit 14 takes the Request ID and the timestamp as input to calculate the processing duration of the service Request, and writes the processing duration into the log file 15. The conversion module 16 converts the collected log into lightweight text. The lightweight log collector 13 monitors the variation condition of the log file 15, converts the collected log into lightweight text according to a data template corresponding to the configuration file, and sends the lightweight text to the search server 12. The log micro service unit 11 queries the lightweight text from the query interface configured by the search server 12 and is presented externally in the visualization interface 17. The search server 12 receives and saves the log converted into the lightweight text, and responds to the log micro service unit 11. The lightweight log collector 13 and search server 12 are deployed in an elastic search cluster 20.
The lightweight text comprises a JSON file, an XML file, a TXT file or a Binlog file; the log file 15 includes a service Request time stamp, service Request call information, service Request processing result, and Request ID. In the present embodiment, the search server 12 sets the batch processing maximum event number.
The lightweight log collector 13 judges whether the number of events of the lightweight text sent to the search server 12 exceeds the maximum number of events of batch processing set by the search server 12;
if yes, the transceiving speed between the lightweight text and the search server 12 is reduced;
if not, the speed of sending and receiving the lightweight text to and from the search server 12 is increased to the maximum number of batch processing events set by the search server 12.
The lightweight log collector 13 is selected from Filebeat, flume, fluentd, logagent, rsyslog or Syslog-ng and the search server 12 is selected from elastiscearch. In this embodiment, rsysllog is selected. The Rsyslog has good support for Mysql, postgreSQL, oracle and the like, and has the functions of screening log contents, defining log format templates and the like. Besides the default UDP protocol, rsyslog also supports the TCP protocol to receive logs, can be installed not only by yum, but also by source codes, and has good deployment universality.
The cloud platform log management system disclosed in the present embodiment operates the cloud platform log management method disclosed in the first embodiment. The specific technical scheme of the cloud platform log management method disclosed in this embodiment is described in embodiment one, and is not repeated here.
Embodiment III:
a variant embodiment of a cloud platform log management system according to the present invention is disclosed with reference to fig. 4.
The main difference between this embodiment and the first and/or second embodiment, especially compared to the embodiment, is that in this embodiment, in the cloud platform log management system,
the log file 15 which has been converted into the lightweight text is not sent to the search server 12 but to the lightweight log collector 13 in the direction indicated by the arrow 112, and finally the log file 15 which has been converted into the lightweight text is sent to the search server 12 by the lightweight log collector 13; the conversion module 16 may even be deployed in the elastiscearch cluster 20 at the same time.
The technical solutions of the cloud platform log management system disclosed in the present embodiment and the same parts of the first embodiment and/or the second embodiment are referred to above, and are not described herein again.
Embodiment four:
finally, referring to fig. 5, an electronic device 100 is also disclosed. The electronic device 100 may be configured not only as a data center, a cloud computing platform (or simply "cloud platform 10") or a server based on a super fusion architecture, but also as other devices having independent computing storage communication functions, such as a personal computer, tablet computer, or wearable smart device.
The electronic device 100 includes: a processor 51, a storage 52, a communication bus 53 establishing a communication connection between the processor 51 and the storage 52, and an output device 54. The processor 51 is configured to execute one or more programs stored in the storage device 52 to implement a cloud platform log management method according to the first embodiment. The storage device 52 includes a plurality of storage units, i.e. storage unit 521 to storage unit 52i, where the parameter i is a positive integer greater than or equal to 2. The output device 54 may be a display, keyboard and mouse, or an interactive device, such as a touch screen, that may receive user/administrator entered instructions.
The output device 54 has a visual interface 17 through which the user/administrator visually manages recorded files or collections thereof (i.e., log files) of access requests, responses, and events generated during execution of transactions initiated by the user and/or administrator.
The technical solutions of the same parts of the electronic device 100 disclosed in this embodiment and any one of the first to third embodiments are described above, and are not repeated here.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above list of detailed descriptions is only specific to practical embodiments of the present invention, and they are not intended to limit the scope of the present invention, and all equivalent embodiments or modifications that do not depart from the spirit of the present invention should be included in the scope of the present invention.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Furthermore, it should be understood that although the present disclosure describes embodiments, not every embodiment is provided with a separate embodiment, and that this description is provided for clarity only, and that the disclosure is not limited to the embodiments described in detail below, and that the embodiments described in the examples may be combined as appropriate to form other embodiments that will be apparent to those skilled in the art.
Claims (12)
1. The cloud platform log management method is applied to electronic equipment and is characterized by comprising the following steps of:
s1, a service micro-service unit processes a service Request input from a visual interface and binds a Request ID for the service Request;
s2, collecting call information and user information associated with the service Request, and forming a log corresponding to the service Request together with the Request ID so as to write a log file;
s3, monitoring the change condition of the log file through a lightweight log collector, converting the collected log into a lightweight text according to a data template corresponding to the configuration file, sending the lightweight text to a search server, and writing the lightweight text into a search server response body, wherein the data format of the search server response body comprises an index, a type, an id, a version number and a state;
s4, the log micro-service unit queries the lightweight text from a query interface configured by the search server and displays the lightweight text in the visual interface.
2. The cloud platform log management method of claim 1, wherein the lightweight text comprises JSON files, XML files, TXT files, or Binlog files.
3. The cloud platform log management method according to claim 1, wherein the service request in step S1 is processed by a plurality of service micro service units respectively to form a plurality of log files, and the plurality of log files are associated to form log details.
4. The cloud platform log management method of claim 1, wherein the log file comprises a service Request timestamp, service Request invocation information, service Request processing results, and a Request ID.
5. The cloud platform log management method according to claim 1, wherein the step S2 further comprises: and taking the Request ID and the time stamp as the processing time length of the input calculation service Request, and writing the processing time length into a log file.
6. The cloud platform log management method according to claim 1, wherein said step S3 further comprises: detecting the request pressure of a search server, wherein the lightweight log collector adjusts the receiving and transmitting speed between the lightweight text and the search server according to the request pressure of the search server;
the method specifically detects the request pressure of the search server, and comprises the following steps:
judging whether the event number of the lightweight text sent to the search server exceeds the maximum batch processing event number set by the search server or not through the lightweight log collector;
if yes, reducing the receiving and transmitting speed between the lightweight text and the search server;
if not, the transceiving speed between the lightweight text and the search server is increased, and the mass processing maximum event number set by the search server is increased.
7. The cloud platform log management system is applied to electronic equipment and is characterized by comprising the following components:
the service micro-service unit processes the service Request input from the visual interface, binds a Request ID for the service Request, collects call information and user information associated with the service Request, and forms a log corresponding to the service Request together with the Request ID so as to write a log file;
the conversion module is used for converting the collected logs into lightweight texts;
the lightweight log collector monitors the change condition of log files, converts the collected logs into lightweight texts according to a data template corresponding to the configuration files, and then sends the lightweight texts to the search server, and the lightweight texts are written into a search server response body, wherein the data format of the search server response body comprises indexes, types, ids, version numbers and states;
the log micro-service unit inquires a lightweight text from an inquiry interface configured by the search server and displays the lightweight text in a visual interface;
and the search server receives and stores the log converted into the lightweight text and responds to the log micro-service unit.
8. The cloud platform log management system of claim 7, wherein said lightweight text comprises JSON files, XML files, TXT files, or Binlog files; the log file comprises a service Request time stamp, service Request calling information, a service Request processing result and a Request ID.
9. The cloud platform log management system of claim 7, wherein said search server sets a batch processing maximum event number;
the lightweight log collector judges whether the event number of the lightweight text sent to the search server exceeds the maximum batch processing event number set by the search server;
if yes, reducing the receiving and transmitting speed between the lightweight text and the search server;
if not, the transceiving speed between the lightweight text and the search server is increased, and the mass processing maximum event number set by the search server is increased.
10. The cloud platform log management system of claim 7, wherein said lightweight log collector is selected from Filebeat, flume, fluentd, logagent, rsyslog or Syslog-ng; the search server is selected from the group consisting of an elastic search.
11. The cloud platform log management system of claim 7, wherein said lightweight log collector and search server are deployed in an elastesearch cluster.
12. The cloud platform log management system of claim 7, wherein said business micro service unit takes a Request ID and a time stamp as inputs to calculate a processing duration of a business Request, and writes said processing duration into a log file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911280687.8A CN111143286B (en) | 2019-12-13 | 2019-12-13 | Cloud platform log management method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911280687.8A CN111143286B (en) | 2019-12-13 | 2019-12-13 | Cloud platform log management method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111143286A CN111143286A (en) | 2020-05-12 |
| CN111143286B true CN111143286B (en) | 2024-01-09 |
Family
ID=70518237
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911280687.8A Active CN111143286B (en) | 2019-12-13 | 2019-12-13 | Cloud platform log management method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111143286B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111726252A (en) * | 2020-06-17 | 2020-09-29 | 浪潮集团有限公司 | Abnormal data reporting method and tool for industrial equipment |
| CN111767200B (en) * | 2020-06-23 | 2022-12-02 | 平安普惠企业管理有限公司 | Method, device and computer equipment for processing service based on service log |
| CN111930700A (en) * | 2020-07-13 | 2020-11-13 | 车智互联（北京）科技有限公司 | Distributed log processing method, server, system and computing equipment |
| CN113742300A (en) * | 2020-08-11 | 2021-12-03 | 北京沃东天骏信息技术有限公司 | Log management method, device, server and system |
| CN112000564A (en) * | 2020-09-09 | 2020-11-27 | 中盈优创资讯科技有限公司 | Log visual presentation management method and device |
| CN112181929A (en) * | 2020-09-24 | 2021-01-05 | 杭州安恒信息技术股份有限公司 | Cloud management platform log processing method and device, electronic device and storage medium |
| CN112463725A (en) * | 2020-11-19 | 2021-03-09 | 北京思特奇信息技术股份有限公司 | Cloud architecture log file batch processing method and device and storage medium |
| CN112685491B (en) * | 2020-12-30 | 2023-06-06 | 上海掌门科技有限公司 | Log processing method, electronic equipment and computer readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9578063B1 (en) * | 2015-11-20 | 2017-02-21 | International Business Machines Corporation | Application self-service for assured log management in cloud environments |
| CN107861859A (en) * | 2017-11-22 | 2018-03-30 | 北京汇通金财信息科技有限公司 | A kind of blog management method and system based on micro services framework |
| CN109902105A (en) * | 2019-02-28 | 2019-06-18 | 携程旅游网络技术（上海）有限公司 | For the data query system of micro services framework, method, equipment and storage medium |
| CN109933505A (en) * | 2019-03-14 | 2019-06-25 | 深圳市珍爱捷云信息技术有限公司 | Log processing method, device, computer equipment and storage medium |
-
2019
- 2019-12-13 CN CN201911280687.8A patent/CN111143286B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9578063B1 (en) * | 2015-11-20 | 2017-02-21 | International Business Machines Corporation | Application self-service for assured log management in cloud environments |
| CN107861859A (en) * | 2017-11-22 | 2018-03-30 | 北京汇通金财信息科技有限公司 | A kind of blog management method and system based on micro services framework |
| CN109902105A (en) * | 2019-02-28 | 2019-06-18 | 携程旅游网络技术（上海）有限公司 | For the data query system of micro services framework, method, equipment and storage medium |
| CN109933505A (en) * | 2019-03-14 | 2019-06-25 | 深圳市珍爱捷云信息技术有限公司 | Log processing method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111143286A (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111143286B (en) | Cloud platform log management method and system | |
| US11914501B1 (en) | User interface for specifying data stream processing language programs for analyzing instrumented software | |
| US11379475B2 (en) | Analyzing tags associated with high-latency and error spans for instrumented software | |
| US11550829B2 (en) | Systems and methods for load balancing in a system providing dynamic indexer discovery | |
| US9959015B2 (en) | Systems and methods for monitoring and analyzing performance in a computer system with node pinning for concurrent comparison of nodes | |
| US11347577B1 (en) | Monitoring features of components of a distributed computing system | |
| US10585951B2 (en) | Cursored searches in a data fabric service system | |
| CA2969131C (en) | Data stream processing language for analyzing instrumented software | |
| JP2017146994A (en) | Efficient query processing using histograms in columnar database | |
| Li et al. | SparkBench: a spark benchmarking suite characterizing large-scale in-memory data analytics | |
| US20180032387A1 (en) | Predictive Analytics on Database Wait Events | |
| US9442817B2 (en) | Diagnosis of application server performance problems via thread level pattern analysis | |
| US11755531B1 (en) | System and method for storage of data utilizing a persistent queue | |
| JP6423803B2 (en) | Queue monitoring and visualization | |
| CN106796520B (en) | Software-based instrumented real-time reporting | |
| US20160139961A1 (en) | Event summary mode for tracing systems | |
| US11892976B2 (en) | Enhanced search performance using data model summaries stored in a remote data store | |
| US11113243B2 (en) | Methods and apparatus for selective capture of mainframe operating system records | |
| US11841827B2 (en) | Facilitating generation of data model summaries | |
| US11960560B1 (en) | Methods for analyzing recurring accessibility issues with dynamic web site behavior and devices thereof | |
| CN115865645A (en) | Link determination method, device, equipment, medium and product | |
| WO2021217119A1 (en) | Analyzing tags associated with high-latency and error spans for instrumented software | |
| Hu et al. | The Application of Load Runner in Message Broker Software of Financial Fusion in a Stated-Owned Bank |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |