CN112307191A - Multi-system interactive log query method, device, equipment and storage medium - Google Patents
Multi-system interactive log query method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112307191A CN112307191A CN202011207513.1A CN202011207513A CN112307191A CN 112307191 A CN112307191 A CN 112307191A CN 202011207513 A CN202011207513 A CN 202011207513A CN 112307191 A CN112307191 A CN 112307191A
- Authority
- CN
- China
- Prior art keywords
- log
- query
- keywords
- initial
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/335—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/31—Indexing; Data structures therefor; Storage structures
- G06F16/313—Selection or weighting of terms for indexing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/338—Presentation of query results
Abstract
The invention relates to the field of cloud monitoring, and discloses a multi-system interactive log query method, device, equipment and storage medium. The log query method comprises the following steps: performing log collection on all systems based on a preset timing log collection script to obtain a service log data table; acquiring initial query demand information input by a user, and determining a target system and log query keywords; inquiring a search term configuration template example corresponding to the target system in a preset search term configuration template example library; inquiring in the search term configuration template example according to the log inquiry keywords to obtain intermediate output keywords; and performing log query operation in the service log data table according to the intermediate output keywords and the log query keywords to obtain and output all logs corresponding to the log query keywords. The method and the device can be used for accurately inquiring the service, and can be inquired together with the log of the related system when the log of one system is inquired.
Description
Technical Field
The invention relates to the field of cloud monitoring, in particular to a multi-system interactive log query method, device, equipment and storage medium.
Background
With the rapid development of technology and cross-system linkage, the complexity of calling multi-system operation logs is higher and higher, and for complex cross-system calling, one request may need the support of several or hundreds of background nodes. At this time, it is difficult to confirm the complete flow of the processing by manpower specifically to a request, so once an abnormality occurs, how to quickly locate the request becomes a primary problem.
In the prior art, distributed tracking and log aggregation are generally adopted for a cross-system log query solution, and the solution collects call information of an upstream interface and a downstream interface together, wherein a large number of logs which are not related to pain and itch are possible, and analysis is time-consuming when a certain log is retrieved; in addition, the solution is generally to aggregate related logs together through keywords, and cannot meet the accurate query of the service.
Disclosure of Invention
The invention mainly aims to solve the problems that the analysis of the existing cross-system log query technology is long in time consumption and weak in query accuracy.
The invention provides a multi-system interactive log query method in a first aspect, which comprises the following steps:
performing log collection operation on all systems based on a preset timing log collection script to obtain a service log data table;
acquiring initial query demand information input by a user, and determining a target system and log query keywords according to the initial query demand information;
inquiring a search term configuration template example corresponding to the target system in a preset search term configuration template example library;
inquiring in the search term configuration template example according to the log inquiry keywords to obtain intermediate output keywords;
and performing log query operation in the service log data table according to the intermediate output keywords and the log query keywords to obtain and output all logs corresponding to the log query keywords.
Optionally, in a first implementation manner of the first aspect of the present invention, the performing, based on a preset timing log acquisition script, a log acquisition operation on all systems to obtain a service log data table includes:
when a preset log collection time point is reached, calling the timing log collection script to collect logs to obtain log data in all systems;
dividing the log data according to columns to obtain initial log data;
and converting the initial log data into database fields based on a database tool, and then storing the database fields in a segmented manner to obtain a service log data table.
Optionally, in a second implementation manner of the first aspect of the present invention, the determining a target system according to the initial query requirement information includes:
analyzing the initial query demand information, and determining an initial query system to be queried;
detecting upstream and downstream system information of the initial query system, and determining an intermediate query system;
and obtaining the target system according to the initial query system and the intermediate query system.
Optionally, in a third implementation manner of the first aspect of the present invention, the obtaining and outputting all logs corresponding to the log query keyword includes:
setting a log context display threshold value based on a preset display rule configuration module;
and outputting and displaying log information according to the log context display threshold value.
Optionally, in a fourth implementation manner of the first aspect of the present invention, the acquiring initial query requirement information input by the user includes:
generating a data acquisition request;
analyzing the data acquisition request to obtain a data acquisition request analysis result;
and acquiring initial demand query information according to the data acquisition request analysis result.
Optionally, in a fifth implementation manner of the first aspect of the present invention, the dividing the log data by columns to obtain initial log data includes:
analyzing the log data to obtain a segmentation keyword of the log data;
and segmenting the log data according to the segmentation keywords to obtain initial log data.
Optionally, in a sixth implementation manner of the first aspect of the present invention, the invoking the timing log collection script to collect logs includes:
analyzing the log collection script to obtain a log collection path and a log storage path;
and scanning the log file under the log acquisition path, and copying the log file under the log acquisition path to the log storage path.
A second aspect of the present invention provides a multi-system interactive log query apparatus, including:
the data acquisition module is used for carrying out log acquisition operation on all the systems based on a preset timing log acquisition script to obtain a service log data table;
the system comprises a keyword acquisition module, a log query module and a query module, wherein the keyword acquisition module is used for acquiring initial query demand information input by a user and determining a target system and log query keywords according to the initial query demand information; inquiring a search term configuration template example corresponding to the target system in a preset search term configuration template example library; inquiring in the search term configuration template example according to the log inquiry keywords to obtain intermediate output keywords;
and the log output module is used for performing log query operation in the service log data table according to the intermediate output keywords and the log query keywords to obtain and output all logs corresponding to the log query keywords.
Optionally, in a first implementation manner of the second aspect of the present invention, the data acquisition module is specifically configured to:
when a preset log collection time point is reached, calling the timing log collection script to collect logs to obtain log data in all systems;
dividing the log data according to columns to obtain initial log data;
and converting the initial log data into database fields based on a database tool, and then storing the database fields in a segmented manner to obtain a service log data table.
Optionally, in a second implementation manner of the second aspect of the present invention, the data acquisition module is specifically configured to:
analyzing the log collection script to obtain a log collection path and a log storage path;
and scanning the log file under the log acquisition path, and copying the log file under the log acquisition path to the log storage path.
Optionally, in a third implementation manner of the second aspect of the present invention, the keyword obtaining module is specifically configured to:
analyzing the initial query demand information, and determining an initial query system to be queried;
detecting upstream and downstream system information of the initial query system, and determining an intermediate query system;
and obtaining the target system according to the initial query system and the intermediate query system.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the keyword obtaining module is specifically configured to:
generating a data acquisition request;
analyzing the data acquisition request to obtain a data acquisition request analysis result;
and acquiring initial demand query information according to the data acquisition request analysis result.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the data acquisition module is specifically configured to:
analyzing the log data to obtain a segmentation keyword of the log data;
and segmenting the log data according to the segmentation keywords to obtain initial log data.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the log output module is specifically configured to:
setting a log context display threshold value based on a preset display rule configuration module;
and outputting and displaying log information according to the log context display threshold value.
A third aspect of the present invention provides a multi-system interactive log query device, including: a memory and at least one processor, the memory having instructions stored therein; the at least one processor calls the instructions in the memory to cause the log query device to execute the multi-system interactive log query method.
A fourth aspect of the present invention provides a computer-readable storage medium having stored therein instructions, which, when run on a computer, cause the computer to execute the above-mentioned multi-system interactive log querying method.
According to the technical scheme, the method comprises the steps of acquiring log data of each system to obtain a service log data table, determining a target system and log query keywords, querying relevant templates in a preset search term configuration template instance library according to the log query keywords to obtain intermediate output keywords, and finally querying the service log data table by using the log query keywords and the intermediate output keywords as query conditions to obtain all log information corresponding to the log query keywords. The embodiment of the invention can be used for accurately inquiring the service, and can be inquired together with the log of the related system when the log of one system is inquired
Drawings
FIG. 1 is a diagram of an embodiment of a multi-system interactive log query method according to an embodiment of the present invention;
FIG. 2 is a diagram of an embodiment of a multi-system interactive log querying device according to the embodiment of the invention;
FIG. 3 is a diagram of an embodiment of a multi-system interactive log query device in an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a multi-system interactive log query method, a multi-system interactive log query device and a storage medium, which can be used for accurately querying a service and can be queried together with logs of a system related to the system when the logs of the system are queried.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of understanding, a specific flow of the embodiment of the present invention is described below, and referring to fig. 1, an embodiment of a multi-system interactive log query method in the embodiment of the present invention includes:
101. performing log collection operation on logs in all systems based on a preset timing log collection script to obtain a service log data table;
the method comprises the steps that a log platform acquires logs of each system by setting a timing script, the logs are divided according to columns, file data are converted into database fields to be stored, the timing script is an instruction set for executing certain operations at regular time, for example, a shell script is executed at regular time based on a coupon, a python script executed at regular time is realized based on a loop and sleep method, a python script executed at regular time is realized based on a Timer, a python script executed at regular time is realized based on a scheduling module schedule, and a python script realized based on a task framework APScheduler. In the embodiment, a method for executing a timed shell script based on a crottab is adopted, a script main logic statement, namely a function which needs to be realized by the script, is written into a text editor, the written logic statement is a logic statement of an acquisition function and comprises a file type of an acquisition object, such as a log type file ([ log ]), an acquisition path, such as/var/www/log, and a storage path/var/www/logdata, after the script main logic statement is written, the file is named and stored into a file in an sh format, such as getlog. And finally, starting the crontab service, and setting the script file getlog.sh to be executed and the execution time through the editing/etc/crontab file. The embodiment collects log data of all systems, and provides a data support basis for searching the following log data.
In this embodiment, the method 101 further includes the following steps:
when a preset log collection time point is reached, calling the timing log collection script to collect logs to obtain log data in all systems;
the collection time can be collected in fixed time, for example, the collection time is preset to be 13:00 and 15:00, a log collection script is called when the time points are 13:00 and 15:00, the related instructions of log collection are executed, the related instructions of log collection are analyzed by a system kernel to obtain the type ([ log ]) of a collection object, a collection path and a storage path, then the files under the collection path are scanned, and all the files conforming to the collection object are copied to the storage path. The collection time may be set to be periodic collection, and a collection manner of timed polling is adopted, which has the advantage that newly generated log files can be found, and since a certain log file is generated during a certain log file collection and a file is not generated yet in a log collection path, the log file is not collected in the current log collection. In this case, the periodic collection can make up for the defect, for example, the collection period is set to 1 hour, the system will collect the log file in the log collection path every 1 hour, and copy the log file in the storage path.
Dividing the log data according to columns to obtain initial log data;
reading the collected log file into a memory, segmenting the log file by specifying keywords to obtain a string segment of the log, wherein common log segmentation methods such as split method, dd segmentation, head + tail segmentation, sed segmentation and awk segmentation are adopted in the embodiment, the log of each row is sequentially read and segmented into a single string by a space or a separator such as a comma, a vertical line and the like, for example, the first column is a year, month and day, the second column is a time-minute and second column is a log level, and secondary segmentation can be performed on data of the log level, for example, http:// beginning is extracted by a regular expression as a calling interface. For example, a certain log segment a of the memory is read as "2017-12-2710: 56:57,737[ inp ] LogUtils: legischomedelerer-21 com.pic. cgibank.biz.action. cgicoppenecommonosetosys i calls openAPI to start, and the parameter: (EMPR PROP- …", the space is used as a specified segmentation key character, the log segment a is segmented into a new number group b by a split method, namely b ═ a.split ('), the new number group b at this time is [ ' 2017-12-27 ', "10: 56:57,737 ', ' [ inp ] upils: legischomedelerer-21. paic. cgemonomencouch.p. cgopenwork.21. pic. cgibo. cgibo.cgopenwork.ap.36, the log segment a is called as" cgopenapi, the log segment a is started as "cgopentime, the log segment a corresponding log time, the parameter of" 2017-12-2710, 23, the log segment a corresponding time of [ emppr-362, the log segment b.
According to the embodiment, the redundant log is divided into a plurality of data fields based on a common data division method, so that the storability is improved.
And converting the initial log data into database fields based on a database tool, and then storing the database fields in a segmented manner to obtain a service log data table.
In this embodiment, after a database tool, such as a mysql database, creates a new service log database in the mysql database, only a new data table may be created in the service log database, where the table is named as a service log data table, and data segmented by columns is imported into the table to complete creation of the service log data table, please refer to table 1. Besides the log character string segments of all systems are imported into the same data table, a plurality of service log data tables can be established according to the system names, for example, one table name is established for the log of the system A as the service log data table of the system A, one table name is established for the log of the system B as the service log data table of the system B, and one table name is established for the log of the system C as the service log data table of the system C. Therefore, the method can improve the retrieval speed in data retrieval and acquire related log data more quickly by establishing the sublist to store the log character string segments of each system.
TABLE 1
The embodiment stores the log data in a table form, and the log data can be conveniently searched based on the existing database tool.
In this embodiment, the method 101 may further include the following steps:
creating an initial search term configuration template instance base based on a database tool;
and defining query target field information for the data tables in the initial search term configuration template case library in sequence to obtain the search term configuration template case library.
In this embodiment, a database tool, such as a mysql database, newly creates a database named as a search term configuration template instance, and according to business requirements, creates a data table in the database and defines fields and attributes. For example, the table name is an XY system template, and the table includes an input system X, an output system Y, an interface name, a query condition (which may be a single keyword or a plurality of keywords), and an output keyword (which is a keyword required for a link subordinate system query). The query target field information comprises an input system, an output system, an interface name, a query condition and an output keyword. The input system and the output system only include a system which performs log collection, the name of the interface is an interface which is defined in each system, the query condition may be a single keyword or multiple keywords, for example, the single keyword "id" 1112 ", multiple keywords" id "1112, url" apr/appl ", the output keyword is a keyword which is required for query in a link subordinate system, such as SerialNo 2222, and the output keyword may also be a single keyword or multiple keywords.
It is to be understood that the execution subject of the present invention may be a multi-system interactive log query apparatus, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
102. Acquiring initial query demand information input by a user, and determining a target system and log query keywords according to the initial query demand information;
in this embodiment, a user initially queries demand information in a front-end page, where the initial demand query information includes input system information, an interface name, an output system, and query conditions input by the user. The query condition can be a set of a single element or a set of a plurality of elements, the number of the elements in a specific set is determined by the complexity of a business scene, and the query condition can be increased by defaulting to be the single query condition. The query conditions are determined by service differentiation, so that the corresponding target service can be quickly and accurately positioned through filtering of the query conditions, logic expressions such as and or and the like can be set, regular expressions can be designated to extract keyword setting, and keywords can be set according to interface parameter templates preset by various systems.
In this embodiment, the step of determining the target system according to the initial query requirement information in 102 further includes the following steps:
analyzing the initial query demand information, and determining an initial query system to be queried;
detecting upstream and downstream system information of the initial query system, and determining an intermediate query system;
and obtaining the target system according to the initial query system and the intermediate query system.
In this embodiment, data analysis is performed on the initial query requirement information input by the user on the front-end page, and if the input system information is analyzed to be the X system in the initial query requirement information, it is determined that the initial query system is the X system, and then target system information X, Y1, Y2, D1, and D2 are finally obtained by detecting upstream system information and downstream system information of the X system, for example, the upstream system is the Y1 system, the Y2 system, such as the downstream system D1, and the D2 system.
In this embodiment, the step of obtaining the initial query requirement information input by the user in 102 further includes the following steps:
generating a data acquisition request;
analyzing the data acquisition request to obtain a data acquisition request analysis result;
and acquiring initial demand query information according to the data acquisition request analysis result.
In this embodiment, when the user clicks the query button, the data acquisition module in the system sends a data acquisition request to the system, and the system receives the data acquisition request and then analyzes the data acquisition request to obtain an analysis result, for example, what data needs to be acquired, from which path to acquire, and the like, and finally sends the information input by the user, where the information input by the user may be stored in the local storage of the browser.
103. Inquiring a search term configuration template example corresponding to the target system in a preset search term configuration template example library;
in this embodiment, the search term configuration template instance library is composed of search term configuration template instances of respective systems, each two systems having an association relationship correspond to one search term configuration template instance, and the template has a specific naming rule, if the system a has an upstream system B, the name of the corresponding search term configuration template instance is a BA system template, and the upstream system is placed in front during naming; if the A system has a downstream system C, then the template instance name is configured corresponding to the search term as the AC system template. If the target systems X, Y1, Y2, D1, and D2 are the target systems X, Y1, Y2, D1, and D2, a search term configuration template instance library is opened in the database tool, where the lookup table names in the search term configuration template instance are the data tables of the Y1X system template, the Y2X system template, the XD1 system template, and the XD2 system template, respectively, and the query sql is selected from information format schema.
104. Inquiring in the search term configuration template example according to the log inquiry keywords to obtain intermediate output keywords;
in this embodiment, each search term configuration template instance defines an input system, an output system, an interface name, a query condition, and an output keyword, and the query term configuration template instance may be used to quickly obtain an output keyword of the output system corresponding to the query condition, the input system (query system), and the interface name, where the log query keyword is "id 1112, url ═ appr/app", and the intermediate output keyword "code ═ 3568" of the Y1X system template, the intermediate output keyword "attr ═ pl 12" of the Y2X system template, the intermediate output keyword "tel ═ 13561285527" of the XD1 system template, and the intermediate output keyword "dav ═ 645454" of the XD 4 system template are obtained by searching the keyword in the Y1X system template, the Y2X system template, the XD1 system template, and the XD 563 system template.
105. And performing log query operation in the service log data table according to the intermediate output keywords and the log query keywords to obtain and output all logs corresponding to the log query keywords.
In the embodiment of the present invention, according to the log query keyword "id ═ 1112, url ═ appr/appl", and the intermediate output keyword "code ═ 3568", "attr ═ pl 12", "tel ═ 13561285527", "dav ═ 645454, url ═ test/davr" query the corresponding data records in the service log data table and output the content parts of the LOGS fields displaying these data records, the system defaults to set the threshold of the log context, and can control the number of display rows of the context by setting the threshold, thereby further simplifying the query result.
In this embodiment, the method 105 further includes the following steps:
setting a log context display threshold value based on a preset display rule configuration module;
and outputting and displaying log information according to the log context display threshold value.
In this embodiment, the display rule configuration module is called by clicking a front page button, and a display threshold is input to control the number of lines of the context display, and if the display rule is not set, the system adopts a default display threshold. For example, by clicking the "log display setting" button, the number of input display lines is 100, and the type of input value must be an integer. The display module displays the number of the context lines of the log according to the display rule, for example, when the display threshold is set to be 100 when the display rule is configured, the query button is clicked, and the display result is the log information of each of the upper and lower 100 lines of the query keyword; if the display rule is not set, the default display threshold value of the system is 50 lines, the query is directly clicked, and the display result is log information of 50 lines above and below the query keyword. It should be noted that if the log is less than the log line number corresponding to the default display rule, all the information of the log is directly output.
In the embodiment, all the associated logs corresponding to the log keywords in the initial query system X are output and displayed, so that the problem logs are clear at a glance, and the requirements of part of specific services are met.
In the above description of the multi-system interactive log query method in the embodiment of the present invention, referring to fig. 2, a multi-system interactive log query device in the embodiment of the present invention is described below, where an embodiment of the multi-system interactive log query device in the embodiment of the present invention includes:
the data acquisition module 201 is configured to perform log acquisition operations on all systems based on a preset timing log acquisition script to obtain a service log data table;
a keyword obtaining module 202, configured to obtain initial query requirement information input by a user, and determine a target system and a log query keyword according to the initial query requirement information; inquiring a search term configuration template example corresponding to the target system in a preset search term configuration template example library; inquiring in the search term configuration template example according to the log inquiry keywords to obtain intermediate output keywords;
and the log output module 203 is configured to perform a log query operation in the service log data table according to the intermediate output keyword and the log query keyword, so as to obtain and output all logs corresponding to the log query keyword.
Optionally, the data acquisition module 201 may be further specifically configured to:
when a preset log collection time point is reached, calling the timing log collection script to collect logs to obtain log data in all systems;
dividing the log data according to columns to obtain initial log data;
converting the initial log data into database fields based on a database tool and then storing the database fields in a segmented manner to obtain a service log data table;
analyzing the log collection script to obtain a log collection path and a log storage path;
scanning the log file under the log acquisition path, and copying the log file under the log acquisition path to the log storage path;
analyzing the log data to obtain a segmentation keyword of the log data;
and segmenting the log data according to the segmentation keywords to obtain initial log data.
Optionally, the keyword obtaining module 202 may be further specifically configured to:
analyzing the initial query demand information, and determining an initial query system to be queried;
detecting upstream and downstream system information of the initial query system, and determining an intermediate query system;
obtaining the target system according to the initial query system and the intermediate query system;
generating a data acquisition request;
analyzing the data acquisition request to obtain a data acquisition request analysis result;
and acquiring initial demand query information according to the data acquisition request analysis result.
Optionally, the log output module 203 may be further specifically configured to:
setting a log context display threshold value based on a preset display rule configuration module;
and outputting and displaying log information according to the log context display threshold value.
In the embodiment of the invention, the function modularization is realized, the coupling of the system is reduced, and the maintenance is easier.
The multi-system interactive log query device in the embodiment of the present invention is described in detail in the perspective of the above fig. 2 modular functional entity, and the multi-system interactive log query device in the embodiment of the present invention is described in detail in the perspective of hardware processing.
Fig. 3 is a schematic structural diagram of a multi-system interactive logging query device 300 according to an embodiment of the present invention, where the multi-system interactive logging query device 300 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) storing applications 333 or data 332. Memory 320 and storage media 330 may be, among other things, transient or persistent storage. The program stored on the storage medium 330 may include one or more modules (not shown), each of which may include a series of instruction operations for the multi-system interactive log query device 300. Further, the processor 310 may be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the multi-system interactive log query device 300.
The multi-system interactive log querying device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input-output interfaces 360, and/or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, and the like. Those skilled in the art will appreciate that the configuration of the multi-system interactive logging query device shown in fig. 3 does not constitute a limitation of the multi-system interactive logging query device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
The invention also provides a multi-system interactive log query device, which comprises a memory and a processor, wherein computer readable instructions are stored in the memory, and when the computer readable instructions are executed by the processor, the processor executes the steps of the multi-system interactive log query method in the embodiments.
The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein instructions, which, when run on a computer, cause the computer to perform the steps of the multi-system interactive log querying method.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A multi-system interactive log query method is characterized by comprising the following steps:
performing log collection operation on all systems based on a preset timing log collection script to obtain a service log data table;
acquiring initial query demand information input by a user, and determining a target system and log query keywords according to the initial query demand information;
inquiring a search term configuration template example corresponding to the target system in a preset search term configuration template example library;
inquiring in the search term configuration template example according to the log inquiry keywords to obtain intermediate output keywords;
and performing log query operation in the service log data table according to the intermediate output keywords and the log query keywords to obtain and output all logs corresponding to the log query keywords.
2. The multi-system interactive log query method according to claim 1, wherein the log collection operation is performed on all systems based on a preset timing log collection script, and obtaining a service log data table comprises:
when a preset log collection time point is reached, calling the timing log collection script to collect logs to obtain log data in all systems;
dividing the log data according to columns to obtain initial log data;
and converting the initial log data into database fields based on a preset database tool, and then storing the database fields in a segmented manner to obtain a service log data table.
3. The multi-system interactive log query method of claim 1, wherein the determining a target system according to the initial query requirement information comprises:
analyzing the initial query demand information, and determining an initial query system to be queried;
detecting upstream and downstream system information of the initial query system, and determining an intermediate query system;
and obtaining the target system according to the initial query system and the intermediate query system.
4. The multi-system interactive log query method of claim 1, wherein the obtaining and outputting all logs corresponding to the log query keyword comprises:
setting a log context display threshold value based on a preset display rule configuration module;
and outputting and displaying log information according to the log context display threshold value.
5. The multi-system interactive log query method of claim 1, wherein the obtaining of initial query requirement information input by a user comprises:
generating a data acquisition request;
analyzing the data acquisition request to obtain a data acquisition request analysis result;
and acquiring initial demand query information according to the data acquisition request analysis result.
6. The multi-system interactive log querying method according to claim 2, wherein the dividing the log data into columns to obtain initial log data comprises:
analyzing the log data to obtain a segmentation keyword of the log data;
and segmenting the log data according to the segmentation keywords to obtain initial log data.
7. The multi-system interactive log querying method according to claim 2, wherein the calling the timing log collection script for log collection comprises:
analyzing the log collection script to obtain a log collection path and a log storage path;
and scanning the log file under the log acquisition path, and copying the log file under the log acquisition path to the log storage path.
8. A multi-system interactive log query device is characterized in that the multi-system interactive log query device comprises:
the data acquisition module is used for carrying out log acquisition operation on all the systems based on a preset timing log acquisition script to obtain a service log data table;
the system comprises a keyword acquisition module, a log query module and a query module, wherein the keyword acquisition module is used for acquiring initial query demand information input by a user and determining a target system and log query keywords according to the initial query demand information; inquiring a search term configuration template example corresponding to the target system in a preset search term configuration template example library; inquiring in the search term configuration template example according to the log inquiry keywords to obtain intermediate output keywords;
and the log output module is used for performing log query operation in the service log data table according to the intermediate output keywords and the log query keywords to obtain and output all logs corresponding to the log query keywords.
9. A multi-system interactive log query device, characterized in that the multi-system interactive log query device comprises: a memory and at least one processor, the memory having instructions stored therein;
the at least one processor invoking the instructions in the memory to cause the log querying device to perform a multi-system interactive log querying method as recited in any one of claims 1-7.
10. A computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement a multi-system interactive log query method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011207513.1A CN112307191A (en) | 2020-11-03 | 2020-11-03 | Multi-system interactive log query method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011207513.1A CN112307191A (en) | 2020-11-03 | 2020-11-03 | Multi-system interactive log query method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112307191A true CN112307191A (en) | 2021-02-02 |
Family
ID=74334052
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011207513.1A Pending CN112307191A (en) | 2020-11-03 | 2020-11-03 | Multi-system interactive log query method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112307191A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113672488A (en) * | 2021-09-06 | 2021-11-19 | 银清科技有限公司 | Log text processing method and device |
| CN114756901A (en) * | 2022-04-11 | 2022-07-15 | 敏于行(北京)科技有限公司 | Operational risk monitoring method and device |
| CN115757302A (en) * | 2022-10-28 | 2023-03-07 | 支付宝(杭州)信息技术有限公司 | Data analysis method, device, equipment and storage medium |
| CN116610724A (en) * | 2023-04-23 | 2023-08-18 | 北京优特捷信息技术有限公司 | Log data tracking method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010224705A (en) * | 2009-03-23 | 2010-10-07 | Hitachi Software Eng Co Ltd | Log retrieval system |
| CN109739730A (en) * | 2019-01-14 | 2019-05-10 | 平安科技（深圳）有限公司 | Monitoring method, device, equipment and the readable storage medium storing program for executing of multisystem daily record data |
| CN109800223A (en) * | 2018-12-12 | 2019-05-24 | 平安科技（深圳）有限公司 | Log processing method, device, electronic equipment and storage medium |
-
2020
- 2020-11-03 CN CN202011207513.1A patent/CN112307191A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010224705A (en) * | 2009-03-23 | 2010-10-07 | Hitachi Software Eng Co Ltd | Log retrieval system |
| CN109800223A (en) * | 2018-12-12 | 2019-05-24 | 平安科技（深圳）有限公司 | Log processing method, device, electronic equipment and storage medium |
| CN109739730A (en) * | 2019-01-14 | 2019-05-10 | 平安科技（深圳）有限公司 | Monitoring method, device, equipment and the readable storage medium storing program for executing of multisystem daily record data |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113672488A (en) * | 2021-09-06 | 2021-11-19 | 银清科技有限公司 | Log text processing method and device |
| CN114756901A (en) * | 2022-04-11 | 2022-07-15 | 敏于行(北京)科技有限公司 | Operational risk monitoring method and device |
| CN114756901B (en) * | 2022-04-11 | 2022-12-13 | 敏于行(北京)科技有限公司 | Operational risk monitoring method and device |
| CN115757302A (en) * | 2022-10-28 | 2023-03-07 | 支付宝(杭州)信息技术有限公司 | Data analysis method, device, equipment and storage medium |
| CN116610724A (en) * | 2023-04-23 | 2023-08-18 | 北京优特捷信息技术有限公司 | Log data tracking method and device, electronic equipment and storage medium |
| CN116610724B (en) * | 2023-04-23 | 2024-02-09 | 北京优特捷信息技术有限公司 | Log data tracking method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11392604B2 (en) | Designating fields in machine data using templates | |
| CN112307191A (en) | Multi-system interactive log query method, device, equipment and storage medium | |
| US11868717B2 (en) | Multi-page document recognition in document capture | |
| CN110292775B (en) | Method and device for acquiring difference data | |
| US10565208B2 (en) | Analyzing multiple data streams as a single data object | |
| CN107451149B (en) | Monitoring method and device for flow data query task | |
| CN102073726B (en) | Structured data import method and device for search engine system | |
| KR101505858B1 (en) | A templet-based online composing system for analyzing reports or views of big data by providing past templets of database tables and reference fields | |
| US8489643B1 (en) | System and method for automated content aggregation using knowledge base construction | |
| CN112632122A (en) | Report retrieval method, device, equipment and storage medium based on multiple indexes | |
| JPH09259138A (en) | Sort information display method and information retrieval device | |
| CN107291938A (en) | Order Query System and method | |
| JP4927895B2 (en) | Form creation processing system, form creation processing method, and form creation processing program | |
| CN116303427A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114691265A (en) | Batch grabbing method for software interface structured elements | |
| CN114186000A (en) | Business data monitoring method and device based on Flink and storage medium | |
| CN114115831A (en) | Data processing method, device, equipment and storage medium | |
| CN113722296A (en) | Agricultural information processing method and device, electronic equipment and storage medium | |
| CN111143156A (en) | Large data platform garbage task acquisition system and method and computer system | |
| JP2005044087A (en) | Text mining system and program | |
| CN115390853B (en) | Structured analysis method, system, terminal and storage medium for multi-source process file | |
| CN110633430A (en) | Event discovery method, device, equipment and computer readable storage medium | |
| CN117290355A (en) | Metadata map construction system | |
| CN116401031A (en) | Online program table generation method and device based on batch scheduling files | |
| CN115658771A (en) | Data processing method, device, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |