CN114817300A - Log query method based on SQL (structured query language) statements and application thereof - Google Patents

Log query method based on SQL (structured query language) statements and application thereof Download PDF

Info

Publication number
CN114817300A
CN114817300A CN202210538410.6A CN202210538410A CN114817300A CN 114817300 A CN114817300 A CN 114817300A CN 202210538410 A CN202210538410 A CN 202210538410A CN 114817300 A CN114817300 A CN 114817300A
Authority
CN
China
Prior art keywords
log
collection
query
sql
file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210538410.6A
Other languages
Chinese (zh)
Inventor
李圣权
倪林杰
毛云青
黄红叶
林加旺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
CCI China Co Ltd
Original Assignee
CCI China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by CCI China Co Ltd filed Critical CCI China Co Ltd
Priority to CN202210538410.6A priority Critical patent/CN114817300A/en
Publication of CN114817300A publication Critical patent/CN114817300A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/242Query formulation
    • G06F16/2433Query languages
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems

Abstract

The application provides a log query method based on SQL sentences and application thereof, comprising the following steps: configuring the output configuration of the log file according to the running environment of the application program; collecting log files according to a log collection mode, wherein the log collection mode comprises log file collection, structured database collection and unstructured database collection; performing query after SQL sentences are correspondingly converted and processed according to a log collection mode; and (6) adaptively packaging and returning the query result. According to the method and the system, only SQL sentences are used for searching and inquiring the log data, log configuration is simplified, resource consumption is remarkably reduced, invasiveness is weak, operation and maintenance personnel can quickly check the logs and locate the reasons of problems of the program, and working efficiency is greatly improved.

Description

Log query method based on SQL (structured query language) statements and application thereof
Technical Field
The application relates to the field of log search query, in particular to a log query method based on SQL statements and application thereof.
Background
Currently, log data of software applications can be used in a number of areas such as problem location, troubleshooting, monitoring, forensics, and the like. Generally, a log plug-in is used in a software program, a log file format and an output format are configured before the software program runs, and log data can be continuously written into a log file through the log plug-in the running process of the software program. When a program encounters a fault and needs to check logs, a server needs to be logged to check log files, and for some hidden problems, the log files need to be downloaded and some analysis tools need to be used for positioning and checking.
Although many tools are currently available for log collection and analysis, such as ELK (Elasticsearch, Logstash, and Kibana), in order to monitor and analyze logs of an application program, log collection in this way requires installation and deployment of multiple middleware, resource consumption, modification of log configuration in a software program, and is very invasive.
Therefore, a method and an application thereof for querying a log file by using SQL are needed, which can Query a specified log file by using an SQL (structured Query language) structured Query language on a page, and quickly screen out log data required to be related by writing SQL statements according to key information.
Disclosure of Invention
The embodiment of the application provides a log query method based on SQL statements and application thereof, and aims to solve the problems that in the prior art, a plurality of middleware needs to be installed and deployed, resources are consumed, log configuration needs to be modified in a software program, and the invasion is high.
The core technology of the invention mainly provides a query mode of SQL sentences in a unified way, supports the analysis of the SQL sentences into different query modes, can effectively reduce the threshold of operation and maintenance personnel, can query the data only by mastering the general SQL language, and does not need to care about the specific storage mode of the data and master other various languages and expressions.
In a first aspect, the present application provides a log query method based on SQL statements, the method including the following steps:
configuring the output configuration of the log file according to the running environment of the application program;
collecting log files according to a log collection mode, wherein the log collection mode comprises log file collection, structured database collection and unstructured database collection;
performing query after SQL sentences are correspondingly converted and processed according to a log collection mode;
and (6) adaptively packaging and returning the query result.
Further, if the log collection mode is log file collection, splitting SQL sentences according to SQL standard keywords and syntax, and converting the split arrays into regular expressions for matching query.
Further, if the log collection mode is structured database collection, the query is directly performed by using an SQL statement.
Further, if the log collection mode is unstructured database collection, the SQL statement is converted and adapted to be queried by a query API corresponding to the unstructured database.
Further, the output configuration comprises log levels, log file storage paths, log file naming formats, log content formats in files, the maximum size of the total log file, the maximum size of a single log file and the maximum retention period of the log file.
Furthermore, in the collection of the log file according to the log collection mode, a plurality of collection fields of relevant configuration output by the log file are generated according to the configuration of the user-defined output field, the configuration is stored, and the query result is output according to the configuration.
Further, the operating environment at least comprises an operating environment memory and hard disk information.
In a second aspect, the present application provides a log query device based on SQL statements, including:
the log output unit is used for configuring the output configuration of the log file according to the running environment of the application program;
the log collection unit is used for collecting log files according to a log collection mode, wherein the log collection mode comprises log file collection, structured database collection and unstructured database collection;
the SQL query conversion unit is used for carrying out query after SQL sentences are correspondingly converted and processed according to a log collection mode;
and the query result unit is used for adaptively packaging and returning the query result.
In a third aspect, the present application provides an electronic device, including a memory and a processor, where the memory stores a computer program, and the processor is configured to execute the computer program to execute the SQL statement based log query method.
In a fourth aspect, the present application provides a readable storage medium having stored therein a computer program comprising program code for controlling a process to execute a process, the process comprising the SQL statement-based log query method according to the above.
The main contributions and innovation points of the invention are as follows: 1. compared with the prior art, the log file is not required to be manually configured, a log collection mode is selected to collect the log file through configuring the log output field, and then the log data is searched and inquired by using SQL statements, so that the log configuration is simplified, the resource consumption is obviously reduced, the invasiveness is weak, and operation and maintenance personnel can quickly check the log and locate the reason of the problem of the program;
2. compared with the prior art, the method and the device have the advantages that the query mode of the SQL statement is provided uniformly, the SQL statement is analyzed into different query modes, the threshold of operation and maintenance personnel can be effectively reduced, the data can be queried only by mastering the general SQL language, the specific storage mode of the data is not concerned, and other various languages and expressions are not needed to be mastered.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a flow chart of a method according to the present application;
FIG. 2 is a diagram of the steps of one implementation of a method according to the method of the present application;
FIG. 3 is a flow chart of an implementation of one implementation of a method according to the method of the present application;
FIG. 4 is a diagram of an exemplary application scenario in accordance with the method of the present application;
FIG. 5 is a schematic structural diagram of an apparatus for querying a log based on an SQL statement according to an embodiment of the present application;
fig. 6 is a schematic hardware structure diagram of an electronic device according to an embodiment of the application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of the present specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of one or more embodiments of the specification, as detailed in the claims which follow.
It should be noted that: in other embodiments, the steps of the corresponding methods are not necessarily performed in the order shown and described herein. In some other embodiments, the method may include more or fewer steps than those described herein. Moreover, a single step described in this specification may be broken down into multiple steps for description in other embodiments; multiple steps described in this specification may be combined into a single step in other embodiments.
Currently, in the prior art, in order to monitor and analyze the log of the application program, many tools may be used to perform log collection and analysis, such as ELK (elastic search, Logstash, and Kibana), however, performing log collection in this way requires installing and deploying multiple middleware, consumes resources, requires modifying the log configuration in the software program, and is highly invasive.
The Logstash is mainly used for collecting, analyzing and filtering logs and supports a large number of data acquisition modes. The general working mode is c/s architecture, the client end is installed on a host computer needing log collection, and the server end is responsible for filtering, modifying and the like operations of received node logs and sending the node logs to the elastic search. The ElasticSearch is used for being responsible for storing final data, establishing indexes and providing search logs for the outside. The distributed search engine is an open-source distributed search engine and provides three functions of collecting, analyzing and storing data. It is characterized in that: distributed, zero configuration, automatic discovery, index automatic fragmentation, index copy mechanism, restful style interface, multiple data sources, automatic search load, etc. Kibana is an excellent front-end log display framework, which can convert logs into various charts in great detail and provide powerful data visualization support for users.
Based on the method, the log output field is configured, the log collection mode is selected to collect the log file, then the SQL statement is used for searching and inquiring the log data, the log configuration is simplified, the resource consumption is reduced, the invasiveness is weak, and the operation and maintenance personnel can quickly investigate the log and locate the reason of the problem of the program.
Example one
Specifically, an embodiment of the present application provides a log query method based on an SQL statement, and may specifically refer to fig. 1, where the method includes the following steps:
configuring the output configuration of the log file according to the running environment of the application program;
the output configuration comprises log levels, log file storage paths, a log file naming format, a log content format in a file, the maximum size of a total log file, the maximum size of a single log file and the maximum retention period of the log file;
the operation environment at least comprises an operation environment memory and hard disk information;
collecting log files according to a log collection mode, wherein the log collection mode comprises log file collection, structured database collection and unstructured database collection;
performing query after SQL sentences are correspondingly converted and processed according to a log collection mode;
the method comprises the steps that a plurality of collection fields of relevant configuration of log file output are generated according to custom output field configuration, the configuration is stored, and a query result is output according to the configuration;
if the log collection mode is log file collection, splitting SQL sentences according to SQL standard keywords and syntax, and converting the split arrays into regular expressions for matching query;
if the log collection mode is structured database collection, directly querying by using SQL statements;
if the log collection mode is unstructured database collection, the SQL statement is converted and adapted to be queried by a query API corresponding to the unstructured database;
and (6) adaptively packaging and returning the query result.
In this embodiment, as shown in fig. 2-3, the specific method of this embodiment includes:
s1, automatically calculating the saving configuration of the log file according to the running environment of the system (application program), and generating several collection fields of the related configuration of the log file output according to the field configuration of the output log, the device will save the configuration, and output according to the several fields when inquiring the output result.
Here, fields that need to be output by self-definition are supported, for example, logType (log type), line _ number (line number), message (content), log _ time (log time) are configured.
Specifically, the automatic calculation process of the storage configuration includes first obtaining information of the memory and the hard disk in the operating environment, and calculating the maximum size of the total log file, the maximum size of the single log file, and the maximum retention period of the log file according to the information, where the calculation formula is the maximum size (GB) of the total log file, which is the calculation formula of the memory size (GB) 20% + the hard disk size (GB) 80%, the maximum size (MB) of the single log file, which is the hard disk size (MB)/1024 × 10%, and the maximum retention period (day) of the log file, which is the maximum size of the total log file/the maximum size of the single log file/the number of log levels.
The log level is fixed to five levels of TRACE, DEBUG, INFO, WARN and ERROR, a log storage path creates logs directory under the current application program package path, sub-directories are respectively created according to the log level, the name format of the log file is named according to log- { level } - { date } - { timestamp }, and the log content format in the file is output by rows according to configured field values with spaces.
And generating configuration according to the format, storing the configuration for effectiveness, and outputting the system operation to a log file according to the configured format.
And S2, selecting a log collection mode to collect the logs, and entering the log data into different storage media according to different collection modes.
Outputting the log data to a log file by the log output configuration generated in the previous step under the default condition according to the three provided collection modes;
if the structured database collection mode is selected, the logs generated in the program are sent to a structured database message queue, a consumer collects a certain number of logs, the logs are connected to a database according to the configured database connection address, a table structure is created according to the naming mode log- { level } - { date } - { timestamp } of the log file, if the table structure exists, the table structure does not need to be created, and then the log data is stored in a designated table;
if the unstructured database collection mode is selected, the logs generated in the program are sent to an unstructured database message queue, a consumer collects a certain number of logs, the logs are connected to an unstructured database according to the configured unstructured database connection address, an index is created according to the naming mode log- { level } - { date } - { timestamp } of the log file, if the index exists, the index does not need to be created, and then log data are stored in the designated index.
And S3, analyzing and splitting the SQL sentences edited in the SQL editor according to the selected file names, converting the SQL sentences into query methods corresponding to the collection modes according to the log collection mode, and querying the target data. The specific implementation mode is that SQL sentences are split according to SQL standard keywords and grammar, query fields are arranged after split select, all fields are queried if the query fields are wildcard characters, target file names are arranged after split from, screening conditions are arranged after split where, association table names are arranged after split join, association field names are arranged after split on, and the split fields are temporarily stored in a key value pair array form and used as the basis of the subsequent conversion. If the log collection mode is file collection, the conversion to the regular expression is realized through the information in the split array, for example, the log data with the row number of 211 of log-info-2022.03.30.10 is queried, and the SQL statement is written as follows:
Select*from log-info-2022.03.30.10where line_number=211
the conversion to regular expression is:
(?<log_time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2})(?<mills>.\d{3})\s*%{LOGLEVEL:log_type}\s*(?<program_file_name>([\S+]*))\s*Line:211\s*-\s*%{GREEDYDATA:message}
if the log collection mode is Mysql, the Mysql is directly inquired through SQL, and if the log collection mode is ElasticSearch, the conversion of the ElasticSearch inquiry SDK is carried out, and then the inquiry is carried out. The other databases are queried after being converted according to the corresponding statements or APIs, and are not described in detail.
And S4, after the SQL conversion query is completed, carrying out adaptive packaging on the query results of various types of collection modes, and ensuring the uniformity of the returned data structure. And generating a query result field according to the configuration, and encapsulating the queried data into an array according to field matching for returning.
Wherein, the Structured Query Language is a Structured Query Language. The system is a special purpose programming language, is a database query and programming language, is used for accessing data and querying, updating and managing a relational database system, and is called SQL for short;
regular expressions are a logical formula for operating on character strings (including common characters (e.g., letters between a and z) and special characters (called meta characters)), and a "regular character string" is formed by using specific characters and combinations of the specific characters defined in advance, and is used for expressing a filtering logic for the character string. A regular expression is a text pattern that describes one or more strings of characters to be matched when searching for text.
Example two
Based on the first embodiment, as shown in fig. 4, a typical application scenario of this embodiment is as follows:
when a user uses a certain application system, a system abnormal error is prompted, and operation and maintenance personnel are required to investigate the reason of the error. The log collection mode of the current application system is file collection, and several collection fields of logType, line _ number, message and log _ time are configured, so that an operator enters a log management page, screens three log files of war, error and info generated on the same day and enters an SQL editing page, and performs associated search query according to prompt information 'system exception' generated in the info log, wherein the SQL statement is as follows:
select
b.*
from log-info-2022-03-30a
join log-error-2022-03-30b on a.line_number=b.line_number
join log-warn-2022-03-30c on a.line_number=c.line_number
where a. message like "% system anomaly";
according to the query result display, in the data display with the logType of Error, the key position of the program with problems can be obtained from the message field, so that the quick troubleshooting, positioning and repairing are realized.
In the third embodiment, the first step is that,
based on the same concept, as shown in fig. 5, the present application further provides a log query device based on SQL statements, including:
a log output unit 101 configured to configure an output configuration of a log file according to an execution environment of an application;
the log output unit 101 is configured to configure a log file output format (output configuration), where the log output format (output configuration) includes a log level, a log file storage path, a log file naming format, a file internal log content format, a total log file maximum size, a single log file maximum size, and a log file maximum retention period, and these configuration items do not need to be configured, and the unit automatically identifies an operating environment where a current application program is located to perform calculation, and performs default configuration;
therefore, the log output unit 101 can automatically identify the operating environment of the application system and calculate the default configuration without the configuration of a user, and the problems of complexity and intrusiveness caused by log configuration of operation and maintenance engineering personnel are solved.
A log collection unit 102, configured to collect log files according to a log collection manner, where the log collection manner includes log file collection, structured database collection, and unstructured database collection;
the log collection unit 102 is configured to collect log data, synchronize a printout log to a designated structured database or an unstructured database according to log collection configuration, and perform analysis statistics on the log data after the log is placed in a database, where the log collection configuration may select three data collection modes, namely a log file (default), a structured database, and an unstructured database, where the structured database includes Mysql, Oracle, and the like, the unstructured database includes MongoDB, elastic search, and the like, and the structured and unstructured database configurations require configuration of a data source connection address and an account password, thereby ensuring successful connection. When the collection mode is configured to collect log files, the log information fields needing to be output are selected, and the log information is directly output to the specified path file according to the configuration of the output unit; when the collection mode is a structured database, the log information is sent to a message queue, a consumer creates database connection according to the configuration of the database, automatically creates a log table, monitors the log information and then stores the log information into the created table; when the collection mode is the unstructured database, the log information is also sent to the message queue, the consumer establishes connection according to the unstructured database configuration, automatically establishes an index, and monitors the log information and then stores the log information into the established index.
Thus, the log collection unit 102 provides three collection modes, namely log file collection, structured database collection and unstructured database collection, and can select one collection mode according to actual business requirements, if only the requirements of troubleshooting system failure and problem location are needed to select the log file collection mode, if the requirements of troubleshooting system failure, problem location and log analysis are needed to select the structured database collection, and if the requirements of troubleshooting system failure, problem location, log analysis, log monitoring and high query performance are needed to select the unstructured database collection. The three collection modes cover most requirements of current log scene application, the problem of collecting log data under different scene application is solved, a lot of configurations are simplified, and the operation of letting operation and maintenance engineering staff collect logs manually is avoided.
The SQL query conversion unit 103 is used for performing query after SQL sentences are correspondingly converted and processed according to a log collection mode;
the SQL query conversion unit 103 is configured to parse and convert the SQL statements, query target data according to the configuration of the log output unit and the log collection unit, convert the SQL statements into a regular expression to perform matching query if the data is in a log file collection manner, directly connect configured database addresses if the data is a structured database, and query relevant table data, connect configured unstructured database addresses if the data is an unstructured database, and convert and adapt the SQL statements to query Api of each unstructured database and then query relevant index data.
Therefore, conversion adaptation of the query in various log collection modes can be realized by directly using SQL sentences, for example, the log collection mode is file collection, the search mode of file data is a regular expression, the unit realizes conversion from SQL to the regular expression, the log collection mode is Mysql, Mysql is directly queried through SQL, and the log collection mode is ElasticSearch, so that conversion from SQL to ElasticSearch is realized for querying SDK. The threshold of inquiring and analyzing the log is reduced, and the efficiency of system problem troubleshooting and positioning is improved.
And the query result unit 104 is configured to structure the query result, package the query result data into an array object according to the field to be queried in the sql, and return the array object.
Example four
The present embodiment also provides an electronic device, referring to fig. 6, comprising a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to execute the computer program to perform the steps in any of the above method embodiments.
Specifically, the processor 402 may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
Memory 404 may include, among other things, mass storage 404 for data or instructions. By way of example, and not limitation, memory 404 may include a hard disk drive (hard disk drive, HDD for short), a floppy disk drive, a solid state drive (SSD for short), flash memory, an optical disk, a magneto-optical disk, tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Memory 404 may include removable or non-removable (or fixed) media, where appropriate. The memory 404 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 404 is a Non-Volatile (Non-Volatile) memory. In particular embodiments, memory 404 includes Read-only memory (ROM) and Random Access Memory (RAM). The ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or FLASH memory (FLASH), or a combination of two or more of these, where appropriate. The RAM may be a Static Random Access Memory (SRAM) or a Dynamic Random Access Memory (DRAM), where the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an Extended Data Out Dynamic Random Access Memory (EDODRAM), a Synchronous Dynamic Random Access Memory (SDRAM), or the like.
Memory 404 may be used to store or cache various data files for processing and/or communication use, as well as possibly computer program instructions for execution by processor 402.
The processor 402 implements any of the SQL statement-based log query methods in the above embodiments by reading and executing computer program instructions stored in the memory 404.
Optionally, the electronic apparatus may further include a transmission device 406 and an input/output device 408, where the transmission device 406 is connected to the processor 402, and the input/output device 408 is connected to the processor 402.
The transmitting device 406 may be used to receive or transmit data via a network. Specific examples of the network described above may include wired or wireless networks provided by communication providers of the electronic devices. In one example, the transmission device includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmitting device 406 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
The input and output devices 408 are used to input or output information. In this embodiment, the input information may be SQL statements or the like, and the output information may be query results or the like.
EXAMPLE five
The present embodiment also provides a readable storage medium in which a computer program is stored, where the computer program includes program code for controlling a process to execute the process, and the process includes the SQL statement-based log query method according to the first embodiment.
It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiment and optional implementation manners, and details of this embodiment are not described herein again.
In general, the various embodiments may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
Embodiments of the invention may be implemented by computer software executable by a data processor of the mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and/or macros can be stored in any device-readable data storage medium and they include program instructions for performing particular tasks. The computer program product may comprise one or more computer-executable components configured to perform embodiments when the program is run. The one or more computer-executable components may be at least one software code or a portion thereof. Further in this regard it should be noted that any block of the logic flow as in the figures may represent a program step, or an interconnected logic circuit, block and function, or a combination of a program step and a logic circuit, block and function. The software may be stored on physical media such as memory chips or memory blocks implemented within the processor, magnetic media such as hard or floppy disks, and optical media such as, for example, DVDs and data variants thereof, CDs. The physical medium is a non-transitory medium.
It should be understood by those skilled in the art that various features of the above embodiments can be combined arbitrarily, and for the sake of brevity, all possible combinations of the features in the above embodiments are not described, but should be considered as within the scope of the present disclosure as long as there is no contradiction between the combinations of the features.
The above examples are merely illustrative of several embodiments of the present application, and the description is more specific and detailed, but not to be construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims (10)

1. The log query method based on the SQL statement is characterized by comprising the following steps of:
configuring the output configuration of the log file according to the running environment of the application program;
collecting the log files according to a log collection mode, wherein the log collection mode comprises log file collection, structured database collection and unstructured database collection;
performing query after SQL sentences are correspondingly converted and processed according to the log collection mode;
and (6) adaptively packaging and returning the query result.
2. The SQL-statement-based log query method according to claim 1, wherein if the log collection mode is the log file collection, the SQL statement is split according to SQL standard keywords and syntax, and the split array is converted into a regular expression for matching query.
3. The SQL statement-based log query method according to claim 1, wherein if the log collection manner is the structured database collection, the SQL statement is directly used for query.
4. The SQL statement-based log query method according to claim 1, wherein if the log collection manner is the unstructured database collection, the SQL statement conversion is adapted to a query API corresponding to the unstructured database for query.
5. The SQL statement-based log query method according to claim 1, wherein the output configuration includes a log level, a log file storage path, a log file naming format, an in-file log content format, a total log file maximum size, a single log file maximum size, and a log file maximum retention period.
6. The SQL statement-based log query method according to claim 1, wherein, in collecting the log file according to a log collection method, a plurality of collection fields of relevant configuration for log file output are generated and stored according to a custom output field configuration, and the query result is output in the configuration.
7. The SQL statement-based log query method according to claim 1, wherein the execution environment at least includes execution environment memory and hard disk information.
8. An apparatus for querying a log based on an SQL statement, comprising:
the log output unit is used for configuring the output configuration of the log file according to the running environment of the application program;
the log collection unit is used for collecting the log files according to a log collection mode, wherein the log collection mode comprises log file collection, structured database collection and unstructured database collection;
the SQL query conversion unit is used for carrying out query after SQL sentences are correspondingly converted and processed according to the log collection mode;
and the query result unit is used for adaptively packaging and returning the query result.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to execute the SQL statement-based log query method according to any one of claims 1 to 7.
10. A readable storage medium having stored therein a computer program comprising program code for controlling a process to execute a process, the process comprising the SQL statement-based log query method according to any one of claims 1 to 7.
CN202210538410.6A 2022-05-18 2022-05-18 Log query method based on SQL (structured query language) statements and application thereof Pending CN114817300A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210538410.6A CN114817300A (en) 2022-05-18 2022-05-18 Log query method based on SQL (structured query language) statements and application thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210538410.6A CN114817300A (en) 2022-05-18 2022-05-18 Log query method based on SQL (structured query language) statements and application thereof

Publications (1)

Publication Number Publication Date
CN114817300A true CN114817300A (en) 2022-07-29

Family

ID=82514551

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210538410.6A Pending CN114817300A (en) 2022-05-18 2022-05-18 Log query method based on SQL (structured query language) statements and application thereof

Country Status (1)

Country Link
CN (1) CN114817300A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116991877A (en) * 2023-09-25 2023-11-03 城云科技(中国)有限公司 Method, device and application for generating structured query statement

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116991877A (en) * 2023-09-25 2023-11-03 城云科技(中国)有限公司 Method, device and application for generating structured query statement
CN116991877B (en) * 2023-09-25 2024-01-02 城云科技(中国)有限公司 Method, device and application for generating structured query statement

Similar Documents

Publication Publication Date Title
CN111522922B (en) Log information query method and device, storage medium and computer equipment
US20110191394A1 (en) Method of processing log files in an information system, and log file processing system
CN105183860B (en) Method of data synchronization and system
CN110213207B (en) Network security defense method and equipment based on log analysis
US20220358023A1 (en) Method And System For The On-Demand Generation Of Graph-Like Models Out Of Multidimensional Observation Data
CN111367760B (en) Log collection method and device, computer equipment and storage medium
CN108228322B (en) Distributed link tracking and analyzing method, server and global scheduler
CN113704790A (en) Abnormal log information summarizing method and computer equipment
CN111581232B (en) ELK-based slow SQL real-time analysis method and system
US11301475B1 (en) Transmission handling of analytics query response
CN114817300A (en) Log query method based on SQL (structured query language) statements and application thereof
CN111666344A (en) Heterogeneous data synchronization method and device
CN111177239B (en) Unified log processing method and system based on HDP big data cluster
CN113760677A (en) Abnormal link analysis method, device, equipment and storage medium
CN112732663A (en) Log information processing method and device
CN112579406B (en) Log call chain generation method and device
CN111858280A (en) SQL information processing method, device, equipment and system
CN117271584A (en) Data processing method and device, computer readable storage medium and electronic equipment
CN110716912B (en) SQL performance detection method and server
CN110727565B (en) Network equipment platform information collection method and system
CN114385551B (en) Log time-sharing management method, device, equipment and storage medium
CN115686535A (en) Inspection method and device for Kubernets cluster and application
US20210141796A1 (en) System and methods for performing updated query requests in a system of multiple database engine
CN113204558A (en) Method and device for automatically updating data table structure
CN117349323B (en) Database data processing method and device, storage medium and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination