CN112100138A - Log query method and device, storage medium and electronic equipment - Google Patents
Log query method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN112100138A CN112100138A CN202010976706.7A CN202010976706A CN112100138A CN 112100138 A CN112100138 A CN 112100138A CN 202010976706 A CN202010976706 A CN 202010976706A CN 112100138 A CN112100138 A CN 112100138A
- Authority
- CN
- China
- Prior art keywords
- query
- log
- time period
- target
- searching
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000004590 computer program Methods 0.000 claims description 8
- 230000011218 segmentation Effects 0.000 claims description 5
- 230000008901 benefit Effects 0.000 abstract description 5
- 235000019580 granularity Nutrition 0.000 description 35
- 238000010586 diagram Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 238000007781 pre-processing Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000002829 reductive effect Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 241000700605 Viruses Species 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000000586 desensitisation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Abstract
The embodiment of the application provides a log query method, a log query device, a storage medium and electronic equipment, wherein the log query method comprises the following steps: acquiring a log query request; according to the log query request, searching target log data matched with the log query request from a ClickHouse database, and generating a log query result comprising the target log data; and outputting a log query result. According to the method and the device, log data can be queried by utilizing the advantage that the ClickHouse database is high in query speed, so that query efficiency can be improved, and user experience is improved.
Description
Technical Field
The application relates to the technical field of large-data-volume query, in particular to a log query method, a log query device, a storage medium and electronic equipment.
Background
Due to the limitation of the practical application scenario, the log data related to the application scenario can be stored only in the architecture of a single server, not in the architecture of a distributed type.
In the process of implementing the invention, the inventor finds that the following problems exist in the prior art: under the condition that the data volume of log data to be queried is large, the pressure of a single server is increased, and the performance reaches the bottleneck, so that the problem of low log query efficiency is easily caused, and the user experience is influenced.
Disclosure of Invention
An object of the embodiments of the present application is to provide a log query method, an apparatus, a storage medium, and an electronic device, so as to solve the problem in the prior art that log query efficiency is relatively low.
In a first aspect, an embodiment of the present application provides a log query method, where the log query method includes: acquiring a log query request; according to the log query request, searching target log data matched with the log query request from a ClickHouse database, and generating a log query result comprising the target log data; and outputting a log query result.
Therefore, according to the method and the device for querying the log data, the log data can be queried by utilizing the advantage that the query speed of the ClickHouse database is high, so that the query efficiency can be improved, and the user experience is improved.
In one possible embodiment, the ClickHouse database includes at least one table, each table corresponding to a time granularity.
Therefore, the data is stored in a table dividing mode, so that the table in the relevant time range can be quickly searched, and the log query efficiency can be further improved.
In one possible embodiment, the log query request includes a query time period and a query condition, and searching the target log data matched with the log query request from the ClickHouse database according to the log query request includes: according to the query time period, searching a target table matched with the query time period from all tables of the ClickHouse database; and finding out target log data matched with the query condition from the target table.
In one possible embodiment, according to the query time period, searching a target table matching the query time period from all tables in the ClickHouse database, including: according to the preset time granularity, performing time segmentation on the query time period to obtain at least two time periods; the target table matching each time period is looked up from all tables.
Therefore, by dividing the query time period, the embodiment of the application can quickly find the corresponding target table, and further can further improve the log query efficiency.
In a second aspect, an embodiment of the present application provides a log query apparatus, where the log query apparatus includes: the acquisition module is used for acquiring a log query request; the searching module is used for searching target log data matched with the log query request from the ClickHouse database according to the log query request and generating a log query result comprising the target log data; and the output module is used for outputting the log query result.
In one possible embodiment, the ClickHouse database includes at least one table, each table corresponding to a time granularity.
In one possible embodiment, the log query request includes a query time period and a query condition, and the lookup module includes: the first searching submodule is used for searching a target table matched with the query time period from all tables of the ClickHouse database according to the query time period; and the second searching submodule is used for searching the target log data matched with the query condition from the target table.
In a possible embodiment, the first lookup submodule is specifically configured to: according to the preset time granularity, performing time segmentation on the query time period to obtain at least two time periods; the target table matching each time period is looked up from all tables.
In a third aspect, an embodiment of the present application provides a storage medium, where a computer program is stored on the storage medium, and when the computer program is executed by a processor, the computer program performs the method according to the first aspect or any optional implementation manner of the first aspect.
In a fourth aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating via the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the method of the first aspect or any of the alternative implementations of the first aspect.
In a fifth aspect, the present application provides a computer program product which, when run on a computer, causes the computer to perform the method of the first aspect or any possible implementation manner of the first aspect.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a schematic diagram illustrating an application scenario provided in an embodiment of the present application;
fig. 2 is a flowchart illustrating a log query method according to an embodiment of the present application;
fig. 3 shows a specific flowchart of a log query method provided by an embodiment of the present application;
fig. 4 shows a block diagram of a log query device according to an embodiment of the present application;
fig. 5 is a block diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
In order to solve the problem of low log query efficiency in the prior art, the embodiment of the application skillfully provides a log query scheme, and by acquiring a log query request, then searching target log data matched with the log query request from a ClickHouse database according to the log query request, generating a log query result comprising the target log data, and finally outputting the log query result.
Therefore, according to the method and the device for querying the log data, the log data can be queried by utilizing the advantage that the query speed of the ClickHouse database is high, so that the query efficiency can be improved, and the user experience is improved.
To facilitate understanding of the embodiments of the present application, some terms in the embodiments of the present application are first explained herein as follows:
"ClickHouse": the system is a columnar database management system for Online Analytical Processing (OLAP), and solves the problem that a traditional database is slow to query under the condition of large data volume.
For example, when the data size of the query log data required by a single log query request is relatively large (for example, a network log of a certain company is to be queried, or the data size required to be queried exceeds a preset data size, etc.), the ClickHouse engine can reach up to billions of lines per second at most when processing the log query request, and the writing speed is very fast and can reach 50-200M/s.
Referring to fig. 1, fig. 1 is a schematic diagram illustrating an application scenario provided in an embodiment of the present application. The application scenario shown in fig. 1 includes a server 110, a capture device 120, and a ClickHouse database 130.
It should be understood that the server 110 may be a single server.
It should also be understood that the specific device of the acquisition device 120 may be set according to actual requirements, as long as it is ensured that the acquisition device 120 can acquire the original log data, and the embodiment of the present application is not limited thereto.
For example, the collecting device 120 may be a switch, a router, or the like.
In order to facilitate understanding of the embodiments of the present application, the following description will be given by way of specific examples.
Specifically, the collection device 120 may collect all raw log data in the intranet, and send the collected raw log data to the server 110. All the raw log data may include mail related data, chat related data, and the like.
Subsequently, the server 110 may perform a first pre-processing on the received raw log data. Subsequently, the server 110 may store the first preprocessed raw log data in the clickwouse database 130. The clickwouse database 130 may be stored in a table according to different time granularities.
It should be understood that the specific operations involved in the first preprocessing may be set according to actual requirements, and the embodiments of the present application are not limited thereto.
For example, the first pre-processing may include a screening operation, may also include a sorting operation, may also include an aggregation operation, may also include a desensitization operation, and the like.
It should be noted that, since the clickwouse database 130 may perform table storage according to different time granularities, the server 110 may store the original log data after the first preprocessing into a table of the corresponding time granularity.
Alternatively, in a case where the ClickHouse database 130 may include a minute table corresponding to a time granularity of minutes, an hour table corresponding to a time granularity of hours, a day table corresponding to a time granularity of days, a week table corresponding to a time granularity of weeks, a month table corresponding to a time granularity of months, a year table corresponding to a time granularity of years, and the like, the server 110 may analyze a time granularity corresponding to original log data to be currently stored, and store the time granularity into a specific table of the corresponding time granularity.
Further, the specific storage manner may be determined according to the data amount of the current object (e.g., company, etc.).
Optionally, when the data amount of the current object per day is greater than or equal to the preset data amount, performing table-splitting storage by taking "day" as granularity may be selected; in the case that the data amount of the current object per day is smaller than the preset data amount, the table storage with the granularity of 'month' can be selected.
It should be understood that the specific value of the preset data amount may be set according to actual requirements, and the embodiment of the present application is not limited thereto.
For example, for a first company, the amount of data generated by the first company per day is almost 1000 thousands, and the amount of data per day is greater than a preset amount of data (e.g., 50 thousands or 100 thousands, etc.), then the data storage manner of the first company may select to perform table division storage with "day" as granularity, and the name of each table may be composed of specific days and the identifier of an object (e.g., a certain day table has a name of Y20200801 and stores data of day 1/8/2020, where Y is the identifier of an object corresponding to the stored data, and another day table has a name of B20200802 and stores data of day 2/8/2020, where B is the identifier of an object corresponding to the stored data).
For another example, for a second company, the amount of data generated by the second company per day is only about 50 ten thousand, about 1500 ten thousand per month, and the amount of data per day is less than a preset amount of data (e.g., 50 ten thousand, etc.), then the data storage manner of the second company may select to perform sub-table storage with "month" as granularity, and the name of each table may be a specific number of months (e.g., a certain month table is named Z202008 and stores data of 8 months in 2020, where Z is an identifier of an object corresponding to the stored data, and another month table is named N202009 and stores data of 9 months in 2020, where N is an identifier of an object corresponding to the stored data).
In addition, it should be noted that, for the same object, the object may use the same granularity of storing tables (for example, for a certain company, it can only store tables with the granularity of "day" or "month").
In addition, it should be noted that, although the table names formed by specific days and the identifiers of the objects (i.e., the data of different objects are stored in different tables) are taken as an example for description, those skilled in the art should understand that the table names may also be taken as specific days, so that the ClickHouse database 130 may store the data of the time corresponding to the internal network (i.e., the data of different objects may be stored in the same table), and the embodiment of the present application is not limited thereto.
Further, in a case where the server 110 acquires the log query request, the server 110 may find target log data matching the log query request from the ClickHouse database 130 according to the log query request, and generate a log query result including the target log data. Subsequently, the server 110 may output the log query result for the user to browse.
It should be noted that the log query scheme provided in the embodiment of the present application may be further extended to other suitable application scenarios, and is not limited to the application scenario shown in fig. 1.
Referring to fig. 2, fig. 2 is a flowchart illustrating a log query method according to an embodiment of the present disclosure. The log query method as shown in fig. 2 may be applied to a single server (hereinafter, described as a server for convenience of description), and includes:
in step S210, the server obtains a log query request.
It should be understood that the specific content included in the log query request may be set according to actual needs, and the embodiment of the present application is not limited thereto.
For example, the log query request may include a query time period and a query condition. The query time period may be a time range of log data to be queried; the query condition may be a condition for processing (e.g., sorting, filtering, or screening, etc.) all log data corresponding to the query time period.
For example, the query time period may be 5 months of 2020.
For another example, the query condition may be a rank, or may be a query for data related to a certain device, or may be a query for data related to a certain user, or the like.
It should be noted that, for the query time period and the query condition, the user may set according to actual requirements, and the embodiment of the present application is not limited to this.
For another example, the log query request may include an interface address and the like in addition to the query time period and the query condition. The interface address may refer to an address of an interface related to the clickwouse database, so that the server can query the log through the interface corresponding to the interface address.
It should also be understood that the specific manner in which the server obtains the log query request can also be set according to actual requirements.
For example, the server may receive a log query request sent by the client, obtain the log query request through a browser on the client, or obtain the log query request triggered at regular time.
In order to facilitate understanding of the embodiments of the present application, the following description will be given by way of specific examples.
Specifically, since the log query page may be provided with an input box of query time, an input box of query conditions (for example, data related to the type to be filtered may be displayed in a highlighted manner so that a subsequent user may perform a filtering operation on highlighted log data, etc.), a selection box of a format for deriving target log data, etc., after the user submits a click, the server may obtain a log query request including a plurality of items of content set by the user.
Step S220, the server searches target log data matched with the log query request from the ClickHouse database according to the log query request. Wherein the log query request may include a query time range and a query condition. Wherein the query condition may be a related query statement containing the query object.
It should be understood that the ClickHouse database may include at least one table, each table corresponding to a time granularity, and that there may be multiple tables for each table.
For example, the ClickHouse database may include a day table associated with a first object, may include a month table associated with a second object, may include a year table associated with a third object, and so on.
It should also be understood that the specific time unit corresponding to the time granularity may be set according to actual requirements, and the embodiment of the present application is not limited thereto.
For example, the specific time unit corresponding to the time granularity may be minutes, hours, days, months, years, or the like.
For another example, in the case that the specific time unit corresponding to the time granularity is a minute, the ClickHouse database may include a minute table corresponding to the minute, and in the ClickHouse database, the amount of data generated by a certain object for one minute may correspond to one minute table; in the case that the specific time unit corresponding to the time granularity is an hour, the ClickHouse database may include an hour table corresponding to the hour, and in the ClickHouse database, the amount of data generated by another object for one hour may correspond to one hour table; in the case that the specific time unit corresponding to the time granularity is a day, the ClickHouse database may include a day table corresponding to the day, and in the ClickHouse database, the amount of data generated by another object for one day may correspond to one day table; in the case where the specific time unit corresponding to the time granularity is a month, the clickwouse database may include a month table corresponding to the month, and in the clickwouse database, the data amount generated by another object for one month may correspond to a month table, and the like.
It should also be understood that the specific process of the server searching the target log data matched with the log query request from the clickwouse database according to the log query request may be set according to actual requirements, and the embodiment of the present application is not limited thereto.
In order to facilitate understanding of the embodiments of the present application, the following description will be given by way of specific examples.
Optionally, since the server may store a plurality of tables, the server may find the target table matching the query time period (or the target table matching the query time period and the query object) from the plurality of tables according to the query time period (in a case where the M tables store data of M objects, the server may find the query object and the query time period included in the query condition in the target query request, where M is a positive integer), that is, the time period corresponding to the target table is the query time period, so that the search range can be greatly reduced. And the server can also find out the target log data matched with the query condition from the target table according to the query condition in the log query request, so that the related content of each table is not required to be searched, and only the target table is required to be searched, and the log query efficiency can be improved.
For example, in the case where the query time period is 2020 and 8 months and the query condition is a relevant condition to the first query object, the server may query the log data relevant to the first query object based on the query time period.
In addition, since the query time period may be a longer time period, in order to further improve the log query efficiency, the server may time-divide the query time period according to a preset time granularity to obtain at least two time periods, and then the server may find the target table matching each of the at least two time periods from all the tables.
For example, in a case where the query time period is from 13: 14: 8/30/2020 to 15: 18: 9/1/2020 and the query condition is a related condition to the first query object, the "day" may be used as a granularity, the query time range may be divided into three time periods, i.e., from 13: 14: 8/30/2020 to 23: 59: 8/30/2020, from 0: 8/31: 2020 to 23: 59: 8/31: 2020, and from 0: 9/1: 2020 to 15: 18: 9/1: 2020, and the server may find the log data corresponding to the three time periods from the month table corresponding to 8 months and the month table corresponding to 9 months of the first object, respectively. And, while presenting the relevant log data, presenting may be performed according to different time periods (e.g., a user may select a different time period from a date selection box to display data corresponding to the different time period).
It should be understood that the specific time unit corresponding to the preset time granularity may also be set according to actual requirements, and the embodiment of the present application is not limited thereto.
For example, the specific time unit corresponding to the preset time granularity may be at least one of minutes, hours, days, weeks, months, and years.
In addition, it should be noted that, the server may first summarize tables corresponding to all time periods (for example, a table is set for 4 months when the query time period is 4 months, and a table is set for one month when the time period of each month corresponds to each month), and then find out the target log data from the summarized tables, or first find out the log data from the tables corresponding to each time period, and then summarize all the found log data to obtain the target log data, which is not limited in this embodiment of the application.
Alternatively, the server may store a table for recording historical query records (for example, for the first object, the historical query record information may include historical query time and historical query results), so that, when a new log query request is received by the server, the new log query request may be matched with the table for recording the historical query records, so that the historical query record with the highest association degree with the new log query request may be obtained, and then the target log data matched with the new log query request may be quickly searched according to the historical query record with the highest association degree, thereby also improving query efficiency. The historical query record with the highest association degree with the new log query request refers to the historical query record closest to the query time and the query object of the new log query request.
For example, when the server receives a new log query request as log data for querying the first object from 13 o 'clock 14/30/2020 to 15 o' clock 15/1/2020, the new log query request may be matched with the table for recording the historical query records, so as to obtain the historical query record with the highest relevance of the new log query request as the historical query record for querying the first object from 0 o 'clock 0/29/2020 to 15 o' clock 1/9/2020, and then the target log data matched with the new log query request may be quickly searched according to the historical query record with the highest relevance, so that the query range may also be reduced, and each table does not need to be traversed, thereby increasing the query efficiency.
In addition, considering the situation that the storage space of the server is occupied by the historical query records and the query is dead due to the fact that the disk is full as time increases, the server can count the number of times of use of each historical query record in all the historical query records.
The server keeps the current historical query record under the condition that the used times of the current historical query record are determined to be more than or equal to the preset times; and under the condition that the used times of the current historical query records are determined to be less than the preset times by the server, further determining whether the storage time of the current historical query records exceeds the preset time, deleting the current historical query records if the storage time of the current historical query records exceeds the preset time, and returning to continue counting the used times of the current historical query records if the storage time of the current historical query records does not exceed the preset time.
That is, in order to avoid a case where a newly stored history query record is immediately deleted, the history query record is deleted only in a case where the storage time exceeds a preset time and the number of times of use is made smaller than a preset number of times.
It should be understood that the specific number of the preset times and the specific time of the preset time can be set according to actual requirements, and the embodiment of the application is not limited thereto.
For example, the preset number of times for different historical query records may be different or the same.
In step S230, the server generates a log query result including the target log data.
Specifically, the server may perform second preprocessing on the acquired target log data, and then generate a log data query result according to the target log data subjected to the second preprocessing.
It should be understood that the operations involved in the second preprocessing may be set according to actual requirements, and the embodiments of the present application are not limited thereto.
For example, the second pre-processing may include format conversion (e.g., conversion of time format, which may be converted from a specific time to a timestamp that the log browsing interface is capable of displaying, etc.), and may also include generating a table. Wherein the table is generated by the format-converted log data.
In step S240, the server outputs a log query result.
For example, the server may send the log query results to the client so that the client can display the log query results.
Therefore, according to the method and the device for querying the log data, the log data can be queried by utilizing the advantage that the query speed of the ClickHouse database is high, so that the query efficiency can be improved, and the user experience is improved.
In order to facilitate understanding of the embodiments of the present application, the following description will be given by way of specific examples.
Referring to fig. 3, fig. 3 is a specific flowchart illustrating a log query method according to an embodiment of the present application. The log query method as shown in fig. 3 includes:
step S310, storing the acquired original log data into a ClickHouse database through a storage component in the server.
It should be understood that the specific devices of the storage component may be set according to actual requirements, and the embodiments of the present application are not limited thereto.
For example, the storage component may be a Stream interface in Java.
It should be noted here that, although step S310 is described in fig. 3, it should be understood by those skilled in the art that, in the case where log data corresponding to the log query request is stored in the ClickHouse database in advance, step S320 may be directly performed without performing step S310.
Step S320, a log query request is obtained through a log component in the server.
It should be noted that, for the log component in the embodiment of the present application, it may correspond to different types of logs (for example, a plurality of logs such as a virus log and an alarm log may share one log component). In addition, when a new type of log needs to be added, a new log component does not need to be developed, and only parameters related to the new type of log (such as an interface of a corresponding clickwouse database) need to be imported into the log component, so that great convenience is brought to developers.
In addition, the embodiment of the application can also realize completely autonomous service, and developers can master all technical details of the realization of log audit, which is more convenient when dealing with the increasing user demands.
Step S330, searching target log data corresponding to the log query request from the ClickHouse database through a searching component in the server.
It should be understood that the specific apparatus of the search component may be set according to actual requirements, and the embodiments of the present application are not limited thereto.
For example, the search component may be a RESTful interface in Java.
Step S340, generating a log query result including the target log data through the log component in the server, and outputting the log query result.
It should be understood that the above log query method is only exemplary, and those skilled in the art can make various changes, modifications or variations according to the above method and also fall within the scope of the present application.
Referring to fig. 4, fig. 4 shows a block diagram of a log query apparatus 400 according to an embodiment of the present application, it should be understood that the log query apparatus 400 can perform the steps related to the above method embodiment, and specific functions of the log query apparatus 400 may be referred to the above description, and a detailed description is appropriately omitted herein to avoid redundancy. The log query device 400 includes at least one software function module that can be stored in a memory in the form of software or firmware (firmware) or solidified in an Operating System (OS) of the log query device 400. Specifically, the log query apparatus 400 includes:
an obtaining module 410, configured to obtain a log query request; the searching module 420 is configured to search, according to the log query request, target log data matched with the log query request from the clickwouse database, and generate a log query result including the target log data; and an output module 430, configured to output the log query result.
In one possible embodiment, the ClickHouse database includes at least one table, each table corresponding to a time granularity.
In one possible embodiment, the log query request includes a query time period and a query condition, and the lookup module 420 includes: a first searching sub-module (not shown) for searching a target table matched with the query time period from all tables of the ClickHouse database according to the query time period; and a second lookup sub-module (not shown) for looking up the target log data matching the query condition from the target table.
In a possible embodiment, the first lookup submodule is specifically configured to: according to the preset time granularity, performing time segmentation on the query time period to obtain at least two time periods; the target table matching each time period is looked up from all tables.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working process of the apparatus described above may refer to the corresponding process in the foregoing method, and will not be described in too much detail herein.
Fig. 5 is a block diagram of an electronic device 500 according to an embodiment of the present disclosure, as shown in fig. 5. Electronic device 500 may include a processor 510, a communication interface 520, a memory 530, and at least one communication bus 540. Wherein the communication bus 540 is used for realizing direct connection communication of these components. The communication interface 520 in the embodiment of the present application is used for communicating signaling or data with other devices. Processor 510 may be an integrated circuit chip having signal processing capabilities. The Processor 510 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor 510 may be any conventional processor or the like.
The Memory 530 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like. The memory 530 stores computer readable instructions that, when executed by the processor 510, enable the electronic device 500 to perform the various steps involved in the above-described method embodiments.
The electronic device 500 may further include a memory controller, an input-output unit, an audio unit, and a display unit.
The memory 530, the memory controller, the processor 510, the peripheral interface, the input/output unit, the audio unit, and the display unit are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements may be electrically coupled to each other via one or more communication buses 540. The processor 510 is used to execute executable modules stored in the memory 530. Also, the electronic device 500 is configured to perform the following method: acquiring a log query request; according to the log query request, searching target log data matched with the log query request from a ClickHouse database, and generating a log query result comprising the target log data; and outputting the log query result.
The input and output unit is used for providing input data for a user to realize the interaction of the user and the server (or the local terminal). The input/output unit may be, but is not limited to, a mouse, a keyboard, and the like.
The audio unit provides an audio interface to the user, which may include one or more microphones, one or more speakers, and audio circuitry.
The display unit provides an interactive interface (e.g. a user interface) between the electronic device and a user or for displaying image data to a user reference. In this embodiment, the display unit may be a liquid crystal display or a touch display. In the case of a touch display, the display can be a capacitive touch screen or a resistive touch screen, which supports single-point and multi-point touch operations. The support of single-point and multi-point touch operations means that the touch display can sense touch operations simultaneously generated from one or more positions on the touch display, and the sensed touch operations are sent to the processor for calculation and processing.
It will be appreciated that the configuration shown in FIG. 5 is merely illustrative and that the electronic device 500 may include more or fewer components than shown in FIG. 5 or may have a different configuration than shown in FIG. 5. The components shown in fig. 5 may be implemented in hardware, software, or a combination thereof.
The present application also provides a storage medium having a computer program stored thereon, which, when executed by a processor, performs the method of the method embodiments.
The present application also provides a computer program product which, when run on a computer, causes the computer to perform the method of the method embodiments.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the system described above may refer to the corresponding process in the foregoing method, and will not be described in too much detail herein.
It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the device-like embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes. It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A log query method, comprising:
acquiring a log query request;
according to the log query request, searching target log data matched with the log query request from a ClickHouse database, and generating a log query result comprising the target log data;
and outputting the log query result.
2. The log query method of claim 1, wherein the ClickHouse database comprises at least one table, each of the tables corresponding to a time granularity.
3. The log query method of claim 2, wherein the log query request comprises a query time period and a query condition, and the searching target log data matched with the log query request from a ClickHouse database according to the log query request comprises:
according to the query time period, searching a target table matched with the query time period from all tables of the ClickHouse database;
and finding out target log data matched with the query condition from the target table.
4. The log query method as claimed in claim 3, wherein said searching a target table matching the query time period from all tables of the ClickHouse database according to the query time period comprises:
according to the preset time granularity, performing time segmentation on the query time period to obtain at least two time periods;
and searching a target table matched with each time period from all the tables.
5. A log querying device, comprising:
the acquisition module is used for acquiring a log query request;
the searching module is used for searching target log data matched with the log query request from a ClickHouse database according to the log query request and generating a log query result comprising the target log data;
and the output module is used for outputting the log query result.
6. The log querying device of claim 5, wherein the ClickHouse database comprises at least one table, each table corresponding to a time granularity.
7. The log query device of claim 6, wherein the log query request comprises a query time period and a query condition, and the search module comprises:
the first searching submodule is used for searching a target table matched with the query time period from all tables of the ClickHouse database according to the query time period;
and the second searching submodule is used for searching the target log data matched with the query condition from the target table.
8. The log querying device of claim 7, wherein the first lookup sub-module is specifically configured to: according to the preset time granularity, performing time segmentation on the query time period to obtain at least two time periods; and searching a target table matched with each time period from all the tables.
9. A storage medium, having stored thereon a computer program which, when executed by a processor, performs the log query method according to any one of claims 1 to 4.
10. An electronic device, characterized in that the electronic device comprises: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating over the bus when the electronic device is operating, the machine-readable instructions when executed by the processor performing the log querying method of any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010976706.7A CN112100138A (en) | 2020-09-16 | 2020-09-16 | Log query method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010976706.7A CN112100138A (en) | 2020-09-16 | 2020-09-16 | Log query method and device, storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112100138A true CN112100138A (en) | 2020-12-18 |
Family
ID=73758709
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010976706.7A Pending CN112100138A (en) | 2020-09-16 | 2020-09-16 | Log query method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112100138A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112988798A (en) * | 2021-03-29 | 2021-06-18 | 成都卫士通信息产业股份有限公司 | Log processing method, device, equipment and medium |
| CN113112158A (en) * | 2021-04-13 | 2021-07-13 | 青岛海尔科技有限公司 | Method and device for processing equipment use data, storage medium and electronic device |
| CN113343023A (en) * | 2021-06-01 | 2021-09-03 | 广州欢网科技有限责任公司 | Method and system for rapidly inquiring channel ranking in time period |
| CN115662607A (en) * | 2022-12-13 | 2023-01-31 | 四川大学 | Internet online inquiry recommendation method based on big data analysis and server |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104657387A (en) * | 2013-11-22 | 2015-05-27 | 华为技术有限公司 | Data query method and device |
| CN104765800A (en) * | 2015-03-30 | 2015-07-08 | 浪潮集团有限公司 | Big data based efficient search method |
| CN105389352A (en) * | 2015-10-30 | 2016-03-09 | 北京奇艺世纪科技有限公司 | Log processing method and apparatus |
| CN106407190A (en) * | 2015-07-27 | 2017-02-15 | 阿里巴巴集团控股有限公司 | Event record querying method and device |
| CN107506140A (en) * | 2017-08-16 | 2017-12-22 | 郑州云海信息技术有限公司 | A kind of data access method and system for storage system |
| CN108304527A (en) * | 2018-01-25 | 2018-07-20 | 杭州哲信信息技术有限公司 | A kind of data extraction method |
| CN109788307A (en) * | 2019-02-11 | 2019-05-21 | 北京字节跳动网络技术有限公司 | Processing method, device, storage medium and the electronic equipment of video list |
| CN109933645A (en) * | 2019-01-28 | 2019-06-25 | 平安科技（深圳）有限公司 | Information query method, device, computer equipment and storage medium |
| CN110008228A (en) * | 2019-03-26 | 2019-07-12 | 北京字节跳动网络技术有限公司 | Acquisition methods and device, the storage medium and electronic equipment of user group's data |
| CN110096489A (en) * | 2019-04-30 | 2019-08-06 | 阿里巴巴集团控股有限公司 | A kind of data query method, system, device and electronic equipment |
-
2020
- 2020-09-16 CN CN202010976706.7A patent/CN112100138A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104657387A (en) * | 2013-11-22 | 2015-05-27 | 华为技术有限公司 | Data query method and device |
| CN104765800A (en) * | 2015-03-30 | 2015-07-08 | 浪潮集团有限公司 | Big data based efficient search method |
| CN106407190A (en) * | 2015-07-27 | 2017-02-15 | 阿里巴巴集团控股有限公司 | Event record querying method and device |
| CN105389352A (en) * | 2015-10-30 | 2016-03-09 | 北京奇艺世纪科技有限公司 | Log processing method and apparatus |
| CN107506140A (en) * | 2017-08-16 | 2017-12-22 | 郑州云海信息技术有限公司 | A kind of data access method and system for storage system |
| CN108304527A (en) * | 2018-01-25 | 2018-07-20 | 杭州哲信信息技术有限公司 | A kind of data extraction method |
| CN109933645A (en) * | 2019-01-28 | 2019-06-25 | 平安科技（深圳）有限公司 | Information query method, device, computer equipment and storage medium |
| CN109788307A (en) * | 2019-02-11 | 2019-05-21 | 北京字节跳动网络技术有限公司 | Processing method, device, storage medium and the electronic equipment of video list |
| CN110008228A (en) * | 2019-03-26 | 2019-07-12 | 北京字节跳动网络技术有限公司 | Acquisition methods and device, the storage medium and electronic equipment of user group's data |
| CN110096489A (en) * | 2019-04-30 | 2019-08-06 | 阿里巴巴集团控股有限公司 | A kind of data query method, system, device and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| 马刚: "《商务智能》", vol. 1, 东北财经大学出版社, pages: 118 - 128 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112988798A (en) * | 2021-03-29 | 2021-06-18 | 成都卫士通信息产业股份有限公司 | Log processing method, device, equipment and medium |
| CN112988798B (en) * | 2021-03-29 | 2023-05-23 | 成都卫士通信息产业股份有限公司 | Log processing method, device, equipment and medium |
| CN113112158A (en) * | 2021-04-13 | 2021-07-13 | 青岛海尔科技有限公司 | Method and device for processing equipment use data, storage medium and electronic device |
| CN113343023A (en) * | 2021-06-01 | 2021-09-03 | 广州欢网科技有限责任公司 | Method and system for rapidly inquiring channel ranking in time period |
| CN115662607A (en) * | 2022-12-13 | 2023-01-31 | 四川大学 | Internet online inquiry recommendation method based on big data analysis and server |
| CN115662607B (en) * | 2022-12-13 | 2023-04-07 | 四川大学 | Internet online inquiry recommendation method based on big data analysis and server |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112100138A (en) | Log query method and device, storage medium and electronic equipment | |
| US10338977B2 (en) | Cluster-based processing of unstructured log messages | |
| JP5661104B2 (en) | Method and system for search using search engine indexing and index | |
| US8990241B2 (en) | System and method for recommending queries related to trending topics based on a received query | |
| JP5575902B2 (en) | Information retrieval based on query semantic patterns | |
| JP5264740B2 (en) | Time series search engine | |
| CN107451149B (en) | Monitoring method and device for flow data query task | |
| US20120124047A1 (en) | Managing log entries | |
| TWI564737B (en) | Web search methods and devices | |
| CN107861981B (en) | Data processing method and device | |
| CN112988863A (en) | Elasticissearch-based efficient search engine method for heterogeneous multiple data sources | |
| CN110750581A (en) | Data display method and device of table component, storage medium and electronic equipment | |
| CN111258819A (en) | Data acquisition method, device and system for MySQL database backup file | |
| US20190034247A1 (en) | Creating alerts associated with a data storage system based on natural language requests | |
| CN112380416A (en) | Method for updating course index, course searching method and device | |
| CN110347922B (en) | Recommendation method, device, equipment and storage medium based on similarity | |
| CN112597168A (en) | Processing method, device and platform of multi-source customer data and storage medium | |
| WO2016027364A1 (en) | Topic cluster selection device, and search method | |
| CN111680072B (en) | System and method for dividing social information data | |
| TWI547888B (en) | A method of recording user information and a search method and a server | |
| CN111259274A (en) | Information processing method, device, equipment and information display device | |
| CN113434607A (en) | Behavior analysis method and device based on graph data, electronic equipment and storage medium | |
| CN104951869A (en) | Workflow-based public opinion monitoring method and workflow-based public opinion monitoring device | |
| US11720591B1 (en) | Virtual metrics | |
| WO2024079833A1 (en) | Information processing device, output method, and output program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |