CN116126795A - Log retrieval method and device, electronic equipment and storage medium - Google Patents

Log retrieval method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN116126795A
CN116126795A CN202310098348.8A CN202310098348A CN116126795A CN 116126795 A CN116126795 A CN 116126795A CN 202310098348 A CN202310098348 A CN 202310098348A CN 116126795 A CN116126795 A CN 116126795A
Authority
CN
China
Prior art keywords
record
search
target
keywords
retrieval
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310098348.8A
Other languages
Chinese (zh)
Inventor
苑小军
郭锐
安文然
王国乾
刘慧慧
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanxi Saidun Network Security Evaluation Technology Co ltd
Original Assignee
Shanxi Saidun Network Security Evaluation Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanxi Saidun Network Security Evaluation Technology Co ltd filed Critical Shanxi Saidun Network Security Evaluation Technology Co ltd
Priority to CN202310098348.8A priority Critical patent/CN116126795A/en
Publication of CN116126795A publication Critical patent/CN116126795A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/14Details of searching files based on file metadata
    • G06F16/148File search processing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/14Details of searching files based on file metadata
    • G06F16/144Query formulation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/14Details of searching files based on file metadata
    • G06F16/156Query results presentation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Library & Information Science (AREA)
  • Mathematical Physics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application relates to a log retrieval method, a log retrieval device, electronic equipment and a storage medium. The log retrieval method comprises the following steps: acquiring a search keyword; determining all search keywords matched with the search keywords in a search library according to the search keywords; acquiring corresponding search conditions according to the search keywords; for each search condition, acquiring a record brief introduction in the search condition; generating a record profile list according to all record profiles, and displaying the profile list; acquiring a designated record brief introduction; the specified record profile is a record profile selected according to the profile list; determining target retrieval conditions according to the designated record profile; determining that the corresponding record is positioned at a specific position of the log file according to the target retrieval condition; and acquiring a target record according to the specific position, and displaying the target record.

Description

Log retrieval method and device, electronic equipment and storage medium
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a method and apparatus for retrieving logs, an electronic device, and a storage medium.
Background
When each system operates, event records named as logs are generated, and the logs mainly record the operating state and the operating content of the system during operation. When a system loopholes or a system accident occurs, a log is often used for fault investigation, or when system data is manually operated, the log is used for personnel operation analysis.
However, for the log generated by the monitoring network system, massive text information is often recorded in one log file, and when analyzing some text information of the log file, the whole log file needs to be read, so that the problem that the log reading time is too long and the system resource is wasted is caused.
Disclosure of Invention
The application provides a log retrieval method, a device, electronic equipment and a storage medium, which are used for solving the problems that when a certain text message is retrieved in a log at present, a system is required to read the whole log file, then the text message is retrieved by a little by manpower, and the system resource and time are wasted too, and achieving the effect of quickly retrieving target text messages by inputting search keywords.
In a first aspect, the present application provides a log retrieval method, including:
Acquiring a search keyword;
determining all search keywords matched with the search keywords in a search library according to the search keywords;
acquiring corresponding search conditions according to the search keywords;
for each search condition, acquiring a record brief introduction in the search condition;
generating a record profile list according to all record profiles, and displaying the profile list;
acquiring a designated record brief introduction; the specified record profile is a record profile selected according to the profile list;
determining target retrieval conditions according to the designated record profile;
determining that the corresponding record is positioned at a specific position of the log file according to the target retrieval condition;
and acquiring a target record according to the specific position, and displaying the target record.
According to the scheme, the record list is returned by inputting the search keywords generated by the search keyword matching system to the user, so that the screening difficulty of the user is reduced, the user determines the needed target log record by the record list, and finally, the target log is accurately read through the specific position of the log record in the search condition, so that the need of reading the whole log file is avoided, and the efficiency of searching the target log is improved.
Optionally, monitoring a target log file, and obtaining a new record of the target log file;
determining a retrieval condition and a retrieval keyword corresponding to the new record according to the new record;
and correspondingly storing the search conditions and the search keywords into a search library.
According to the scheme, the setting monitoring unit is utilized to acquire the newly-added record of each writing log file, corresponding operation can be carried out on each piece of newly-added information, the retrieval key words and the retrieval conditions of each piece of newly-added information are correspondingly obtained and correspondingly stored in the retrieval library, the reading conditions are provided for accurately reading each record in the log file, and the efficiency of retrieving the log is improved to a certain extent.
Optionally, the monitoring the target log file, obtaining a new record of the target log file includes:
acquiring the storage position of the target log file, and monitoring the size of the target log file;
when the target log file is newly added, recording the data of the size before the new addition and the data of the size after the new addition of the target log file;
reading the changed target log file in reverse order according to the newly increased size data to obtain a newly increased record;
the determining, according to the new record, the search condition and the search keyword corresponding to the new record includes:
Combining keywords in the newly added record to generate a record brief introduction;
generating search conditions according to the data of the size before the new increase, the data of the size after the new increase and the record brief introduction;
and extracting different types of search keywords according to the record brief introduction.
According to the scheme, the log file is read by using the newly increased size of the target log file, so that the latest newly increased record content can be obtained quickly each time, then different types of keywords are generated by the newly increased record content, and the keywords are generated into record brief introduction, so that the aim of summarizing the core content of the newly increased record is achieved, a user can judge whether the newly increased record is needed according to the record brief introduction, and then the size data before the new increase, the size data after the new increase and the record brief introduction are recorded by the monitoring unit to generate search conditions and search keywords according to the record brief introduction, so that the efficiency is improved for subsequent search of the newly increased record, and the time for searching the target record is reduced.
Optionally, the type of the search keyword includes: generating time, operator, operated, and operating state; the search keyword includes: inputting characters, screening time and screening state in a search box;
And determining all the search keywords matched with the search keywords in the search library according to the search keywords, wherein the method comprises the following steps:
according to the search keywords, matching the input characters in the search box with the search keywords of the operator type and the operated type, and confirming the search keywords with the same characters; matching the screening time with the search keywords of the generation time type, and confirming the search keywords with the time coincidence; matching the screening state with the search keywords of the operation state type, and confirming the search keywords with the same characters describing the state;
the obtaining the corresponding search condition according to the search keyword comprises the following steps:
classifying the search keywords according to types;
respectively acquiring corresponding search conditions according to different types of search keywords;
and comparing the search conditions corresponding to the search keywords of different types, and picking out the search conditions which repeatedly appear in the search conditions corresponding to the search keywords of all types as the search conditions corresponding to the search keywords.
According to the scheme, the search keywords and the search keywords are classified and refined, search matching is carried out according to each category, so that keywords input by a user can be searched efficiently, finally, search conditions corresponding to no category are compared and screened, search conditions which simultaneously accord with the search keywords are selected, and the search accuracy of the target log record is improved.
Optionally, storing the target record and the retrieval condition corresponding to the target record in a cache in an associated manner;
the determining that the corresponding record is located in the specific position of the log file according to the target retrieval condition comprises the following steps:
confirming whether the target retrieval condition is contained in a cache according to the target retrieval condition, and if so, confirming the specific position of the target record in the cache;
if not, determining that the corresponding record is positioned at the specific position of the log file according to the target retrieval condition.
According to the scheme, the target records searched by the user are stored in the cache, so that when the target records are searched for two or more times, the target records can be obtained through the cache, when certain target records are searched for a high frequency, the searching time is shortened, the reading pressure of a disk is also reduced, and the retrieval efficiency of the log is further optimized.
Optionally, the storing the target record and the search condition association corresponding to the target record in a cache includes:
judging whether the number of the storage target records in the cache exceeds the preset storage number or not;
if yes, deleting the record with the least queried times in the cache and the search condition corresponding to the record, and storing the target record and the search condition corresponding to the target record in the cache in a correlated manner;
If not, storing the target record and the search condition association corresponding to the target record into a cache.
According to the scheme, the normal use of the memory space is protected by setting the preset storage number of the cache, and the target records stored in the cache are all high-frequency search records in a mode of replacing the target records with fewer inquired times in the cache, so that the search efficiency of the target log records is improved to a certain extent.
Optionally, setting a cache update time;
inquiring the inquired times of each record in the cache when the set cache updating time is reached;
and deleting all the records with the queried times smaller than the preset times.
According to the scheme, the cache update time can be set to delete the target records with low queried times in one period at regular time, so that the target records in the cache are all target records with more queried times in the last period.
In a second aspect, the present application provides a log retrieval device, including:
the first acquisition module is used for acquiring the search keywords;
the matching module is used for determining all search keywords matched with the search keywords in the search library according to the search keywords;
The retrieval module is used for acquiring corresponding retrieval conditions according to the retrieval keywords; for each search condition, acquiring a record brief introduction in the search condition; generating a record profile list according to all record profiles, and calling a display module to display the profile list;
the second acquisition module is used for acquiring the appointed record brief introduction; the specified record profile is a record profile selected according to the profile list;
the reading module is used for determining target retrieval conditions according to the designated record brief introduction; determining that the corresponding record is positioned at a specific position of the log file according to the target retrieval condition; and acquiring a target record according to the specific position, and calling a display module to display the target record.
Optionally, the log searching device further includes: a monitoring module;
the monitoring module is used for monitoring the target log file and acquiring a new record of the target log file;
determining a retrieval condition and a retrieval keyword corresponding to the new record according to the new record;
and correspondingly storing the search conditions and the search keywords into a search library.
Optionally, the monitoring module monitors the target log file, and when obtaining a new record of the target log file, the monitoring module is specifically configured to: acquiring the storage position of the target log file, and monitoring the size of the target log file;
When the target log file is newly added, recording the data of the size before the new addition and the data of the size after the new addition of the target log file;
reading the changed target log file in reverse order according to the newly increased size data to obtain a newly increased record;
the monitoring module is specifically configured to, when determining, according to the new record, a search condition and a search keyword corresponding to the new record:
combining keywords in the newly added record to generate a record brief introduction;
generating search conditions according to the data of the size before the new increase, the data of the size after the new increase and the record brief introduction;
and extracting different types of search keywords according to the record brief introduction.
Optionally, the type of the search keyword includes: generating time, operator, operated, and operating state; the search keyword includes: inputting characters, screening time and screening state in a search box;
the matching module is specifically used for determining all search keywords matched with the search keywords in the search library according to the search keywords: according to the search keywords, matching the input characters in the search box with the search keywords of the operator type and the operated type, and confirming the search keywords with the same characters; matching the screening time with the search keywords of the generation time type, and confirming the search keywords with the time coincidence; matching the screening state with the search keywords of the operation state type, and confirming the search keywords with the same characters describing the state;
The retrieval module is specifically configured to, when obtaining the corresponding retrieval condition according to the retrieval keyword:
classifying the search keywords according to types;
respectively acquiring corresponding search conditions according to different types of search keywords;
and comparing the search conditions corresponding to the search keywords of different types, and picking out the search conditions which repeatedly appear in the search conditions corresponding to the search keywords of all types as the search conditions corresponding to the search keywords.
Optionally, the log searching device further includes: a cache module;
the cache module is used for storing the target record and the retrieval condition corresponding to the target record in a cache in an associated manner;
the reading module is specifically configured to, when determining that the corresponding record is located at a specific position of the log file according to the target search condition:
confirming whether the target retrieval condition is contained in a cache according to the target retrieval condition, and if so, confirming the specific position of the target record in the cache;
if not, determining that the corresponding record is positioned at the specific position of the log file according to the target retrieval condition.
Optionally, when the cache module stores the target record and the search condition association corresponding to the target record in the cache, the cache module is specifically configured to:
Judging whether the number of the storage target records in the cache exceeds the preset storage number or not;
if yes, deleting the record with the least queried times in the cache and the search condition corresponding to the record, and storing the target record and the search condition corresponding to the target record in the cache in a correlated manner;
if not, storing the target record and the search condition association corresponding to the target record into a cache.
Optionally, the log searching device further includes: updating a module;
the updating module is used for setting cache updating time;
inquiring the inquired times of each record in the cache when the set cache updating time is reached;
and deleting all the records with the queried times smaller than the preset times.
In a third aspect, the present application provides an electronic device, comprising: a memory and a processor, the memory having stored thereon a computer program capable of being loaded by the processor and performing the method of the first aspect.
In a fourth aspect, the present application provides a computer readable storage medium storing a computer program capable of being loaded by a processor and performing the method of the first aspect.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief description will be given below of the drawings that are needed in the embodiments or the prior art descriptions, it being obvious that the drawings in the following description are some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort to a person skilled in the art.
Fig. 1 is a schematic view of an application scenario provided in an embodiment of the present application;
FIG. 2 is a flowchart of a log searching method according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a log searching device according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It will be apparent that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
In addition, the term "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In this context, unless otherwise specified, the term "/" generally indicates that the associated object is an "or" relationship.
Embodiments of the present application are described in further detail below with reference to the drawings attached hereto.
With the development of the internet, more and more services are responsible for each network system, and when the request of each service is abnormal, a log file retrieval mode is often adopted to check the processing record of the service request, so as to analyze the reason of the abnormality. However, the number of service requests per day of the network system is very large, so that massive text information is often recorded in the log file, and in the current technology, when a certain piece of log information is searched, all log files need to be traversed, so that the problem that the time for reading the log files is long and the searching efficiency is very low exists.
Based on the above, the application provides a log retrieval method, a device, an electronic device and a storage medium, which can accurately read the log file according to the search keywords input by the user, so that the steps of full-text traversal and manual retrieval of the log file are avoided. So as to improve the log retrieval efficiency and the user experience.
Fig. 1 is a schematic view of an application scenario provided in the present application. As shown in fig. 1, a user sends a search keyword to a server through a device, the server searches according to the search keyword, returns a record profile list conforming to the search keyword to the user device, and after the user selects any record profile, the server returns a target record to the user device according to the record profile selected by the user. Reference may be made to the following examples for specific implementation.
Fig. 2 is a flowchart of a log searching method according to an embodiment of the present application, where the method of the present embodiment may be applied to a server in the above scenario. As shown in fig. 2, the method includes:
s201, acquiring search keywords, and determining search keywords matched with the search keywords in a search library according to the search keywords.
The search keywords refer to information content which is input by a user and can furthest summarize log records needing to be searched. The search key refers to information content generated from the main content of each record in the log file.
The search library, specifically a database, stores the search key words of each record of the log file. Matching, including precision matching, fuzzy matching, range matching.
Specifically, after the user inputs the search keywords on the search page, the search keywords are searched in the search library, and all the search keywords which meet the matching conditions of the search keywords are searched out. For example, if the keyword of the search inputted by the user is the date 2023/01/29, the search keyword conforming to the search keyword is 2023/01/29-00:00:00 to 2023/01/29-23:59:59, and all search keywords in the search table. Or, if the keyword of the search input by the user is abnormal, all abnormal search keywords are searched out.
S202, acquiring corresponding search conditions according to the search keywords, and acquiring record brief introduction in the search conditions according to each search condition.
The search conditions and the search keywords are stored in a search library in a corresponding relation, each search keyword is corresponding to one search condition, the search conditions are specifically in a JSON format, and the search conditions comprise the storage position of the target record in the log file and the record brief introduction of the target record. The record profile, which is based on a combination of key information in the recorded content, indicates the main content of the record.
Specifically, the search key words and the search conditions are stored as a piece of information in a search library, all the searched search key words can correspondingly obtain a search condition, and after the search condition is obtained, a record brief introduction in the search condition is obtained by analyzing the JSON format. For example, the above-mentioned searched-out search keyword is abnormal, and the information on the search keyword and the search condition is "ID: 1. search key: abnormal, search conditions: { ' storage location ': 0kb ', ' record size ': '2kb', 'recording profile',: '2023/01/29-00:00:00 System cms anomaly' }, the available search conditions are specifically "search conditions: { ' storage location ': 0kb ', ' record size ': '2kb', 'recording profile',: '2023/01/29-00:00:00 System cms anomaly' }, and then recording the profile as '2023/01/29-00:00:00 System cms anomaly' by parsing the JSON format.
S203, a record profile list is generated according to all record profiles, and the profile list is displayed.
Specifically, the search conditions corresponding to all the search keywords searched from the search library are traversed, all the corresponding record brief introduction is searched, all the record brief introduction is displayed to the user pages in a list form, and the user pages are paged according to the brief introduction number. When the record profiles are displayed to the user in the form of a list, the list comprises information IDs corresponding to each record profile, and only the record profile is selected for display through page processing. For example, the search list is [ 2023/01/29-00:00:00 System cms abnormal ', '1', '2023/01/29-00:00:01 System cms abnormal ', '2', '2023/01/29-00:00:02 System cms abnormal ', '3', '1', '2', and '3' are IDs among the above information containing the search key and the search condition, and each information has a unique ID, and when the list is returned to the user page, each information ID is hidden, and only the profile list is displayed.
S204, acquiring a designated record profile, and determining target retrieval conditions according to the designated record profile.
Wherein, the designated record profile is a record profile selected by a user according to the profile list, and the target search condition is a search condition corresponding to the designated record profile.
Specifically, after displaying all the record profiles to the user page in the form of a list, the user selects one record profile from the record profiles, the page returns the information ID corresponding to the selected record profile to the server, and then the information is accurately searched from the search library through the information ID, so that the corresponding search condition is found.
S205, determining that the corresponding record is positioned at a specific position of the log file according to the target retrieval condition, acquiring a target record according to the specific position, and displaying the target record
The specific location is a 'storage location' in the above information, representing the byte location of the target record stored in the log file.
Specifically, the information of the 'storage position' and the information of the 'record size' are obtained by analyzing the JSON data of the target retrieval condition, then when the log file is read, the information of the 'record size' is read from the 'storage position', the read information is the information of the target record, and finally the target record is sent to a user page for display. For example, the target search condition is "search condition: { ' storage location ': 1kb ', ' record size ': '2kb', 'recording profile',: when the '2023/01/29-00:00:00 System datebase is abnormal ' } ', the ' storage position ' is read to be 1kb, the ' record size ' is read to be 2kb, and when the log file is read, 2kb of record information is read from the 1kb part, and the record information is the target record information.
According to the scheme, the record list is returned by inputting the search keywords generated by the search keyword matching system to the user, so that the screening difficulty of the user is reduced, the user determines the needed target log record by the record list, and finally, the target log is accurately read through the specific position of the log record in the search condition, so that the need of reading the whole log file is avoided, and the efficiency of searching the target log is improved.
In some embodiments, a target log file is monitored, a new record of the target log file is obtained, a search condition and a search keyword corresponding to the new record are determined according to the new record, and the search condition and the search keyword are correspondingly stored in a search library.
Specifically, a monitoring unit is arranged in the log storage server, the monitoring unit is a triggered monitoring unit, and when the monitored log file is newly added, the monitoring unit is triggered and reads the log file, and the reading content is specifically the newly added record of the log file.
Then analyzing the new record to determine the content information of the search key words and the search conditions corresponding to the new record,
and correspondingly storing the search keywords and the search conditions in the search library in the format of the information.
According to the scheme, the setting monitoring unit is utilized to acquire the newly-added record of each writing log file, corresponding operation can be carried out on each piece of newly-added information, the retrieval key words and the retrieval conditions of each piece of newly-added information are correspondingly obtained and correspondingly stored in the retrieval library, the reading conditions are provided for accurately reading each record in the log file, and the efficiency of retrieving the log is improved to a certain extent.
In some embodiments, monitoring the target log file, obtaining a new record of the target log file, includes: the method comprises the steps of obtaining a storage position of a target log file, monitoring the size of the target log file, recording the data of the size before the target log file is newly increased and the newly increased data, reading the changed target log file in reverse order according to the newly increased data to obtain a newly increased record, and determining the retrieval condition and the retrieval key word corresponding to the newly increased record according to the newly increased record, wherein the method comprises the following steps: and combining the keywords in the newly added record to generate a record brief introduction, generating search conditions according to the data of the size before the new addition, the data of the new size after the new addition and the record brief introduction, and extracting different types of search keywords according to the record brief introduction.
Specifically, a file path and a file name of the target log file are set for the monitoring unit, after the monitoring unit starts normal monitoring, the monitoring unit is triggered when the size of the target log file is newly increased in daily monitoring, at this time, the monitoring unit records the size data before the new increase of the target log file and the size data newly increased, then the target log file is read in a reverse mode, the newly increased size is read together, and at this time, the newly increased record information can be completely recorded.
Filtering and word segmentation are carried out on the newly added record information by using a word segmentation device, the whole text of the newly added record content is converted into a series of words or phrases, then each word or phrase is classified according to time class, state description class, name word class and other classes to obtain keywords of different types, or filtering and word segmentation are carried out on the newly added record by using a regular expression to obtain keywords of different types, and then the obtained keywords are combined in a specific sequence to obtain a record brief. The state description class is matched with the input specific words, and the same words are classified into the state description class, wherein the specific words generally mean words such as success, abnormality, failure and the like.
And finally, combining the size data before the new increase, the size data after the new increase and the record profile recorded by the monitoring unit to generate data information in a JSON format, wherein the data information is a retrieval condition, then intercepting according to the record profile to generate different types of retrieval keywords, and storing each retrieval keyword in a retrieval library corresponding to the retrieval condition.
For example, when the monitored log file is a 1kb file and the log file becomes 2kb, the monitoring unit is triggered, the monitoring unit records that the size data before the new addition is 1kb, the size data newly added this time is 1kb, and the content of the log file 2kb is read in a reverse manner, so as to obtain "system. Datebase Connection is not available, request timed out 2023/01/29-00:00:00", then filtering and classifying by using a word segmentation device to obtain a time class '2023/01/29-00:00', a noun class 'System', 'Datebase', a state description class being 'Exception', and combining to obtain a record brief of '2023/01/29-00:00:00 System Datebase Exception', wherein the generated search condition is as follows: { ' storage location ': 1kb ', ' record size ': '1kb', 'recording profile',: '2023/01/29-00:00:00 System Datebase Exception' }, generating '2023/01/29-00:00:00', 'System', 'Datebase', 'Exception' search keywords according to the record profile, and respectively storing the four search keywords and the search conditions into a search library for summarization.
According to the scheme, the log file is read by using the newly increased size of the target log file, so that the latest newly increased record content can be obtained quickly each time, then different types of keywords are generated by the newly increased record content, and the keywords are generated into record brief introduction, so that the aim of summarizing the core content of the newly increased record is achieved, a user can judge whether the newly increased record is needed according to the record brief introduction, and then the size data before the new increase, the size data after the new increase and the record brief introduction are recorded by the monitoring unit to generate search conditions and search keywords according to the record brief introduction, so that the efficiency is improved for subsequent search of the newly increased record, and the time for searching the target record is reduced.
In some embodiments, retrieving the type of keyword includes: generating time, operator, operated, operating state, search keywords, including: inputting characters, screening time and screening state in a search box, determining search keywords matched with all the search keywords in a search library according to the search keywords, wherein the method comprises the following steps: according to the search keywords, matching the input characters in the search box with the search keywords of the operator type and the operated type, confirming the search keywords with the same characters, matching the screening time with the search keywords of the generated time type, confirming the search keywords with the same time, matching the screening state with the search keywords of the operation state type, confirming the search keywords with the same characters of the description state, and obtaining the corresponding search conditions according to the search keywords, wherein the method comprises the following steps: classifying the search keywords according to types, respectively acquiring corresponding search conditions according to the search keywords of different types, comparing the search conditions corresponding to the search keywords of different types, and picking out the search conditions repeatedly appearing in the search conditions corresponding to the search keywords of all types as the search conditions corresponding to the search keywords.
The operator can be a user or a certain system, the operator can be a certain system or certain data, the screening time and the screening state are checked input, the screening time can be a time period or a specific time point, and the screening state is words of success, failure, abnormality and the like.
The search input box is provided with prompt information for prompting the operator or the operator in the log to be queried, and when the format input by the user in the search box does not accord with the preset input information, prompt input errors are performed and the user inputs again.
Specifically, when the user inputs the search keyword in the search page, the user may select to input the operator or the information about the operator in the search input box, or select the occurrence time of the log record and the state of the log record, and the three may be input simultaneously, or may input only any one of them.
And then searching respectively according to the search keywords input by the user, searching the search keywords of the search input box in the category of the search keywords of the operator and the operator, searching the search keywords of the checked log record in the category of the generation time search keywords, and searching the search keywords of the checked log record in the category of the operation state search keywords to obtain three types of search results.
When the search keywords of the search input box are searched in the category of the search keywords of the operator and the searched keywords of the operator, the accurate query of character comparison is adopted, namely the search keywords are completely the same as the search keywords, and the matching is successful.
When searching the search keywords for checking the occurrence time of the log record in the generation time search keyword category, adopting a range searching mode, if the checked occurrence time is a time period, matching all the search keywords with the generation time in the time period successfully, and if the checked occurrence time is a specific time point, adopting accurate inquiry, wherein the time points are identical and the matching is successful.
When the search keywords for checking the state of the log record are searched in the operation state search keyword category, the search keywords are completely the same as the search keywords by using the accurate search of character comparison, and the matching is successful.
The three types of search keywords may be multiple, the search keywords of the three types are all corresponding to the obtained search conditions, three groups of search conditions can be obtained, the obtained three groups of search conditions are compared in pairs, and the search conditions appearing in the three groups are selected, so that the search conditions corresponding to the search keywords are obtained.
If the input search keyword contains only one or two of the search keywords, the type of input is not selected as the search.
According to the scheme, the search keywords and the search keywords are classified and refined, search matching is carried out according to each category, so that keywords input by a user can be searched efficiently, finally, search conditions corresponding to no category are compared and screened, search conditions which simultaneously accord with the search keywords are selected, and the search accuracy of the target log record is improved.
In some embodiments, storing the target record and the search condition associated with the target record in the cache, and determining that the corresponding record is located in a specific location of the log file according to the target search condition includes: and according to the target search condition, confirming whether the target search condition is contained in the cache, if so, confirming the specific position of the target record in the cache, and if not, confirming that the corresponding record is positioned at the specific position of the log file according to the target search condition.
Wherein the cache is a temporary storage in the memory, and the specific search is performed in the memory, and the speed is faster than the reading speed of the magnetic disk.
Specifically, after the server returns and displays the target record to the user, the target record and the search condition corresponding to the target record are stored in the cache in a one-to-one correspondence.
After the user designates the record profile, when the search condition corresponding to the designated record profile is acquired, searching the search condition in the cache, if the search condition is contained in the cache, acquiring the target record corresponding to the search condition in the cache, and if the search condition is not contained in the cache, reading the log file according to the 'storage position' contained in the search condition to acquire the corresponding target record.
According to the scheme, the target records searched by the user are stored in the cache, so that when the target records are searched for two or more times, the target records can be obtained through the cache, when certain target records are searched for a high frequency, the searching time is shortened, the reading pressure of a disk is also reduced, and the retrieval efficiency of the log is further optimized.
Further, storing the target record and the search condition association corresponding to the target record in the cache, including: judging whether the number of the stored target records in the cache exceeds the preset storage number, if so, deleting the record with the least queried times in the cache and the search condition corresponding to the record, storing the target record and the search condition corresponding to the target record in the cache in an associated manner, and if not, storing the target record and the search condition corresponding to the target record in the cache in an associated manner.
The preset storage number is the number of storable records which are set in advance for the cache.
When the target record corresponding to the search condition is obtained in the cache, the target record is digitally marked, 1 is added to each time the target record is obtained, 1 is obtained when the target record is marked for the first time, and the target record which is not obtained in the cache has no digital mark.
Specifically, before each time the target records and the corresponding search conditions are stored in the cache, judging the storage record quantity of the cache, if the storage record quantity in the current cache reaches the preset storage quantity, inquiring the digital mark of each target record, selecting the target record without the digital mark for random deletion, and if the target record without the digital mark is not contained, selecting the target record with the minimum digital mark for deletion. If not, storing the target record and the corresponding retrieval condition into a cache.
According to the scheme, the normal use of the memory space is protected by setting the preset storage number of the cache, and the target records stored in the cache are all high-frequency search records in a mode of replacing the target records with fewer inquired times in the cache, so that the search efficiency of the target log records is improved to a certain extent.
In some embodiments, a cache update time is set, and when the set cache update time is reached, the queried times of each record in the cache are queried, and all records with queried times less than a preset time are deleted.
The preset times are the times of inquiring the target records set in advance, and the cache update time is set to be the time of cleaning the target records at the cache timing set in advance.
Specifically, when the set cache update time is reached, the digital marks of all the target records are queried, and if the digital marks are smaller than or equal to the preset times, the target records corresponding to the digital marks are deleted.
For example, the preset times are 2 times, the cache update time is set to be one month, and when each month is reached, the digital marks of all target records are queried, and if the digital marks are less than or equal to 2, the target records are deleted
According to the scheme, the cache update time can be set to delete the target records with low queried times in one period at regular time, so that the target records in the cache are all target records with more queried times in the last period.
Fig. 3 is a schematic structural diagram of a log searching device according to an embodiment of the present application, and as shown in fig. 3, the log searching device 300 includes:
The first obtaining module 301 is configured to obtain a search keyword.
And the matching module 302 is configured to determine, according to the search keyword, all search keywords matching the search keyword in the search library.
A retrieval module 303, configured to obtain corresponding retrieval conditions according to the retrieval keywords; for each search condition, acquiring a record brief introduction in the search condition; and generating a record profile list according to all record profiles, and calling a display module to display the profile list.
A second obtaining module 304, configured to obtain a specified record profile; the specified record profile is a record profile selected based on the profile list.
A reading module 305, configured to determine a target retrieval condition according to the specified record profile; determining that the corresponding record is positioned at a specific position of the log file according to the target retrieval condition; and acquiring a target record according to the specific position, and calling a display module to display the target record.
In some embodiments, log retrieval apparatus 300 further comprises: a monitoring module 306;
the monitoring module 306 is configured to monitor a target log file, and obtain a new record of the target log file;
Determining a retrieval condition and a retrieval keyword corresponding to the new record according to the new record;
and correspondingly storing the search conditions and the search keywords into a search library.
In some embodiments, the monitoring module 306 monitors the target log file, and when obtaining the new record of the target log file, the monitoring module is specifically configured to: acquiring the storage position of the target log file, and monitoring the size of the target log file;
when the target log file is newly added, recording the data of the size before the new addition and the data of the size after the new addition of the target log file;
reading the changed target log file in reverse order according to the newly increased size data to obtain a newly increased record;
the monitoring module 306 is specifically configured to, when determining, according to the new record, a search condition and a search keyword corresponding to the new record:
combining keywords in the newly added record to generate a record brief introduction;
generating search conditions according to the data of the size before the new increase, the data of the size after the new increase and the record brief introduction;
and extracting different types of search keywords according to the record brief introduction.
In some embodiments, the type of the search key includes: generating time, operator, operated, and operating state; the search keyword includes: inputting characters, screening time and screening state in a search box;
The matching module 302 is specifically configured to, when determining, according to the search keywords, all search keywords matching the search keywords in the search library: according to the search keywords, matching the input characters in the search box with the search keywords of the operator type and the operated type, and confirming the search keywords with the same characters; matching the screening time with the search keywords of the generation time type, and confirming the search keywords with the time coincidence; matching the screening state with the search keywords of the operation state type, and confirming the search keywords with the same characters describing the state;
the retrieving module 303 is specifically configured to, when obtaining the corresponding retrieving condition according to the retrieving keyword:
classifying the search keywords according to types;
respectively acquiring corresponding search conditions according to different types of search keywords;
and comparing the search conditions corresponding to the search keywords of different types, and picking out the search conditions which repeatedly appear in the search conditions corresponding to the search keywords of all types as the search conditions corresponding to the search keywords.
In some embodiments, log retrieval apparatus 300 further comprises: a cache module 307;
A cache module 307, configured to store the target record and a search condition corresponding to the target record in a cache in an associated manner;
the reading module 305 determines that the corresponding record is located in a specific location of the log file according to the target search condition, and is specifically configured to:
confirming whether the target retrieval condition is contained in a cache according to the target retrieval condition, and if so, confirming the specific position of the target record in the cache;
if not, determining that the corresponding record is positioned at the specific position of the log file according to the target retrieval condition.
In some embodiments, when the cache module 307 stores the target record and the search condition association corresponding to the target record in the cache, the method is specifically used for:
judging whether the number of the storage target records in the cache exceeds the preset storage number or not;
if yes, deleting the record with the least queried times in the cache and the search condition corresponding to the record, and storing the target record and the search condition corresponding to the target record in the cache in a correlated manner;
if not, storing the target record and the search condition association corresponding to the target record into a cache.
In some embodiments, log retrieval apparatus 300 further comprises: an update module 308;
an update module 308, configured to set a cache update time;
inquiring the inquired times of each record in the cache when the set cache updating time is reached;
and deleting all the records with the queried times smaller than the preset times.
The apparatus of this embodiment may be used to perform the method of any of the foregoing embodiments, and its implementation principle and technical effects are similar, and will not be described herein again.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application, as shown in fig. 4, an electronic device 400 according to the present embodiment may include: a memory 401 and a processor 402.
The memory 401 has stored thereon a computer program that can be loaded by the processor 402 and that performs the methods of the above-described embodiments.
Wherein the processor 402 is coupled to the memory 401, e.g. via a bus.
Optionally, the electronic device 400 may also include a transceiver. It should be noted that, in practical applications, the transceiver is not limited to one, and the structure of the electronic device 400 is not limited to the embodiments of the present application.
The processor 402 may be a CPU (Central Processing Unit ), general purpose processor, DSP (Digital Signal Processor, data signal processor), ASIC (Application Specific Integrated Circuit ), FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic device, transistor logic device, hardware components, or any combination thereof. Which may implement or perform the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. The processor 602 may also be a combination that performs computing functions, such as including one or more microprocessors, a combination of a DSP and a microprocessor, and the like.
A bus may include a path that communicates information between the components. The bus may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture ) bus, or the like. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
Memory 401 may be, but is not limited to, a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory ) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory ), a CD-ROM (Compact Disc Read Only Memory, compact disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
The memory 401 is used for storing application program codes for executing the present application and is controlled to be executed by the processor 402. The processor 402 is configured to execute the application code stored in the memory 401 to implement what is shown in the foregoing method embodiment.
Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. But may also be a server or the like. The electronic device shown in fig. 4 is only an example and should not be construed as limiting the functionality and scope of use of the embodiments herein.
The electronic device of the present embodiment may be used to execute the method of any of the foregoing embodiments, and its implementation principle and technical effects are similar, and will not be described herein.
The present application also provides a computer-readable storage medium storing a computer program capable of being loaded by a processor and executing the method in the above embodiments.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the method embodiments described above may be performed by hardware associated with program instructions. The foregoing program may be stored in a computer readable storage medium. The program, when executed, performs steps including the method embodiments described above; and the aforementioned storage medium includes: various media that can store program code, such as ROM, RAM, magnetic or optical disks.

Claims (10)

1. A log retrieval method, comprising:
acquiring a search keyword;
determining all search keywords matched with the search keywords in a search library according to the search keywords;
acquiring corresponding search conditions according to the search keywords;
for each search condition, acquiring a record brief introduction in the search condition;
generating a record profile list according to all record profiles, and displaying the profile list;
acquiring a designated record brief introduction; the specified record profile is a record profile selected according to the profile list;
determining target retrieval conditions according to the designated record profile;
determining that the corresponding record is positioned at a specific position of the log file according to the target retrieval condition;
and acquiring a target record according to the specific position, and displaying the target record.
2. The method as recited in claim 1, further comprising:
monitoring a target log file and acquiring a new record of the target log file;
determining a retrieval condition and a retrieval keyword corresponding to the new record according to the new record;
and correspondingly storing the search conditions and the search keywords into a search library.
3. The method of claim 2, wherein the monitoring the target log file, obtaining a new record of the target log file, comprises:
acquiring the storage position of the target log file, and monitoring the size of the target log file;
when the target log file is newly added, recording the data of the size before the new addition and the data of the size after the new addition of the target log file;
reading the changed target log file in reverse order according to the newly increased size data to obtain a newly increased record;
the determining, according to the new record, the search condition and the search keyword corresponding to the new record includes:
combining keywords in the newly added record to generate a record brief introduction;
generating search conditions according to the data of the size before the new increase, the data of the size after the new increase and the record brief introduction;
and extracting different types of search keywords according to the record brief introduction.
4. The method of claim 1, wherein the type of search key comprises: generating time, operator, operated, and operating state; the search keyword includes: inputting characters, screening time and screening state in a search box;
And determining all the search keywords matched with the search keywords in the search library according to the search keywords, wherein the method comprises the following steps:
according to the search keywords, matching the input characters in the search box with the search keywords of the operator type and the operated type, and confirming the search keywords with the same characters; matching the screening time with the search keywords of the generation time type, and confirming the search keywords with the time coincidence; matching the screening state with the search keywords of the operation state type, and confirming the search keywords with the same characters describing the state;
the obtaining the corresponding search condition according to the search keyword comprises the following steps:
classifying the search keywords according to types;
respectively acquiring corresponding search conditions according to different types of search keywords;
and comparing the search conditions corresponding to the search keywords of different types, and picking out the search conditions which repeatedly appear in the search conditions corresponding to the search keywords of all types as the search conditions corresponding to the search keywords.
5. The method of any one of claims 1-4, further comprising:
Storing the target record and the retrieval condition corresponding to the target record in a cache in an associated manner;
the determining that the corresponding record is located in the specific position of the log file according to the target retrieval condition comprises the following steps:
confirming whether the target retrieval condition is contained in a cache according to the target retrieval condition, and if so, confirming the specific position of the target record in the cache;
if not, determining that the corresponding record is positioned at the specific position of the log file according to the target retrieval condition.
6. The method of claim 5, wherein storing the target record, the search condition association corresponding to the target record, in a cache, comprises:
judging whether the number of the storage target records in the cache exceeds the preset storage number or not;
if yes, deleting the record with the least queried times in the cache and the search condition corresponding to the record, and storing the target record and the search condition corresponding to the target record in the cache in a correlated manner;
if not, storing the target record and the search condition association corresponding to the target record into a cache.
7. The method of claim 6, wherein the method further comprises:
Setting a cache updating time;
inquiring the inquired times of each record in the cache when the set cache updating time is reached;
and deleting all the records with the queried times smaller than the preset times.
8. A log retrieval apparatus, comprising:
the first acquisition module is used for acquiring the search keywords;
the matching module is used for determining all search keywords matched with the search keywords in the search library according to the search keywords;
the retrieval module is used for acquiring corresponding retrieval conditions according to the retrieval keywords; for each search condition, acquiring a record brief introduction in the search condition; generating a record profile list according to all record profiles, and calling a display module to display the profile list;
the second acquisition module is used for acquiring the appointed record brief introduction; the specified record profile is a record profile selected according to the profile list;
the reading module is used for determining target retrieval conditions according to the designated record brief introduction; determining that the corresponding record is positioned at a specific position of the log file according to the target retrieval condition; and acquiring a target record according to the specific position, and calling a display module to display the target record.
9. An electronic device, comprising: a memory and a processor;
the memory is used for storing program instructions;
the processor is configured to call and execute program instructions in the memory to perform the log retrieval method according to any one of claims 1-7.
10. A computer-readable storage medium, wherein the computer-readable storage medium has a computer program stored therein; the computer program, when executed by a processor, implements the log retrieval method according to any one of claims 1-7.
CN202310098348.8A 2023-02-10 2023-02-10 Log retrieval method and device, electronic equipment and storage medium Pending CN116126795A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310098348.8A CN116126795A (en) 2023-02-10 2023-02-10 Log retrieval method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310098348.8A CN116126795A (en) 2023-02-10 2023-02-10 Log retrieval method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116126795A true CN116126795A (en) 2023-05-16

Family

ID=86299031

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310098348.8A Pending CN116126795A (en) 2023-02-10 2023-02-10 Log retrieval method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116126795A (en)

Similar Documents

Publication Publication Date Title
US8145617B1 (en) Generation of document snippets based on queries and search results
US10372718B2 (en) Systems and methods for enterprise data search and analysis
US11321336B2 (en) Systems and methods for enterprise data search and analysis
CN106407360B (en) Data processing method and device
CN111258966A (en) Data deduplication method, device, equipment and storage medium
CN112487150B (en) File management method, system, storage medium and electronic equipment
US8010528B2 (en) Problem isolation through weighted search of knowledge bases
CN107870915B (en) Indication of search results
CN111324689A (en) Index updating method, device, equipment and storage medium of question-answering system
US20150206101A1 (en) System for determining infringement of copyright based on the text reference point and method thereof
CN111400323A (en) Data retrieval method, system, device and storage medium
CN111258819A (en) Data acquisition method, device and system for MySQL database backup file
US20120254166A1 (en) Signature Detection in E-Mails
CN113722296A (en) Agricultural information processing method and device, electronic equipment and storage medium
CN110287338B (en) Industry hotspot determination method, device, equipment and medium
CN115080684B (en) Network disk document indexing method and device, network disk and storage medium
CN107908724B (en) Data model matching method, device, equipment and storage medium
CN116126795A (en) Log retrieval method and device, electronic equipment and storage medium
CN113742291A (en) File saving method and device and computer storage medium
US20160203573A1 (en) System and Method for Retrieving and Displaying a Patent Family
US10810236B1 (en) Indexing data in information retrieval systems
CN112631905A (en) Execution process data management method and device, computer equipment and storage medium
CN113779193B (en) Text quotation method and device and electronic equipment
CN114064638B (en) Information processing method, device, equipment and storage medium
CN114254081B (en) Enterprise big data search system, method and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination