CN111639016A - Big data log analysis method and device and computer storage medium - Google Patents
Big data log analysis method and device and computer storage medium Download PDFInfo
- Publication number
- CN111639016A CN111639016A CN202010478170.6A CN202010478170A CN111639016A CN 111639016 A CN111639016 A CN 111639016A CN 202010478170 A CN202010478170 A CN 202010478170A CN 111639016 A CN111639016 A CN 111639016A
- Authority
- CN
- China
- Prior art keywords
- database
- log
- logs
- search request
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
- G06F11/3433—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment for load management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Computer Hardware Design (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application provides a big data log analysis method and device, electronic equipment and a computer storage medium. The big data log analysis method comprises the following steps: acquiring a search request; searching database logs which are matched with the search request from a plurality of database logs with uniform preset formats; and analyzing the database logs matched with the search request to obtain an analysis result. According to the method and the device, the database logs matched with the search request are searched from a large number of database logs with a uniform preset format, the searched database logs are closely related to the search request, so that the database logs are analyzed, when the search request of a user changes, the database logs only need to be searched again according to the changed search request, then the database logs are analyzed, other data processing steps are not needed, the time is saved, the operation is simple and convenient, the database logs are easy to master by the user, and the user experience is improved.
Description
Technical Field
The embodiment of the application relates to the field of big data log analysis, in particular to a big data log analysis method and device, electronic equipment and a computer storage medium.
Background
With the advent of the big data era, network data is growing rapidly, and a large amount of data logs generated by equipment such as a server in the operation process or the business processing process need to be collected, sorted, audited, analyzed and the like. The big data log analysis is used for mining the value of data, and many enterprises change the traditional operation and maintenance into the big data analysis operation and maintenance and apply a big data analysis technology in the aspects of safety and business.
The whole process of big data log analysis generally comprises data acquisition, data storage, data search and data analysis, wherein four open source software of Hadoop and ELK are used for carrying out integration development on database log analysis, specifically, Logstash is used for collecting, analyzing and converting data, then a distributed system Hadoop component is used for storing the collected data, a full-text search engine Elasticisearch is used for carrying out data search, and finally an open source analysis platform Kibana is used for carrying out visual display on analysis results.
However, with the rapid development of the big data era, the number of devices such as servers and the like is rapidly increasing, the number of generated data logs is increasing, and a large number of data logs are generated by business systems, application supports and hardware devices, so that when the business systems, the application supports and the hardware devices have a plurality of network devices with different models, a large number of servers with different models, various virtualization schemes, different operating systems, diversified application software and databases, a large number of data logs are mutually progressive and stacked. When the prior art is adopted to analyze the big data log and present the result to the user, all data needs to be processed each time, and the steps of data acquisition, storage, search, analysis and the like consume a large amount of time, are complex to operate and have low search response speed to the user.
Disclosure of Invention
In view of this, one of the technical problems solved by the embodiments of the present invention is to provide a method and an apparatus for analyzing a big data log, an electronic device, and a computer storage medium, so as to overcome the defects that when a big data log is analyzed in the prior art, data is acquired, stored, searched, and analyzed each time, so that analysis of the big data log is time-consuming and complex to operate.
In a first aspect, an embodiment of the present application provides a big data log analysis method, where the method includes:
acquiring a search request;
searching database logs matched with the search request from a plurality of database logs with a uniform preset format, wherein the database logs with the uniform preset format are obtained by preprocessing the collected original database logs;
and analyzing the database log matched with the search request to obtain an analysis result.
Optionally, in an embodiment of the present application, searching a database log matching a search request from a plurality of database logs having a uniform predetermined format includes:
determining a database log to be selected from a plurality of database logs with uniform preset formats according to the source type and/or the time period in the search request;
and determining a target database log in the database logs to be selected according to the condition characteristics in the search request, wherein the target database log is matched with the condition characteristics.
Optionally, in an embodiment of the present application, the method further includes:
collecting log information of logs of an original database;
and preprocessing the original database log according to the log information to obtain the database log with the uniform preset format, wherein the database log with the uniform preset format is used for matching with the search request.
Optionally, in an embodiment of the present application, the preprocessing the original database log according to the log information includes:
cleaning the original database log according to the log information to obtain an effective database log;
and analyzing and converting the effective database logs to obtain the database logs with a uniform preset format.
Optionally, in an embodiment of the present application, the method further includes:
and displaying the analysis result in a preset form, wherein the preset form corresponds to the search request.
Optionally, in an embodiment of the present application, the preset form includes at least one of: graphs, tables, data, information.
Optionally, in an embodiment of the present application, the log information includes at least one of: device information, system information, process information, disk information, state information, load information, service information, application information, fault information.
In a second aspect, an embodiment of the present application provides a big data log analysis apparatus, including an obtaining module and a data searching module;
the acquisition module is used for acquiring a search request;
the data searching module is used for searching database logs matched with the searching request from a plurality of database logs with a uniform preset format, wherein the database logs with the uniform preset format are obtained by preprocessing the collected original database logs;
and the analysis module is used for analyzing the database logs matched with the search request to obtain an analysis result.
In a third aspect, an embodiment of the present application provides an electronic device, including: at least one processor; storage means for storing at least one program; the at least one program, when executed by the at least one processor, causes the at least one processor to implement a method as described in the first aspect or any one of the embodiments of the first aspect.
In a fourth aspect, embodiments of the present application provide a computer storage medium storing a computer program, which when executed by a processor implements the method as described in the first aspect or any one of the embodiments of the first aspect.
The embodiment of the application provides a big data log analysis method and device, electronic equipment and a computer storage medium. The big data log analysis method comprises the following steps: acquiring a search request; searching database logs matched with the search request from a plurality of database logs with a uniform preset format, wherein the database logs with the uniform preset format are obtained by preprocessing the collected original database logs; and analyzing the database logs matched with the search request to obtain an analysis result. According to the method and the device, the database logs matched with the search request are searched from a large number of database logs with a uniform preset format, the searched database logs are closely related to the search request, so that the database logs are analyzed to obtain an analysis result, and when the search request of a user changes, the database logs still have the uniform preset format, so that the database logs only need to be searched again according to the changed search request and then are analyzed, other data processing steps are not needed, the time is saved, the operation is simple and convenient, the database logs are easy to master by the user, and the user experience is improved.
Drawings
Some specific embodiments of the present application will be described in detail below by way of illustration and not limitation with reference to the accompanying drawings. The same reference numbers in the drawings identify the same or similar elements or components. Those skilled in the art will appreciate that the drawings are not necessarily drawn to scale. In the drawings:
fig. 1 is a flowchart of a big data log analysis method according to an embodiment of the present disclosure;
fig. 2 is an application scenario diagram of a big data log analysis method according to an embodiment of the present application;
FIG. 3 is a schematic diagram illustrating a database log load trend according to an embodiment of the present disclosure;
fig. 4 is a schematic diagram of a big data log analysis apparatus according to an embodiment of the present application;
fig. 5 is a structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following further describes specific implementation of the embodiments of the present invention with reference to the drawings.
The first embodiment,
Fig. 1 is a flowchart of a big data log analysis method provided in an embodiment of the present application, and as shown in fig. 1, the big data log analysis method includes steps 101 to 103. First, an application scenario of the embodiment of the present application is explained, fig. 2 is an application scenario diagram of a big data log analysis method provided by the embodiment of the present application, fig. 2 shows an application scenario of the big data log analysis method, in which an original database log is collected first in the application scenario, the original database log is preprocessed to obtain a database log with a uniform predetermined format, taking a search request of × year × month × day 8:00-9:00 and load information of a × type database log as an example, a search is performed in the database log with the uniform predetermined format to obtain a database log corresponding to the search request, then the database log is analyzed, when the search request changes, a search is performed again in the database log with the uniform predetermined format according to the changed search request, so that a database log corresponding to the changed search request can be obtained, and analyzing the changed database log.
Step 101, obtaining a search request.
It should be noted that the types of big data logs are many, and may include a system log, an application log, a security log, a database log, and so on. Each log records the relevant description of a timestamp, a relevant equipment name, a user, an operation behavior and the like, system operation and development personnel can know software and hardware information of the server and check errors in the configuration process and the reasons of the errors through the logs, and can know the load and performance safety of the server by frequently analyzing the logs, analyze relevant problems in time, trace the root causes of the errors and correct the errors. The present application describes an example of searching and analyzing a database log in a big data log.
The search request is a requirement provided by a user according to an actual situation, for example, the search request can be a load trend of a plurality of databases in a certain time period, the search request can be a detailed comparison of load data of the database A and the database B in the certain time period, the search request can be a load occupied by the database A in different processing services in the certain time period, fault information can be generated when the database fails, the work of the database cannot be directly influenced by some fault information, the search request can be information of monitoring the database in real time, and early warning is performed on the database when the database fails; it will be appreciated that the search request may vary from one requirement to another.
Step 102, searching database logs matching the search request in a plurality of database logs with uniform preset formats.
The database logs with the uniform preset format are obtained by preprocessing the collected original database logs.
It should be noted that, in the present application, the database log matched with the search request refers to the database log that satisfies the search condition. The database log is used for representing logs generated in the running process of the database, the logs are one of the most important data of the database, and the database is provided with the logs for recording all transactions and messages and modifications made to the database by each transaction and message. For enterprises, various devices such as switches, calculators, routers and the like can generate a large amount of logs in the production and operation processes, the running condition and the safety degree of the devices can be obtained through analyzing the logs, and some faults existing in enterprise networks and the devices can be searched through analyzing the logs; a large amount of logs can be generated in the process of doing business, and enterprises can find problems in the business by analyzing the logs; the operation and maintenance management department of an enterprise needs to store some necessary data for auditing while ensuring the normal operation of the machine; therefore, the value of the data can be mined by analyzing the database logs, and database management personnel and operation and maintenance personnel can master the use condition of the database according to the analysis result, such as the load condition of the database in a certain time period, the fault condition of the database when a fault occurs, and the like.
In the prior art, when analyzing database logs, the analysis is generally performed according to the steps of data collection, data storage, data search and data analysis, however, when the number of database logs is very large and the user's demand changes, when analyzing the database logs by using the four open source software, all data is processed and then analyzed each time, for example, when locating load information in a certain time period according to the user's demand, the collected database logs are analyzed and converted according to the four open source software, and then the database logs are searched, so as to analyze the load information of the database logs; when the user requirement changes, the database logs can be analyzed only by executing the steps again according to the four open source software, the operation is complex, a large amount of time is consumed, a large burden is brought to the database, and the efficiency of data analysis and processing is seriously influenced.
In the embodiment of the application, the search request is obtained, and the desired database log can be searched from the database logs with the uniform preset format according to the search request, so that the searched database log is analyzed, a series of steps of collecting, storing, searching, analyzing and the like are not required to be performed on the database log, the time is saved, the workload is small, the database log searching method is simple and easy to operate and grasp by a user, the efficiency of searching the database log is improved, and the user experience is improved.
Optionally, in an embodiment of the present application, step 102 may include determining a candidate database log among a plurality of database logs having a uniform predetermined format according to a source type and/or a time period in the search request; and determining a target database log in the database logs to be selected according to the condition characteristics in the search request, wherein the target database log is matched with the condition characteristics.
When determining the target database log, the embodiment of the application firstly determines the database log to be selected in the database log according to the source type and/or the time period, namely, preliminarily narrows the range of the database log, and then continues to determine the target database log in the database log to be selected according to the condition characteristics in the search request; in an implementation manner, if the condition characteristic is fault information of the database log, and the database a has no fault in the operation process, that is, the log information of the database a does not include the fault information, the database a is filtered, that is, the database log irrelevant to the condition characteristic is filtered, so as to obtain a target database log; in another implementation manner, if the condition characteristic is load information of the database log, that is, the load condition that the user only needs a certain period of the database log, and the log information in the database log includes not only the load information but also other operation information, the log information irrelevant to the load information is filtered, only the log information relevant to the load information is retained, and the database log including only the load information is used as the target database log.
Here, the Search analysis of the database log is described by taking OSL (oraca Search Language) as an example, where the OSL Language is a Structured Query Language (SQL) Language and can be used for analysis, association, and analysis of the big data log. In the embodiment of the present application, the collected database log is stored in a local database oracle, and it should be understood that the collected database log may also be stored in other databases.
index ═ weichai _ db sourcetype ═ oracle: osPerf; the index function index in the embodiment of the application is used for specifying the index and the source type, and the range of the called database log is narrowed, so that the searching efficiency is improved.
LOAD; the characteristic conditions in the embodiment of the application are that load information of database logs is inquired, keywords are used for specifying log information needing to be searched, the keywords in the embodiment of the application are load loads, and load information of each database log in an Oracle database is inquired.
And 103, analyzing the database log matched with the search request to obtain an analysis result.
After the database logs matched with the search request are searched out according to the search request, the database logs obtained through searching are analyzed. Analyzing the database logs matched with the search request to obtain an analysis result; here, the OSL language is taken as an example to analyze the database log to obtain an analysis result.
timehart VALUEs (VALUE) as VALUE; according to the embodiment of the application, the load condition of the database is analyzed through time aggregation analysis, the load condition of each database log in a certain time period is analyzed, and the time aggregation is used for analyzing the database logs by taking time as an abscissa and taking load as an ordinate to obtain an analysis result.
The big data log analysis method in the embodiment of the application is simple and easy to operate, can realize quick positioning, and when the user requirement changes, the target database log corresponding to the user requirement can be found only by reducing the search range of the database log through the index function index, modifying the keyword, and setting the keyword to be associated with the user requirement, and then analyzing the database log. Compared with the prior art that the steps of collecting, storing, searching and analyzing the database logs need to be executed again according to four open source software, the big data log analyzing method is small in workload, simple and convenient to operate, easy to master and capable of improving user experience, and data analyzing and processing efficiency is improved.
Example II,
Before searching out the matched database log according to the search request, the big data log analysis method further comprises preprocessing the database log, and optionally comprises the steps 100a and 100 b.
Step 100a, collecting log information of an original database log.
In the embodiment of the present application, log information of an original database log is collected from a database device, and optionally, the original database log may be collected into a local storage by using a script, a collector, a Transmission Control Protocol (TCP), a User Datagram Protocol (UDP) port, or the like, where the log information is used to reflect related information generated in an operation process of a database.
Optionally, in an embodiment of the present application, the log information includes at least one of: device information, system information, process information, disk information, state information, load information, service information, application information, fault information.
It should be noted here that the fault information may be, but is not limited to, used to represent information generated when a device in a database log fails, and the database with the failure may be found according to the fault information, or the database log may be detected in real time according to information characteristics reflected when the fault information is generated, and when a corresponding information characteristic exists in the database log, an early warning is given according to the information characteristic; the disk information may be, but is not limited to, the disk type and disk space occupied by the database log.
And step 100b, preprocessing the original database log according to the log information to obtain the database log with a uniform preset format.
Wherein database logs having a uniform predetermined format are used for matching with search requests.
The collected original database logs are very various, the number of log information contained in the original database logs is large, the confidence is complex, a lot of space is occupied, the operation speed is influenced, and some database logs are invalid, so that the original database logs need to be preprocessed, and usable and effective database logs are obtained. When the original database logs are preprocessed, the data filtering and screening are carried out on the original database logs according to the log information, and the method and the device have pertinence and comprehensiveness, so that the number of the preprocessed database logs is reduced, and the effectiveness of the database logs is guaranteed.
Optionally, in an embodiment of the present application, the step 100b may include: cleaning the original database log according to the log information to obtain an effective database log; and analyzing and converting the effective database logs to obtain the database logs with a uniform preset format.
The database log is also a collection of data including but not limited to valid data, invalid data and chaotic data, the washing of the database log means that the data is washed, and the washing of the data can obtain a usable and valid database log, wherein the invalid data may not be in a given range or have no meaning for actual services, the chaotic data may indicate that the data format is illegal, and there is irregular coding and ambiguous service logic. When the data is cleaned, related technologies such as mathematical statistics, data mining or predefined cleaning rules can be used to filter invalid data in the data, for example, to delete duplicate data information, and to convert chaotic data in the data into data meeting data quality requirements, where the chaotic data may include, but is not limited to, incomplete data and erroneous data, for example, to correct errors existing in the chaotic data, to complement the incomplete data completely, and to ensure validity and consistency of the data. In the embodiment of the present application, any one or a combination of several ways of methods such as deleting, sorting, converting, and supplementing may be used when the database log is cleaned, which is not limited in the embodiment of the present application, and any method for cleaning the database log is within the protection scope of the present application.
The method and the device have the advantages that the effective database logs are obtained after the original database logs are cleaned, the effective database logs are analyzed, the data in the effective database logs are analyzed into the data which can be analyzed and identified, and the database logs are searched and analyzed. The method and the device also perform format conversion on the effective database logs, wherein the format conversion is a process of integrating the database logs, so that the original formats of the effective database logs are all converted into the database logs with the uniform preset format, the format conversion on the database logs is to convert the database logs with the original formats inconsistent with the preset formats, and the database logs with the original formats consistent with the preset formats are reserved, so that the uniform formats of the database logs are ensured, and the searching and analyzing of the database logs are facilitated.
Example III,
In an embodiment, after analyzing the database log matching the search request and obtaining the analysis result, the present application further displays the analysis result, that is, after step 103, the method may further include displaying the analysis result in a preset form, where the preset form corresponds to the search request.
In the embodiment of the application, the analysis result of the database log matching the search request is shown in a preset form, and the preset form corresponds to the search request, for example, if the search request is a load trend of a plurality of databases within a certain time period, the preset form takes time as an abscissa and load as an ordinate as a coordinate system, and the preset form may be a bar graph, a line graph, a scatter diagram, a trend diagram, or the like, and is for showing the corresponding database load to a user; if the search request is a detailed comparison of the load data of the database A and the load data of the database B in a certain time period, the preset form can be shown in a table form, and the load data of the two databases are displayed in a list form, so that a user can compare the difference of the loads of the two databases from the table; if the search request is the load of the database a in different processing services within a certain time period, the preset form may be shown in the form of a pie chart; if the search request is to search specific information of the load of the database A in a certain time period, the preset form can be directly shown in a data form; if the search request is information for monitoring the database in real time, and the database is early warned when the database fails, the preset form can be directly shown in the form of information, so that a user can locate the time and the position of the database failure according to the information; it can be understood that when the search request changes, the analysis result obtained by the search can be displayed in different forms, and the analysis result in the application is displayed in a preset form corresponding to the search request, so that a user can conveniently and intuitively know the analysis result comprehensively.
Here, the analysis result of the database log will be described by using OSL as an example.
span is 1m by host; setting the analysis result to be output by taking the database as a unit, and setting the time interval to be 1 minute; it is to be understood that the time interval may be set according to practical situations, and the embodiment of the present application is only an exemplary illustration and does not represent that the embodiment of the present application is limited thereto.
As shown in fig. 3, fig. 3 is a schematic diagram of a load trend of a database log provided in an embodiment of the present application, where an abscissa is time, an ordinate is load, 3 databases are shown, a load condition of a database Bi, a database ERP, and a database FSSC in a time period from 14:50 to 15:50 is shown, a time interval is 1min, it should be noted here that the database Bi, the database ERP, and the database FSSC all have 60 points in the time period from 14:50 to 15:50, each point corresponds to a database load at a certain time point, the 60 points are too dense, and if 60 time points are marked on an abscissa to affect a display effect, the coordinate on the abscissa is marked at an interval of 5 min. In the database load analysis result of fig. 3, the load information of the database is displayed once in 1min, for example, a database manager or a user can know that the load of the database Bi is too large at 15:05 according to the analysis result of fig. 3, preliminarily judge that the database Bi may have a problem at the time point of 15:05, and can specifically look up the operation condition of the database Bi at 15:05, thereby further judging whether the database Bi is in a state, for example, the writing speed is slow, the database Bi is not displayed, and the database is maintained in time.
Optionally, in an embodiment of the present application, the preset form includes at least one of: graphs, tables, data, information. The analysis result of the database in the embodiment of the application can have various display forms; for example, when the user needs to compare the load conditions of a plurality of databases, the comparison and analysis results can be displayed in a data table form, so that the user can intuitively and comprehensively know the load conditions of the databases; when a user needs to search a time point when a certain database fails, the analysis result of the database can be displayed in a data form, so that the user can directly obtain a desired result according to the data; therefore, the embodiments of the present application are not limited.
The graphic in this application may be, but is not limited to, a pie chart, a bar chart, a line chart, a scatter chart, a map, a thermodynamic diagram, a trend diagram, and the like. The analysis result is displayed on a visual interface in a graph, so that a user can intuitively and comprehensively know the analysis result.
The database logs are common mass data, for example, a large number of user group platforms such as Jingdong and Taobao are provided, the number of the logs per hour can reach billions, when the number of the database logs is very large and the requirements of users change, all data are processed and analyzed each time when four open source software is used for analyzing the database logs, the time is consumed, and the positioning speed of the database logs is reduced; and with different user requirements, the presented analysis result cannot be adaptively changed along with the user requirements, so that the user experience is reduced. The method and the device for analyzing the database logs get rid of the use of a main stream open source tool and an open source frame in a general scheme, custom development is carried out on the basis of independent research and development in combination with user requirements, a search request is obtained, a desired database log can be searched from a plurality of database logs with a unified preset format according to the search request, the database logs obtained through searching are analyzed, an analysis result is obtained, the analysis object is obtained through screening in a large number of database logs according to the search request, the presentation form of the analysis result corresponds to the search request, and the analysis result is comprehensive and visual and is closely related to the search request. When the search request of the user changes, the database logs only need to be re-screened according to the changed search request, then the screened database logs are analyzed, other data processing steps are not needed, the time is saved, the operation is simple and convenient, the database logs are easy to master by the user, the user experience is improved, and the speed of analyzing and analyzing the big data logs is improved.
Example four,
The embodiment of the present application provides a schematic diagram of a big data log analysis apparatus, as shown in fig. 4, fig. 4 is a big data log analysis apparatus 40 provided in the embodiment of the present application, where the apparatus 40 includes an obtaining module 401, a data searching module 402, and an analysis module 403;
an obtaining module 401, configured to obtain a search request;
a data searching module 402, configured to search a database log matching the search request from a plurality of database logs in a unified predetermined format, where the database log in the unified predetermined format is obtained by preprocessing the acquired original database log;
the analysis module 403 is configured to analyze the database log matching the search request to obtain an analysis result.
The data search module 402 may be composed of an embedded data search analyzer, and the user may input an OSL language to perform a custom search.
Optionally, in an embodiment of the present application, the data search module 402 is further configured to determine a database log to be selected from a plurality of database logs having a uniform predetermined format according to a source type and/or a time period in the search request; and determining a target database log in the database logs to be selected according to the condition characteristics in the search request, wherein the target database log is matched with the condition characteristics.
Optionally, in an embodiment of the present application, the apparatus 40 further includes a data acquisition and storage module, where the data acquisition and storage module is configured to acquire log information of the original database log, and store the database log with a uniform predetermined format obtained by preprocessing the original database log according to the log information.
Optionally, in an embodiment of the present application, the data acquisition and storage module is further configured to clean the original database log according to the log information to obtain an effective database log; and analyzing and converting the effective database logs to obtain the database logs with a uniform preset format.
The required database logs can be collected from the database equipment by a data collection tool, such as a script, a collector and the like, the collected database logs are stored locally, and then the database logs are cleaned, sorted and processed to change complex data into usable effective data.
Optionally, in an embodiment of the present application, the log information includes at least one of: device information, system information, process information, disk information, state information, load information, service information, application information, fault information.
Optionally, in an embodiment of the present application, the apparatus 40 further includes a presentation module, configured to present the analysis result in a preset form, where the preset form corresponds to the search request.
Optionally, in an embodiment of the present application, the preset form includes at least one of: graphs, tables, data, information.
After the searched database logs are analyzed to obtain the analysis results, the display module visually displays the analysis results according to the preset form corresponding to the search request, and a user can be helped to know the analysis results more intuitively.
Example V,
Based on the big data log analysis method described in the first to third embodiments, an embodiment of the present application provides an electronic device, as shown in fig. 5, fig. 5 is a structural diagram of an electronic device provided in an embodiment of the present application, where the electronic device 50 includes: at least one processor 501; the memory 502, the memory storing at least one program 512, the processor 501 and the memory 502 being electrically connected, when the at least one program is executed by the at least one processor 501, to cause the at least one processor 501 to implement the method as described in embodiments one to three.
Optionally, in an embodiment of the present application, the electronic device 50 further includes: a bus 503 and a communication interface 504, and the at least one processor 501, the memory 502, and the communication interface 504 communicate with each other through the bus 503.
Example six,
Based on the big data log analysis methods described in the first to third embodiments, the present application provides a computer storage medium storing a computer program, and the computer program, when executed by a processor, implements the methods described in the first to third embodiments.
The image similarity calculation device of the embodiment of the present application exists in various forms, including but not limited to:
(1) a mobile communication device: such devices are characterized by mobile communications capabilities and are primarily targeted at providing voice, data communications. Such terminals include: smart phones (e.g., iphones), multimedia phones, functional phones, and low-end phones, among others.
(2) Ultra mobile personal computer device: the equipment belongs to the category of personal computers, has calculation and processing functions and generally has the characteristic of mobile internet access. Such terminals include: PDA, MID, and UMPC devices, etc., such as ipads.
(3) A portable entertainment device: such devices can display and play multimedia content. This type of device comprises: audio, video players (e.g., ipods), handheld game consoles, electronic books, and smart toys and portable car navigation devices.
(4) And other electronic equipment with data interaction function.
Thus, particular embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may be advantageous.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable computer storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus, and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media for a computer include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable computer storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular transactions or implement particular abstract data types. The application may also be practiced in distributed computing environments where transactions are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A big data log analysis method, the method comprising:
acquiring a search request;
searching database logs matched with the search request from a plurality of database logs with a uniform preset format, wherein the database logs with the uniform preset format are obtained by preprocessing collected original database logs;
and analyzing the database log matched with the search request to obtain an analysis result.
2. The method of claim 1, wherein searching for a database log matching the search request among a plurality of database logs having a uniform predetermined format comprises:
determining a database log to be selected in the database logs with the uniform preset format according to the source type and/or the time period in the search request;
and determining a target database log in the database logs to be selected according to the condition characteristics in the search request, wherein the target database log is matched with the condition characteristics.
3. The method of claim 1, further comprising:
collecting log information of the original database log;
and preprocessing the original database log according to the log information to obtain a database log with a uniform preset format, wherein the database log with the uniform preset format is used for matching with the search request.
4. The method of claim 3, wherein pre-processing the raw database log according to the log information comprises:
cleaning the original database log according to the log information to obtain an effective database log;
and analyzing and converting the effective database log to obtain the database log with the uniform preset format.
5. The method of claim 3 or 4, wherein the log information comprises at least one of: device information, system information, process information, disk information, state information, load information, service information, application information, fault information.
6. The method of claim 1, further comprising:
and displaying the analysis result according to a preset form, wherein the preset form corresponds to the search request.
7. The method of claim 1, wherein the preset pattern comprises at least one of: graphs, tables, data, information.
8. The device for searching the database logs is characterized by comprising an acquisition module, a data searching module and an analysis module;
the acquisition module is used for acquiring a search request;
the data searching module is used for searching database logs matched with the searching request from a plurality of database logs with a uniform preset format, wherein the database logs with the uniform preset format are obtained by preprocessing collected original database logs;
and the analysis module is used for analyzing the database logs matched with the search request to obtain an analysis result.
9. An electronic device, comprising: a processor and a memory, the memory having a computer program stored thereon; the processor is configured to execute the computer program stored in the memory to implement the method of any one of claims 1-7.
10. A computer storage medium, characterized in that it stores a computer program which, when executed by a processor, implements the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010478170.6A CN111639016A (en) | 2020-05-29 | 2020-05-29 | Big data log analysis method and device and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010478170.6A CN111639016A (en) | 2020-05-29 | 2020-05-29 | Big data log analysis method and device and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111639016A true CN111639016A (en) | 2020-09-08 |
Family
ID=72331658
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010478170.6A Pending CN111639016A (en) | 2020-05-29 | 2020-05-29 | Big data log analysis method and device and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111639016A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112765114A (en) * | 2021-01-21 | 2021-05-07 | 神州数码融信云技术服务有限公司 | Financial system log file processing method and system |
| CN113839952A (en) * | 2021-09-27 | 2021-12-24 | 深信服科技股份有限公司 | Threat tracking method and device for log access relationship and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105302827A (en) * | 2014-06-30 | 2016-02-03 | 华为技术有限公司 | Event search method and device |
| CN106209405A (en) * | 2015-05-06 | 2016-12-07 | 中国移动通信集团内蒙古有限公司 | Method for diagnosing faults and device |
| CN106503079A (en) * | 2016-10-10 | 2017-03-15 | 语联网（武汉）信息技术有限公司 | A kind of blog management method and system |
| CN107506422A (en) * | 2017-08-17 | 2017-12-22 | 北京理工大学 | The distributed information log processing system and method for a kind of multi-data source |
| CN109582551A (en) * | 2018-10-11 | 2019-04-05 | 平安科技（深圳）有限公司 | Daily record data analytic method, device, computer equipment and storage medium |
| CN109800223A (en) * | 2018-12-12 | 2019-05-24 | 平安科技（深圳）有限公司 | Log processing method, device, electronic equipment and storage medium |
| CN111026727A (en) * | 2019-10-24 | 2020-04-17 | 贝壳技术有限公司 | Table dimension retrieval data synchronization method, system and device based on log file |
-
2020
- 2020-05-29 CN CN202010478170.6A patent/CN111639016A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105302827A (en) * | 2014-06-30 | 2016-02-03 | 华为技术有限公司 | Event search method and device |
| CN106209405A (en) * | 2015-05-06 | 2016-12-07 | 中国移动通信集团内蒙古有限公司 | Method for diagnosing faults and device |
| CN106503079A (en) * | 2016-10-10 | 2017-03-15 | 语联网（武汉）信息技术有限公司 | A kind of blog management method and system |
| CN107506422A (en) * | 2017-08-17 | 2017-12-22 | 北京理工大学 | The distributed information log processing system and method for a kind of multi-data source |
| CN109582551A (en) * | 2018-10-11 | 2019-04-05 | 平安科技（深圳）有限公司 | Daily record data analytic method, device, computer equipment and storage medium |
| CN109800223A (en) * | 2018-12-12 | 2019-05-24 | 平安科技（深圳）有限公司 | Log processing method, device, electronic equipment and storage medium |
| CN111026727A (en) * | 2019-10-24 | 2020-04-17 | 贝壳技术有限公司 | Table dimension retrieval data synchronization method, system and device based on log file |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112765114A (en) * | 2021-01-21 | 2021-05-07 | 神州数码融信云技术服务有限公司 | Financial system log file processing method and system |
| CN113839952A (en) * | 2021-09-27 | 2021-12-24 | 深信服科技股份有限公司 | Threat tracking method and device for log access relationship and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10761687B2 (en) | User interface that facilitates node pinning for monitoring and analysis of performance in a computing environment | |
| US10205643B2 (en) | Systems and methods for monitoring and analyzing performance in a computer system with severity-state sorting | |
| US10243818B2 (en) | User interface that provides a proactive monitoring tree with state distribution ring | |
| CN110472068B (en) | Big data processing method, equipment and medium based on heterogeneous distributed knowledge graph | |
| CN107251024B (en) | Database query execution tracking and data generation for diagnosing execution problems | |
| US20150325017A1 (en) | Proactive monitoring tree providing distribution stream chart with branch overlay | |
| US10713070B2 (en) | Systems and methods for capturing and visualizing user interactions across devices | |
| CN113420009B (en) | Electromagnetic data analysis device, system and method based on big data | |
| CN113360554A (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| US9706005B2 (en) | Providing automatable units for infrastructure support | |
| CN111639016A (en) | Big data log analysis method and device and computer storage medium | |
| CN114428822A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113965389A (en) | Network security management method, equipment and medium based on firewall log | |
| CN108733543B (en) | Log analysis method and device, electronic equipment and readable storage medium | |
| US9727666B2 (en) | Data store query | |
| CN112579578A (en) | Metadata-based data quality management method, device and system and server | |
| CN113987086A (en) | Data processing method, data processing device, electronic device, and storage medium | |
| CN114428705A (en) | Network data monitoring method, device, equipment and storage medium | |
| CN111125045B (en) | Lightweight ETL processing platform | |
| CN115168474B (en) | Internet of things central station system building method based on big data model | |
| CN115576834A (en) | Software test multiplexing method, system, terminal and medium for supporting fault recovery | |
| CN115757174A (en) | Database difference detection method and device | |
| CN114860851A (en) | Data processing method, device, equipment and storage medium | |
| CN115168297A (en) | Bypassing log auditing method and device | |
| US20150081735A1 (en) | System and method for fast identification of variable roles during initial data exploration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200908 |