CN110489464B - Exploration type graph fusion visualization method and device - Google Patents
Exploration type graph fusion visualization method and device Download PDFInfo
- Publication number
- CN110489464B CN110489464B CN201910589334.XA CN201910589334A CN110489464B CN 110489464 B CN110489464 B CN 110489464B CN 201910589334 A CN201910589334 A CN 201910589334A CN 110489464 B CN110489464 B CN 110489464B
- Authority
- CN
- China
- Prior art keywords
- data
- log
- fusion
- field
- selecting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses an exploratory graph fusion visualization method and device, wherein the method comprises the following steps: sending an HTTP request to acquire data from a log pushing platform to obtain multidimensional log data; preprocessing the multidimensional log data and storing the preprocessed multidimensional log data in a database; acquiring query conditions of a current user to query the preprocessed log data from the database, and screening the preprocessed log data according to the screening conditions to preliminarily display the log data and realize log exploration type visual fusion. The method greatly improves the precision and the comprehensiveness of log analysis, and can provide certain help for enterprise decision making.
Description
Technical Field
The invention relates to the technical field of log data analysis, in particular to an exploration type graph fusion visualization method and device.
Background
With the rapid development of internet application technology, the amount of data generated by various servers and network devices is showing explosive growth. The complicated and large-scale log data and the textualization and disorder of the log format make it difficult for an analyst to gain insight into useful information, and even impossible to perform comparative analysis on a certain index from the log data in multiple sources, so that the technology of performing comparative analysis, processing and visualization on massive log data in multiple sources at the same time is becoming a technical hotspot of current research, and it is very important to apply the heuristic graph fusion technology to time-series log data.
Enterprises and public institutions often install various network security devices at the entrances and exits of the network to record various network events to secure the internal network. Due to the multi-source heterogeneity, the space-time relevance and the mass of the equipment log data, a data analyzer or operation and maintenance personnel can hardly sense the situation of a certain index comprehensively, so that accurate network decision, prediction analysis and risk control can not be made. In addition, through benchmarking analysis of the industrial business intelligent visual platform, the method generally has the advantages of high deployment operation threshold, complex and unstable data source access, and weak multi-dimensional data screening capability, so that multi-source data display is not flexible or even a multi-source data display function is not provided.
Related technologies, for example, a data query method for a heterogeneous storage multi-source data management and visualization system provides a heterogeneous storage-oriented multi-source data management and visualization system, which includes a storage layer for storing structured data and unstructured data; the service layer is used for extracting, processing, fusing and abstracting the stored data; the application layer realizes the visual link table query of the heterogeneous data source in a visual mode, and the query result reorganizes the service data through a visual technology to form special data used in a specific scene. For another example, a visual display method and system for detecting log collection stability, the system includes obtaining log data of a user terminal, and constructing a log file according to the log data; constructing a data table based on the log file; acquiring a detection time period; calculating and acquiring detection data of a detection time period according to the data table; the detection data are visually displayed, and a visual display system for detecting log collection stability is further disclosed.
However, in most existing log data visualization systems, log data of a user terminal is acquired in a configuration manner to form a log data table, and the configuration manner is uneven, most of the log data only support log access of a single device, even if log access of a plurality of devices is supported, the steps are complicated, and the visualization portion is only simple display in a table format, which is not beneficial to quick viewing and analysis of a certain index situation in the log. And the multi-source log fusion only supports the file level, and is not detailed to the atomic level fusion of a specific index in the multi-source log, so that an accurate result can not be obtained according to the log, and the decision making of an analyst is not facilitated.
Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art.
Therefore, one purpose of the invention is to provide a heuristic graph fusion visualization method, which greatly improves the accuracy and the comprehensiveness of log analysis and can provide certain help for enterprise decision making.
Another object of the present invention is to provide an exploratory graph fusion visualization apparatus.
In order to achieve the above object, an embodiment of the invention provides an exploratory graph fusion visualization method, which sends an HTTP request to acquire data from a log push platform to obtain multidimensional log data; preprocessing the multidimensional log data and storing the multidimensional log data in a database; acquiring query conditions of a current user to query the preprocessed log data from the database, and screening the preprocessed log data according to the screening conditions to preliminarily display the log data and realize log exploration type visual fusion.
The exploration type graph fusion visualization method disclosed by the embodiment of the invention can be used for comparing and analyzing different log indexes of the same source or different aggregation dimensions of the same log index, and supporting fusion of multiple index items; the same or a plurality of log index situations of different sources can be displayed at the same time, and multi-source fusion is supported; and provides three broad categories of statistical charts: the basic statistical chart, the fusion statistical chart and the single-value statistical chart meet the display characteristics of log data, and provide good interactivity in the display process, so that the log analysis accuracy and the comprehensiveness are greatly improved, and certain help can be provided for enterprise decision making.
In addition, the exploratory graph fusion visualization method according to the above embodiment of the present invention may further have the following additional technical features:
further, in an embodiment of the present invention, the collecting data from the log pushing platform includes: initiating a data source collection logging service to collect data from a plurality of dimensions; collecting log file data, system service indexes and module indexes, and updating the data.
Further, in an embodiment of the present invention, the preprocessing the multidimensional log data includes: performing data conversion on the multidimensional log data according to a preset conversion strategy; and/or deleting invalid data in the multidimensional log data; and/or adding a label to the data meeting the preset condition in the multidimensional log data; and/or counting the data meeting the counting in the multi-dimensional log data according to a preset requirement, and recording the data into a database; and/or detecting abnormal data of the multidimensional log data.
Further, in an embodiment of the present invention, the implementing log-exploration visualization fusion includes: visually displaying the same index item under the multi-source condition according to the log data; visually displaying a plurality of index items under the homologous condition according to the log data; and dynamically combining and displaying two views among a histogram, a broken line graph, a pie chart and a map through view category fusion.
Further, in an embodiment of the present invention, the method further includes: receiving an operation instruction of a current user; and freely customizing according to the operation instruction to realize visual monitoring.
In order to achieve the above object, another embodiment of the present invention provides an exploratory graph fusion visualization apparatus, including: the data acquisition module is used for sending an HTTP request to acquire data from the log pushing platform to obtain multidimensional log data; the data processing module is used for preprocessing the multidimensional log data and storing the multidimensional log data in a database; and the data visualization fusion module is used for acquiring the query conditions of the current user so as to query the preprocessed log data from the database, and screening the preprocessed log data according to the screening conditions so as to preliminarily display the log data and realize log exploration type visualization fusion.
The exploration type graph fusion visualization device provided by the embodiment of the invention can compare and analyze different log indexes of the same source or different aggregation dimensions of the same log index, and supports multi-index item fusion; the same or a plurality of log index situations of different sources can be displayed at the same time, and multi-source fusion is supported; and provides three broad categories of statistical charts: the basic statistical chart, the fusion statistical chart and the single-value statistical chart meet the display characteristics of log data, and provide good interactivity in the display process, so that the log analysis accuracy and the comprehensiveness are greatly improved, and certain help can be provided for enterprise decision making.
In addition, the exploratory graph fusion visualization apparatus according to the above embodiment of the present invention may further have the following additional technical features:
further, in an embodiment of the present invention, the data collection module is further configured to initiate a data source collection log service to collect data from multiple dimensions, collect log file data and system service indexes and module indexes, and update the data.
Further, in an embodiment of the present invention, the data processing module is further configured to perform data conversion on the multidimensional log data according to a preset conversion policy; and/or deleting invalid data in the multidimensional log data; and/or adding labels to the data meeting preset conditions in the multidimensional log data; and/or counting the data meeting the counting in the multi-dimensional log data according to a preset requirement, and recording the data into a database; and/or detecting abnormal data of the multidimensional log data.
Further, in an embodiment of the present invention, the data visualization fusion module is further configured to support dynamic combination display of two or more of a histogram, a line graph, a pie graph, and a map by view category fusion according to the visualization display of the log data on the same index item under the multi-source condition and the visualization display of the log data on multiple index items under the homologous condition.
Further, in an embodiment of the present invention, the method further includes: and the visual monitoring module is used for receiving the operation instruction of the current user and carrying out free customization according to the operation instruction so as to realize visual monitoring.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flow chart of a method for heuristic graph fusion visualization according to an embodiment of the present invention;
FIG. 2 is a flow chart of a heuristic graph fusion visualization method according to an embodiment of the present invention;
FIG. 3 is a flowchart of an exploratory view fusion category, according to an embodiment of the invention;
FIG. 4 is an exploratory view fusion category according to an embodiment of the invention;
FIG. 5 is a flow chart of multi-index item fusion according to an embodiment of the present invention;
FIG. 6 is a flow diagram of dynamic class diagram fusion according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a probe map fusion visualization apparatus according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative and intended to explain the present invention and should not be construed as limiting the present invention.
The following describes a heuristic graph fusion visualization method and apparatus proposed according to an embodiment of the present invention with reference to the accompanying drawings, and first, the heuristic graph fusion visualization method proposed according to an embodiment of the present invention will be described with reference to the accompanying drawings.
Fig. 1 is a flowchart of a heuristic graph fusion visualization method according to an embodiment of the present invention.
As shown in fig. 1, the exploration map fusion visualization method includes the following steps:
in step S101, an HTTP request is sent to collect data from the log pushing platform, so as to obtain multidimensional log data.
It can be understood that, as shown in fig. 2, when data collection is performed, the embodiment of the present invention provides a collector management function, supports a custom tag type generation script, executes the script on a node to which data is to be collected, implements a function of adding a data collection manager, is responsible for managing a log file collector and a system monitoring index collector, and provides a start and stop function to implement full life cycle management on the collector.
Further, in one embodiment of the present invention, collecting data from a log pushing platform comprises: initiating a data source collection logging service to collect data from a plurality of dimensions; collecting log file data, system service indexes and module indexes, and updating the data.
Specifically, as shown in fig. 2, the embodiment of the present invention performs data acquisition by sending an HTTP request to a log pushing platform, and mainly includes:
(1) a log collection manager: the embodiment of the invention designs a mode for running a script adding collector manager, generates a custom script by adding a label with practical significance and distinctiveness, executes the custom script on a node of data to be collected, starts a service of collecting logs from a data source and provides data collection from multiple dimensions, and comprises collection of HTTP request data, data of multiple servers, system monitoring data, log file data and various module data (Kafka, k8s, elastic search and the like). The collector manager is responsible for managing the log file collector and the system monitoring index collector and providing starting and stopping functions to realize the full life cycle management of the collector.
(2) A log file collector: collecting common log files, and configuring data sources including log paths, types, fields, whether to start, matching modes, merging modes and other fields; the configuration destination comprises the configuration of host address, index and ES version number; confirm and add collector: and filling in the name, description and type, providing modification to the beat information, completing the collection configuration of the log data, distributing the configuration to a specified log collector manager after the configuration is completed, starting a log file collector, and starting log collection. At this time, after the node log file data is changed, the node log file data is pushed to a log file collector through a collector manager, so that data updating is realized.
(3) The system service index collector is used for collecting system service indexes and module indexes, and selecting the type of monitoring information by configuring system information; configuring a sending mode, and providing three types of sending data destinations, including an Elasticsearch service, Kafka and a local file; and the configuration confirms and sends the collector, fills in the name, describes and provides modification to the module information and the beat information to complete the collection configuration of the system monitoring data, and distributes the configuration to a specified log collector manager to start the system service index collector and start the collection of the system service index after the configuration is completed. And then, after the node system service index data is changed, the node system service index data is pushed to a system service index collector through a collector manager, so that data updating is realized.
In step S102, the multidimensional log data is preprocessed and stored in a database.
It can be understood that, as shown in fig. 2, the embodiment of the present invention performs data cleaning on collected log data, processes missing values, and performs error correction checking to ensure data consistency.
Further, in an embodiment of the present invention, the preprocessing the multidimensional log data includes: performing data conversion on the multi-dimensional log data according to a preset conversion strategy; and/or deleting invalid data in the multidimensional log data; and/or adding labels to the data meeting the preset conditions in the multi-dimensional log data; and/or counting the data meeting the counting in the multi-dimensional log data according to a preset requirement, and recording the data into a database; and/or, detecting abnormal data of the multidimensional log data.
Specifically, as shown in fig. 2, data processing is divided into data conversion, data deletion, data enhancement, data statistics, and data anomaly detection, specifically:
(1) data conversion, including time conversion, geographical position conversion and user-defined data conversion according to functional requirements; (2) data deletion, including processing of duplicate data and useless data; (3) data enhancement, namely adding Tag labels to certain types of data; (4) data statistics, namely counting corresponding index data according to requirements such as index increment and recording the data into a database; (5) and data exception detection, including detecting whether the basic information of the server is abnormal or not, detecting whether each basic system (such as kafka) is abnormal or not, detecting stack exception of an operating system, detecting data exception of a service log and the like.
In step S103, query conditions of the current user are obtained to query the preprocessed log data from the database, and the preprocessed log data are screened according to the screening conditions, so as to preliminarily display the log data and implement exploratory visualization fusion of the log.
It can be appreciated that, as shown in fig. 2, the embodiment of the present invention provides a heuristic graph fusion visualization in three dimensions for the log data that is processed: the method comprises the steps of visual fusion of the same one or more log index items in multiple data sources, and visual fusion and dynamic view category fusion of different aggregation dimensions of multiple log index items or the same log index item in the same data source. The method provides powerful and flexible exploratory analysis and display functions for complex log data.
Further, in an embodiment of the present invention, a log-exploration visualization fusion is implemented, including: visually displaying the same index item under the multi-source condition according to the log data; visually displaying a plurality of index items under the homologous condition according to the log data; and dynamically combining and displaying two views among a histogram, a broken line graph, a pie chart and a map through view category fusion.
Specifically, as shown in fig. 2, in the embodiment of the present invention, a user selects a log source, a field type, a monitoring time period, and a number of user-defined data to be subjected to data analysis, to obtain a query condition, and transmits the query condition to a corresponding interface, so as to query a database as required to obtain log data after data processing, and provide a strong log screening function for the log data, where the screening rule is as follows:
wherein, the parent condition: must represent that this condition must be included, must _ not represents that this condition must be excluded, should represents that this condition can be included; and (3) sublevel conditions: equal stands for strictly equal to, gte stands for more than or equal to, lte stands for less than or equal to, regexp stands for a regular expression; any combination and hierarchical nesting are supported.
And performing preliminary two-dimensional table display on the logs screened according to specific service scenes, such as user behavior analysis indexes, big data cluster indexes, AI computing platform indexes, k8s cluster indexes and the like, so that the user can sense the log data preliminarily and provide a function of supporting log exploration type visual fusion.
Further, as shown in fig. 3, the heuristic graph fusion is divided into multi-source fusion, multi-index item fusion, and dynamic class graph fusion, specifically:
the multi-source data visualization fusion provides a data analyzer with visualization display of the same index item under the multi-source condition, is favorable for comparison and observation, and is favorable for comparison and analysis of each node of a specific index under a large data cluster in an actual production environment; meanwhile, visual display of different index items under the multi-source condition is provided for a data analyst, and multi-source correlation observation and analysis are facilitated. As shown in fig. 4, the steps are as follows:
1. the analyst configures the filter rule (e.g., host name best quality master represents that the data source must be a server master) by selecting the log field (e.g., system
2. Clicking the add field, configuring the next log field, wherein the name of the log field may be the same as that in step one, selecting an aggregation indicator (such as count, sum, maximum, minimum, average), filling out a custom legend name, and configuring a filter rule (such as host. name must equal slave1, which represents that the data source must be server slave 1). The method mainly comprises two scenes: the same log field under different sources is visible; different log fields under different sources are visible.
3. The above steps are repeated based on the configuration of the Y-axis dimension.
4. Configuration based on X-axis metrics: and filling a custom coordinate interval, selecting interval dimensions (such as seconds, minutes and hours), and automatically displaying the number of the log entries.
5. All the query conditions are converted into data objects, the data objects are transmitted to corresponding interfaces, the database is queried to obtain original data of a generated chart, a visual data structure with a uniform format is formed through processing of a data conversion model, then data is converted into a view through the view conversion model, and finally rendering is performed and presented to a user.
The multi-index visual integration provides visual display of a plurality of index items under the homologous condition for a data analyzer, facilitates simultaneous observation of associated indexes, and facilitates comparison and analysis of the same node of different specific indexes under a certain large data cluster in an actual production environment; meanwhile, visual display of different polymerization index dimensions of the same index item under the same source condition is provided for a data analyst, and comparison observation and analysis are facilitated, such as comparison analysis of maximum usage, minimum usage and average usage of a memory (master _ hdfs _ memory) at a specified time interval. As shown in fig. 5, the steps are as follows:
1. the analyst configures the filter rule (e.g., host. name best quality master represents that the host name must be master) by selecting the log field (e.g., system. process. memory. size) desired to be analyzed, aggregating metrics (e.g., count, sum, maximum, minimum, average), filling out custom legend names, and configuring the filter rule (e.g., host. name best quality master represents that the host name must be master)
2. Click add field, configure next log field desired to be analyzed (e.g., system. process. cpu. total. value), select aggregation indicator (e.g., count, sum, maximum, minimum, average), fill out custom legend name, configure filter rule (e.g., host. name best answer master represents that the host name must be master). The method mainly comprises two scenes: the same log field under different aggregation index dimensions; different log fields in the same aggregation indicator dimension.
3. Repeating the above steps based on the configuration of the Y-axis dimension
4. Configuration based on X-axis metrics: filling in the self-defined coordinate interval, selecting interval dimension (such as second, minute and hour), and automatically displaying the number of the log
5. Converting all the query conditions into data objects, transmitting the data objects to corresponding interfaces, querying a database to obtain original data for generating a chart, processing the data objects by a data conversion model to form a visual data structure with a uniform format, converting data into a view by a view conversion model, and finally rendering and presenting the data objects to a user.
The dynamic class diagram fusion mainly provides visual convenience for a data analyzer through view type fusion and supports pairwise dynamic combination among a bar chart, a broken line chart, a pie chart and a map. The combination type can be a column folding fusion graph, a map column fusion graph, a map pie fusion graph and the like, and the specific type is determined according to the dimension of log index data. As shown in fig. 6, the steps are as follows:
1. the analyst configures the filter rules (e.g., host name must be master or otherwise) by selecting the log field to be analyzed, aggregating metrics (e.g., count, sum, maximum, minimum, average), filling in custom legend names, selecting chart types that match the log field dimensions, and configuring the filter rules (e.g., host name must be master or otherwise)
2. Clicking the add field, configuring the next log field to be analyzed, selecting an aggregation index (such as count, sum, maximum, minimum, average), filling in a custom legend name, selecting a chart type matched with the log field dimension, and configuring a screening rule (such as host. name must equal master or other).
3. Configuration based on X-axis metrics: filling in the self-defined coordinate interval, selecting interval dimension (such as second, minute and hour), and automatically displaying the number of the log
5. Converting all the query conditions into data objects, transmitting the data objects to corresponding interfaces, querying a database to obtain original data for generating a chart, processing the data objects by a data conversion model to form a visual data structure with a uniform format, converting data into a view by a view conversion model, and finally rendering and presenting the data objects to a user.
In the visual fusion implementation process, the problem of data non-uniformity caused by different indexes under the condition of multiple sources or homologous sources is solved through a data conversion model. The premise of showing multiple types of data in a chart in Echarts is that data items of dimensions and measurement need to be kept consistent and need to be processed into an ARRAY format in a unified way, but under the condition of multiple sources or multiple indexes, the data items have great randomness, and the technical problem of consistency of the data items and the formats is solved. The scheme is as follows:
1. data entry and format inconsistency can occur in the scene of the same or different log indexes of the same source or the same source, different log index data with the legend name as the unique identifier are returned in the API by transmitting the corresponding query conditions to the corresponding API, and meanwhile, a dynamic legend name array is maintained at the front end, namely the self-defined legend name filled when the log field is dynamically added, so that the log data form a front-end and back-end mapping relation with the legend name as the unique identifier.
2. Dimension axis data consistency: the traversal API returns each piece of log data and a legend name array, and maintains a data object, wherein the attribute in the object is named as KEY by the legend name, and the array formed by all log data under the legend name is VALUE. If a piece of log data under a certain legend name is undefined, it is assigned as the average value of the first two pieces of log data of the current log data.
3. Measuring the consistency of the axis data: dimension and measured data can express correct semantics only by strict one-to-one correspondence, and dimension magnitude data maintains an array and stores time sequence data strictly corresponding to the sequence of dimension axis data.
4. The data item and format consistency effect is achieved, and the visual data structure is obtained.
The data source inputs are as follows:
the consistency output is as follows:
further, in the visual fusion implementation process, the visual data structure and the visual Option are connected through a view conversion model, and the visual fusion is completed by calling a self-developed Dhart plug-in. The plug-in abstract parameters comprise a selector, data, a diagram type, real-time or not and the like, and the supported diagram types comprise a multi-value diagram (a line diagram, a bar diagram, a scatter diagram, a stack diagram, a pie diagram and a ring diagram), a single-value diagram (a percentage diagram, a character and number diagram), a fusion diagram (a bar-pie fusion diagram, a bar-folding fusion diagram and a folding-pie fusion diagram) and a table. Each drawing function (subfunction) comprises acquiring a diagram storage container DOM, initializing an Echarts instance, setting a custom Option, transmitting the custom Option to the Echarts instance through a setOption API, and returning to the custom Option. The parent function uses switch syntax to call each drawing function by judging the chart type. Wherein, table 1 is a chart type table, and the pseudo code is as follows:
TABLE 1
The exploration type graph fusion technology combines the functions of data calculation, data screening, data visualization, chart configuration and the like. The data calculation means that the log index is provided with an aggregation calculation function such as count, average, sum, maximum, minimum, and the like, and the aggregated value is used as data for generating a graph to observe the correlation of the calculated log index. The data screening is to provide a powerful screening mechanism for log data, and by selecting log indexes, parent rules (such as must, must _ not and should) and child rules (equal, gte, lte and regexp), filling specific screening conditions, and then converting the screening rules into a JSON format to serve as parameters to be transmitted to a corresponding API, so that a data source for generating a diagram is changed. The data visualization supports free selection among three major graphs, namely a basic statistical graph, a fusion statistical graph and a single-value statistical graph, which accord with the data dimension characteristics, and supports switching between the graphs and the tables. The chart can be configured to support the change of colors of different Y-axis values, the change of coordinate axis names, legend names and the like, so that the generated chart is simpler, clearer and more attractive.
The exploration type graph fusion technology is basically realized by interactive operations such as clicking, selecting and the like, the threshold of a user for visual analysis of log data is greatly reduced, the type of a generated data graph can be freely switched according to intelligent recommendation, and the efficiency of the user for data analysis is greatly improved.
Further, in an embodiment of the present invention, the method of an embodiment of the present invention further includes: receiving an operation instruction of a current user; and freely customizing according to the operation instruction to realize visual monitoring.
It can be understood that, as shown in fig. 2, the log situation graph generated in the data visualization fusion is added to the specified log report in the specified folder in the embodiment of the present invention, so as to provide functions of free typesetting, real-time refreshing, data source migration, dynamic carousel, and the like.
Specifically, as shown in fig. 2, visual monitoring provides a situation map free customization mechanism for an analyst, the analyst can rearrange and freely stretch the situation map by dragging, click and store the dragged position information and transmit the position information to a corresponding interface in a parameter form, so as to realize layout persistence; a template mechanism is provided based on the principle that a data source can migrate, switching of display data is performed by selecting different log indexes, and re-mapping is not needed to realize multiplexing, so that the efficiency is improved; each situation map supports data dynamic carousel, dynamic performance is achieved, floating layer display is conducted on the data in the carousel process, and observation of a user is facilitated; providing a precisely editable function for each situation map, log monitoring reports of various styles can be formed according to the aesthetic quality of the analysts.
In summary, the embodiment of the invention combines the exploratory analysis technology and the data visualization technology based on the exploration-type graph fusion technology on the basis of the existing visualization system, designs an extremely simple interaction mode by the thinking of the ordinary user, has low operation threshold, and can be used by non-technical personnel. The embodiment of the invention comprises the steps of acquiring a log data source, exploring a log index, fusing and visualizing a graph, and adding the graph to a log report, specifically: establishing a data transmission channel with a log source; adding a log collector and distributing to a collector manager; acquiring index, type, monitoring time period and the like of a collector for analyzing log data; multi-source fusion analysis among log sources, and fusion analysis among indexes of the same log source; visualization plots, added to the log report.
That is to say, the embodiment of the invention conforms to the behavior habits of most users through a friendly and simple interaction mode. After the log collector is distributed to a plurality of or a single collector manager, all log data are subjected to two-dimensional table display according to log index field names by selecting retrieval conditions such as log indexes, log types, monitoring time periods and the like, file (molecular) level multi-source data fusion is completed, table visualization is carried out, and a strong file level fusion data screening mechanism is provided; after data to be analyzed is screened out, multi-source or multi-index item map fusion is realized by configuring a plurality of Y-axes, the specific configuration indexes of the Y-axes are provided with log data fields, aggregation indexes and legend names, and a log index field (atomic) level fusion data screening mechanism is provided for data on each Y-axis dimension; the X-axis measurement is time sequence representation of time, and specific configuration indexes comprise coordinate intervals and interval dimensions; finally, selecting a display graph suitable for the data dimension characteristics to complete the exploration type graph fusion visualization of the log indexes; and provides an add to log reporting function. The embodiment of the invention constructs a set of intelligent log visualization process with data acquisition, data processing, data mapping and log reporting functions, provides a configurable, reusable and extensible log monitoring visualization solution, and has the characteristics of aesthetic feeling, flexibility and multiple sources.
According to the exploration type graph fusion visualization method provided by the embodiment of the invention, not only can different log indexes of the same source or different aggregation dimensions of the same log index be contrastively analyzed, but also multiple index item fusion is supported; the same or a plurality of log index situations of different sources can be displayed at the same time, and multi-source fusion is supported; and provides three broad categories of statistical charts: the basic statistical chart, the fusion statistical chart and the single-value statistical chart meet the display characteristics of log data, and provide good interactivity in the display process, so that the log analysis accuracy and the comprehensiveness are greatly improved, and certain help can be provided for enterprise decision making.
Next, a heuristic graph fusion visualization apparatus proposed according to an embodiment of the present invention is described with reference to the drawings.
FIG. 7 is a schematic structural diagram of a probe chart fusion visualization apparatus according to an embodiment of the present invention
As shown in fig. 7, the heuristic graph fusion visualization device 10 includes: the system comprises a data acquisition module 100, a data processing module 200 and a data visualization fusion module 300.
The data acquisition module 100 is configured to send an HTTP request to acquire data from the log push platform, so as to obtain multidimensional log data. The data processing module 200 is configured to pre-process the multidimensional log data and store the pre-processed multidimensional log data in a database. The data visualization fusion module 300 is configured to obtain a query condition of a current user, to query the preprocessed log data from the database, and to filter the preprocessed log data according to a filtering condition, to preliminarily display the log data, and to implement exploratory visualization fusion of the log. The device 10 of the embodiment of the invention greatly improves the accuracy and the comprehensiveness of log analysis and can provide certain help for enterprise decision making.
Further, in an embodiment of the present invention, the data collection module 100 is further configured to initiate a data source collection log service to collect data from multiple dimensions, collect log file data and system service indexes and module indexes, and update the data.
Further, in an embodiment of the present invention, the data processing module 200 is further configured to perform data conversion on the multidimensional log data according to a preset conversion policy; and/or deleting invalid data in the multidimensional log data; and/or adding labels to the data meeting the preset conditions in the multi-dimensional log data; and/or counting the data meeting the counting in the multi-dimensional log data according to a preset requirement, and recording the data into a database; and/or detecting anomalous data of the multidimensional log data.
Further, in an embodiment of the present invention, the data visualization fusion module 300 is further configured to support dynamic combination display of two or more of the histogram, the line graph, the pie graph, and the map through view category fusion according to the visualization display of the log data on the same index item under the multi-source condition, and according to the visualization display of the log data on multiple index items under the homologous condition.
Further, in an embodiment of the present invention, the apparatus 10 of the embodiment of the present invention further includes: and a visual monitoring module. The visual monitoring module is used for receiving an operation instruction of a current user and carrying out free customization according to the operation instruction so as to realize visual monitoring.
It should be noted that the above explanation of the embodiment of the exploratory graph fusion visualization method is also applicable to the exploratory graph fusion visualization apparatus of the embodiment, and is not repeated here.
According to the exploration type graph fusion visualization device provided by the embodiment of the invention, not only can different log indexes of the same source or different aggregation dimensions of the same log index be contrasted and analyzed, but also fusion of multiple index items is supported; the same or a plurality of log index situations of different sources can be displayed at the same time, and multi-source fusion is supported; and provides three broad categories of statistical charts: the basic statistical chart, the fusion statistical chart and the single-value statistical chart meet the display characteristics of log data, and provide good interactivity in the display process, so that the log analysis accuracy and the comprehensiveness are greatly improved, and certain help can be provided for enterprise decision making.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or to implicitly indicate the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In the description of the specification, reference to the description of "one embodiment," "some embodiments," "an example," "a specific example," or "some examples" or the like means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Moreover, various embodiments or examples and features of various embodiments or examples described in this specification can be combined and combined by one skilled in the art without being mutually inconsistent.
Although embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are exemplary and not to be construed as limiting the present invention, and that changes, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (8)
1. An exploratory graph fusion visualization method is characterized by comprising the following steps of:
sending an HTTP request to acquire data from a log pushing platform to obtain multidimensional log data;
preprocessing the multidimensional log data and storing the multidimensional log data in a database; and
acquiring query conditions of a current user to query the preprocessed log data from the database, and screening the preprocessed log data according to the screening conditions to preliminarily display the log data and realize log exploration type visual fusion;
wherein, the realization of the exploration-type visual fusion of the log comprises the following steps: multi-source fusion, multi-index item fusion and dynamic fusion;
wherein, the multi-source fusion comprises the visual display of the same index item under the multi-source condition according to the log data, and further comprises:
step one, filling a custom legend name and configuring a screening rule by selecting an analyzed log field and an analyzed aggregation index;
clicking an added field, configuring a next log field, selecting a polymerization index, filling a custom legend name, and configuring a screening rule, wherein the field name is the same as that of the first step; the method mainly comprises two scenes: the same log field under different sources can be seen, and different log fields under different sources can be seen;
step three, repeating the step one and the step two based on the configuration of the Y-axis dimension;
step four, based on the configuration of the X-axis measurement: filling a custom coordinate interval, selecting interval dimensions and automatically displaying the number of the logs;
converting all the query conditions into data objects, transmitting the data objects to corresponding interfaces, querying a database to obtain original data of a generated chart, processing the original data by a data conversion model to form a visual data structure with a uniform format, converting the data into a view by a view conversion model, and finally rendering and presenting the data to a user;
wherein the fusion of the multiple index items comprises the visual display of the multiple index items under the homologous condition according to the log data, and further comprises:
step one, filling a custom legend name and configuring a screening rule by selecting an analyzed log field and an analyzed aggregation index;
clicking an added field, configuring a next log field, selecting a polymerization index, filling a custom legend name, and configuring a screening rule, wherein the field name is the same as that of the first step; the method mainly comprises two scenes: the same log field under different aggregation index dimensions, and different log fields under the same aggregation index dimension;
step three, repeating the step one and the step two based on the configuration of the Y-axis dimension;
step four, based on the configuration of the X-axis measurement: filling a custom coordinate interval, selecting interval dimensions and automatically displaying the number of the logs;
converting all the query conditions into data objects, transmitting the data objects to corresponding interfaces, querying a database to obtain original data for generating a chart, processing the original data to form a visual data structure in a uniform format through a data conversion model, converting data into views through the view conversion model, and finally rendering and presenting the data to a user;
wherein, the dynamic fusion comprises supporting dynamic combined display of two of the histogram, the broken line graph, the pie chart and the map through view type fusion, and further comprises:
step one, filling a custom legend name by selecting an analyzed log field and an analyzed aggregation index, selecting a chart type matched with the log field, and configuring a screening rule;
clicking an adding field, configuring a next log field, selecting a polymerization index, filling a self-defined legend name, selecting a chart type matched with the log field, and configuring a screening rule, wherein the field name is the same as that in the step one;
step three, repeating the step one and the step two based on the configuration of the Y-axis dimension;
step four, based on the configuration of the X-axis measurement: filling a custom coordinate interval, selecting interval dimensions and automatically displaying the number of the logs;
and step five, converting all the query conditions into data objects, transmitting the data objects to corresponding interfaces, querying a database to obtain original data for generating a chart, forming a visual data structure in a uniform format through the processing of a data conversion model, converting data into views through the view conversion model, and finally rendering and presenting the data to a user.
2. The method of claim 1, wherein the collecting data from a log push platform comprises:
initiating a data source collection logging service to collect data from a plurality of dimensions;
collecting log file data, system service indexes and module indexes, and updating the data.
3. The method of claim 1, wherein the preprocessing the multidimensional log data comprises:
performing data conversion on the multidimensional log data according to a preset conversion strategy;
and/or deleting invalid data in the multidimensional log data;
and/or adding a label to the data meeting the preset condition in the multidimensional log data;
and/or counting the data meeting the counting in the multi-dimensional log data according to a preset requirement, and recording the data into a database;
and/or detecting abnormal data of the multidimensional log data.
4. The method of claim 1, further comprising:
receiving an operation instruction of a current user;
and freely customizing according to the operation instruction to realize visual monitoring.
5. An exploratory graph fusion visualization device, comprising:
the data acquisition module is used for sending an HTTP request to acquire data from the log pushing platform to obtain multidimensional log data;
the data processing module is used for preprocessing the multidimensional log data and storing the multidimensional log data in a database; and
the data visualization fusion module is used for acquiring the query conditions of the current user, querying the preprocessed log data from the database, screening the preprocessed log data according to the screening conditions, preliminarily displaying the log data and realizing log exploration type visualization fusion;
wherein, the realization of the exploration-type visual fusion of the log comprises the following steps: multi-source fusion, multi-index item fusion and dynamic fusion;
wherein, the multi-source fusion comprises the visual display of the same index item under the multi-source condition according to the log data, and further comprises:
step one, filling a custom legend name and configuring a screening rule by selecting an analyzed log field and an analyzed aggregation index;
clicking an added field, configuring a next log field, selecting a polymerization index, filling a custom legend name, and configuring a screening rule, wherein the field name is the same as that of the first step; the method mainly comprises two scenes: the same log field under different sources can be seen, and different log fields under different sources can be seen;
step three, repeating the step one and the step two based on the configuration of the Y-axis dimension;
step four, based on the configuration of the X-axis measurement: filling a custom coordinate interval, selecting interval dimensions and automatically displaying the number of the logs;
converting all the query conditions into data objects, transmitting the data objects to corresponding interfaces, querying a database to obtain original data for generating a chart, processing the original data to form a visual data structure in a uniform format through a data conversion model, converting data into views through the view conversion model, and finally rendering and presenting the data to a user;
wherein the fusion of the multiple index items comprises the visual display of the multiple index items under the homologous condition according to the log data, and further comprises:
step one, filling a custom legend name and configuring a screening rule by selecting an analyzed log field and an analyzed aggregation index;
clicking an added field, configuring a next log field, selecting a polymerization index, filling a custom legend name, and configuring a screening rule, wherein the field name is the same as that of the first step; the method mainly comprises two scenes: the same log field under different aggregation index dimensions, and different log fields under the same aggregation index dimension;
step three, repeating the step one and the step two based on the configuration of the Y-axis dimension;
step four, based on the configuration of the X-axis measurement: filling a custom coordinate interval, selecting interval dimensions and automatically displaying the number of the logs;
converting all the query conditions into data objects, transmitting the data objects to corresponding interfaces, querying a database to obtain original data for generating a chart, processing the original data to form a visual data structure in a uniform format through a data conversion model, converting data into views through the view conversion model, and finally rendering and presenting the data to a user;
wherein, the dynamic fusion comprises supporting dynamic combined display of two of the histogram, the broken line graph, the pie chart and the map through view type fusion, and further comprises:
step one, filling a custom legend name by selecting an analyzed log field and an analyzed aggregation index, selecting a chart type matched with the log field, and configuring a screening rule;
clicking an adding field, configuring a next log field, selecting a polymerization index, filling a self-defined legend name, selecting a chart type matched with the log field, and configuring a screening rule, wherein the field name is the same as that in the step one;
step three, repeating the step one and the step two based on the configuration of the Y-axis dimension;
step four, based on the configuration of the X-axis measurement: filling a custom coordinate interval, selecting interval dimensions and automatically displaying the number of the logs;
and step five, converting all the query conditions into data objects, transmitting the data objects to corresponding interfaces, querying a database to obtain original data for generating a chart, forming a visual data structure in a uniform format through the processing of a data conversion model, converting data into views through the view conversion model, and finally rendering and presenting the data to a user.
6. The apparatus of claim 5, wherein the data collection module is further configured to initiate a data source collection log service to collect data from multiple dimensions, collect log file data and system service metrics and module metrics, and update the data.
7. The apparatus of claim 5, wherein the data processing module is further configured to perform data transformation on the multidimensional log data according to a preset transformation policy; and/or deleting invalid data in the multidimensional log data; and/or adding a label to the data meeting the preset condition in the multidimensional log data; and/or counting the data meeting the counting in the multi-dimensional log data according to a preset requirement, and recording the data into a database; and/or detecting abnormal data of the multidimensional log data.
8. The apparatus of claim 5, further comprising:
and the visual monitoring module is used for receiving the operation instruction of the current user and carrying out free customization according to the operation instruction so as to realize visual monitoring.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910589334.XA CN110489464B (en) | 2019-07-02 | 2019-07-02 | Exploration type graph fusion visualization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910589334.XA CN110489464B (en) | 2019-07-02 | 2019-07-02 | Exploration type graph fusion visualization method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110489464A CN110489464A (en) | 2019-11-22 |
| CN110489464B true CN110489464B (en) | 2022-05-31 |
Family
ID=68546394
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910589334.XA Active CN110489464B (en) | 2019-07-02 | 2019-07-02 | Exploration type graph fusion visualization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110489464B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111324582B (en) * | 2020-02-18 | 2023-05-02 | 支付宝（中国）网络技术有限公司 | Visual backtracking processing method and device for business processing behaviors |
| CN111400565A (en) * | 2020-03-19 | 2020-07-10 | 北京三维天地科技股份有限公司 | Visualized dragging online data processing method and system |
| CN112148700A (en) * | 2020-10-12 | 2020-12-29 | 平安科技（深圳）有限公司 | Log data processing method and device, computer equipment and storage medium |
| CN112187550B (en) * | 2020-10-16 | 2022-09-30 | 温州职业技术学院 | Log analysis method based on density peak value multi-attribute clustering |
| CN113961518B (en) * | 2021-09-08 | 2022-09-23 | 北京百度网讯科技有限公司 | Log visual display method and device, electronic equipment and storage medium |
| CN114448672A (en) * | 2021-12-27 | 2022-05-06 | 奇安信科技集团股份有限公司 | Multi-source network security data processing method and device |
| CN114567498B (en) * | 2022-03-04 | 2024-02-02 | 科来网络技术股份有限公司 | Metadata extraction and processing method and system for network behavior visualization |
| CN114860734B (en) * | 2022-05-27 | 2022-11-15 | 河北省科学技术情报研究院(河北省科技创新战略研究院) | Processing method for data presentation of multi-source index structure fusion and scene reduction |
| CN115801262B (en) * | 2022-08-31 | 2023-07-18 | 重庆市规划和自然资源信息中心 | Intersection operator space retrieval method based on elastic search technology |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2918243A1 (en) * | 2013-06-28 | 2014-12-31 | Life Technologies Corporation | Methods and systems for visualizing data quality |
| CN109376532A (en) * | 2018-10-31 | 2019-02-22 | 云南电网有限责任公司 | Power network security monitoring method and system based on the analysis of ELK log collection |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
| CN109902072A (en) * | 2019-02-21 | 2019-06-18 | 云南电网有限责任公司红河供电局 | A kind of log processing system |
-
2019
- 2019-07-02 CN CN201910589334.XA patent/CN110489464B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2918243A1 (en) * | 2013-06-28 | 2014-12-31 | Life Technologies Corporation | Methods and systems for visualizing data quality |
| CN109376532A (en) * | 2018-10-31 | 2019-02-22 | 云南电网有限责任公司 | Power network security monitoring method and system based on the analysis of ELK log collection |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
| CN109902072A (en) * | 2019-02-21 | 2019-06-18 | 云南电网有限责任公司红河供电局 | A kind of log processing system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110489464A (en) | 2019-11-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110489464B (en) | Exploration type graph fusion visualization method and device | |
| US11238033B1 (en) | Interactive location queries for raw machine data | |
| US20200167350A1 (en) | Loading queries using search points | |
| US20220398243A1 (en) | Key name synthesis | |
| CN108416620B (en) | Portrait data intelligent social advertisement putting platform based on big data | |
| US11151761B2 (en) | Analysing Internet of Things | |
| CN101971165B (en) | Graphic representations of data relationships | |
| CN108460087A (en) | Heuristic high dimensional data visualization device and method | |
| EP2098967A1 (en) | Apparatus and method for positioning user-created data in OLAP data sources | |
| US9547646B2 (en) | User-created members positioning for OLAP databases | |
| Van Ham et al. | Honeycomb: Visual analysis of large scale social networks | |
| CN110442550B (en) | Log screen-gathering real-time visualization method and device | |
| WO2015039046A1 (en) | Data flow exploration | |
| CN109254901B (en) | A kind of Monitoring Indexes method and system | |
| CN103605651A (en) | Data processing showing method based on on-line analytical processing (OLAP) multi-dimensional analysis | |
| CN109213747A (en) | A kind of data managing method and device | |
| Xia et al. | Dimscanner: A relation-based visual exploration approach towards data dimension inspection | |
| EP1966721A1 (en) | Multi-dimensional aggregation on event streams | |
| EP3561688A1 (en) | Hierarchical tree data structures and uses thereof | |
| CN110968624A (en) | Animal epidemic disease monitoring data statistical analysis geographic information system and implementation method | |
| US8850321B2 (en) | Cross-domain business service management | |
| CN115858526A (en) | Multidimensional visual test data management system based on uncertain data source formats | |
| US20130232158A1 (en) | Data subscription | |
| CN109558194B (en) | One-stop universal industry report visualization tool | |
| US11308104B2 (en) | Knowledge graph-based lineage tracking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |