CN110489464A - Heuristic figure fusion visualization method and device - Google Patents
Heuristic figure fusion visualization method and device Download PDFInfo
- Publication number
- CN110489464A CN110489464A CN201910589334.XA CN201910589334A CN110489464A CN 110489464 A CN110489464 A CN 110489464A CN 201910589334 A CN201910589334 A CN 201910589334A CN 110489464 A CN110489464 A CN 110489464A
- Authority
- CN
- China
- Prior art keywords
- data
- log
- daily record
- fusion
- multidimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a kind of heuristic figure fusion visualization method and devices, wherein method is the following steps are included: send HTTP request, to obtain multidimensional log data from log push platform acquisition data；Multidimensional log data are pre-processed, and are stored to database；The querying condition for obtaining active user, to inquire pretreated daily record data from database, and screens pretreated daily record data according to screening conditions, tentatively to be shown to daily record data, and realizes the visualization fusion of log heuristic.This method greatly improves log analysis precision, Quan Miandu, and can provide business decision certain help.
Description
Technical field
The present invention relates to daily record data analysis technical field, in particular to a kind of heuristic figure fusion visualization method and dress
It sets.
Background technique
With the rapid development of Internet application technology, the data volume that various servers and the network equipment generate just shows
Explosive growth.The textual of the complication of daily record data, scale and journal format, disordering make analyst be difficult from
In have insight into useful information, can not analyze from the daily record data in multi-source for a certain index, therefore, how together
Massive logs data in Shi Duoyuan compare and analyze, handle and visual technology is becoming the technology heat of current research
Point, heuristic figure integration technology, which can be applied in sequential log data, is particularly important.
Multiple network safety equipment is installed usually at the entrance of network to record various network events by enterprises and institutions
To ensure the safety of internal network.The multi-source of device log data, space-time relationship, magnanimity make data analyst
Either operation maintenance personnel is difficult the situation of a certain index of complete perception, so that accurately network decision, forecast analysis can not be made
And risk control.In addition, generally there is deployment operation threshold by analyzing mark to industry business intelligence Visualization Platform
Height, data source access is complicated unstable, and data multidimensional screening capacity is weak, causes to show multi-source data not flexible or even not provide
Multi-source data shows function.
The relevant technologies, for example, a kind of data query method that multi-source data management and visualization system are stored towards isomery,
There is provided it is a kind of towards isomery storage multi-source data management and visualization system, the system include accumulation layer, be used for storage organization
Change data and unstructured data；Service layer is for extracting storing data, being processed, being merged and being abstracted；Application layer passes through
Visual mode realizes the visualization company table inquiry of heterogeneous data source, query result by visualization technique to business datum into
Row organizes the formation of the proprietary data that special scenes use again.For another example, a kind of for detecting the visualization of log collection stability
Methods of exhibiting and system, the system include the daily record data for obtaining user terminal, and construct journal file according to daily record data；Base
Tables of data is constructed in journal file；Obtain detection time section；According to tables of data, the detection data for obtaining detection time section is calculated；
Detection data is visualized, and is also disclosed a kind of for detecting the visual presentation system of log collection stability.
However, being all to obtain user terminal by way of configuring mostly in existing daily record data visualization system
Daily record data, form daily record data table, and configuration mode is irregular, most of only the log of individual equipment to be supported to connect
Enter, even supporting the log access of multiple equipment, step is also cumbersome, and visualization portion is also all tabular simple exhibition
Show, is unfavorable for quickly checking and analyzing to index situation a certain in log.And heterogeneous event logs fusion is only supported to arrive file layers
Face, does not refine to the atom level fusion of a certain specific targets in heterogeneous event logs, therefore can not precisely be tied according to log
Fruit is unfavorable for analyst and makes a policy.
Summary of the invention
The present invention is directed to solve at least some of the technical problems in related technologies.
For this purpose, an object of the present invention is to provide a kind of heuristic figure fusion visualization method, this method is greatly mentioned
High log analysis precision, Quan Miandu, and certain help can be provided business decision.
It is another object of the present invention to propose a kind of heuristic figure fusion visualization device.
In order to achieve the above objectives, one aspect of the present invention embodiment proposes a kind of heuristic figure fusion visualization method, hair
HTTP request is sent, to acquire data from log push platform, obtains multidimensional log data；The multidimensional log data are carried out pre-
Processing, and store to database；The querying condition of active user is obtained, to inquire pretreated log from the database
Data, and the pretreated daily record data is screened according to screening conditions, it is preliminary to be carried out to the daily record data
It shows, and realizes the visualization fusion of log heuristic.
The heuristic figure fusion visualization method of the embodiment of the present invention can not only be referred to the homologous different logs of comparative analysis
The different polymerization dimensions of mark or same log index, support the fusion of multi objective item；Also it can show simultaneously not homologous same
Or multiple log index situation, support multi-source fusion；And provide three categories statistical graph: basic statistical chart, fusion statistics
Chart, monodrome statistical graph meet the displaying characteristic of daily record data, and good interactivity is provided during displaying, thus
Log analysis precision, Quan Miandu are greatly improved, and certain help can be provided business decision.
In addition, heuristic figure fusion visualization method according to the above embodiment of the present invention can also have it is following additional
Technical characteristic:
Further, in one embodiment of the invention, described to acquire data from log push platform, comprising: starting
The service of data source collector journal, to carry out data collection from multiple dimensions；Acquire log file data and system service index
And module index, and more new data.
Further, in one embodiment of the invention, described that the multidimensional log data are pre-processed, packet
It includes: data conversion is carried out to the multidimensional log data according to default switching strategy；And/or delete the multidimensional log data
Middle invalid data；And/or label is increased to the data for meeting preset condition in the multidimensional log data；And/or according to pre-
If demand counts the data for meeting statistics in the multidimensional log data, and is included in database；And/or described in detection
The abnormal data of multidimensional log data.
Further, in one embodiment of the invention, the realization log heuristic visualization fusion, comprising: root
According to the daily record data to the visual presentation of index item same in the case of multi-source；According to the daily record data under the conditions of homologous
The visual presentation of multiple index item；Merged by view classification support histogram, line chart, pie chart, between map two-by-two
Dynamic combined is shown.
Further, in one embodiment of the invention, further includes: receive the operational order of active user；According to institute
It states operational order and carries out free customization, to realize visual control.
In order to achieve the above objectives, another aspect of the present invention embodiment proposes a kind of heuristic figure fusion visualization device,
Include: data acquisition module, for sending HTTP request, to acquire data from log push platform, obtains multidimensional log data；
Data processing module for pre-processing to the multidimensional log data, and is stored to database；Data visualization merges mould
Block, for obtaining the querying condition of active user, to inquire pretreated daily record data from the database, and according to sieve
It selects condition to screen the pretreated daily record data, tentatively to be shown to the daily record data, and realizes day
The visualization fusion of will heuristic.
The heuristic figure fusion visualization device of the embodiment of the present invention can not only be referred to the homologous different logs of comparative analysis
The different polymerization dimensions of mark or same log index, support the fusion of multi objective item；Also it can show simultaneously not homologous same
Or multiple log index situation, support multi-source fusion；And provide three categories statistical graph: basic statistical chart, fusion statistics
Chart, monodrome statistical graph meet the displaying characteristic of daily record data, and good interactivity is provided during displaying, thus
Log analysis precision, Quan Miandu are greatly improved, and certain help can be provided business decision.
In addition, heuristic figure fusion visualization device according to the above embodiment of the present invention can also have it is following additional
Technical characteristic:
Further, in one embodiment of the invention, the data acquisition module is further used for log-on data source
The service of collector journal to carry out data collection from multiple dimensions, and acquires log file data and system service index and mould
Block index, and more new data.
Further, in one embodiment of the invention, the data processing module is further used for being turned according to default
It changes strategy and data conversion is carried out to the multidimensional log data；And/or delete invalid data in the multidimensional log data；With/
Or, increasing label to the data for meeting preset condition in the multidimensional log data；And/or it will be described more according to preset need
The data for meeting statistics in dimension daily record data are counted, and are included in database；And/or detect the multidimensional log data
Abnormal data.
Further, in one embodiment of the invention, the data visualization Fusion Module is further used for basis
The daily record data is to the visual presentation of index item same in the case of multi-source, according to the daily record data to more under the conditions of homologous
The visual presentation of a index item passes through moving between view classification fusion support histogram, line chart, pie chart, map two-by-two
State combination is shown.
Further, in one embodiment of the invention, further includes: visual control module, for receiving current use
The operational order at family, and free customization is carried out according to the operational order, to realize visual control.
The additional aspect of the present invention and advantage will be set forth in part in the description, and will partially become from the following description
Obviously, or practice through the invention is recognized.
Detailed description of the invention
Above-mentioned and/or additional aspect and advantage of the invention will become from the following description of the accompanying drawings of embodiments
Obviously and it is readily appreciated that, in which:
Fig. 1 is the flow chart according to the heuristic figure fusion visualization method of the embodiment of the present invention；
Fig. 2 is the flow chart according to the heuristic figure fusion visualization method of a specific embodiment of the invention；
Fig. 3 is to merge classification flow chart according to the heuristic view of the embodiment of the present invention；
Fig. 4 is to merge classification according to the heuristic view of the embodiment of the present invention；
Fig. 5 is to merge flow chart according to the multi objective item of the embodiment of the present invention；
Fig. 6 is to merge flow chart according to the dynamic class figure of the embodiment of the present invention；
Fig. 7 is the structural schematic diagram according to the heuristic figure fusion visualization device of the embodiment of the present invention.
Specific embodiment
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, it is intended to is used to explain the present invention, and is not considered as limiting the invention.
The heuristic figure fusion visualization method proposed according to embodiments of the present invention and device are described with reference to the accompanying drawings, it is first
The heuristic figure fusion visualization method proposed according to embodiments of the present invention is first described with reference to the accompanying drawings.
Fig. 1 is the flow chart of the heuristic figure fusion visualization method of one embodiment of the invention.
As shown in Figure 1, the heuristic figure fusion visualization method the following steps are included:
In step s101, HTTP request is sent, to acquire data from log push platform, obtains multidimensional log data.
It is understood that as shown in Fig. 2, the embodiment of the present invention provides collector and manages function when carrying out data acquisition
Can, it supports customized label formula to generate script, in executing the script on the node of data to be collected, realizes addition data collection pipe
Device function is managed, management journal file collector and system monitoring index collector are responsible for, and starting is provided, stops function realization pair
The lifecycle management of collector.
Further, in one embodiment of the invention, data are acquired from log push platform, comprising: log-on data
The service of source collector journal, to carry out data collection from multiple dimensions；Acquire log file data and system service index and mould
Block index, and more new data.
Specifically, as shown in Fig. 2, the embodiment of the present invention pushes platform progress data to log by sending HTTP request
Acquisition, specifically includes that
(1) log collection manager: the embodiment of the present invention devises the mode of Run Script addition collector manager, leads to
Cross addition be of practical significance, have distinctive label generate custom script, execute, open on the node of data to be collected
The service of dynamic data source collector journal is simultaneously provided from multiple dimensions progress data collection, including to HTTP request data, more clothes
It is engaged in device data, system monitoring data, log file data, various module datas (Kafka, k8s, Elasticsearch etc.)
It collects.Collector manager is responsible for managing journal file collector and system monitoring index collector, and provides starting, stops function
It is able to achieve the lifecycle management to collector.
(2) journal file collector: being the collection to generic log file, by disposition data source, including to log road
Whether diameter type, field, starts, the configuration of the fields such as match pattern, merging mode；Configuration purpose, including to host
The configuration of location, index, ES version number；Confirm and add collector: filling in title, description, type and provide to beat information
Modification is completed the collection to daily record data and is configured, and specified log concentrator manager is distributed to after the completion of configuration and carries out log
File collector starts and starts to carry out log collection.At this point, node log file data will pass through collection after changing
Device manager pushes to journal file collector, realizes that data update.
(3) system service index collector is the collection to system service index and module index, by configuring system letter
Breath selects monitoring information classification；Sending method is configured, three classes are provided and send data destination, including Elasticsearch clothes
Business, Kafka, local file；Configure-ack simultaneously sends collector, fills in title, description and provides to module information, beat letter
Breath modification complete system monitoring data collection configuration, configuration after the completion of be distributed to specified log concentrator manager into
Row system service index collector starts and starts to carry out the collection of system service index.Hereafter, node system service indication data
System service index collector will be pushed to after changing by collector manager, realizes that data update.
In step s 102, multidimensional log data are pre-processed, and stored to database.
It is understood that as shown in Fig. 2, the embodiment of the present invention carries out data cleansing, place to the daily record data gathered
It manages missing values and carries out error checking and correction, guarantee data consistency.
Further, in one embodiment of the invention, multidimensional log data are pre-processed, comprising: according to pre-
If switching strategy carries out data conversion to multidimensional log data；And/or delete invalid data in multidimensional log data；And/or
Label is increased to the data for meeting preset condition in multidimensional log data；And/or it will be in multidimensional log data according to preset need
The data for meeting statistics are counted, and are included in database；And/or the abnormal data of detection multidimensional log data.
Specifically, as shown in Fig. 2, being divided into data conversion to data processing, data are deleted, data enhancing, data statistics
With data abnormality detection, specifically:
(1) data conversion, including time conversion, geographical location are converted and carry out self-defining data according to functional requirement and turn
It changes；(2) data are deleted, including the processing to repeated data, hash；(3) data enhance, and refer to and increase Tag for certain class data
Label；(4) data statistics such as index increment, counts corresponding index data, is included in database according to demand；(5) data exception
Whether detection, including detection service device basic information are abnormal, detect whether each basic system (such as: kafka) is abnormal, and detection is transported
The storehouse of row system is abnormal, detects the data exception etc. of business diary.
In step s 103, the querying condition of active user is obtained, to inquire pretreated log number from database
According to, and pretreated daily record data is screened according to screening conditions, tentatively to be shown to daily record data, and realize
The visualization fusion of log heuristic.
It is understood that as shown in Fig. 2, the daily record data that the embodiment of the present invention is completed for processing provides three dimensions
On heuristic figure fusion visualization: in multi-data source for one or more log index item visualization merge, same number
Visualization fusion and dynamic view according to the different polymerization dimensions for being directed to multiple log index item or same log index item in source
Classification fusion.Powerful, flexible heuristic analysis is provided to numerous and complicated daily record data and shows function.
Further, in one embodiment of the invention, the visualization fusion of log heuristic is realized, comprising: according to day
Visual presentation of the will data to index item same in the case of multi-source；According to daily record data to multiple index item under the conditions of homologous
It visualizes；It is shown by the dynamic combined two-by-two between view classification fusion support histogram, line chart, pie chart, map.
Specifically, as shown in Fig. 2, the embodiment of the present invention by user select want carry out data analysis Log Source,
Field type, monitoring time section and self-defining data item number obtain querying condition, pass it to corresponding interface, thus according to
Demand inquiry database obtains the daily record data after data processing, and provides it powerful Log Filter function, screening rule
It is as follows:
Wherein, parent condition: it must include this condition that must, which is represented, and must_not representative must exclude this condition,
Should representative may include this condition；Sub- grade condition: equal is represented exactly equal to gte representative is more than or equal to, and lte represents small
In being equal to, regexp represents regular expression；Support any combination, level nested.
Will according to specific transactions scene, as user behavior analysis index, big data cluster index, AI computing platform index,
The preliminary two-dimensional table displaying of log progress after the screenings such as k8s cluster index, the preliminary perception so as to user to daily record data,
And it provides and supports log heuristic visualization fusion function.
Further, as shown in figure 3, the fusion of heuristic figure is divided into multi-source fusion, multi objective item merges and dynamic class figure melts
It closes, specifically:
Multi-source data visualization is fused to data analyst and provides to the visual presentation of index item same in the case of multi-source,
Conducive to comparative observation, in actual production environment, compared conducive to each node to a certain specific targets under certain big data cluster
Analysis；The visual presentation to index item different in the case of multi-source also is provided for data analyst simultaneously, is conducive to multi-source correlation
Observation and analysis.As shown in figure 4, steps are as follows:
1, analyst wants the log field (such as system.process.memory.size) of analysis, polymerization by selection
Index (such as counting, summation, maximum value, minimum value, average value) fills in customized legend title, and configuration screening rule is (such as
Host.name must equal master represents data source and is necessary for server master)
2, addition field is clicked, configures next log field, log field name can be identical as step 1, and selective polymerization refers to
Mark (such as counting, summation, maximum value, minimum value, average value) fills in customized legend title, and configuration screening rule is (such as
Host.name must equal slave1 represents data source and is necessary for server slave1).It is broadly divided into two kinds of scenes:
Same log field under not homologous is visual；Different log fields under not homologous are visual.
3, the configuration based on Y-axis dimension repeats the above steps.
4, the configuration based on X-axis measurement: filling in customized coordinate interval, select spacing dimension (such as the second, point, when), it is automatic
Show log item number.
5, above-mentioned all querying conditions are converted into data object, pass it to corresponding interface, inquiry database obtains
To the initial data for generating chart, the visualized data structure of unified format is formed by the processing of Data Translation model, then is led to
The conversion that view transformation model carries out data direction view is crossed, final rendering is presented to the user.
The visualization of multi objective item is fused to data analyst and provides visual presentation to multiple index item under the conditions of homologous,
It observes while conducive to associated index, in actual production environment, is referred specifically to conducive to difference under certain big data cluster
The same node comparative analysis of target；It is provided simultaneously for data analyst and indexs is polymerize to the different of same index item under the conditions of homologous
The visual presentation of dimension is conducive to comparative observation and analyzes, as memory (master_hdfs_memory) is at the appointed time spaced
The comparative analysis of maximum usage amount, minimum usage amount, average usage amount.As shown in figure 5, steps are as follows:
1, analyst wants the log field (such as system.process.memory.size) of analysis, polymerization by selection
Index (such as counting, summation, maximum value, minimum value, average value) fills in customized legend title, and configuration screening rule is (such as
Host.name must equal master represents Hostname and is necessary for master)
2, addition field is clicked, configures the log field of next desired analysis (such as
System.process.cpu.total.value), selective polymerization index (such as counting, summation, maximum value, minimum value, is averaged
Value), customized legend title is filled in, (such as host.name must equal master represents Hostname to configuration screening rule
It is necessary for master).It is broadly divided into two kinds of scenes: the same log field under difference polymerization index dimension；Same polymerization index
Different log fields under dimension.
3, the configuration based on Y-axis dimension repeats the above steps
4, the configuration based on X-axis measurement: filling in customized coordinate interval, select spacing dimension (such as the second, point, when), it is automatic
Show log item number
5, above-mentioned all querying conditions are converted into data object, pass it to corresponding interface, inquiry database obtains
To the initial data for generating chart, the visualized data structure of unified format is formed by the processing of Data Translation model, then is led to
The conversion that view transformation model carries out data direction view is crossed, final rendering is presented to the user.
The fusion of dynamic class figure is mainly fused to data analyst by view classification and provides visual convenience, supporting pillar
Shape figure, line chart, pie chart, the dynamic combined two-by-two between map.Composite type can be column folding fusion figure, the fusion of map cylindricality
Figure, map pie fusion figure etc., concrete type is determined according to log achievement data dimension.As shown in fig. 6, steps are as follows:
1, analyst wants the log field of analysis, polymerization index (such as counting, summation, maximum value, minimum by selection
Value, average value), fill in customized legend title, the subtype that selection matches with this log field dimension, configuration screening rule
Then (such as host.name must equal master represent Hostname be necessary for master or other)
2, click addition field, the log field of the next desired analysis of configuration, selective polymerization index (as counted, summing,
Maximum value, minimum value, average value), fill in customized legend title, the chart class that selection matches with this log field dimension
Type, configuration screening rule (such as host.name must equal master represent Hostname be necessary for master or other).
3, the configuration based on X-axis measurement: filling in customized coordinate interval, select spacing dimension (such as the second, point, when), it is automatic
Show log item number
5, above-mentioned all querying conditions are converted into data object, pass it to corresponding interface, inquiry database obtains
To the initial data for generating chart, the visualized data structure of unified format is formed by the processing of Data Translation model, then is led to
The conversion that view transformation model carries out data direction view is crossed, final rendering is presented to the user.
During above-mentioned visualization is merged and realized, solve by Data Translation model due to multi-source or homologous situation
Data disunity problem caused by lower index is different.The premise that multi-class data is shown in a chart in Echarts is dimension
Degree and the data entry of measurement need being consistent property and ARRAY format are uniformly processed into, but in multi-source or multi objective
In the case where, there is very big randomness in data entry, it solves data entry and format consistency technical problems.Side
Case is as follows:
1, will appear data entry in the case where multi-source is same or different log indexs or homologous different log index scenes to pass
Formula inconsistency is returned by the way that respective queries condition is transmitted to corresponding API in API with the difference of the entitled unique identification of legend
Log achievement data, while front end can safeguard oneself filled in when dynamic legend title array, i.e. dynamic addition log field
Legend title is defined, such daily record data is formed a kind of front and back end mapping relations with the entitled unique identification of legend.
2, dimension axis data consistency: traversal API returns to each daily record data and legend title array, safeguards a number
According to object, attribute is in object with the entitled KEY of legend, and the array formed with all daily record datas under the legend title is VALUE.
If a daily record data under a certain legend title is undefined, it is assigned a value of first two of current log data
The average value of daily record data.
3, measure axis data consistency: the data of dimension and measurement need stringent one-to-one relationship that could express correctly
Semanteme, one array of measurement level data maintenance, storage with the dimension number of axle according to sequence strictly corresponding timing class data.
4, reach data entry and format consistency effect, obtain visualized data structure.
Data source input is as follows:
Consistency output is as follows:
Further, during above-mentioned visualization is merged and realized, visualized data is connected to by view transformation model
Structure and visualization Option, are completed by calling from the Dchart plug-in unit ground.The plug-in unit abstract parameter includes selector, number
According to subtype, if in real time etc., support that subtype includes that (line chart, scatter plot, stacks figure, cake at histogram to multivalue figure
Figure, circular chart), chart for individual values (chart of percentage comparison, text & digitized map), fusion figure (column cake fusion figure, column folding fusion figure, folding cake fusion
Figure) and table.Each mapping function (subfunction) includes obtaining chart storing containers DOM, initializes Echarts example, setting
Customized Option is transmitted to Echarts example by setOption API, returns to customized Option by customized Option.
Switch grammer is used in father's function, and each mapping function is called by decision chart type respectively.Wherein, table 1 is chart
Type list, pseudocode are as follows:
Table 1
Heuristic figure integration technology combines the functions such as data calculating, data screening, data visualization, chart be configurable.
Data calculating refers to that providing counting, average value, summation, maximum value, minimum value etc. for log index polymerize computing function, poly- using it
For value after conjunction as the data for generating chart, observation calculates the correlation of log index.Data screening is provided daily record data
Powerful Filtering system, sub- grade rule regular (such as must, must_not, should) by selection log index, parent
(equal, gte, lte, regexp) fills in specific screening conditions, is then converted into the screening rule of JSON format as parameter
It is transmitted to corresponding API, to change the data source for generating chart.Data visualization is supported in the basis system for meeting data dimension characteristic
Chart, fusion statistical graph, unrestricted choice in monodrome statistical chart three categories chart are counted, and supports to switch between chart and Table.
The configurable change supported to different Y-axis value colors of chart, change coordinate Spindle Name, legend title etc., make the chart generated more
Add simple, clear, beautiful.
Above-mentioned heuristic figure integration technology substantially by the interactive operations such as clicking, selecting realize, greatly reduces use
To the threshold of daily record data visual analysis, the type for generating data drawing list can also be according to intelligent recommendation free switching, greatly at family
Improve user carry out data analysis efficiency.
Further, in one embodiment of the invention, the method for the embodiment of the present invention further include: receive active user
Operational order；Free customization is carried out according to operational order, to realize visual control.
It is understood that as shown in Fig. 2, the embodiment of the present invention by data visualization merge in the log situation map that generates
It is added in the specified Log Report under specified folder, free typesetting is provided, is refreshed in real time, data source migration, dynamic carousel
Etc. functions.
Specifically, analyst can as shown in Fig. 2, visual control provides situation map free customizing mechanism for analyst
To carry out situation map rearrangement and free extension by pulling, clicks preservation and be transmitted to the location information after dragging by parametric form
The corresponding interface realizes layout persistence；Template mechanism is provided based on the transportable principle of data source, by selecting different days
Will index, is shown the switching of data, does not need to be mapped again to realize multiplexing, improve efficiency；For each situation map
It supports data dynamic carousel, realizes dynamic, data are subjected to floating layer displaying during carousel, be conducive to user and observe；It provides
It, can be according to the log monitoring report of the aesthetic formation different style of analyst to the accurate editable function of each situation map.
To sum up, the embodiment of the present invention is based on heuristic figure integration technology, will visit on the basis of existing visualization system
Cable-styled analytical technology is combined with data visualization technique, and extremely simple interactive mode, operation are devised with the thinking of ordinary user
Threshold is low, and non-technical personnel can also using while learning.The embodiment of the present invention from obtain daily record data source, to the exploration of log index
Formula, figure fusion visualization arrive be added to Log Report again, specifically: establishing data transmission channel with Log Source；Log is added to receive
Storage is simultaneously distributed to collector manager；Collector index is obtained, type, monitoring time section etc. is for dividing daily record data
Analysis；Multi-source fusion is analyzed between Log Source, convergence analysis between same Log Source index；Visualization mapping, is added to log report
It accuses.
That is, the embodiment of the present invention meets the behavioural habits of most of user by friendly extremely simple interactive mode.
After log concentrator is distributed to multiple or single collector manager, pass through selection log index, Log Types, when monitoring
Between the search conditions such as section all daily record datas are subjected to two-dimensional table displaying according to log index field name, complete file (molecule)
The multisource data fusion of rank simultaneously carries out table visualization, and provides powerful file-level fused data Filtering system；Screening
After the data for wanting analysis out, realize that multi-source or multi objective term diagram merge by configuring multiple Y-axis, Y-axis concrete configuration index has
Daily record data field, polymerization index, legend title, provide log index field (atom) for the data in each Y-axis dimension
Rank fused data Filtering system；X-axis measurement is the temporal representation of time, and concrete configuration index has coordinate interval, spacing dimension；
The present graphical for being suitble to data dimension feature is finally selected to complete the heuristic figure fusion visualization of log index；And provide addition
To log function of reporting.The embodiment of the present invention construct it is a set of have both data acquisition, data processing, data drawing, Log Report
Intelligent log visible process, provide configurable, a reusable, expansible log monitoring visibility solution,
Have the characteristics that aesthetic feeling, flexible, multi-source.
The heuristic figure fusion visualization method proposed according to embodiments of the present invention, not only can be homologous not with comparative analysis
With the different polymerization dimensions of log index or same log index, the fusion of multi objective item is supported；Also it can show simultaneously not homologous
The same or multiple log index situation, support multi-source fusion；And provide three categories statistical graph: basic statistical chart,
Statistical graph, monodrome statistical graph are merged to meet the displaying characteristic of daily record data, and good friendship is provided during displaying
Mutual property, so that log analysis precision is greatly improved, Quan Miandu, and certain help can be provided business decision.
The heuristic figure fusion visualization device proposed according to embodiments of the present invention is described referring next to attached drawing.
Fig. 7 is the structural schematic diagram of the heuristic figure fusion visualization device of one embodiment of the invention
As shown in fig. 7, the heuristic figure fusion visualization device 10 includes: data acquisition module 100, data processing module
200 and data visualization Fusion Module 300.
Wherein, data acquisition module 100, to acquire data from log push platform, obtains more for sending HTTP request
Tie up daily record data.Data processing module 200 is stored for pre-processing to multidimensional log data to database.Data can
The querying condition for being used to obtain active user depending on changing Fusion Module 300, to inquire pretreated daily record data from database,
And pretreated daily record data is screened according to screening conditions, tentatively to be shown to daily record data, and realize day
The visualization fusion of will heuristic.The device 10 of the embodiment of the present invention greatly improves log analysis precision, Quan Miandu, and can be with
Certain help is provided to business decision.
Further, in one embodiment of the invention, data acquisition module 100 is further used for log-on data source receipts
Collect the service of log, to carry out data collection from multiple dimensions, and acquires log file data and system service index and module
Index, and more new data.
Further, in one embodiment of the invention, data processing module 200 is further used for according to default conversion
Strategy carries out data conversion to multidimensional log data；And/or delete invalid data in multidimensional log data；And/or to multidimensional
The data for meeting preset condition in daily record data increase label；And/or it will be met in multidimensional log data according to preset need and united
The data of meter are counted, and are included in database；And/or the abnormal data of detection multidimensional log data.
Further, in one embodiment of the invention, data visualization Fusion Module 300 was further used for according to day
Will data are to the visual presentation of index item same in the case of multi-source, according to daily record data to multiple index item under the conditions of homologous
It visualizes, is shown by the dynamic combined two-by-two between view classification fusion support histogram, line chart, pie chart, map.
Further, in one embodiment of the invention, the device 10 of the embodiment of the present invention further include: visual control
Module.Wherein, visual control module is used to receive the operational order of active user, and is freely determined according to operational order progress
System, to realize visual control.
It should be noted that the aforementioned explanation to heuristic figure fusion visualization method embodiment is also applied for the reality
The heuristic figure fusion visualization device of example is applied, details are not described herein again.
The heuristic figure fusion visualization device proposed according to embodiments of the present invention, not only can be homologous not with comparative analysis
With the different polymerization dimensions of log index or same log index, the fusion of multi objective item is supported；Also it can show simultaneously not homologous
The same or multiple log index situation, support multi-source fusion；And provide three categories statistical graph: basic statistical chart,
Statistical graph, monodrome statistical graph are merged to meet the displaying characteristic of daily record data, and good friendship is provided during displaying
Mutual property to greatly improve log analysis precision, Quan Miandu, and can provide business decision certain help.
In addition, term " first ", " second " are used for descriptive purposes only and cannot be understood as indicating or suggesting relative importance
Or implicitly indicate the quantity of indicated technical characteristic.Define " first " as a result, the feature of " second " can be expressed or
Implicitly include at least one this feature.In the description of the present invention, the meaning of " plurality " is at least two, such as two, three
It is a etc., unless otherwise specifically defined.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is included at least one embodiment or example of the invention.In the present specification, schematic expression of the above terms are not
It must be directed to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be in office
It can be combined in any suitable manner in one or more embodiment or examples.In addition, without conflicting with each other, the skill of this field
Art personnel can tie the feature of different embodiments or examples described in this specification and different embodiments or examples
It closes and combines.
Although the embodiments of the present invention has been shown and described above, it is to be understood that above-described embodiment is example
Property, it is not considered as limiting the invention, those skilled in the art within the scope of the invention can be to above-mentioned
Embodiment is changed, modifies, replacement and variant.
Claims (10)
1. a kind of heuristic figure fusion visualization method, which comprises the following steps:
HTTP request is sent, to acquire data from log push platform, obtains multidimensional log data；
The multidimensional log data are pre-processed, and are stored to database；And
The querying condition for obtaining active user, to inquire pretreated daily record data from the database, and according to screening
Condition screens the pretreated daily record data, tentatively to be shown to the daily record data, and realizes log
Heuristic visualization fusion.
2. the method according to claim 1, wherein described acquire data from log push platform, comprising:
The service of log-on data source collector journal, to carry out data collection from multiple dimensions；
Acquire log file data and system service index and module index, and more new data.
3. being wrapped the method according to claim 1, wherein described pre-process the multidimensional log data
It includes:
Data conversion is carried out to the multidimensional log data according to default switching strategy；
And/or delete invalid data in the multidimensional log data；
And/or label is increased to the data for meeting preset condition in the multidimensional log data；
And/or the data for meeting statistics in the multidimensional log data are counted according to preset need, and be included in data
Library；
And/or detect the abnormal data of the multidimensional log data.
4. the method according to claim 1, wherein realization log heuristic visualization fusion, comprising:
According to the daily record data to the visual presentation of index item same in the case of multi-source；
According to the daily record data to the visual presentation of multiple index item under the conditions of homologous；And
It is shown by the dynamic combined two-by-two between view classification fusion support histogram, line chart, pie chart, map.
5. the method according to claim 1, wherein further include:
Receive the operational order of active user；
Free customization is carried out according to the operational order, to realize visual control.
6. a kind of heuristic figure fusion visualization device characterized by comprising
Data acquisition module, to acquire data from log push platform, obtains multidimensional log data for sending HTTP request；
Data processing module for pre-processing to the multidimensional log data, and is stored to database；And
Data visualization Fusion Module, for obtaining the querying condition of active user, to inquire pretreatment from the database
Daily record data afterwards, and the pretreated daily record data is screened according to screening conditions, to the daily record data
It is tentatively shown, and realizes the visualization fusion of log heuristic.
7. device according to claim 6, which is characterized in that the data acquisition module is further used for log-on data source
The service of collector journal to carry out data collection from multiple dimensions, and acquires log file data and system service index and mould
Block index, and more new data.
8. device according to claim 6, which is characterized in that the data processing module is further used for being turned according to default
It changes strategy and data conversion is carried out to the multidimensional log data；And/or delete invalid data in the multidimensional log data；With/
Or, increasing label to the data for meeting preset condition in the multidimensional log data；And/or it will be described more according to preset need
The data for meeting statistics in dimension daily record data are counted, and are included in database；And/or detect the multidimensional log data
Abnormal data.
9. device according to claim 6, which is characterized in that the data visualization Fusion Module is further used for basis
The daily record data is to the visual presentation of index item same in the case of multi-source, according to the daily record data to more under the conditions of homologous
The visual presentation of a index item passes through moving between view classification fusion support histogram, line chart, pie chart, map two-by-two
State combination is shown.
10. device according to claim 6, which is characterized in that further include:
Visual control module carries out free customization for receiving the operational order of active user, and according to the operational order,
To realize visual control.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910589334.XA CN110489464B (en) | 2019-07-02 | 2019-07-02 | Exploration type graph fusion visualization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910589334.XA CN110489464B (en) | 2019-07-02 | 2019-07-02 | Exploration type graph fusion visualization method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110489464A true CN110489464A (en) | 2019-11-22 |
| CN110489464B CN110489464B (en) | 2022-05-31 |
Family
ID=68546394
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910589334.XA Active CN110489464B (en) | 2019-07-02 | 2019-07-02 | Exploration type graph fusion visualization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110489464B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111324582A (en) * | 2020-02-18 | 2020-06-23 | 支付宝(杭州)信息技术有限公司 | Method and device for visually backtracking and processing business processing behaviors |
| CN111400565A (en) * | 2020-03-19 | 2020-07-10 | 北京三维天地科技股份有限公司 | Visualized dragging online data processing method and system |
| CN112148700A (en) * | 2020-10-12 | 2020-12-29 | 平安科技（深圳）有限公司 | Log data processing method and device, computer equipment and storage medium |
| CN112187550A (en) * | 2020-10-16 | 2021-01-05 | 温州职业技术学院 | Log analysis method based on density peak value multi-attribute clustering |
| CN113961518A (en) * | 2021-09-08 | 2022-01-21 | 北京百度网讯科技有限公司 | Log visual display method and device, electronic equipment and storage medium |
| CN114448672A (en) * | 2021-12-27 | 2022-05-06 | 奇安信科技集团股份有限公司 | Multi-source network security data processing method and device |
| CN114567498A (en) * | 2022-03-04 | 2022-05-31 | 科来网络技术股份有限公司 | Metadata extraction and processing method and system for network behavior visualization |
| CN114860734A (en) * | 2022-05-27 | 2022-08-05 | 河北省科学技术情报研究院(河北省科技创新战略研究院) | Processing method for data presentation of multi-source index structure fusion and scene restoration |
| CN115801262A (en) * | 2022-08-31 | 2023-03-14 | 重庆市规划和自然资源信息中心 | Intersection operator space retrieval method based on ElasticSearch technology |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2918243A1 (en) * | 2013-06-28 | 2014-12-31 | Life Technologies Corporation | Methods and systems for visualizing data quality |
| CN109376532A (en) * | 2018-10-31 | 2019-02-22 | 云南电网有限责任公司 | Power network security monitoring method and system based on the analysis of ELK log collection |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
| CN109902072A (en) * | 2019-02-21 | 2019-06-18 | 云南电网有限责任公司红河供电局 | A kind of log processing system |
-
2019
- 2019-07-02 CN CN201910589334.XA patent/CN110489464B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2918243A1 (en) * | 2013-06-28 | 2014-12-31 | Life Technologies Corporation | Methods and systems for visualizing data quality |
| CN109376532A (en) * | 2018-10-31 | 2019-02-22 | 云南电网有限责任公司 | Power network security monitoring method and system based on the analysis of ELK log collection |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
| CN109902072A (en) * | 2019-02-21 | 2019-06-18 | 云南电网有限责任公司红河供电局 | A kind of log processing system |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111324582A (en) * | 2020-02-18 | 2020-06-23 | 支付宝(杭州)信息技术有限公司 | Method and device for visually backtracking and processing business processing behaviors |
| CN111400565A (en) * | 2020-03-19 | 2020-07-10 | 北京三维天地科技股份有限公司 | Visualized dragging online data processing method and system |
| CN112148700A (en) * | 2020-10-12 | 2020-12-29 | 平安科技（深圳）有限公司 | Log data processing method and device, computer equipment and storage medium |
| CN112187550B (en) * | 2020-10-16 | 2022-09-30 | 温州职业技术学院 | Log analysis method based on density peak value multi-attribute clustering |
| CN112187550A (en) * | 2020-10-16 | 2021-01-05 | 温州职业技术学院 | Log analysis method based on density peak value multi-attribute clustering |
| CN113961518A (en) * | 2021-09-08 | 2022-01-21 | 北京百度网讯科技有限公司 | Log visual display method and device, electronic equipment and storage medium |
| CN114448672A (en) * | 2021-12-27 | 2022-05-06 | 奇安信科技集团股份有限公司 | Multi-source network security data processing method and device |
| CN114567498A (en) * | 2022-03-04 | 2022-05-31 | 科来网络技术股份有限公司 | Metadata extraction and processing method and system for network behavior visualization |
| CN114567498B (en) * | 2022-03-04 | 2024-02-02 | 科来网络技术股份有限公司 | Metadata extraction and processing method and system for network behavior visualization |
| CN114860734A (en) * | 2022-05-27 | 2022-08-05 | 河北省科学技术情报研究院(河北省科技创新战略研究院) | Processing method for data presentation of multi-source index structure fusion and scene restoration |
| CN114860734B (en) * | 2022-05-27 | 2022-11-15 | 河北省科学技术情报研究院(河北省科技创新战略研究院) | Processing method for data presentation of multi-source index structure fusion and scene reduction |
| CN115801262A (en) * | 2022-08-31 | 2023-03-14 | 重庆市规划和自然资源信息中心 | Intersection operator space retrieval method based on ElasticSearch technology |
| CN115801262B (en) * | 2022-08-31 | 2023-07-18 | 重庆市规划和自然资源信息中心 | Intersection operator space retrieval method based on elastic search technology |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110489464B (en) | 2022-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110489464A (en) | Heuristic figure fusion visualization method and device | |
| US11386156B1 (en) | Threshold establishment for key performance indicators derived from machine data | |
| US11238033B1 (en) | Interactive location queries for raw machine data | |
| CN108804513A (en) | Automatic visual analysis method for big data platform | |
| CN103532739B (en) | A kind of monitoring analysis system based on network service with application | |
| CN104794113B (en) | Data processing method and device | |
| CN109254901B (en) | A kind of Monitoring Indexes method and system | |
| CN110442550B (en) | Log screen-gathering real-time visualization method and device | |
| CN103902537A (en) | Multi-service log data storage processing and inquiring system and method thereof | |
| CN114791846B (en) | Method for realizing observability aiming at cloud-originated chaos engineering experiment | |
| CN102156739A (en) | GIS (Geographic Information System) platform processing method for mass lightning data | |
| CN106547622B (en) | Network resource situation presentation data optimization processing method based on computing module | |
| Li et al. | A distributed parallel alarm management strategy for alarm reduction in chemical plants | |
| CN108710347A (en) | A kind of monitoring cloud platform | |
| CN105956141A (en) | Medicine data storage method based on Internet of things | |
| CN111400278B (en) | Method and system for constructing multi-level target crowd based on dragging of labels | |
| CN107515913A (en) | A kind of multivariate data model integrated construction method and its virtual interactive interface system | |
| CN110032560B (en) | Method and device for generating monitoring chart | |
| Farid et al. | Implementing Advanced Visualization Techniques for CIT Business Intelligence. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |