Invention content
In view of the defects existing in the prior art, the present invention, which provides a kind of visualization data and collects, shows system, can be effective
It solves the above problems.
The technical solution adopted by the present invention is as follows:
The present invention, which provides a kind of visualization data and collects, shows system, including:
Data source definitions module, for defining different types of data source;
Data source connects configuration module, and the connection for setting the data source of each data source definitions module definition is matched
Confidence ceases, and by the connection configuration information, may have access to the corresponding data source;
Data source link block for the connection configuration information according to the data source, passes through with the data source and communicates
Network is attached;
Tables of data catalogue acquisition module after being attached with the data source by communication network, is got described
The tables of data catalogue for all tables of data that data source includes;The tables of data catalogue is by multiple tables of data title forms;
Tables of data previewing module, for after the tables of data catalogue is got, for each data table name, to
The data source transmission carries the tables of data preview request of data table name;And receive the corresponding number that the data source returns
According to table, tables of data preview function is realized;
Data model creation module, for creating data model;Specifically, for the tables of data catalogue acquisition module
The multiple data table names got are operated, and selection needs to establish at least two tables of data of table incidence relation;Allocation list
The type of incidence relation;Then, the literary name section of every tables of data chosen is obtained;It is reconfigured the attribute letter of each literary name section
Breath;The attribute information includes the attribute of literary name section show or hide;Then, it establishes between each tables of data between literary name section
Relevance, the relevance between table incidence relation and literary name section form the incidence relation between final tables of data, that is, are formed
Data model;
Data acquisition configuration module, for configuration data acquisition parameter;The data acquisition parameters are acquired including data
Begin time and data acquiring frequency;
Data storage method configuration module, for configuration data storage mode;
Execution module, for pressing the data acquisition initial time of the data acquisition configuration module configuration and data acquisition
Frequency, data model described in recursive call obtain every data table name in the data model with table incidence relation, then
Obtain the associate field of every tables of data in the data model with the relevance between literary name section;The data are obtained again
The attribute information of the type of table incidence relation and literary name section in model;
According to the attribute information of the data table name, associate field, the type of table incidence relation and literary name section, to institute
It states data source and sends corresponding not hiding object table field information and not hiding target literary name section in inquiry corresponding data table
The token record information having, that is, inquire the data table information in corresponding data table after Screening Treatment;
Then, according to the type of the table incidence relation, each data after the Screening Treatment that the data source is returned
Table information is merged, the tables of data after being merged;And by the data storage of data storage method configuration module configuration
Mode, the tables of data after storage fusion;
Display module for the tables of data after the fusion to be analyzed, and shows analysis result.
Preferably, the different types of data source of the data source definitions module definition, including:File type data source,
Web interface categorical data source, database type data source and data service type data source.
Preferably, the file type data source includes:Excel file type data sources and csv file type data source;
The web interface categorical data source includes:Webservice interface types data source, http interface types data source, Splunk
Interface type data source and RESTful interface type data sources;The database type data source includes:Oracle categorical datas
Source, MySql categorical datas source, SQLServer categorical datas source, DB2 categorical datas source, PostgreSQL categorical datas source,
SyBase categorical datas source, Informix categorical datas source and H2 categorical datas source;The data service type data source includes
VBI data reception service categorical datas source.
Preferably, the connection configuration information of each data source that the data source connection configuration module is configured includes:
For file type data source, the corresponding file in the file type data source is parsed, and will be after parsing
File data imported into a new data table of database;The connection configuration information of the data source includes:New data table
Title and the storage new data table database address;
For web interface categorical data source, connection configuration information includes:The link address in web interface categorical data source
Information;
For database type data source, connection configuration information includes:Database-name, host address, port numbers with
And the username and password for accessing database;
For data service type data source, connection configuration information includes:Data service title and data service address
Information.
Preferably, the type of the table incidence relation includes:Left connection, right connection and interior connection.
Preferably, the data storage method includes:Real-time section storage mode covers storage mode, accumulates and go every time
Weight storage mode accumulates not duplicate removal storage mode and accumulation and updates acquisition time storage mode.
Preferably, the execution module is specifically used for:
Assuming that every data table name with table incidence relation is respectively tables of data A and tables of data B in data model;Number
According to n field is shared in Table A, n1 field is to hide attribute not hide attribute, n-n1 field;M are shared in tables of data B
Field, m1 field are to hide attribute not hide attribute, m-m1 field;It needs to establish in tables of data A not in data model
Hide the association do not hidden in field C and tables of data B in attribute field in attribute field between field D;Each word in tables of data A
Section has x record value;Each field has y record value in tables of data B;Assuming that the table incidence relation being currently configured is left company
It connects;Tables of data A is current data table;Tables of data B is the left data table of tables of data A;
Then:When reaching data acquisition time point, execution module obtains tables of data B to corresponding data source gathered data
M1 do not hide the fields of attributes and y record value of each field for not hiding attribute, be denoted as sublist B;Number is obtained again
According to the corresponding x record value of the field C of Table A;The x of the y record value of the field D of comparison data table B and the field C of data Table A
A record value, screening obtain the record value of the field C of the tables of data A equal with the record value of the field D of tables of data B, it is assumed that sieve
It is respectively the i-th row record value and jth row record value to select the obtained record value of the field C of tables of data A;Then, the execution mould
Block sends all record values for not hiding fields of the i-th row of inquiry tables of data A to corresponding data source and jth row is all does not hide
The record value of field, and the query result of data source return is received, query result is denoted as sublist A;
Then, the sublist B and sublist A that the execution module fusion is got, obtain and store the tables of data after fusion.
Visualization data provided by the invention, which collect, to be showed system and has the following advantages:
Visualizing rear end acquisition module can use java technologies to be directly associated A tables and B tables in gathered data
The fused data after association is obtained afterwards, and modeling can be associated between various types data source, applied widely, behaviour
Make simply, volume of transmitted data is small, consumption resource is small and easy to use.
Specific embodiment
In order to which technical problem solved by the invention, technical solution and advantageous effect is more clearly understood, below in conjunction with
Accompanying drawings and embodiments, the present invention will be described in further detail.It should be appreciated that specific embodiment described herein only to
It explains the present invention, is not intended to limit the present invention.
In the traditional forms of enterprises, if interface goes out associated data or goes to take the data of interface immediately with interface conjunctionn,
It will go to be customized exploitation for this interface, then data are shown again, if the figure variation of displaying or category
Property variation, it is also necessary to change code, take time and effort.The present invention can be abstracted the data-interface of most of type inside enterprise, make
Data between interface and interface can carry out being associated in field, also, need to only change the page shows attribute, does not need to repair
Change code, basic zero code is suitble to staff to use operation.
The present invention provides a kind of visualization data and collects and shows system, with reference to figure 1, including:
Data source definitions module, for defining different types of data source;Different types of data source includes but not limited to:
File type data source, web interface categorical data source, database type data source and data service type data source.
Wherein, file type data source includes but not limited to:Excel file type data sources and csv file type data
Source;
The web interface categorical data source includes but not limited to:Webservice interface types data source, http interface classes
Type data source, Splunk interface types data source and RESTful interface type data sources;
The database type data source includes but not limited to:Relevant database data source and column storage database;
Wherein, relevant database data source includes but not limited to Oracle categorical datas source, MySql categorical datas source, SQLServer
Categorical data source, DB2 categorical datas source, PostgreSQL categorical datas source, SyBase categorical datas source, Informix number of types
According to source and H2 categorical datas source;
The data service type data source includes but not limited to VBI data reception service categorical datas source.
Data source connects configuration module, and the connection for setting the data source of each data source definitions module definition is matched
Confidence ceases, and by the connection configuration information, may have access to the corresponding data source;
For example, for file type data source, the corresponding file in the file type data source is parsed, and will solution
File data after analysis is imported into a new data table of database;The connection configuration information of the data source includes:New number
According to the title of table and the database address of the storage new data table;
For web interface categorical data source, connection configuration information includes:The link address in web interface categorical data source
Information;
For database type data source, connection configuration information includes:Database-name, host address, port numbers with
And the username and password for accessing database;
For data service type data source, connection configuration information includes:Data service title and data service address
Information.
Data source link block for the connection configuration information according to the data source, passes through with the data source and communicates
Network is attached;
Tables of data catalogue acquisition module after being attached with the data source by communication network, is got described
The tables of data catalogue for all tables of data that data source includes;The tables of data catalogue is by multiple tables of data title forms;
Tables of data previewing module, for after the tables of data catalogue is got, for each data table name, to
The data source transmission carries the tables of data preview request of data table name;And receive the corresponding number that the data source returns
According to table, tables of data preview function is realized;
Data model creation module, for creating data model;Specifically, for the tables of data catalogue acquisition module
The multiple data table names got are operated, and selection needs to establish at least two tables of data of table incidence relation;Allocation list
The type of incidence relation, the type of table incidence relation include but not limited to left connection, right connection and interior connection;Then, choosing is obtained
The literary name section of every tables of data selected;It is reconfigured the attribute information of each literary name section;The attribute information is shown including literary name section
The attribute for showing or hiding;In addition, for literary name section, it is also an option that modification field needs title to be shown, and specific field
Index;Then, the relevance between literary name section, the relevance between table incidence relation and literary name section between each tables of data are established
The incidence relation between final tables of data is formd, that is, the data model formed;
Data acquisition configuration module, for configuration data acquisition parameter;The data acquisition parameters are acquired including data
Begin time and data acquiring frequency;
Data storage method configuration module, for configuration data storage mode;The data storage method includes:It is disconnected in real time
Face storage mode covers storage mode, accumulation duplicate removal storage mode, accumulates not duplicate removal storage mode and accumulation and update and adopt every time
Collect time storage mode.
Execution module, for pressing the data acquisition initial time of the data acquisition configuration module configuration and data acquisition
Frequency, data model described in recursive call obtain every data table name in the data model with table incidence relation, then
Obtain the associate field of every tables of data in the data model with the relevance between literary name section;The data are obtained again
The attribute information of the type of table incidence relation and literary name section in model;
According to the attribute information of the data table name, associate field, the type of table incidence relation and literary name section, to institute
It states data source and sends corresponding not hiding object table field information and not hiding target literary name section in inquiry corresponding data table
The token record information having, that is, inquire the data table information in corresponding data table after Screening Treatment;
Then, according to the type of the table incidence relation, each data after the Screening Treatment that the data source is returned
Table information is merged, the tables of data after being merged;And by the data storage of data storage method configuration module configuration
Mode, the tables of data after storage fusion;
Execution module and each Data Source communication, getting the mode of query messages can be:Execution module is connected by jdbc
All relevant databases are connect, and different SQL statements is generated according to relational database difference and fetches data, other number of types
Data are individually fetched according to source, the thought progress according still further to SQL is interrelated.
The execution module is specifically used for:
Assuming that every data table name with table incidence relation is respectively tables of data A and tables of data B in data model;Number
According to n field is shared in Table A, n1 field is to hide attribute not hide attribute, n-n1 field;M are shared in tables of data B
Field, m1 field are to hide attribute not hide attribute, m-m1 field;It needs to establish in tables of data A not in data model
Hide the association do not hidden in field C and tables of data B in attribute field in attribute field between field D;Each word in tables of data A
Section has x record value;Each field has y record value in tables of data B;Assuming that the table incidence relation being currently configured is left company
It connects;Tables of data A is current data table;Tables of data B is the left data table of tables of data A;Wherein, n, n1, m, m1 are natural number;
Then:When reaching data acquisition time point, execution module obtains tables of data B to corresponding data source gathered data
M1 do not hide the fields of attributes and y record value of each field for not hiding attribute, be denoted as sublist B;Number is obtained again
According to the corresponding x record value of the field C of Table A;The x of the y record value of the field D of comparison data table B and the field C of data Table A
A record value, screening obtain the record value of the field C of the tables of data A equal with the record value of the field D of tables of data B, it is assumed that sieve
It is respectively the i-th row record value and jth row record value to select the obtained record value of the field C of tables of data A;Then, the execution mould
Block sends all record values for not hiding fields of the i-th row of inquiry tables of data A to corresponding data source and jth row is all does not hide
The record value of field, and the query result of data source return is received, query result is denoted as sublist A;
Then, the sublist B and sublist A that the execution module fusion is got, obtain and store the tables of data after fusion.
During the principles illustrated of above-mentioned execution module, described by taking left connection as an example.Certainly, in practical application,
Also right connection and interior connection, realization principle and left connection are essentially identical, if it is right connection, return to the institute in right table
There is the record value of the field for not hiding attribute and each field for not hiding attribute, form a sublist 1;It is false for left-handed watch
If needing to establish the incidence relation in field a and right table between field b in left-handed watch, then compare in left-handed watch the record value of field a and
The record value of field b in right table, screening obtain the record value of the field a equal with the record value of field b in right table, example
Such as, it is the 1st record value;Then, all record values for not hiding field of the first row in left-handed watch are obtained to data source, forms one
Sublist 2;Sublist 2 and sublist 1 are merged again.
For interior connection, it is assumed that need to establish the incidence relation in field a and table B between field b in Table A, first
The record value of field a in Table A is only obtained, then obtains the record value of field b in table B, compares and obtains the equal row of record value, example
Such as, the 2nd row record value of field a is equal with the 3rd row record value of field b in table B in Table A, then is obtained the in Table A to data source
All record values for not hiding field of 2 rows, form sublist 1;All record values for not hiding field of the 3rd row in table B, shape are obtained again
Into sublist 2;Sublist 2 and sublist 1 are merged again.
The implementation principle of execution module can be briefly described for:Execution module according to timing configured recycle to data model
Content is parsed, according to the corresponding data source information of tables of data inside data model, into corresponding data source according to existing
Rule be acquired data, be then associated according to the associate field of data source and data source with data filtering (according to master
Table and the associate field of sublist carry out recycle ratio pair, are combined according to incidence relation is equal, are finally synthesizing one group of data), add
Enter screening conditions, the data of return are processed further according to field information defined in model, finally treated data
It is stored in an independent table.
Display module for the tables of data after the fusion to be analyzed, and shows analysis result.
In Fig. 1,13 steps are described altogether, and each step is described below:
Step 1:Data source is defined, fills in the information of data source;
Step 2:Define the connection configuration information of data source;
Step 3:Data modeling;
Step 4:Define data model;
Step 5-9:Data model is parsed according to the content to data model that timing configured recycles, according in model
Table corresponding data source information in face data source goes in each data source to be acquired data according to existing rule, then according to number
It is associated according to the associate field in source and data source and (recycle ratio is carried out according to main table and the associate field of sublist with data filtering
Right, be combined according to incidence relation is equal, be finally synthesizing one group of data), add in screening conditions, the data of return further according to
Field information is processed defined in model, and finally treated, data are stored in an independent table.
Step 10:Timer-triggered scheduler task, by data model preserve scheduler task information, execution module will automatically according to
The timing of scheduler task information goes to call data model function, and timing returned data information is stored in database, while is also notified that
Multidimensional analysis engine removes the table of this preservation of data base querying, and is inquired according to the querying condition of some control in picture.
Step 11:Data in the data model information inquiry local data base bound according to front end, and it is pushed to multidimensional
Analysis engine, at present local data base only support MySql and oracle database.
Step 12:Multidimensional analysis, multidimensional analysis, which refers to be put into space coordinate more than a bidimensional, to be analyzed.Mainly
It is when analysis is established in picture, to be handled, including the conversion to field, and pass through ZMQ and push data into front end.
Step 13:ZMQ is a transport layer, is a Message Processing bank of queues, can be in multiple threads, kernel and host box
Between elastic telescopic.
ZMQ points are three patterns:Answer model subscribes to release model and based on distributed treatment, and the present invention uses subscription
Release model.Publishing side is unidirectional transmission data, and is indifferent to and whether whole information is all sent to subscription end.If
When publishing side starts to release news, subscription end not yet successful connection, these information directly abandon.But, once subscription end
Successful connection, centre can ensure that no information is lost.Equally, subscription end is then only responsible for reception, and cannot feed back.If publishing side
Interaction (such as on confirming whether subscriber has connected) is needed with subscription end, then is used and asked back using additional socket
Model is answered to meet this demand.
Visualization data provided by the invention collect and show system, before data are showed, can from various real-time interfaces,
Data, and preparatory processing data are obtained in the data formats such as relevant database, file, columnar database, in modeling, to not
With data modeling, data correlation is carried out between the data in source, then condition filter carries out multidimensional analysis or is united by mathematics
PC large-size screen monitors, WEB page or cell phone application application are pushed to when meter is by fructufy to be shown.
Visualization data provided by the invention, which collect, shows system, has the following advantages:
Visualizing rear end acquisition module can use java technologies to be directly associated A tables and B tables in gathered data
The fused data after association is obtained afterwards, and modeling can be associated between various types data source, applied widely, behaviour
Make simply, volume of transmitted data is small, consumption resource is small and easy to use.
The above is only the preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications also should
Depending on protection scope of the present invention.