CN116225381A - Multi-data source access method and system for data visualization - Google Patents

Abstract

The invention belongs to the technical field of data visualization, and particularly relates to a multi-data source access method and system for data visualization, comprising the following steps: obtaining a visual component of a visual large screen; designing the type of the data source of the visual component accessing the data source, and determining the accessing mode of each type of data source; selecting a data source type of an access data source for the visual component, configuring data source information, and acquiring a data set; binding a data set to the visualization component, and converting data in the data set according to the data requirement of the visualization component; the data source type comprises at least two data types of JSON static data type, FILE data type, JDBC data type, API interface data type and SCRIPT data type; therefore, the invention solves the problems of low development work efficiency and high cost when the data visualization function is realized in the prior art.

Description

Multi-data source access method and system for data visualization

Technical Field

The invention belongs to the technical field of data visualization, and particularly relates to a multi-data source access method and system for data visualization.

Background

Data visualization (english : data visualization) is viewed by many disciplines as a modern concept with the same meaning as visual communication. It relates to the creation and study of visual representations of data.

For clear and efficient transfer of information, data visualization uses statistical graphs, charts, information charts, and other tools. The digital data may be encoded using points, lines or stripes to visually convey quantitative information. An effective visualization may help the user analyze and infer data and evidence. It makes complex data easier to understand and use. The user may have a particular analysis task (e.g., making a comparison or understanding causal relationships) and the graphical design rules that the task follows. Tables are typically used for users to find specific metrics, while various types of charts are used to display patterns or relationships in the data of one or more variables.

Data visualization is both an art and a science. Some consider it a branch of descriptive statistics, but some consider it a root-taking theoretical development tool. The increase in the amount of data generated by internet activity and the increase in the number of sensors in the environment is referred to as big data or the internet of things. Processing, analyzing, and communicating such data is an analytical challenge for data visualization.

The data visualization is about the research of the visual expression form of the data, the visual expression form of the data is also developed along with the continuous rapid development of the industry, and compared with the traditional chart and the BI data instrument panel, the data visualization effect is presented in a more time-efficient, diversified, visual and vivid mode, and is an urgent requirement in daily work; but the presentation of the data visualization effect also shows one side of its non-reusability. The existing data visualization mostly adopts a disposable design, the data source of the presentation effect is specific, the data type is specific, once the data source changes, reduces or increases the data type, the originally developed data visualization delivery object cannot truly reflect the data value of the data visualization delivery object, and data research personnel are required to re-customize and develop the data visualization component corresponding to the data type for the new data source data type, so that the access processing supporting the data type is realized, the period is long, the workload is large, and the problems of low development efficiency and high cost are caused.

For example, the visualized data are the same for different clients, but the data sources are different, so that the modification is always performed for the data sources of the clients in the prior art, and extra workload is brought, which results in low working efficiency.

On the other hand, the data source types are different at different stages of the visual delivery of data, such as:

1) In the development phase, the visualization component is typically accessing static JSON data;

2) In the test phase, the visual component is dynamic data in the test database environment;

3) In the delivery implementation phase, the data source of the visualization component may be low-code data in a data set in a data warehouse, or may be SQL data, or may be websocket data or MQTT real-time data.

In summary, at present, aiming at different data sources, data research personnel are required to re-customize and develop the corresponding data type of the data visualization component for the new data source data type again so as to realize the access processing supporting the data type, which results in long development period and large workload, namely, the problems of low development efficiency and high cost when the data visualization function is realized due to the few data sources supported in the prior art.

Disclosure of Invention

The invention aims to provide a multi-data source access method and system for data visualization, which are used for solving the problems of low development work efficiency and high cost when the data visualization function is realized in the prior art.

In order to solve the technical problems, the technical scheme and the corresponding beneficial effects of the technical scheme provided by the invention are as follows:

the invention discloses a multi-data source access method for data visualization, which is characterized by comprising the following steps of:

the method comprises the following steps:

1) Obtaining a visual component of a visual large screen;

2) Designing the type of the data source of the visual component accessing the data source, and determining the accessing mode of each type of data source;

3) Selecting a data source type of an access data source for the visual component, configuring data source information, and acquiring a data set;

4) Binding a data set to the visualization component, and converting data in the data set according to the data requirement of the visualization component;

the data source type comprises at least two data types of JSON static data type, FILE data type, JDBC data type, API interface data type and SCRIPT data type; the API interface data is used for acquiring the data to be requested in an interface mode according to the basic information of the data to be requested so as to establish a data set; the SCRIPT data is data obtained by obtaining data to be requested in a JavaScript and/or Python SCRIPT mode and carrying out secondary processing on the obtained data to be requested according to service requirements, and is used for establishing a data set.

The beneficial effects of the technical scheme are as follows: when the data visualization function is realized, the invention supports multiple data type access; the invention only needs to configure and support various forms such as a relational database, a system interface service, a memory database, a static file, static data and the like once according to the data requirement of the visual component, and when the data visualization is realized, the data of the type can be obtained by selecting the data source type and configuring the data source information without excessive modification of developers. Based on heterogeneous data source integration, each business system of an enterprise can be easily accessed, workload of workers is reduced, development cost and labor cost for manufacturing a large screen are reduced, development efficiency is improved, and data island is thoroughly broken.

Further, to support multiple data sources, converting the data in the data set in step 4) includes: cutting the data in the data set according to the target data required in the visual component; and performing type conversion on the data in the data set according to the data type of the target data required in the visualization component.

Further, the API interface data type is accessed in the following manner:

selecting a data model, creating a data set, configuring API interface basic information, and obtaining data requested by an API interface for generating a corresponding data set; the API basic information comprises a request address, a request mode and request data; the request data includes a request header, a request parameter, and a request body.

Further, in order to improve flexibility of the presentation data, the SCRIPT data type data source is accessed in the following manner: selecting a data model, creating a data set, capturing data by adopting a script, and associating the data obtained by script processing with a template variable to obtain data for generating the data set.

Further, the clipping data is performed by setting the template variable.

Further, the JSON static data type data source is accessed in the following manner: selecting a JSON static data type, and compiling chart data in an editing area, wherein the data format is determined according to the data format in the corresponding visual component; storing the written chart data and generating a data set corresponding to the JSON static data;

the FILE data type data source is accessed in the following way: the data source FILE corresponding to the FILE data type comprises Excel or CSV; acquiring a selected data source file, establishing a data model, and establishing a corresponding data set;

the JDBC data type data source is accessed in the following manner: creating a JDBC data source, selecting a data source driver, and setting configuration information of a connection database; the configuration information comprises a JDBC source address, a JDBC user name and a JDBC password; verifying the communication condition with the database; after the connectivity verification is passed, JDBC data is obtained, a data model is established, and a corresponding data set is obtained.

Further, the database connected with the JDBC data source is any one of MySQL, SQL Server, postgreSQL, oracle, redis and MongoDB.

Further, the setting of the access mode of the type data source includes setting a refresh frequency of the access data source for refreshing the data according to the set refresh frequency.

In order to solve the above problems, the present invention further provides a multiple data source access system for data visualization, which includes a processor for executing computer instructions to implement a multiple data source access method for data visualization as in the present invention.

Drawings

FIG. 1 is a flow chart of a multiple data source access method for data visualization in a method embodiment of the present invention;

FIG. 2 is a schematic diagram of a data large screen template in an embodiment of the method of the present invention;

FIG. 3 is a schematic diagram of data conversion in an embodiment of the method of the present invention;

FIG. 4 is a schematic diagram of a configuration interface for JSON static data in a method embodiment of the present invention;

fig. 5 is a schematic diagram of a data format of JSON static data in an embodiment of the method of the present invention.

Detailed Description

The invention aims to provide a multi-data source access method for data visualization, which ensures the stability of upper layer assembly construction under the condition of not changing assembly construction, realizes the analysis and display of multi-element heterogeneous data, and supports various forms such as a relational database, a system interface service, a memory database, a static file, static data and the like. Based on heterogeneous data source integration, each business system of an enterprise can be easily accessed, the data island is thoroughly broken, and the problems of long development period, large workload, high cost and low efficiency for realizing data visualization in the prior art are solved.

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent.

Method embodiment:

an embodiment of a multi-data source access method for data visualization of the present invention, as shown in fig. 1, includes the following steps:

step one, based on a visual designer, manufacturing a visual large screen, and designing a visual component, as shown in fig. 2.

And step two, selecting a visual component, and configuring a data source access mode of various data types for the visual component.

First kind: JSON static data: some static data are manually input to establish a data set, the data set is bound to a visual component chart, and the data type is mainly used for testing, demonstration and emergency occasions.

The visualization component is in effect through binding with the dataset, and the data of the dataset can be provided in the form of JSON static data. The values for the parameters may be replaced by variables in the format $ { xxx }, requesting the content support vector template engine to replace the variables. The method has the advantages that the static JSON static data mode is used for binding the data of the chart, the system demonstration, debugging and testing can be conveniently carried out in the mode, and the data visualization effect can be quickly built through the manual data input mode under the condition that a data source is not required to be connected.

The method is configured by a visualization tool and comprises the following steps:

s11: the selected component, as shown in fig. 4, is illustrated by taking a line diagram as an example, selects a data type, where the data type includes static data, dynamic data and SQL data, sets a refresh time, and may edit the static data.

S12: editing the data value. In a specific operation, a code editing box is opened, chart data is written, and the data format is defined by referring to the data format in each component, as shown in fig. 5.

Second kind: FILE data source: uploading an Excel/CSV file to obtain Excel/CSV file data, establishing a data model and a data set, and then performing visual display on the data in the Excel/CSV file data.

The method is configured by a visualization tool and comprises the following steps:

(1) and selecting a FILE data source name, and selecting a corresponding FILE source to upload to the platform FILE server.

(2) After the file is successfully uploaded, the prompt of the uploaded file is increased below the pop-up page, and the download is provided.

(3) And (5) confirming that the uploading is successful, and completing the FILE data source addition by clicking the submission.

(4) The newly added data source is automatically displayed in the FILE data source list.

Third kind: JDBC data source: and establishing connection with various databases, and establishing a data model and a data set through which data in the databases can be bound to a visualization chart and other visualization components for visualization analysis. MySQL, SQL Server, postgreSQL, oracle, redis, mongoDB database types are supported. The visualization platform supports the adaptation connection expansion of the JDBC data source, can add the JDBC data source configuration through the data dictionary function, and can conveniently expand the adaptation of various data sources. JDBC (Java Database Connectivity, short) Java database connection) is some Java-provided interfaces, which are mostly database vendor-provided (jar package).

The method is configured by a visualization tool and comprises the following steps:

(1) clicking [ new ], defaulting to create JDBC data source, selecting new data source driver in pop-up box, filling in JDBC data source name when adding data source, selecting JDBC data source driver and providing necessary information (JDBC source address, JDBC user name, JDBC cipher) for connecting database.

(2) After the information of the database is filled, the test button is clicked to carry out test connection, and whether the database can be communicated normally or not is verified.

(3) After the test connection is successful, the data source addition is completed by clicking and submitting.

(4) Newly added data sources are automatically displayed in the data source list.

Fourth kind: API interface service: the data set is established by means of an API interface, and the data is bound to the visualization chart and other visualization components through the data set. The API interfaces are all interfaces set according to the data requirements and are used for acquiring the requirement data. The method is configured by a visualization tool and comprises the following steps:

(1) and finding an operation column on a corresponding data model item on the data model list, clicking a new data set, popping up a page of the new data set, selecting a data source type as an API interface service, and filling necessary API basic information (request address, request mode [ post/get ], request data).

(2) The request data needs to fill in the request header, the request parameter, the request Body, the request header and the request parameter respectively according to the provided API request requirement, the request Body is directly written with the key value parameter by using the JSON data format.

(3) After the necessary information of the API interface service is filled, clicking a calling button at the right lower corner of the data conversion function pops up a data preview page, and at the same time, the upper right corner displays log information of interface calling.

Fifth: and a SCRIPT data source supporting the processing of capturing by using JavaScript or Python SCRIPT, and binding the data to the visualization chart and other visualization components. The SCRIPT data source is data obtained by grabbing the obtained original data through JavaScript or Python SCRIPT pair and carrying out logic processing according to service requirements. For example, the captured original data is the consumption of each consumer, and the total consumption of all consumers is displayed in the visualization component, so that the total consumption of each consumer needs to be accumulated to obtain the total consumption. The SCRIPT data source currently supports JavaScrip, python two scripting methods for processing data sources, charts and other visualization components on the datacolor dvp are validated through binding with the data set, the data of which can be provided in the form of SCIIPT data sources. The values for the parameters may be replaced by variables in the format $ { xxx }, requesting the content support vector template engine to replace the variables.

The method is configured by a visualization tool and comprises the following steps:

(1) and finding an operation column on a corresponding data model item on the data model list, clicking the newly built data set, popping up a page of the newly built data set, and then selecting a data source type as an SCRIPT data source.

(2) The user JavaScript/Python processes the business data.

(3) After the data is processed by the JavaScript/Python script, clicking a calling button at the right lower corner of the related variable function, popping up a data preview page, and displaying log information called by an interface at the right upper corner.

Step three, after the data source (static data, dynamic data, SQL data, etc.) is mounted, the format of the component corresponding to the designer may not be adapted to the data format, and then the process needs to clip and adapt the data through the function of data processing, as shown in fig. 3. In the case of dynamic data and SQL data, the data can be queried at regular time by setting the refresh frequency.

Data processing one: variable association, cropping data for display by the visualization component.

And extracting target information from a plurality of information of each data set according to the data information required to be displayed in the visual assembly, and finishing cutting the data so as to be displayed in the corresponding visual assembly.

In this embodiment, variables support the associated use of four data source types, namely JSON static data, JDBC data source, API interface service, SCRIPT data source, SQL expression, JSON data, API (request header, request parameter, request Body), and Python/Javascript data processing SCRIPT all support the vector template engine to replace variables. In the following, a detailed description will be given of an example of a system, where system user variables include: user_id (USER ID), dept_id (department ID), tenant_id (TENANT ID), row_id_list (ROLE ID array), row_id_string (ROLE LIST STRING, format 1,2, 3)

(1) SQL expressions.

For JDBC data types, SQL expressions are used to filter the data sources and drill down the queries through variables.

By setting three variables of departments, starting time and ending time, the SQL expression is used for acquiring target data corresponding to the variables, and data dynamic filtering query is supported.

(2) API (request address, request header, request parameters, request Body).

When dynamic data is selected, the request address, the request header, the request parameters and the request Body of the API interface transmit parameters to the interface through variables, so that the dynamic data query is realized.

(3) JSON data.

When static data is selected, the JSON data returns different static data through a vector template engine and variable substitution.

(4) Python/Javascript data processing script.

The Python/Javascript data processing script is replaced by a vector template engine and variables, so that data processing is facilitated.

And II, data processing: data is converted to adapt the visualization component display.

The data conversion function supports JavaScript programming, is convenient for the front-end component to render the secondary processing of the data, and meets better data display effect. Specifically, the method comprises the steps of converting the data type and converting the interface type. And converting the data in the data source according to the type of the data and the type of the interface required in the assembly so as to adapt to the visual assembly. For example, a video stream in a format obtained from a data source, which is not supported by the visualization component, may need to be converted to a video stream format supported by the visualization component.

The beneficial effects of the scheme of the invention include:

the visual component can be accessed into data sources with various data types, and various forms such as a relational database, a system interface service, a memory database, a static file, static data and the like are supported by one-time configuration of the visual component. In different stages, when facing the data sources of different clients, the data source types and configuration data source information are simply adjusted. The invention is based on heterogeneous data source integration, can easily access each business system of enterprises, reduces the workload of staff, reduces the development cost and the labor cost for manufacturing a large screen, and thoroughly breaks the data island. Compared with the prior art, the method requires a plurality of professionals to work cooperatively for a long time, and the method can complete the large data screen in a short time by only one professional, thereby greatly reducing the labor cost.

On the other hand, the method has stronger adaptability and compatibility. The data connection of the visual graphic assembly is dynamically updated, so that the data can be updated according to the set refresh frequency, and once the data source changes, increases or decreases the data and the data type.

The SCRIPT data source is arranged in the invention, corresponding data can be flexibly obtained according to the requirements, the expansion function is strong, the invention can adapt to different data scenes of enterprises, solves the problem of data island of each business system of the enterprises, helps the enterprises to establish a data center, provides various data service capabilities for upper-layer application, and builds foundation stones for data center construction.

In a word, the invention effectively solves the data analysis requirements of different roles under the enterprise organization architecture, helps enterprises to quickly establish strategic decision analysis capacity, adapts to the variable market requirements, and accelerates the business innovation of the enterprises.

System embodiment:

the embodiment of the invention relates to a multi-data source access system for data visualization, which is a visual editable system and is used for configuring the data source type of data accessed by a visual component. The memory includes at least one software functional module stored in the memory, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory to implement a multi-data source access method for data visualization in the method embodiment of the present invention.

The processor may be a microprocessor MCU, a programmable logic device FPGA, or other processing device. The memory may be a variety of memories that store information using electrical energy, such as RAM, ROM, etc.

Claims (9)

1. A multi-data source access method for data visualization is characterized in that: the method comprises the following steps:

1) Obtaining a visual component of a visual large screen;

2) Designing the type of the data source of the visual component accessing the data source, and determining the accessing mode of each type of data source;

3) Selecting a data source type of an access data source for the visual component, configuring data source information, and acquiring a data set;

4) Binding a data set to the visualization component, and converting data in the data set according to the data requirement of the visualization component;

the data source type comprises at least two data types of JSON static data type, FILE data type, JDBC data type, API interface data type and SCRIPT data type; the API interface data is used for acquiring the data to be requested in an interface mode according to the basic information of the data to be requested so as to establish a data set; the SCRIPT data is data obtained by obtaining data to be requested in a JavaScript and/or Python SCRIPT mode and carrying out secondary processing on the obtained data to be requested according to service requirements, and is used for establishing a data set.

2. The multiple data source access method for data visualization of claim 1, wherein: converting the data in the dataset in step 4) comprises: cutting the data in the data set according to the target data required in the visual component; and performing type conversion on the data in the data set according to the data type of the target data required in the visualization component.

3. The multiple data source access method for data visualization of claim 1, wherein: the API interface data type data source is accessed in the following way:

selecting a data model, creating a data set, configuring API interface basic information, and obtaining data requested by an API interface for generating a corresponding data set; the API basic information comprises a request address, a request mode and request data; the request data includes a request header, a request parameter, and a request body.

4. The multiple data source access method for data visualization of claim 1, wherein: the SCRIPT data type data source is accessed in the following way: selecting a data model, creating a data set, capturing data by adopting a script, and associating the data obtained by script processing with a template variable to obtain data for generating the data set.

5. The multiple data source access method for data visualization of claim 2, wherein: cutting data is carried out by setting template variables.

6. The multiple data source access method for data visualization of claim 1, wherein: the JSON static data type data source is accessed by the following modes: selecting a JSON static data type, and compiling chart data in an editing area, wherein the data format is determined according to the data format in the corresponding visual component; storing the written chart data and generating a data set corresponding to the JSON static data;

the FILE data type data source is accessed in the following way: the data source FILE corresponding to the FILE data type comprises Excel or CSV; acquiring a selected data source file, establishing a data model, and establishing a corresponding data set;

the JDBC data type data source is accessed in the following manner: creating a JDBC data source, selecting a data source driver, and setting configuration information of a connection database; the configuration information comprises a JDBC source address, a JDBC user name and a JDBC password; verifying the communication condition with the database; after the connectivity verification is passed, JDBC data is obtained, a data model is established, and a corresponding data set is obtained.

7. The multiple data source access method for data visualization of claim 6, wherein: the database connected with the JDBC data source is any one of MySQL, SQL Server, postgreSQL, oracle, redis and MongoDB.

8. The multiple data source access method for data visualization according to any of claims 1 to 7, wherein: the setting of the access modes of the various types of data sources comprises setting of the refresh frequency of the access data sources, and the setting of the refresh frequency is used for refreshing the data according to the set refresh frequency.

9. A multiple data source access system for data visualization, characterized by: the system comprising a processor for executing computer instructions to implement a multiple data source access method for data visualization as claimed in any of claims 1-8.

Images