CN110737712A

CN110737712A - InfluxDB time sequence data storage system

Info

Publication number: CN110737712A
Application number: CN201910996509.9A
Authority: CN
Inventors: 陈凯园; 石鹏程; 张成伟; 张苏; 陈晨
Original assignee: Nanjing Kisen International Engineering Co Ltd
Current assignee: Nanjing Kisen International Engineering Co Ltd
Priority date: 2019-10-19
Filing date: 2019-10-19
Publication date: 2020-01-31

Abstract

The invention discloses an InfluxDB time sequence data storage system which comprises an acquisition module, a storage module, an analysis module, an inquiry module and a visualization module, wherein the acquisition module is used for acquiring production data in a production process and sending the production data information to the storage unit through a network, the storage module is used for receiving the production data information from the acquisition module and compressing and storing the production data information, and also provides an API (application programming interface) for the analysis module, the inquiry module and the visualization module to call, the analysis module is used for acquiring data in the storage module at regular time for analysis and processing and storing an analysis result to the storage module, the inquiry module is used for providing an inquiry API (application programming interface) and providing an inquiry interface for the analysis module and the visualization module, and the visualization module is used for displaying the analysis result of the analysis module.

Description

InfluxDB time sequence data storage system

Technical Field

The invention relates to the technical field of cement production, in particular to InfluxDB time sequence data storage systems.

Background

At present, a large amount of information data generated in the production and operation process of the cement factory is not provided with a production data full-period acquisition system which is targeted, efficient and easy to use.

Disclosure of Invention

This section is intended to summarize aspects of the embodiments of the present invention and to briefly introduce preferred embodiments in this section, as well as in the abstract and title of the specification of the application, which are intended to be simplified or omitted so as to not obscure the purpose of this section, abstract and title of the specification, and are not intended to limit the scope of the invention.

The present invention has been made in view of the problems occurring in the prior art.

Therefore, objectives of the present invention are to provide a infiluxdb time series data storage system, which facilitates to know real-time production status and historical statistical information in the plant, and effectively improves the production management level.

In order to solve the technical problems, the invention provides an InfluxDB time sequence data storage system which comprises an acquisition module, a storage module, an analysis module, an inquiry module and a visualization module, wherein the acquisition module is used for acquiring production data in a production process and sending the production data information to the storage unit through a network, the storage module is used for receiving the production data information from the acquisition module and compressing and storing the production data information, and also provides an API (application programming interface) for the analysis module, the inquiry module and the visualization module to call, the analysis module is used for acquiring data in the storage module at regular time to analyze and process, and stores an analysis result to the storage module, the inquiry module is used for providing an inquiry API and providing an inquiry interface for the analysis module and the visualization module, and the visualization module is used for displaying the analysis result of the analysis module.

The invention relates to preferred schemes of an InfluxDB time sequence data storage system, wherein an acquisition module (100) acquires label variable information in OPCSERVER in the industry by using an OPC-DA protocol, the service provides a function of adding or deleting labels, the label information needing to be acquired is selected by the function of adding or deleting the labels, whether the labels need to be calculated can be set for a production management system calculation service to use, after the label information is added, the acquisition service can acquire numerical information of the OPCSERVER labels and send the numerical information to a storage module by an HTTP request, the storage module receives the request, and data is stored to a database in a Point structure according to db information and table information in the request.

The preferred schemes of the InfluxDB time sequence data storage system are that after the acquisition module receives data, the acquisition module inserts the data into the storage module by calling the api of the InfluxDB to insert the data into the storage module in a Point structure, and the storage module writes the data into the cache and wal simultaneously when inserting the data.

The preferred schemes are used as the InfluxDB time sequence data storage system, wherein when data are inserted into a storage module, the data are written into caches and wal at the same time, the caches are caches of data in wal files in an internal memory, when InfluxDB is started, all wal files are traversed, the caches are reconstructed, the caches enable the data to be written into tsm files after the data in the caches occupy a certain amount of the internal memory through maxSize parameters, if the data are not configured, the default upper limit is 25MB, when the data in the caches reach a threshold value, times of snapshot are carried out on the current caches, then the content in the current caches are emptied, new wal files are created for writing, the rest wal files are deleted, and the data in the snapshots are written into new tsm files through sorting.

As optimal schemes of the InfluxDB time sequence data storage system, the analysis module acquires data information stored in the InfluxDB by the acquisition service through the query module, periodically calculates the maximum value, the minimum value, the average value, the th value, the last th value, the difference value and the standard deviation of each variable, and rewrites the calculation result into the storage module.

The preferred schemes of the InfluxDB time sequence data storage system are that the visualization module provides a function of configuring a data source, the function can be friendly and compatible with an InfluxDB data source, the service is based on a web service developed by a go language, a common report component is provided on an html page, a user can drag the component on a design interface, the report display effect is set on a report attribute column, data source information is configured for the report, the data information and a chart are bound, fixed unique access http links are generated after the designed chart is stored, the designed report page is conveniently embedded into a third party, and meanwhile, the chart data can be exported in an EXCEL format file.

The invention has the beneficial effects that: the invention completes the full-period collection of production data by integrating the informatization technology into the daily maintenance operation work of a factory, and can compress and regularly archive massive data due to the high-efficiency compression algorithm and the storage strategy. Due to the excellent performance of the system, second-level access of user historical data can be realized, and user experience is greatly improved. Meanwhile, by matching with the Grafana suite, a user can finish production data visualization independently, so that real-time production conditions and historical statistical information in a factory can be conveniently known, and the production management level is effectively improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and other drawings can be obtained by those skilled in the art without inventive exercise, wherein:

FIG. 1 is a schematic diagram of the system of the present invention.

Fig. 2 is a schematic diagram of the connection relationship of modules according to the present invention.

FIG. 3 is a schematic diagram of data transmission relationship between modules according to the present invention.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and one skilled in the art may make similar reference without departing from the spirit of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.

Furthermore, the appearances of the phrase " embodiments" or "embodiments" in various places in the specification are not necessarily all referring to the same embodiments, nor are separate or alternative embodiments mutually exclusive of other embodiments.

Referring to fig. 1 to 3, an th embodiment of the present invention provides infiluxdb time series data storage systems, which facilitate understanding of real-time production conditions and historical statistical information in a plant, and effectively improve production management level.

The InfluxDB time series data storage system comprises an acquisition module 100, a storage module 200, an analysis module 300, a query module 400 and a visualization module 500.

The acquisition module 100 is used for acquiring production data in the production process and transmitting production data information to the storage unit 200 through a network; the storage module 200 is configured to receive the production data information from the acquisition module 100, compress and store the production data information, and provide an API for the analysis module 300, the query module 400, and the visualization module 500 to call; the analysis module 300 is configured to obtain data in the storage module 200 at regular time for analysis, and store an analysis result in the storage module 200; the query module 400 provides a query API and provides a query interface for the analysis module 300 and the visualization module 500; the visualization module 500 is used for displaying the analysis result of the analysis module 300.

The overall working process of the InfluxDB time sequence data storage system is as follows: the acquisition module 100 is used for acquiring data from the optserver and storing the acquired data in the storage module 200. The storage module 200 is used for storing data, and is responsible for receiving data collected by the collection module 100 and providing query data support for query services. The analysis module 300 is used for historical data analysis metrics, which obtains data of the storage module 200 through the query service, calculates the data through an algorithm, and writes the result into the storage service 200. The query module 400 is used for displaying the historical data in real time, and the user can quickly see the historical trend of the archived production data from the storage module 200 by calling the query service. The visualization module 500 is a BI tool for presenting real-time, historical, and analysis results, which takes data through the query module 400 and presents the data in a graphical form.

When a system is actually built, the acquisition module 100 and 1 can be deployed on a data server computer, and an OPC upper computer is connected to the acquisition module 100 in a configuration mode. 2. The acquisition module 100 is connected with a storage module 200. 3. The memory module 200 is connected in a display module configuration connection. 4. The connection memory module address is configured at the analysis module 300. The acquisition module 100 monitors data change of the OPC upper computer through the monitoring module and receives a data notification. And writes the data to the storage module 200 through the network. And visually configuring connection statements and display column charts to finish the display of the collected data and the analyzed data.

Specifically, the collection module 100 obtains tag variable information in the OPCSERVER in the industry by using the OPC-DA protocol, and the service provides the functions of adding and deleting tags. The user can select the label information required to be collected through the function, and can set whether the label needs to be calculated or not so as to be used by the calculation service of the production management system. After the user adds the label information, the collecting service can obtain the numerical information of the OPCSERVER label, and sends the numerical information to the storage module 200 through the HTTP request, and the storage module 200 receives the request and stores the data to the database in a Point structure according to the db information and the table information in the request.

After receiving the data, the acquisition module 100 inserts the data into the storage module 200 in a Point structure by calling the api of the infiluxdb to insert the data, the storage module 200 writes the data into the caches and wal simultaneously when inserting the data, and the caches can be considered as caches of data in wal files in the memory, so that the query speed of the query module 400 can be improved in aspect, in addition, when the infiluxdb is started, all wal files can be traversed, and the caches are reconstructed, so that even if a system fails, the data cannot be lost, and meanwhile, the analysis module 300 also stores the analysis result into the storage module 200 for the visualization module 500 to use.

Step , when the storage module 200 inserts data, data is actually written into the caches and wal at the same time, the caches can be considered as caches of data in wal files in an internal memory, when the infilux xdb is started, all wal files are traversed, the caches are reconstructed, and thus even if a system fails, data loss cannot be caused.

The analysis module 300 obtains the data information stored in the infiluxdb by the acquisition service by calling the query module 400, periodically calculates the maximum value, the minimum value, the average value, the th value, the last value, the difference value, and the standard deviation of each variable, and rewrites the calculation result into the storage module 200.

The visualization module 500 provides a function of configuring a data source, which is a web service developed based on a go language, provides visualization components such as a line graph, a pie graph, a bar graph, a thermodynamic diagram, a dashboard and the like for common report components on an html page, a user can drag the components on a design interface, sets a report display effect on a report attribute column, configures data source information for the report and binds the data information with the chart, fixed unique access http links are generated after the designed chart is stored, the designed report page is conveniently embedded into a third party, and meanwhile, a export button is provided on the chart, and the user can export the data of the report in an EXCEL format file by clicking the export button.

The invention completes the full-period collection of production data by integrating the informatization technology into the daily maintenance operation work of a factory, and can compress and regularly archive massive data due to the high-efficiency compression algorithm and the storage strategy. Due to the excellent performance of the system, second-level access of user historical data can be realized, and user experience is greatly improved. Meanwhile, by matching with the Grafana suite, a user can finish production data visualization independently, so that real-time production conditions and historical statistical information in a factory can be conveniently known, and the production management level is effectively improved.

It is important to note that the construction and arrangement of the present application as shown in the various exemplary embodiments is illustrative only. Although only a few embodiments have been described in detail in this disclosure, those skilled in the art who review this disclosure will readily appreciate that many modifications are possible (e.g., variations in sizes, dimensions, structures, shapes and proportions of the various elements, values of parameters (e.g., temperatures, pressures, etc.), mounting arrangements, use of materials, colors, orientations, etc.) without materially departing from the novel teachings and advantages of the subject matter recited in this application. For example, elements shown as integrally formed may be constructed of multiple parts or elements, the position of elements may be reversed or otherwise varied, and the nature or number of discrete elements or positions may be altered or varied. Accordingly, all such modifications are intended to be included within the scope of this invention. The order or sequence of any process or method steps may be varied or re-sequenced according to alternative embodiments. In the claims, any means-plus-function clause is intended to cover the structures described herein as performing the recited function and not only structural equivalents but also equivalent structures. Other substitutions, modifications, changes and omissions may be made in the design, operating conditions and arrangement of the exemplary embodiments without departing from the scope of the present inventions. Therefore, the present invention is not limited to a particular embodiment, but extends to various modifications that nevertheless fall within the scope of the appended claims.

Moreover, in an effort to provide a concise description of the exemplary embodiments, all features of an actual implementation may not be described (i.e., those unrelated to the presently contemplated best mode of carrying out the invention, or those unrelated to enabling the invention).

It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions may be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may be related to the development of design, fabrication, and manufacture, without undue experimentation, will be readily apparent to those of ordinary skill in the art having the benefit of this disclosure.

It should be noted that the above-mentioned embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, which should be covered by the claims of the present invention.

Claims

1, InfluxDB time sequence data storage system, which is characterized by comprising,

the acquisition module (100) is used for acquiring production data in the production process and transmitting the production data information to the storage unit (200) through a network;

the storage module (200) is used for receiving the production data information from the acquisition module (100), compressing and storing the production data information, and providing an API (application programming interface) for the analysis module (300), the query module (400) and the visualization module (500) to call;

the analysis module (300) is used for acquiring the data in the storage module (200) at regular time for analysis processing and storing the analysis result in the storage module (200);

a query module (400) providing a query API and providing a query interface for the analysis module (300) and the visualization module (500); and the number of the first and second groups,

a visualization module (500) for presenting the analysis results of the analysis module (300).

2. The InfluxDB time-sequential data storage system of claim 1, wherein: the acquisition module (100) acquires label variable information in OPCSERVER in industry by using OPC-DA protocol, and the service provides a function of adding or deleting labels;

selecting the label information to be collected through the label adding or deleting function, and setting whether the label needs to be calculated or not so as to be used by the calculation service of the production management system;

after the label information is added, the collecting service can obtain the numerical value information of the OPCSERVER label, and sends the numerical value information to the storage module (200) through an HTTP request, and the storage module (200) receives the request and stores the data to the database in a Point structure according to db information and table information in the request.

3. The InfluxDB time-sequential data storage system of claim 2, wherein: after the acquisition module (100) receives the data, the data is inserted into the storage module (200) in a Point structure by calling the api of the InfluxDB to insert the data, and the storage module (200) writes the data into the caches and wal simultaneously when the data is inserted.

4. The InfluxDB time sequence data storage system according to claim 3, wherein when data are inserted into the storage module (200), data are written into the caches and wal at the same time, the caches are caches of wal files in an internal memory, when the InfluxDB is started, all wal files are traversed, the caches are reconstructed, the caches write data into tsm files after occupying much internal memory by the data in the caches through maxSize parameters, if the data are not configured, the default upper limit is 25MB, when the data in the caches reach a threshold value, the current caches are subjected to times of snapshot, then the content in the current caches is emptied, new wal files are created for writing, the rest wal files are deleted, and the data in the snapshots are written into new tsm files through sorting.

5. The InfluxDB time-series data storage system according to claim 4, wherein the analysis module (300) obtains the data information stored in the InfluxDB by the collection service through the query module (400), periodically calculates according to the maximum value, the minimum value, the average value, the th value, the last values, the difference value and the standard deviation of each variable, and rewrites the calculation result into the storage module (200).

6. The InfluxDB time sequence data storage system according to claim 5, wherein the visualization module (500) provides a function of configuring a data source, the function is friendly and compatible with an InfluxDB data source, the service is based on a web service developed by a go language, a common report component is provided on an html page, a user can drag the component on a design interface, the report display effect is set on a report attribute column, data source information is configured for the report, the data information and a chart are bound, fixed unique access http links are generated after the designed chart is stored, the designed report page is conveniently embedded into a third party, and meanwhile, graph data can be exported in an EXCEL format file.