CN110019247A - Data storage and querying method, device and monitoring system - Google Patents
Data storage and querying method, device and monitoring system Download PDFInfo
- Publication number
- CN110019247A CN110019247A CN201810934580.XA CN201810934580A CN110019247A CN 110019247 A CN110019247 A CN 110019247A CN 201810934580 A CN201810934580 A CN 201810934580A CN 110019247 A CN110019247 A CN 110019247A
- Authority
- CN
- China
- Prior art keywords
- data
- stored
- type
- data type
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
The present invention proposes a kind of storage of data and querying method, device and monitoring system, wherein this method comprises: receiving the data storage request of external equipment, data storage request includes data to be stored；Whether the data type for judging data to be stored is preset data type；If judging to know, as preset data type, data to be stored is saved to time series database for the data type of data to be stored；If judgement knows that the data type of data to be stored is not preset data type, data to be stored is then saved in relational database, thus, unified data memory interface is provided for data acquisition analysis system, and corresponding data are saved using the advantage of respective database, improve the storage efficiency of data.
Description
Technical field
The present invention relates to monitoring technology field more particularly to a kind of storage of data and querying methods, device and monitoring system
System.
Background technique
Comprehensive monitoring system is based on modern computer information technology, network technology, communication and automatic control technology research and development
Modularization, scalable distributed mainframe computer integrated technology.
Comprehensive monitoring system is monitoring operation people by functions such as data acquisition, data processing, data storage, implementing monitorings
Member provides field device and service operation state real-time status monitoring, alarm, trend analysis of profession etc..
In the related technology, comprehensive monitoring system generally pass through traditional Relational DataBase (such as MySql database,
Oracle database, SQL Server database etc.) save all data.Wherein, relevant database has stronger data
Model tormulation ability can establish complex data model by way of creation table name and field and express business model, general
It spends relatively high.However, for the data acquired in real time in monitoring system and being not suitable for, the data in monitoring system have data
Amount is big, and duration generates, and generates the features such as frequency is high, and temporal associativity is strong, and data model is single, this shows that its data characteristics is inclined
High-performance read-write is overweighted, and not on complex data model, the strong modeling ability mismatch of this and relevant database, Er Qiezheng
It is since the complex data model of relevant database makes it on reading and writing data and does not have advantage.
Summary of the invention
The present invention is directed to solve at least some of the technical problems in related technologies.
For this purpose, being data present invention aims at a kind of storage of data and querying method, device and monitoring system is proposed
Acquisition provides unified data memory interface with monitoring system, and is saved by the determination of the data type of data to be stored wait store
The database of data so as to be suitble in the data rapid saving of time series database preservation to time series database, and will be suitble to
In the data rapid saving that relevant database saves into relevant database, is saved and corresponded to using the advantage of respective database
Data, improve the storage efficiency of data.
In order to achieve the above object, first aspect present invention embodiment proposes a kind of date storage method, the method application
In data acquisition and comprehensive monitoring system, the data acquisition and comprehensive monitoring system include time series database and relation data
Library, which comprises receive the data storage request of external equipment, the data storage request includes data to be stored；Sentence
Whether the data type for the data to be stored of breaking is preset data type；If the data class of the data to be stored is known in judgement
Type is preset data type, then saves the data to be stored to the time series database；If judgement is known described wait store
The data type of data is not preset data type, then the data to be stored is saved in the relational database.
The date storage method of the embodiment of the present invention, after getting data to be stored, in the number for determining data to be stored
According to type be preset data type when, data to be stored is saved by time series database；In the data class for determining data to be stored
When type is not preset data type, data to be stored is saved by relational database.It is mentioned as a result, for data acquisition analysis system
For unified data memory interface, and the database for saving data to be stored is determined by the data type of data to be stored, from
And it can will be suitble in the data rapid saving of time series database preservation to time series database, and will be suitble to protect in relevant database
The data rapid saving deposited saves corresponding data into relevant database, using the advantage of respective database, improves number
According to storage efficiency.
In order to achieve the above object, second aspect of the present invention embodiment proposes a kind of data query method, the method application
In data acquisition and comprehensive monitoring system, the data acquisition and comprehensive monitoring system include time series database and relation data
Library, which comprises receive the inquiry request of user's input；It is pre- to judge whether the user inquires according to the inquiry request
If the data of data type；If the data of the user query preset data type are known in judgement, from the time series database
It is middle to obtain the inquiry data for meeting the inquiry request；If judgement knows that the user does not inquire the data of preset data type,
The inquiry data for meeting the inquiry request are then obtained from the relational database.
The data query method of the embodiment of the present invention passes through inquiry request after the inquiry request for receiving user's input
Determine whether user inquires the data of preset data type, and in the data for determining user query preset data type, when passing through
The inquiry data for meeting inquiry request are obtained in sequence database；And determining that user do not inquire the data of preset data type,
The inquiry data for meeting inquiry request are then obtained from relational database.Unification is provided as a result, for data acquisition analysis system
Query interface, facilitate the data of type needed for user is inquired by unified query interface, improve user and adopted in data
Collection and the efficiency that data are inquired in monitoring system, improve the Experience Degree of user.
In order to achieve the above object, third aspect present invention embodiment proposes a kind of data storage device, described device application
In data acquisition and comprehensive monitoring system, the data acquisition and comprehensive monitoring system include time series database and relation data
Library, described device include: the first receiving module, for receiving the data storage request of external equipment, the data storage request
Including data to be stored；First judgment module, for judging whether the data type of the data to be stored is preset data class
Type；First processing module, if for judging to know that the data type of the data to be stored, will be described as preset data type
Data to be stored is saved to the time series database；Second processing module, if for judging to know the number of the data to be stored
It is not preset data type according to type, then the data to be stored is saved in the relational database.
The data storage device of the embodiment of the present invention, after getting data to be stored, in the number for determining data to be stored
According to type be preset data type when, data to be stored is saved by time series database；In the data class for determining data to be stored
When type is not preset data type, data to be stored is saved by relational database.It is mentioned as a result, for data acquisition analysis system
For unified data memory interface, and the database for saving data to be stored is determined by the data type of data to be stored, from
And it can will be suitble in the data rapid saving of time series database preservation to time series database, and will be suitble to protect in relevant database
The data rapid saving deposited saves corresponding data into relevant database, using the advantage of respective database, improves number
According to storage efficiency.
In order to achieve the above object, fourth aspect present invention embodiment proposes a kind of data query device, described device application
In data acquisition and comprehensive monitoring system, the data acquisition and comprehensive monitoring system include time series database and relation data
Library, described device include: the second receiving module, for receiving the inquiry request of user's input；Second judgment module is used for basis
The inquiry request judges whether the user inquires the data of preset data type；Third processing module, if for judging to obtain
The data for knowing the user query preset data type, then obtain from the time series database and meet looking into for the inquiry request
Ask data；Fourth processing module, if for judging to know the data that the user does not inquire preset data type, from the pass
It is that the inquiry data for meeting the inquiry request are obtained in database.
The data query device of the embodiment of the present invention passes through inquiry request after the inquiry request for receiving user's input
Determine whether user inquires the data of preset data type, and in the data for determining user query preset data type, when passing through
The inquiry data for meeting inquiry request are obtained in sequence database；And determining that user do not inquire the data of preset data type,
The inquiry data for meeting inquiry request are then obtained from relational database.Unification is provided as a result, for data acquisition analysis system
Query interface, facilitate the data of type needed for user is inquired by unified query interface, improve user and adopted in data
Collection and the efficiency that data are inquired in monitoring system, improve the Experience Degree of user.
In order to achieve the above object, fifth aspect present invention embodiment proposes a kind of acquisition of data and comprehensive monitoring system, institute
It states data acquisition and comprehensive monitoring system includes time series database and relational database, the system comprises:
The data storage device of third aspect present invention embodiment；
The data query device of fourth aspect present invention embodiment.
To achieve the goals above, sixth aspect present invention embodiment proposes a kind of computer readable storage medium,
On be stored with computer program, when which is executed by processor realize first aspect present invention embodiment described in data storage
Method.
To achieve the goals above, seventh aspect present invention embodiment proposes a kind of computer readable storage medium,
On be stored with computer program, when which is executed by processor realize first aspect present invention embodiment described in data query
Method.
The additional aspect of the present invention and advantage will be set forth in part in the description, and will partially become from the following description
Obviously, or practice through the invention is recognized.
Detailed description of the invention
Above-mentioned and/or additional aspect and advantage of the invention will become from the following description of the accompanying drawings of embodiments
Obviously and it is readily appreciated that, in which:
Fig. 1 is the flow diagram of date storage method according to an embodiment of the invention；
Fig. 2 is the flow diagram of data query method according to an embodiment of the invention；
Fig. 3 is to judge whether user inquires the refinement flow diagram of the data of preset data type according to inquiry request；
Fig. 4 is the structural schematic diagram of data storage device according to an embodiment of the invention；
Fig. 5 is the structural schematic diagram of data storage device in accordance with another embodiment of the present invention；
Fig. 6 is the structural schematic diagram of data storage device in accordance with another embodiment of the present invention；
Fig. 7 is the structural schematic diagram of data query device according to an embodiment of the invention；
Fig. 8 is the structural schematic diagram of data acquisition and comprehensive monitoring system according to an embodiment of the invention；
Fig. 9 is the structural schematic diagram of data acquisition and comprehensive monitoring system accord to a specific embodiment of that present invention；
Figure 10 is the structural schematic diagram of openTSDB.
Specific embodiment
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, it is intended to is used to explain the present invention, and is not considered as limiting the invention.
Below with reference to the accompanying drawings the data storage and querying method, device and monitoring system of the embodiment of the present invention are described.
Fig. 1 is the flow diagram of date storage method according to an embodiment of the invention.
As shown in Figure 1, the date storage method the following steps are included:
Step 101, the data storage request of external equipment is received.
Wherein, data storage request can include but is not limited to data to be stored, for example, data storage request can also wrap
Include device identification, the temporal information of external equipment acquisition data to be stored etc. of external equipment.
Wherein, it should be noted that the date storage method of the embodiment of the present invention is applied to data acquisition and comprehensively monitoring
SCADA (Supervisory Control And Data Acquisition) system.
Wherein, it should be noted that can include but is not limited to time series data in data acquisition and comprehensive monitoring system
Library and relational database.
Step 102, whether the data type for judging data to be stored is preset data type.
Wherein, preset data type can be the pre-set data type that data can be saved by time series database.
Wherein, preset data type can be with the data of time series, for example, preset data type can in monitoring system
Think original index data.
Wherein, original index data monitor initial data collected, ordinal number when having to system equipment or service condition
According to the part that feature and data volume are most huge.
Wherein, it should be noted that under different application scenarios, judge whether the data type of data to be stored is pre-
If there are many kinds of the modes of data type, it is illustrated below:
As an example, it can extract the data characteristics of data to be stored, and by data characteristics and preset data type pair
The data characteristics answered is matched, and determines whether the data type of data to be stored is preset data class by matching result
Type.
Wherein, if knowing data characteristics data characteristics matching corresponding with preset data type, it is determined that data to be stored
Data type be preset data type.
If knowing, data characteristics data characteristics corresponding with preset data type is mismatched, it is determined that the number of data to be stored
It is not preset data type according to type.
Specifically, can be by the progress of data characteristics corresponding to the data characteristics of data to be stored and preset data type
Match, if knowing, matching degree between the two is more than preset threshold, it is determined that the data type of data to be stored is preset data class
Type.
Wherein, if knowing, matching degree between the two is less than preset threshold, it is determined that the data type of data to be stored
It is not preset data type.
As another example, data storage request can also include that the device identification of external equipment ceases, can determine whether in advance
If whether including device identification in the corresponding default device identification set of data type.
Wherein, if judgement is known in default device identification set comprising device identification, it is determined that the data of data to be stored
Type is preset data type.
Wherein, if judgement, which is known, does not include device identification in default device identification set, it is determined that the number of data to be stored
It is not preset data type according to type.
Step 103, if judging to know, as preset data type, data to be stored is protected for the data type of data to be stored
It deposits to time series database.
Wherein, time series database mainly follows time series and carrys out storing data, and data format is relatively easy, can provide efficiently
Literacy is thus more suitable for the data storage requirement in monitoring system.
Wherein, it should be noted that time series database has high I/O performance, supports cluster expansion, is suitble to large-scale data
Store and have the characteristics that high search efficiency.
Wherein, it is to be understood that the data format of disparate databases storing data is different, in order to can be according to time series data
Data format corresponding to library saves data to be stored, and the embodiment of property, saves by data to be stored as an example
To before time series database, data structure packing rule corresponding with preset data type can also be obtained, and according to data knot
Structure packing rule carries out construction packages to data to be stored, then, calls access memory interface corresponding with time series database will
Data write timing database after encapsulation.
For example, it is assumed that external equipment is information collecting device, and preset data type is time series data type, is being received
To after information collecting device original index data collected, in order to save information collecting device in systems sequentially in time
Original index data collected, the data of data to be stored in determining data storage request transmitted by information collecting device
Type is time series data type, can be the data format that time series database requires by data to be stored, and be sent to time series data
Listening port corresponding to library.Specifically, can using equipment identification information as metric, and call time by as
Timestamp, reporting numerical value is value, is encapsulated as JSON data mode, uses HTTP (HyperText Transfer
Protocol, hypertext transfer protocol) data after encapsulation are sent to listening port corresponding to time series database by agreement.
The occupied memory space of data is saved in order to reduce, the embodiment of property as an example, will be wait store
Data are saved to time series database, can also be used preset data compression algorithm, be compressed to data to be stored, and will
Compressed data are saved to time series database.
Wherein, preset data compression algorithm is for being compressed on time dimension to the data in time series database
Algorithm.
Wherein, preset data compression algorithm can include but is not limited to average value, maximum value, minimum value compression, and be based on
Swinging door compression algorithm etc., the embodiment are not construed as limiting this.
The embodiment of property as an example can also be time series data lab setting to guarantee the safety of data
Access authority, so that user's accessing time sequence database of only access authority.
Step 104, if judgement knows that the data type of data to be stored is not preset data type, by data to be stored
It is saved in relational database.
Wherein, it is to be understood that time series data can not only be saved in the SCADA system, it can also be to it
Business datum corresponding to his business event saves.
Wherein, other business event can include but is not limited to system event, log in dependent event, control generation event,
Alert event, FEP event, redundancy event etc..
Wherein, it is to be understood that SCADA system, can be square by saving business datum corresponding to other business event
System running state can comprehensively be grasped by continuing administrative staff after an action of the bowels, and the fault messages such as real-time monitoring equipment, communication are simultaneously made in time
Corresponding emergency measure.
The date storage method of the embodiment of the present invention, after getting data to be stored, in the number for determining data to be stored
According to type be preset data type when, data to be stored is saved by time series database；In the data class for determining data to be stored
When type is not preset data type, data to be stored is saved by relational database.It is mentioned as a result, for data acquisition analysis system
For unified data memory interface, and the database for saving data to be stored is determined by the data type of data to be stored, from
And it can will be suitble in the data rapid saving of time series database preservation to time series database, and will be suitble to protect in relevant database
The data rapid saving deposited saves corresponding data into relevant database, using the advantage of respective database, improves number
According to storage efficiency.
Fig. 2 is the flow diagram of data query method according to an embodiment of the invention.
Step 201, the inquiry request of user's input is received.
Step 202, judge whether user inquires the data of preset data type according to inquiry request.
Wherein, the data query method of the embodiment of the present invention is applied to data acquisition and comprehensively monitoring SCADA
(Supervisory Control And Data Acquisition) system.
Wherein, it should be noted that can include but is not limited to time series data in data acquisition and comprehensive monitoring system
Library and relational database.
Step 203, if the data of user query preset data type are known in judgement, met from time series database
The inquiry data of inquiry request.
Wherein, it should be noted that judge whether user inquires present count according to inquiry request in different application scene
Mode according to the data of type is different, is illustrated below:
It as an example, can be by data to be checked if in inquiry request including the data type of data to be checked
Data type is compared with preset data type, if the data type of data to be checked is consistent with preset data type,
Determine the data of user query preset data type.
As another example, as shown in figure 3, judging whether user inquires the number of preset data type according to inquiry request
According to may include:
Step 301, inquiry request is parsed, determines the corresponding query characteristics of inquiry request.
Step 302, default query characteristics corresponding to the data with preset data type are obtained.
Step 303, according to the matching degree between default query characteristics and query characteristics, determine whether user inquires present count
According to the data of type.
Wherein, it should be noted that the sample queries that user is inputted when obtaining inquiry preset data categorical data in advance
Request, and requested according to sample queries, determine query characteristics corresponding when inquiry preset data categorical data, and save
Query characteristics corresponding to data with preset data type.To facilitate it is subsequent can according to the data of preset data type institute
Corresponding query characteristics, determine whether user inquires the data of preset data type.
Step 204, it if judgement knows that user does not inquire the data of preset data type, is accorded with from relational database
Close the inquiry data of inquiry request.
The data query method of the embodiment of the present invention passes through inquiry request after the inquiry request for receiving user's input
Determine whether user inquires the data of preset data type, and in the data for determining user query preset data type, when passing through
It is obtained in sequence database and meets the inquiry data of inquiry request, and determining that user do not inquire the data of preset data type,
The inquiry data for meeting inquiry request are then obtained from relational database.Unification is provided as a result, for data acquisition analysis system
Query interface, facilitate the data of type needed for user is inquired by unified query interface, improve user and adopted in data
Collection and the efficiency that data are inquired in monitoring system, improve the Experience Degree of user.
In order to realize above-described embodiment, the present invention also proposes a kind of data storage device.
Fig. 4 is the structural schematic diagram of data storage device according to an embodiment of the invention.
Wherein, it should be noted that the device is applied to data acquisition and comprehensive monitoring system, and data acquisition is supervised with comprehensive
Control system includes time series database and relational database.
As shown in figure 4, the apparatus may include:
First receiving module 110, for receiving the data storage request of external equipment, data storage request includes wait store
Data.
First judgment module 120, for judging whether the data type of data to be stored is preset data type.
First processing module 130, if for judging to know that the data type of data to be stored, will as preset data type
Data to be stored is saved to time series database.
Second processing module 140, if for judging to know that the data type of data to be stored is not preset data type,
Data to be stored is saved in relational database.
In one embodiment of the invention, on the basis of shown in Fig. 4, as shown in figure 5, the device further include:
Module 150 is obtained, for obtaining data structure packing rule corresponding with preset data type.
Package module 160, for carrying out construction packages to data to be stored according to data structure packing rule.
First processing module 130, specifically for calling access memory interface corresponding with time series database will be after encapsulation
Data write timing database.
In one embodiment of the invention, first judgment module 120 are specifically used for: extracting the data of data to be stored
Feature；Data characteristics data characteristics corresponding with preset data type is matched.
Wherein, if knowing data characteristics data characteristics matching corresponding with preset data type, it is determined that data to be stored
Data type be preset data type.
In one embodiment of the invention, data storage request further includes the device identification breath of external equipment, and first sentences
Whether disconnected module 120, is specifically used for: comprising equipment mark in judgement default device identification set corresponding with preset data type
Know.
Wherein, if judgement is known in default device identification set comprising device identification, it is determined that the data of data to be stored
Type is preset data type.
In one embodiment of the invention, in order to reduce the occupied memory space of storing data, base shown in Fig. 4
On plinth, as shown in fig. 6, the device can also include:
Compression module 170 presses the data for being stored in time series database for using preset data compression algorithm
Contracting.
Wherein, it should be noted that the structure of the compression module 170 in Installation practice shown in fig. 6 also may include
In aforementioned Installation practice shown in fig. 5, which is not construed as limiting this.
Wherein, it should be noted that the aforementioned explanation to date storage method embodiment is also applied for the embodiment
Data storage device, details are not described herein again.
The data storage device of the embodiment of the present invention, after getting data to be stored, in the number for determining data to be stored
According to type be preset data type when, data to be stored is saved by time series database；In the data class for determining data to be stored
When type is not preset data type, data to be stored is saved by relational database.It is mentioned as a result, for data acquisition analysis system
For unified data memory interface, and the database for saving data to be stored is determined by the data type of data to be stored, from
And it can will be suitble in the data rapid saving of time series database preservation to time series database, and will be suitble to protect in relevant database
The data rapid saving deposited saves corresponding data into relevant database, using the advantage of respective database, improves number
According to storage efficiency.
In order to realize above-described embodiment, the present invention also proposes a kind of data query device.
Fig. 7 is the structural schematic diagram of data query device according to an embodiment of the invention.
Wherein, it should be noted that the device is applied to data acquisition and comprehensive monitoring system, and data acquisition is supervised with comprehensive
Control system includes time series database and relational database.
As shown in fig. 7, the apparatus may include:
Second receiving module 210, for receiving the inquiry request of user's input.
Second judgment module 220, for judging whether user inquires the data of preset data type according to inquiry request.
Third processing module 230, if for judging to know the data of user query preset data type, from time series data
The inquiry data for meeting inquiry request are obtained in library.
Fourth processing module 240, if for judging to know the data that user does not inquire preset data type, from relationship number
Meet the inquiry data of inquiry request according to acquisition in library.
In one embodiment of the invention, the second judgment module 220, is specifically used for: parsing to inquiry request, really
Determine the corresponding query characteristics of inquiry request；Obtain default query characteristics corresponding to the data with preset data type；According to pre-
If the matching degree between query characteristics and query characteristics, determine whether user inquires the data of preset data type.
Wherein, it should be noted that the aforementioned explanation to data query embodiment of the method is also applied for the embodiment
Data query device, details are not described herein again.
The data query device of the embodiment of the present invention passes through inquiry request after the inquiry request for receiving user's input
Determine whether user inquires the data of preset data type, and in the data for determining user query preset data type, when passing through
The inquiry data for meeting inquiry request are obtained in sequence database；And determining that user do not inquire the data of preset data type,
The inquiry data for meeting inquiry request are then obtained from relational database.Unification is provided as a result, for data acquisition analysis system
Query interface, facilitate the data of type needed for user is inquired by unified query interface, improve user and adopted in data
Collection and the efficiency that data are inquired in monitoring system, improve the Experience Degree of user.
In order to realize above-described embodiment, the present invention also proposes a kind of data acquisition and comprehensive monitoring system.
Fig. 8 is the structural schematic diagram of data acquisition and comprehensive monitoring system according to an embodiment of the invention.
As shown in figure 8, data acquisition and comprehensive monitoring system may include data storage device 10, data query device
20, time series database 30 and relational database 40.
Wherein, it should be noted that the aforementioned explanation to data query device and data query device embodiment
It is also applied for the embodiment, details are not described herein again.
The data of the embodiment of the present invention acquire and comprehensive monitoring system, are saved by time series database and relational database each
Self-corresponding data, and according to the inquiry request of user, meet inquiry request from database corresponding with inquiry request acquisition
Data.The effect of system storing data is improved as a result, also, is provided unified inquiry for data acquisition analysis system and connect
Mouthful, the data of type needed for user is inquired by unified query interface are facilitated, improve user in data acquisition and monitoring
The efficiency that data are inquired in system, improves the Experience Degree of user.
In order to more clearly illustrate that the data acquisition of the embodiment of the present invention data and is looked into comprehensive monitoring system write-in
The process for asking data is written data with comprehensive monitoring system to data acquisition below with reference to Fig. 9 and Figure 10 and inquires data
Process is illustrated, and is described as follows:
Wherein, it should be noted that Figure 10 is introduced so that time series database is openTSDB as an example.
Wherein, openTSDB is a kind of expansible time series database of open source, provides extensive time series data storage service.
Wherein, openTSDB depends on HBase, supports cluster and extending transversely, bottom even more support big data technology
Hadoop technological frame is very suitable to big data storage and processing.
Wherein, it should be noted that Hadoop is a distributed system foundation frame developed by apache foundation
Structure mainly realizes distributed file system HDFS and distributed computing framework MapReduce.
Wherein, time series database is set in data acquisition and comprehensively monitoring SCADA system, and is protected by time series database
The detailed process of deposit data are as follows:
1. constructing time series database (openTSDB), and start service.
Time series database of the openTSDB based on HBase, therefore need to build HBase service, and HBase is based on Hadoop
Service, should build Hadoop cluster service first.
As an example, minimalist configuration i.e. 3 server is used with cluster in this example, Cloudera CDH is installed
Express software, this Integrated Simulation Hadoop and HBase service, according to official instructs document to install by step.
It is reconfigured openTSDB software based on this, it is specific to download on openTSDB software to any platform of 3 servers,
It decompresses and modifies its configuration file, mainly modify its zookeeper configuration item, being directed toward the IP address of 3 servers then can be with
It completes the dependence to HBase to be associated with, the configuration items such as port numbers keep default value.
Start openTSDB service, using order tsdb tsd&, starts and enters background mode, openTSDB is opened at this time
Default port is moved and monitored, data and inquiry instruction can be received by this port.
2. starting SCADA system, and monitored data changes.
Start SCADA main program, begins listening for the data of its acquisition.
3. giving the correct time when in data, original index data are pushed into time series database by interface.
After receiving acquisition data, encapsulation of data format is the data format that openTSDB is required and is sent to openTSDB prison
Port is listened, nodeID is specifically thought into metric, is above called time as timestamp, reporting numerical value is value, envelope
Dress is JSON data mode, and port both achievable data preservations of openTSDB monitoring are transmitted data to using http protocol.
Wherein, in data acquisition and the detailed process for inquiring data in comprehensively monitoring SCADA system are as follows:
1. upper layer application sends out inquiry request to historical data service according to uniform rules.
SCADA application software specifies start and end time point and index item, that is, nodeID, services to SCADA and initiates inquiry
Request.
2. initiating to inquire to openTSDB.
After receiving inquiry request, request command is switched into openTSDB query grammar, is specifically considered as nodeID
Metric, start and end time point are timestamp timestamp screening conditions, are sent to openTSDB using http agreement and supervise
Port is listened to complete inquiry request.
3. receiving and parsing returning the result
The part response response that query result, that is, http request returns is parsed, this partial data is openTSDB regulation
JSON data format, wherein value be inquiry end value, metric is nodeID, according to JSON rule take out
Every numerical value is converted to SCADA data format.
In order to realize above-described embodiment, the present invention also proposes a kind of computer readable storage medium, is stored thereon with calculating
Machine program, the program realize the date storage method of above-described embodiment when being executed by processor.
In order to realize above-described embodiment, the present invention also proposes a kind of computer readable storage medium, is stored thereon with calculating
Machine program, the program realize the data query method of above-described embodiment when being executed by processor.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is included at least one embodiment or example of the invention.In the present specification, schematic expression of the above terms are not
It must be directed to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be in office
It can be combined in any suitable manner in one or more embodiment or examples.In addition, without conflicting with each other, the skill of this field
Art personnel can tie the feature of different embodiments or examples described in this specification and different embodiments or examples
It closes and combines.
In addition, term " first ", " second " are used for descriptive purposes only and cannot be understood as indicating or suggesting relative importance
Or implicitly indicate the quantity of indicated technical characteristic.Define " first " as a result, the feature of " second " can be expressed or
Implicitly include at least one this feature.In the description of the present invention, the meaning of " plurality " is at least two, such as two, three
It is a etc., unless otherwise specifically defined.
Any process described otherwise above or method description are construed as in flow chart or herein, and expression includes
It is one or more for realizing custom logic function or process the step of executable instruction code module, segment or portion
Point, and the range of the preferred embodiment of the present invention includes other realization, wherein can not press shown or discussed suitable
Sequence, including according to related function by it is basic simultaneously in the way of or in the opposite order, Lai Zhihang function, this should be of the invention
Embodiment person of ordinary skill in the field understood.
Expression or logic and/or step described otherwise above herein in flow charts, for example, being considered use
In the order list for the executable instruction for realizing logic function, may be embodied in any computer-readable medium, for
Instruction execution system, device or equipment (such as computer based system, including the system of processor or other can be held from instruction
The instruction fetch of row system, device or equipment and the system executed instruction) it uses, or combine these instruction execution systems, device or set
It is standby and use.For the purpose of this specification, " computer-readable medium ", which can be, any may include, stores, communicates, propagates or pass
Defeated program is for instruction execution system, device or equipment or the dress used in conjunction with these instruction execution systems, device or equipment
It sets.The more specific example (non-exhaustive list) of computer-readable medium include the following: there is the electricity of one or more wirings
Interconnecting piece (electronic device), portable computer diskette box (magnetic device), random access memory (RAM), read-only memory
(ROM), erasable edit read-only storage (EPROM or flash memory), fiber device and portable optic disk is read-only deposits
Reservoir (CDROM).In addition, computer-readable medium can even is that the paper that can print described program on it or other are suitable
Medium, because can then be edited, be interpreted or when necessary with it for example by carrying out optical scanner to paper or other media
His suitable method is handled electronically to obtain described program, is then stored in computer storage.
It should be appreciated that each section of the invention can be realized with hardware, software, firmware or their combination.Above-mentioned
In embodiment, software that multiple steps or method can be executed in memory and by suitable instruction execution system with storage
Or firmware is realized.Such as, if realized with hardware in another embodiment, following skill well known in the art can be used
Any one of art or their combination are realized: have for data-signal is realized the logic gates of logic function from
Logic circuit is dissipated, the specific integrated circuit with suitable combinational logic gate circuit, programmable gate array (PGA), scene can compile
Journey gate array (FPGA) etc..
Those skilled in the art are understood that realize all or part of step that above-described embodiment method carries
It suddenly is that relevant hardware can be instructed to complete by program, the program can store in a kind of computer-readable storage medium
In matter, which when being executed, includes the steps that one or a combination set of embodiment of the method.
It, can also be in addition, each functional unit in each embodiment of the present invention can integrate in a processing module
It is that each unit physically exists alone, can also be integrated in two or more units in a module.Above-mentioned integrated mould
Block both can take the form of hardware realization, can also be realized in the form of software function module.The integrated module is such as
Fruit is realized and when sold or used as an independent product in the form of software function module, also can store in a computer
In read/write memory medium.
Storage medium mentioned above can be read-only memory, disk or CD etc..Although having been shown and retouching above
The embodiment of the present invention is stated, it is to be understood that above-described embodiment is exemplary, and should not be understood as to limit of the invention
System, those skilled in the art can be changed above-described embodiment, modify, replace and become within the scope of the invention
Type.
Claims (15)
1. a kind of date storage method, which is characterized in that the method is applied to data acquisition and comprehensive monitoring system, the number
It include time series database and relational database according to acquisition and comprehensive monitoring system, which comprises
The data storage request of external equipment is received, the data storage request includes data to be stored；
Whether the data type for judging the data to be stored is preset data type；
If judge to know the data type of the data to be stored as preset data type, by the data to be stored save to
The time series database；
If judgement knows that the data type of the data to be stored is not preset data type, the data to be stored is saved
To the relational database.
2. the method as described in claim 1, which is characterized in that it is described by the data to be stored save to it is described when ordinal number
Before library, further includes:
Obtain data structure packing rule corresponding with the preset data type；
Construction packages are carried out to the data to be stored according to the data structure packing rule；
It is described to save the data to be stored to the time series database, comprising:
Call access memory interface corresponding with the time series database that the time series database is written in the data after encapsulation.
3. the method as described in claim 1, which is characterized in that the data type for judging the data to be stored whether be
Preset data type, comprising:
Extract the data characteristics of the data to be stored；
Data characteristics data characteristics corresponding with the preset data type is matched；
Wherein, if knowing the data characteristics corresponding with preset data type data characteristics matching, it is determined that it is described to
The data type of storing data is preset data type.
4. the method as described in claim 1, which is characterized in that the data storage request further includes setting for the external equipment
Standby mark breath, whether the data type for judging the data to be stored is preset data type, comprising:
Judge in default device identification set corresponding with the preset data type whether to include the device identification；
Wherein, if judgement is known in default device identification set comprising the device identification, it is determined that the data to be stored
Data type is preset data type.
5. method according to any of claims 1-4, which is characterized in that by the data to be stored save to it is described when
After sequence database, further includes:
Using preset data compression algorithm, the data for being stored in the time series database are compressed.
6. a kind of data query method, which is characterized in that the method is applied to data acquisition and comprehensive monitoring system, the number
It include time series database and relational database according to acquisition and comprehensive monitoring system, which comprises
Receive the inquiry request of user's input；
Judge whether the user inquires the data of preset data type according to the inquiry request；
If the data of the user query preset data type are known in judgement, obtained from the time series database described in meeting
The inquiry data of inquiry request；
If judgement knows that the user does not inquire the data of preset data type, is obtained from the relational database and meet institute
State the inquiry data of inquiry request.
7. method as claimed in claim 6, which is characterized in that it is pre- to judge whether the user inquires according to the inquiry request
If the data of data type, comprising:
The inquiry request is parsed, determines the corresponding query characteristics of the inquiry request；
Obtain default query characteristics corresponding to the data with the preset data type；
According to the matching degree between the default query characteristics and the query characteristics, determine whether the user inquires present count
According to the data of type.
8. a kind of data storage device, which is characterized in that described device is applied to data acquisition and comprehensive monitoring system, the number
It include time series database and relational database according to acquisition and comprehensive monitoring system, described device includes:
First receiving module, for receiving the data storage request of external equipment, the data storage request includes number to be stored
According to；
First judgment module, for judging whether the data type of the data to be stored is preset data type；
First processing module, if for judging to know the data type of the data to be stored as preset data type, by institute
Data to be stored is stated to save to the time series database；
Second processing module, if for judging to know that the data type of the data to be stored is not preset data type, it will
The data to be stored is saved in the relational database.
9. device as claimed in claim 8, which is characterized in that further include:
Module is obtained, for obtaining data structure packing rule corresponding with the preset data type；
Package module, for carrying out construction packages to the data to be stored according to the data structure packing rule；
The first processing module, specifically for calling access memory interface corresponding with the time series database will be after encapsulation
The time series database is written in data.
10. device as claimed in claim 8, which is characterized in that the first judgment module is specifically used for:
Extract the data characteristics of the data to be stored；
Data characteristics data characteristics corresponding with the preset data type is matched；
Wherein, if knowing the data characteristics corresponding with preset data type data characteristics matching, it is determined that it is described to
The data type of storing data is preset data type.
11. device as claimed in claim 8, which is characterized in that the data storage request further includes the external equipment
Device identification breath, described is preset data type judging module, is specifically used for:
Judge in default device identification set corresponding with the preset data type whether to include the device identification；
Wherein, if judgement is known in default device identification set comprising the device identification, it is determined that the data to be stored
Data type is preset data type.
12. such as the described in any item devices of claim 8-11, which is characterized in that further include:
Compression module compresses the data for being stored in the time series database for using preset data compression algorithm.
13. a kind of data query device, which is characterized in that described device is applied to data acquisition and comprehensive monitoring system, described
Data acquisition and comprehensive monitoring system include time series database and relational database, and described device includes:
Second receiving module, for receiving the inquiry request of user's input；
Second judgment module, for judging whether the user inquires the data of preset data type according to the inquiry request；
Third processing module, if for judging to know the data of the user query preset data type, from it is described when ordinal number
Meet the inquiry data of the inquiry request according to acquisition in library；
Fourth processing module, if for judging to know the data that the user does not inquire preset data type, from the relationship
The inquiry data for meeting the inquiry request are obtained in database.
14. device as claimed in claim 13, which is characterized in that the second judgment module is specifically used for:
The inquiry request is parsed, determines the corresponding query characteristics of the inquiry request；
Obtain default query characteristics corresponding to the data with the preset data type；
According to the matching degree between the default query characteristics and the query characteristics, determine whether the user inquires present count
According to the data of type.
15. a kind of acquisition of data and comprehensive monitoring system, the data acquisition and comprehensive monitoring system include time series database and
Relational database, the system comprises:
Such as the described in any item data storage devices of claim 8-12；
Such as the described in any item data query devices of claim 13-14.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810934580.XA CN110019247A (en) | 2018-08-16 | 2018-08-16 | Data storage and querying method, device and monitoring system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810934580.XA CN110019247A (en) | 2018-08-16 | 2018-08-16 | Data storage and querying method, device and monitoring system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110019247A true CN110019247A (en) | 2019-07-16 |
Family
ID=67188375
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810934580.XA Pending CN110019247A (en) | 2018-08-16 | 2018-08-16 | Data storage and querying method, device and monitoring system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019247A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110377628A (en) * | 2019-07-23 | 2019-10-25 | 京东方科技集团股份有限公司 | A kind of information acquisition method, device and electronic equipment |
| CN110389967A (en) * | 2019-07-26 | 2019-10-29 | 深圳市腾讯计算机系统有限公司 | Date storage method, device, server and storage medium |
| CN110489445A (en) * | 2019-08-02 | 2019-11-22 | 四川宏力信息科技有限责任公司 | It is a kind of based on polymorphic compound mass data method for quickly querying |
| CN110837516A (en) * | 2019-11-07 | 2020-02-25 | 恩亿科(北京)数据科技有限公司 | Data cutting and connecting method and device, computer equipment and readable storage medium |
| CN110851443A (en) * | 2019-10-28 | 2020-02-28 | 网联清算有限公司 | Database storage management method and device, storage medium and electronic equipment |
| CN111008230A (en) * | 2019-11-22 | 2020-04-14 | 远景智能国际私人投资有限公司 | Data storage method and device, computer equipment and storage medium |
| CN111144926A (en) * | 2019-11-27 | 2020-05-12 | 泰康保险集团股份有限公司 | Service request processing method, device, system, electronic equipment and readable medium |
| CN112506735A (en) * | 2020-11-26 | 2021-03-16 | 中移（杭州）信息技术有限公司 | Service quality monitoring method, system, server and storage medium |
| CN112667604A (en) * | 2021-01-04 | 2021-04-16 | 中车青岛四方车辆研究所有限公司 | Time sequence data and data storage method and device of rail transit TIAS system |
| CN112766713A (en) * | 2021-01-15 | 2021-05-07 | 国家海洋信息中心 | Data processing method, equipment and storage medium for ocean emergency command and dispatch |
| CN112800092A (en) * | 2021-01-26 | 2021-05-14 | 苏州浪潮智能科技有限公司 | Data cache management method, device and equipment and readable storage medium |
| CN114205389A (en) * | 2021-12-08 | 2022-03-18 | 南方电网深圳数字电网研究院有限公司 | Intelligent information screening method and device based on Internet of things |
| CN114205372A (en) * | 2021-12-08 | 2022-03-18 | 南方电网深圳数字电网研究院有限公司 | Data storage method and device of Internet of things |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102968438A (en) * | 2012-09-29 | 2013-03-13 | 南京国电南自轨道交通工程有限公司 | Storage control method of history data in integrated supervisory control system |
| CN106055678A (en) * | 2016-06-07 | 2016-10-26 | 国网河南省电力公司电力科学研究院 | Hadoop-based panoramic big data distributed storage method |
| CN106777108A (en) * | 2016-12-15 | 2017-05-31 | 贵州电网有限责任公司电力科学研究院 | A kind of data query method and apparatus based on mixing storage architecture |
| US20180032567A1 (en) * | 2015-02-27 | 2018-02-01 | Alibaba Group Holding Limited | Method and device for processing data blocks in a distributed database |
-
2018
- 2018-08-16 CN CN201810934580.XA patent/CN110019247A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102968438A (en) * | 2012-09-29 | 2013-03-13 | 南京国电南自轨道交通工程有限公司 | Storage control method of history data in integrated supervisory control system |
| US20180032567A1 (en) * | 2015-02-27 | 2018-02-01 | Alibaba Group Holding Limited | Method and device for processing data blocks in a distributed database |
| CN106055678A (en) * | 2016-06-07 | 2016-10-26 | 国网河南省电力公司电力科学研究院 | Hadoop-based panoramic big data distributed storage method |
| CN106777108A (en) * | 2016-12-15 | 2017-05-31 | 贵州电网有限责任公司电力科学研究院 | A kind of data query method and apparatus based on mixing storage architecture |
Non-Patent Citations (1)
| Title |
|---|
| 单若琦: "一种基于OpenTSDB的海量实时数据存储系统", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110377628A (en) * | 2019-07-23 | 2019-10-25 | 京东方科技集团股份有限公司 | A kind of information acquisition method, device and electronic equipment |
| CN110389967A (en) * | 2019-07-26 | 2019-10-29 | 深圳市腾讯计算机系统有限公司 | Date storage method, device, server and storage medium |
| CN110489445A (en) * | 2019-08-02 | 2019-11-22 | 四川宏力信息科技有限责任公司 | It is a kind of based on polymorphic compound mass data method for quickly querying |
| CN110851443A (en) * | 2019-10-28 | 2020-02-28 | 网联清算有限公司 | Database storage management method and device, storage medium and electronic equipment |
| CN110837516A (en) * | 2019-11-07 | 2020-02-25 | 恩亿科(北京)数据科技有限公司 | Data cutting and connecting method and device, computer equipment and readable storage medium |
| CN111008230B (en) * | 2019-11-22 | 2023-08-04 | 远景智能国际私人投资有限公司 | Data storage method, device, computer equipment and storage medium |
| CN111008230A (en) * | 2019-11-22 | 2020-04-14 | 远景智能国际私人投资有限公司 | Data storage method and device, computer equipment and storage medium |
| CN111144926A (en) * | 2019-11-27 | 2020-05-12 | 泰康保险集团股份有限公司 | Service request processing method, device, system, electronic equipment and readable medium |
| CN112506735A (en) * | 2020-11-26 | 2021-03-16 | 中移（杭州）信息技术有限公司 | Service quality monitoring method, system, server and storage medium |
| CN112667604A (en) * | 2021-01-04 | 2021-04-16 | 中车青岛四方车辆研究所有限公司 | Time sequence data and data storage method and device of rail transit TIAS system |
| CN112766713A (en) * | 2021-01-15 | 2021-05-07 | 国家海洋信息中心 | Data processing method, equipment and storage medium for ocean emergency command and dispatch |
| CN112800092A (en) * | 2021-01-26 | 2021-05-14 | 苏州浪潮智能科技有限公司 | Data cache management method, device and equipment and readable storage medium |
| CN114205389A (en) * | 2021-12-08 | 2022-03-18 | 南方电网深圳数字电网研究院有限公司 | Intelligent information screening method and device based on Internet of things |
| CN114205372A (en) * | 2021-12-08 | 2022-03-18 | 南方电网深圳数字电网研究院有限公司 | Data storage method and device of Internet of things |
| CN114205389B (en) * | 2021-12-08 | 2023-12-01 | 南方电网数字平台科技(广东)有限公司 | Information intelligent screening method and device based on Internet of Things |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019247A (en) | Data storage and querying method, device and monitoring system | |
| CN107577805A (en) | A kind of business service system towards the analysis of daily record big data | |
| CN110263054A (en) | Auditing system, checking method, device and the computer equipment of SQL work order | |
| US20160140235A1 (en) | Real-time distributed in memory search architecture | |
| US10025718B1 (en) | Modifying provisioned throughput capacity for data stores according to cache performance | |
| US11355123B2 (en) | Network data aligning | |
| CN108021809A (en) | A kind of data processing method and system | |
| CN109710612A (en) | Vector index recalls method, apparatus, electronic equipment and storage medium | |
| CN108092813A (en) | Data center's total management system server hardware Governance framework and implementation method | |
| CN111881011A (en) | Log management method, platform, server and storage medium | |
| CN111641520B (en) | Edge computing node device | |
| CN106909641B (en) | Real-time data memory | |
| CN109902126B (en) | Loading system supporting HIVE automatic partition and implementation method thereof | |
| CN112100277A (en) | Method, system, equipment and product for realizing enterprise data chaining prediction machine | |
| CN105989140B (en) | A kind of data block processing method and equipment | |
| CN114817425B (en) | Method, device and equipment for classifying cold and hot data and readable storage medium | |
| CN111046022A (en) | Database auditing method based on big data technology | |
| CN108182263A (en) | A kind of date storage method of data center's total management system | |
| CN113407600A (en) | Enhanced real-time calculation method for dynamically synchronizing multi-source large table data in real time | |
| CN114780335A (en) | Correlation method and device of monitoring data, computer equipment and storage medium | |
| CN107491558B (en) | Metadata updating method and device | |
| CN112905571B (en) | Train rail transit sensor data management method and device | |
| CN109510730A (en) | Distributed system and its monitoring method, device, electronic equipment and storage medium | |
| US20200364211A1 (en) | Predictive database index modification | |
| CN110221929A (en) | A kind of service software system architecture and its application method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190716 |
|
| RJ01 | Rejection of invention patent application after publication |