Disclosure of Invention
In view of this, the invention provides a data query method and device based on an elastic search, which replaces the traditional historical database service with the elastic search, thereby significantly improving the query performance and the retrieval efficiency.
In order to achieve the above purpose, the invention provides the following specific technical scheme:
a data query method based on an ElasticSearch is applied to an SCADA system based on the ElasticSearch, and comprises the following steps:
receiving a data query request;
and calling a real-time service process, and inquiring target data in the ElasticSearch by accessing the public data structure index, the data historical data structure index and the historical data value index.
Optionally, the querying target data in an ElasticSearch by accessing the common data structure index, the data history data structure index, and the history data value index includes:
inquiring a target database unit number corresponding to the filing time of the target data by accessing the public data structure index;
querying a target point name corresponding to the unit number of the target database and a target local point number corresponding to the target point name by accessing the data historical data structure index;
and inquiring the value of the target data corresponding to the target local point number by accessing the historical data value index.
Optionally, the method further includes:
the CCtool configuration page is extended to support the configuration of the ElasticSearch.
Optionally, before the invoking the real-time service process, the method further includes:
acquiring the state of an ElasticSearch service;
if the ElasticSearch service is not available, waiting for the shared disk manager to start the ElasticSearch service in the disk array until the ElasticSearch service is available.
Optionally, the method further includes:
under the condition of starting the SCADA online service, calling the real-time service process, collecting data in real time and recording the data into a local historical data file;
and storing the data in the local historical data file to an ElasticSearch by accessing the public data structure index, the data historical data structure index and the historical data value index.
Optionally, the storing data in the local history data file to an ElasticSearch by accessing the common data structure index, the data history data structure index, and the history data value index includes:
obtaining a database unit number and data archiving time by accessing the public data structure index;
obtaining a mapping relation between the roll names and the local roll numbers by accessing the historical data structure index of the data;
inquiring whether a local point number of data in a local historical data file exists or not;
and if not, self-adding the local point number into the data history data structure index, and storing the data in the local history data file in the ElasticSearch batch.
Optionally, the method further includes:
when the bulk storage fails, the data in the local history data file is stored in the elastic search stripe by stripe.
Optionally, the method further includes:
and periodically carrying out aggregation calculation according to a preset aggregation algorithm, and storing a corresponding aggregation value into the ElasticSearch.
Optionally, the querying target data in the ElasticSearch by accessing the common data structure index and the data history data structure index includes
Judging whether a target aggregation value exists in the ElasticSearch or not by accessing the public data structure index and the data history data structure index;
if yes, determining the target aggregation value as the target data;
if not, inquiring original data corresponding to the target aggregation value in an elastic search, and calculating the target aggregation value according to the original data corresponding to the target aggregation value.
An elastic search based data query device applied to an elastic search based SCADA system, the device comprising:
a query request receiving unit, configured to receive a data query request;
and the data query unit is used for calling the real-time service process and querying the target data in the ElasticSearch by accessing the public data structure index, the data historical data structure index and the historical data value index.
Optionally, the data query unit is specifically configured to:
inquiring a target database unit number corresponding to the filing time of the target data by accessing the public data structure index;
querying a target point name corresponding to the unit number of the target database and a target local point number corresponding to the target point name by accessing the data historical data structure index;
and inquiring the value of the target data corresponding to the target local point number by accessing the historical data value index.
Optionally, the apparatus further comprises:
and the configuration page extension unit is used for extending the CCtool configuration page to support the configuration of the ElasticSearch.
Optionally, the apparatus further comprises:
a service state acquiring unit, configured to acquire a state of an ElasticSearch service; if the ElasticSearch service is not available, waiting for the shared disk manager to start the ElasticSearch service in the disk array until the ElasticSearch service is available.
Optionally, the apparatus further comprises:
the data archiving unit is used for calling the real-time service process under the condition of starting the SCADA online service, collecting data in real time and recording the data into a local historical data file; and storing the data in the local historical data file to an ElasticSearch by accessing the public data structure index, the data historical data structure index and the historical data value index.
Optionally, the data archiving unit is specifically configured to:
obtaining a database unit number and data archiving time by accessing the public data structure index;
obtaining a mapping relation between the roll names and the local roll numbers by accessing the historical data structure index of the data;
inquiring whether a local point number of data in a local historical data file exists or not;
and if not, self-adding the local point number into the data history data structure index, and storing the data in the local history data file in the ElasticSearch batch.
Optionally, the data archiving unit is further configured to store data in the local history data file item by item in the elastic search when the bulk storage fails.
Optionally, the apparatus further comprises:
and the aggregation calculation unit is used for periodically performing aggregation calculation according to a preset aggregation algorithm and storing a corresponding aggregation value into the ElasticSearch.
Optionally, the data query unit is specifically configured to:
judging whether a target aggregation value exists in the ElasticSearch or not by accessing the public data structure index and the data history data structure index;
if yes, determining the target aggregation value as the target data;
if not, inquiring original data corresponding to the target aggregation value in an elastic search, and calculating the target aggregation value according to the original data corresponding to the target aggregation value.
Compared with the prior art, the invention has the following beneficial effects:
the invention discloses a data query method based on an ElasticSearch, which is applied to an SCADA system based on the ElasticSearch, utilizes the ElasticSearch to replace the traditional historical database service, utilizes the characteristics of clusters, nodes, fragments, full text retrieval and the like of the ElasticSearch, and adds a public data structure index, a data historical data structure index and a historical data value index on the ElasticSearch, thereby realizing the query function in the ElasticSearch, obviously improving the query performance and improving the retrieval efficiency.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment discloses a data query method based on an elastic search, which is applied to an SCADA system based on the elastic search, namely the elastic search is used for replacing a traditional historical database service, the configuration of the elastic search is supported by expanding a CCtool configuration page, the management function of a shared disk manager is expanded, the start-stop control of the elastic search in a disk array is supported, and please refer to FIG. 1, the data query method disclosed by the embodiment comprises the following steps:
s101: receiving a data query request;
specifically, the user may send a data query request through an online HMI (Human Machine Interface, or Human Machine Interface), or send a data query request through a webAPI.
The data query request includes query conditions, such as archive time, etc.
It should be noted that before the ElasticSearch service is used, the state of the ElasticSearch service needs to be acquired first, and if the ElasticSearch service is not available, the shared disk manager waits for starting the ElasticSearch service in the disk array until the ElasticSearch service is available.
S102: and calling a real-time service process, and inquiring target data in the ElasticSearch by accessing the public data structure index, the data historical data structure index and the historical data value index.
The method is characterized in that an ElasticSearch is used for replacing a traditional historical database service, the architecture of a SCADA system needs to be redesigned, a holysys _ run service is transversely expanded on the original design of the SCADA system, and the structure of the holysys _ run service comprises a public data structure index, a data historical data structure index and a historical data value index.
The index records data filing time, historical data are filed by taking a database unit as a unit, and relevant fields of the index are shown in table 1.
Table 1 common data structure index
The data history data structure index is only one in the database, a local point number is obtained through a point name NodeId, the local point number is represented by an integer of the point name, when a new record is added, the local point number is +1 of the number of records before the addition, and the structure of the data history data structure index is shown in a table 2.
TABLE 2 data History data Structure indexing
The historical data value index stores analog quantity type data or switching value type data, storage according to year, quarter, month and day (natural year, month and the like, UTC time) is supported, and parameters can be matched.
Document id: PartialNodeId + nodevalueserver time, i.e., local point number + server timestamp. The structure of the historical data value index is shown in table 3.
TABLE 3 historical data value index
When target data is inquired in an ElasticSearch by accessing a public data structure index, a data historical data structure index and a historical data value index, firstly, inquiring a target database unit number corresponding to the filing time of the target data by accessing the public data structure index; then, by accessing the historical data structure index of the data, inquiring a target point name corresponding to the unit number of the target database and a target local point number corresponding to the target point name; and finally, inquiring the value of the target data corresponding to the target local point number by accessing the historical data value index.
It can be seen that the data query method based on the elastic search disclosed in this embodiment is applied to the SCADA system based on the elastic search, replaces the conventional historical database service with the elastic search, and adds the public data structure index, the data historical data structure index, and the historical data value index to the elastic search by using the characteristics of the cluster, node, segment, full-text search, and the like of the elastic search, thereby realizing the query function in the elastic search, significantly improving the query performance, and improving the search efficiency.
Referring to fig. 2, the data query method based on ElasticSearch disclosed in this embodiment may further implement data archiving, and the data archiving method disclosed in this embodiment includes the following steps:
s201: under the condition of starting the SCADA online service, calling a real-time service process, collecting data in real time and recording the data into a local historical data file;
s202: the data in the local history data file is stored to the ElasticSearch by accessing the common data structure index and the data history data structure index.
Obtaining a database unit number and data archiving time by accessing the public data structure index;
obtaining a mapping relation between the roll names and the local roll numbers by accessing the historical data structure index of the data;
inquiring whether a local point number of data in a local historical data file exists or not;
if the local point number does not exist, the local point number is added to the data history data structure index in a self-increment mode, and data in a local history data file is stored in an ElasticSearch batch mode;
when the bulk storage fails, the data in the local history data file is stored in the elastic search stripe by stripe.
Preferably, the data query method based on the ElasticSearch disclosed in this embodiment may further implement a pre-aggregation function, perform aggregation calculation periodically according to a preset aggregation algorithm, such as a maximum value, a minimum value, an average value, an accumulated value, a time value, an increment, a duration, and store a corresponding aggregation value in the ElasticSearch.
Examples are: maximum prepolymerization was carried out every 1 hour for No. 7/11 in 2019, with 24 aggregation values per nodeid on the day of No. 7/11 in 2019.
Judging whether a target aggregation value exists in the ElasticSearch or not by accessing the public data structure index and the data history data structure index;
if yes, determining the target aggregation value as the target data;
if not, inquiring original data corresponding to the target aggregation value in an elastic search, and calculating the target aggregation value according to the original data corresponding to the target aggregation value.
Therefore, through the realization of the pre-polymerization function, if the condition is met during query, the aggregation value is directly obtained from the pre-polymerization, and the query is not from massive original data, so that the query performance is greatly improved.
Based on the data query method based on the above described embodiments, this embodiment correspondingly discloses a data query device based on the ElasticSearch, which is applied to the SCADA system based on the ElasticSearch, please refer to fig. 3, and the device includes:
a query request receiving unit 301, configured to receive a data query request;
and the data query unit 302 is configured to invoke a real-time service process, and query the target data in the ElasticSearch by accessing the public data structure index, the data history data structure index, and the history data value index.
Optionally, the data querying unit 302 is specifically configured to:
inquiring a target database unit number corresponding to the filing time of the target data by accessing the public data structure index;
querying a target point name corresponding to the unit number of the target database and a target local point number corresponding to the target point name by accessing the data historical data structure index;
and inquiring the value of the target data corresponding to the target local point number by accessing the historical data value index.
Optionally, the apparatus further comprises:
and the configuration page extension unit is used for extending the CCtool configuration page to support the configuration of the ElasticSearch.
Optionally, the apparatus further comprises:
a service state acquiring unit, configured to acquire a state of an ElasticSearch service; if the ElasticSearch service is not available, waiting for the shared disk manager to start the ElasticSearch service in the disk array until the ElasticSearch service is available.
Optionally, the apparatus further comprises:
the data archiving unit is used for calling the real-time service process under the condition of starting the SCADA online service, collecting data in real time and recording the data into a local historical data file; and storing the data in the local historical data file to an ElasticSearch by accessing the public data structure index, the data historical data structure index and the historical data value index.
Optionally, the data archiving unit is specifically configured to:
obtaining a database unit number and data archiving time by accessing the public data structure index;
obtaining a mapping relation between the roll names and the local roll numbers by accessing the historical data structure index of the data;
inquiring whether a local point number of data in a local historical data file exists or not;
and if not, self-adding the local point number into the data history data structure index, and storing the data in the local history data file in the ElasticSearch batch.
Optionally, the data archiving unit is further configured to store data in the local history data file item by item in the elastic search when the bulk storage fails.
Optionally, the apparatus further comprises:
and the aggregation calculation unit is used for periodically performing aggregation calculation according to a preset aggregation algorithm and storing a corresponding aggregation value into the ElasticSearch.
Optionally, the data querying unit 302 is specifically configured to:
judging whether a target aggregation value exists in the ElasticSearch or not by accessing the public data structure index and the data history data structure index;
if yes, determining the target aggregation value as the target data;
if not, inquiring original data corresponding to the target aggregation value in an elastic search, and calculating the target aggregation value according to the original data corresponding to the target aggregation value.
The data query device based on the ElasticSearch disclosed by the embodiment is applied to an SCADA system based on the ElasticSearch, replaces the traditional historical database service by utilizing the ElasticSearch, and adds a public data structure index, a data historical data structure index and a historical data value index on the ElasticSearch by utilizing the characteristics of clusters, nodes, fragments, full-text retrieval and the like of the ElasticSearch, so that the query function in the ElasticSearch is realized, the query performance is obviously improved, and the retrieval efficiency is improved.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.