CN111427964A - Industrial cloud data storage model for running timestamp - Google Patents
Industrial cloud data storage model for running timestamp Download PDFInfo
- Publication number
- CN111427964A CN111427964A CN202010296258.6A CN202010296258A CN111427964A CN 111427964 A CN111427964 A CN 111427964A CN 202010296258 A CN202010296258 A CN 202010296258A CN 111427964 A CN111427964 A CN 111427964A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- stamp
- time
- runtime
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2219—Large Object storage; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to an industrial cloud data storage model facing to an operating timestamp. The operation data of the industrial enterprise needs to meet the requirement of life span or permanent storage for the time length; for time intervals, involving sampling timestamps, the temporal scale to meet is within the expertise-aware coverage of the operation and maintenance personnel, with sampling times of approximately from 100 milliseconds to 10 minutes. The invention specifically provides a serialized data storage compression method surrounding an operating timestamp, which is characterized in that a stored data information file is created based on a special micro service program to form an operating timestamp pedigree of a data storage model, and a real-time data set with the operating timestamp is stored. The method can meet the requirements of monitoring, storing and analyzing mass operation data of enterprises in an industrial internet scene, provides the openness and clearness of a basic platform and storage access under the condition of using low-price resources, and promotes the industrial data cloud information technology to realize distributed substitution, reconstruction and expansion of an enterprise information architecture.
Description
Technical Field
The invention relates to a storage model, in particular to an industrial cloud data storage model facing to an operation timestamp, and belongs to the technical field of information of novel real-time/historical data cores of an industrial internet established on a big data cloud computing system architecture.
Background
the industrial enterprise for large-scale continuous output and maintenance by physical or chemical changes of substances mainly comprises electric power, nuclear energy, petroleum, chemical engineering, metallurgy, building materials, water conservancy, environmental protection and the like, and operation operators must monitor and control the production processes through various industrial control systems to realize safe, stable and efficient operation.
Currently, industrial enterprises operate by transmitting field data acquired by an industrial control system to a real-time/historical database for service monitoring, scheduling and long-term data storage, thereby forming a second layer of an enterprise information architecture. The real-time/historical database can meet the requirements of quickly and efficiently acquiring, storing and displaying data, and simultaneously provides data support for the third layer of an enterprise information architecture, namely a management information and operation decision system.
While these existing real-time/historical databases are all specifically designed, non-relational, proprietary databases, most are designed and deployed based on a central database server schema, rather than in a distributed data cluster schema. At present, industrial internet technology is gradually permeated and developed, the total amount of data information is rapidly increased, and the traditional technical scheme based on the real-time/historical database of the central database server mode shows the disadvantages of high cost of reconstruction and maintenance, insufficient data throughput and difficulty in elastic expansion. With rapid progress and gradual maturity of new-generation information technologies such as industrial internet, big data, cloud computing, internet of things, artificial intelligence and the like, design and deployment of a real-time/historical database for operating industrial enterprises are about to meet the opportunity of major technological progress, and the urgent requirements of monitoring, storing and analyzing massive operation data of the enterprises in the industrial internet scene are mainly met by applying the big data and cloud computing technologies.
In a traditional real-time/historical database based on a central database server mode, storage models are designed and compiled by each database manufacturer, data compression technologies are different, and a second-layer data island of an enterprise information framework is formed to a certain extent. In the novel distributed real-time/historical data cluster based on big data cloud computing, the basic platform technology is open-source universal, and the storage model of the novel distributed real-time/historical data cluster is also open-source universal in principle, so that the design with wide applicability and format clearness is required. It should be appreciated that the operational data of these industrial enterprises, although the storage values are very different and the storage locations guide their respective definitions, have two common basic characteristics in terms of time, namely the time length of data storage and the time interval of data storage. For the length of time, 10 years, 20 years, full life, or permanent storage is to be satisfied. For the time interval, referring to the sampling time stamp, it is required for the operation operator and the maintenance personnel who have been specially learned and trained to sense the dynamic change of the process data after being skilled in the control system, that is, the time scale of the data is within the professional ability coverage of the operation operator and the maintenance personnel, and the sampling time is about from 100 milliseconds to 10 minutes. Therefore, sampling time stamping becomes a core of attention of industrial data cloud with wide applicability and clearness of format.
The invention provides an industrial cloud data storage model facing to an operation timestamp, solves the problems of wide applicability and format clearness, promotes the industrial cloud data cluster technology to realize distributed substitution and reconstruction of a second layer of an enterprise information architecture, and has the advantages of data source opening, large throughput, good expansibility and the like.
After searching, the following results are found:
Search result 1, patent application publication No.: CN107229639A, name: storage system for distributed real-time databases, 2017.10.
And (3) abstract: this patent provides a storage system for a distributed real-time database, comprising: a distributed data gateway module: a receiving and querying agent for data; a distributed message queue module: the data distribution and subscription are realized through the distributed message queue; the distributed real-time stream computing service module: the method is used for realizing message subscription service, memory snapshot storage service and persistent storage service based on a storm real-time stream computing framework; the distributed cache service module: and carrying out distributed storage service through a Nosql database HBase, and realizing data retrieval through a distributed search engine Solr. The patent effectively improves the intellectualization and informatization level of enterprises, utilizes a big data technology to mine potential data value, and provides a solid data base for the transformation development of the enterprises.
the distributed real-time stream computing service module is used for realizing message subscription service, memory snapshot storage service and persistent storage service based on a storm real-time stream computing framework, and the distributed cache service module is used for performing distributed storage service through a Nosql database HBase and realizing data retrieval through a distributed search engine Solr.
Search result 2, patent application publication No.: CN110196885A, name: a clouded distributed real-time database system, 2019.9.
the patent discloses a cloud distributed real-time database system which comprises a real-time database and a real-time service bus, wherein the real-time database comprises a storage layer, a positioning layer and an interface layer, the storage layer is based on a distributed NoSQ L database storage model, the real-time model corresponds to a relation model of a real-time relation database of a power grid system, the positioning layer is used for storing a fragment memory snapshot, the fragment memory snapshot comprises a monitoring model snapshot, a numerical section snapshot, a transaction snapshot and the like, the interface layer provides a transaction component for establishing, accessing and processing the fragment snapshot memory, the real-time service bus comprises a model loading service, a fat client service and the like, the patent is based on a CIM model grouping and slicing technology of an intelligent power grid regulation and control system, realizes real-time computing processing of a fat client and quasi real-time access of a thin client, expands model checking and synchronization functions, and improves concurrence control and extension expansibility of the system.
the patent is specially designed for the power grid system, the NoSQ L real-time model is corresponding to the CIM and appoints the NoSQ L as a key value table, maintenance primitive operation from the CIM to the NoSQ L is designed, whether the NoSQ L is from a third party or is designed and compiled by itself is not limited, the patent discloses that the NoSQ is mapping of the CIM, then the fragmentation internal memory is realized, the key value pair of the NoSQ is mapped by itself as the CIM, the value pair of the NoSQ is mapped by itself as storage data, the invention is designed based on the key value pair of the CIM, the key value pair of the NoSQ is mapped by itself as storage data, the invention is similar to the key value pair of the CISQ, the invention is a double key value pair of the NoSQ as a special key value pair of the patent, and the invention is a special key value pair of the multi-state key pair of the NoSQ as a special key value pair of the CIM.
Search result 3, scientific article, name: a cloud storage method facing to industrial internet, which is the university of Beijing aerospace bulletin 2019.1.
And (3) abstract: the industrial internet is the most concerned hotspot in the industrial informatization process, and the management of massive heterogeneous data is one of the key points. The traditional Relational Database (RDB) has performance bottleneck in reading, writing and retrieving mass multi-source heterogeneous data, and recently, the emerging cloud data management method mainly aims at a key-value (K-V) mode and cannot quickly search data by means of data attributes except a main key. The storeCDB realizes unstructured storage management on the basis of a heterogeneous sampling data unified expression data model, and meanwhile, quick retrieval of mass data is realized by using two-stage indexes. Through experiments, a new data management method is provided for the industrial internet on a distributed cluster experiment platform.
The technical points are compared: the thesis discloses a StoreCDB cloud storage data management structure, and a uniform data format D is composed of triplets, which can be expressed as D ═ data (timeStamp), where: propertySet represents a set of attributes, and timeStamp represents a data reception time. The patent firstly maps the attribute value defined as the main key in D to the corresponding paging storage space, and then finds the corresponding storage page according to the data dividing mode. The storage page dataPage is denoted as dataPage ═ (span key, (Di) n, mapldx, comldx), where: the spanKey represents a data division mode and can be divided according to receiving time, value range division of certain attribute, sensor type division and the like. The invention starts from the actual operation control and is oriented to the characteristic of storing data by a timestamp; deployment is carried out around a timestamp in all aspects, and the deployment is not carried out according to a storage page; recording attribute sets every time is not considered in data storage design, and attribute-based division and sensor division are not considered in corresponding division of a storage page; meanwhile, an efficient and complete data compression function facing to the timestamp, which is not related by the paper, is designed. Therefore, the specific method of storing the model of the present invention is different from the patent.
Disclosure of Invention
The invention provides an industrial cloud data storage model facing to an operation timestamp, along with the development of an industrial internet technology, a novel information architecture of a real-time/historical database is designed and deployed according to a distributed data cluster mode, the urgent requirements of monitoring, storing and analyzing mass operation data of an enterprise in an industrial internet scene are met by applying big data and a cloud computing technology, certain reasonable data compression is ensured under the condition of using low-price computer server hardware and storage resources, a basic platform and a storage model technology are provided for being universal, the requirements of rapidly storing, monitoring and analyzing data from an industrial control system in real time are met, and the distributed substitution and reconstruction of a second layer of the enterprise information architecture are promoted by the industrial cloud data cluster technology. The computing processing function of the open source system provided by the big data cloud computing technology platform is used as the basis and the starting point of the invention, and is different from various traditional industrial real-time/historical databases based on a central database server mode.
In order to achieve the aim, the technical scheme of the invention is that the industrial cloud data storage model facing the runtime stamp comprises a client, an application server, a micro-service and a data cluster,
The application server responds to a real-time/historical data storage or query request from a client, receives field industrial control system data from a first layer of an enterprise information architecture, packages data with different running timestamps for communication transmission, and calls a corresponding micro-service program according to the request of the client to process a packed data packet;
The micro service program is divided into two types according to the service, one is a storage micro service, the other is a query micro service, and each type is continuously subdivided according to different specific implementation functions;
The data cluster is responsible for managing, monitoring, storing point information and data information following a cloud data storage model.
The system adopts a distributed master-slave organization structure, and a plurality of servers are connected through a high-speed network to form a cluster. The core of the data cluster is a main node server which is responsible for managing basic information (file name, size, storage location and the like) of data files in the cluster and processing access requests of micro-services to the files. The point information and the data information are physically stored in a plurality of slave node servers and are convenient and flexible to expand corresponding to the division of the subsystem nodes of the industrial enterprise, multiple backup is required to be carried out on the point information and the data information, and data loss possibly caused by breakdown of some servers in a cluster is avoided. The characteristics of multiple redundancy and elastic expansion enable the cluster to have high reliability, high maintainability and high expandability.
As an improvement, the distributed real-time/historical data cluster platform based on the big data cloud computing breaks through the constraint of a storage model facing to the traditional technology of a database from the actual requirement and the basic characteristics of industrial operation, and the distributed storage model of the industrial cloud data facing to the operating timestamp is definitely provided, namely the serialized and comprehensive description of the industrial data storage mode method. The storage model design of the invention surrounds two common basic characteristics of industrial operation in the aspect of a time scale, namely the time length of data storage and the time interval of data storage, namely the dynamic change sampling density meeting the service life. The invention constructs two types of industrial cloud Data distributed physical storage in a micro-service form of a subsystem node on a second layer of an enterprise information architecture, namely a Point record (Point) type and a Data record (Data) type, and Point information and Data information of a corresponding storage recording unit set, a subsystem and the like. The storage point information covers the subsystem node with the order of 1000-100000 bit numbers, and the storage data information covers the service life of the subsystem node to be permanent.
The invention creates a storage point information database table based on an open source HBase database, wherein the storage point table comprises attributes such as a serial number, a point identification number, a point type, a point name, an alias, description, an upper limit, a lower limit, an engineering unit, a display digit, an associated system, a compression function, a compression characteristic and the like. Considering that the storage point information needs to perform the conditions of adding data points, deleting data points, modifying the attribute information of the data points and the like, and the modified storage point information is different from the data point information before modification, the patent requires the design version control to synchronize the data information and directly associate the running time stamp with the version number, thereby ensuring that the data actually stored during data query corresponds to the data points one to one. It is generally recommended that a new storage point table be created in runtime stamp naming.
The invention creates a storage data information file based on a special micro service program and stores a real-time data set with an operation time stamp. The runtime stamp is embodied in the time length of the data storage and the time interval of the data storage, and the design deployment of the data cluster is spread around the core to form the runtime stamp lineage of the data storage model of the invention. For the time length, the cloud computing-based architecture can be expanded transversely and flexibly, the system life cycle or permanent storage needs are met, the storage directory is recommended to be created under the name of the year timestamp, and the storage file is created under the name of the day timestamp. In a compact storage model, if the service life of a subsystem node is 60 years, 60 sub-directories are created, and 365 storage files are created in each sub-directory. For the time interval, the sampling time stamp is related to the minimum running time stamp, and the time scale of the data is within the coverage range of professional perception capability of the running operator and the maintenance personnel and is recorded in the daily storage file. It is generally recommended that the sampling time stamp be from 100 milliseconds to 10 minutes. Of course, storage subdirectories and storage files can also be created with some convenient integer multiple of the runtime stamp.
the invention is based on self-designed and compiled special micro-service program software which is similar to the NoSQ L characteristic, stores a real-time data set with an operation timestamp, is in a key-value pair 'K-V' mode and has double key-value pairs, wherein the first key is to directly use the operation timestamp as a key of the key-value pair and use a polymorphic data set as a value of the key-value pair, the second key-value pair is a polymorphic data set, a storage point is mapped into the key of the key-value pair of the polymorphic data set, and measurement data is the value of the key-value pair of the polymorphic data set.
The invention provides a method for storing real-time data set key value pairs with running time stamps, which follows a simple and clear compression method and can realize the unification of industrial running data compression results on a big data cloud computing architecture platform. Although the cost of the storage space has been greatly reduced, various complex compression algorithms under the traditional real-time/historical database technology system based on the central database server mode seem to be unnecessary. But compression is necessary from the viewpoint of processing efficiency, and compression uniformity and format clearness are more important for the intelligent analysis of large data at the later stage. For the compression result, if data encryption is not carried out, the data in the file can be directly read out, the wide applicability and the format clearness storage mode are realized, meanwhile, the very efficient compression is realized, and the storage space and the time are saved.
The invention provides an internal compression method for running time stamp records, which comprises the following steps: each sampling time stamp of the data storage model is a compressed data storage time stamp, and the sampling time stamp can be set to 1 second. At this time, only data conforming to the compression storage condition is stored, including storage in which the analog quantity change exceeds the compression range, storage in which the switching value change is stored, and the like. In compact storage, the corresponding mapping key-value pairs must be maintained in the polymorphic dataset.
The invention provides a running timestamp abandonment compression method: at the time of storing the compressed data timestamp, if any data which meets the compressed storage condition is not stored at the time, including the range that all analog quantity changes and is not over-compressed, all switching values are not shifted, and the like, the sampling timestamp is abandoned.
The invention provides a running timestamp snapshot compression method: the integral multiple of the sampling time stamp of the data storage model is the snapshot time stamp of the full set of data, for example, the sampling time stamp is 1 second, and the snapshot time stamp can be set to be 60 seconds or 300 seconds. At this time, all data including an analog quantity, a switching quantity, and the like are stored in a snapshot manner. And when the snapshot is stored, default mapping key values are used for compression.
The invention provides a running timestamp event compression method: at the time of storing the compressed data, the event at that time is stored in each sampling time stamp, and character-type storage is adopted. The event store may include code for the occurrence of an event, event descriptions, code for triggering an alarm, alarm descriptions, hyperlinks, multimedia resource entries, and the like. The integrated storage is realized, the additional creation of event records, alarm records, multimedia records and the like is not needed, and the data compression of various event record files, file directories and file guidance is realized.
Compared with the prior art, the method has the advantages that 1) the scheme provides the storage of the real-time data set key value pairs with the running time stamps, a simple and clear compression method is followed, and the unification of industrial running data compression results on a big data cloud computing architecture platform can be realized; 2) the scheme provides an internal compression method of a running time stamp: each sampling time stamp of the data storage model is a compressed data storage time stamp, and the sampling time stamp can be set to 1 second. At this time, only data conforming to the compression storage condition is stored, including storage in which the analog quantity change exceeds the compression range, storage in which the switching value change is stored, and the like. During compression storage, corresponding mapping key value pairs must be maintained in the polymorphic data set; 3) the scheme provides a running timestamp abandonment compression method: at the moment of storing the compressed data timestamp, if any data meeting the compressed storage condition is not stored at the moment, including the range that all analog quantity changes and is not over-compressed, all switching values are not displaced, and the like, the sampling timestamp is abandoned; 4) the scheme provides a running timestamp snapshot compression method: the sampling time stamp of the data storage model is a whole set data snapshot time stamp which is an integral multiple of 1 second, and the snapshot time stamp can be set to be 60 seconds or 300 seconds and the like, so that snapshot storage of all data is performed at the moment, including analog quantity, switching value and the like. And when the snapshot is stored, default mapping key values are used for compression.
Drawings
FIG. 1 is a schematic diagram of a micro-service program software system structure on which the present invention is based;
Fig. 2 is a compression flow chart of implementation of the industrial cloud data collection storage model oriented to the runtime stamp according to the present invention.
The specific implementation mode is as follows:
For the purpose of enhancing an understanding of the present invention, the present embodiment will be described in detail below with reference to the accompanying drawings.
the system adopts a master-slave organization structure of the distributed type, a plurality of servers form a core of the cluster data set through high-speed network connection, the data cluster is responsible for managing, monitoring, storing point information and data information according to the cloud data storage model of the distributed type, the cluster data set is a high-reliability management system, and the system is convenient for expanding data storage nodes and expanding data storage nodes, and the like of a large cluster data set and a small cluster data set.
Before storing the data information of the data cluster and before accessing the data information of the data cluster, the version number of a subsystem point information table is required to be determined, and the point information table not only comprises various attribute information of the storage points, but also specifies the mapping relation between the points and the key value of the polymorphic data set. At each moment, the mapping mechanism is unique to the subsystem node and is also unique to the whole network, but allows for changes in the time development dimension. This patent requires that the design version control synchronize the data information, directly associates the running time stamp with the version number. The point table and version number need to be updated after performing the addition of data points, deletion of data points, modification of data point attribute information, and modification of the attribute information itself. It is generally recommended that a new storage point table be created in runtime stamp naming. Determining the mapping relation between the point and the key according to the updated version number when the data information is stored; before accessing the returned data set, the mapping relation between the point and the key is determined according to the point table version number corresponding to the running timestamp, and the data is accurately restored.
The invention creates a storage data information file based on a special micro service program and stores a real-time data set with an operation time stamp. The naming mode of the storage directory of the data information file takes a year timestamp as an example, and the naming mode of the storage file takes a day timestamp as an example, so that subdirectories of all years related to historical data are included in the unit set directory, and the subdirectories of the current year can be flexibly expanded according to real-time data, so that the system life cycle or permanent storage requirement is met; under each storage directory, if no history data is missing, 365 storage files from 1 month and 1 day to 12 months and 31 days should be included. The hierarchical storage mode has the characteristics of clear hierarchy and wide application, the number of the storage directories and the number of the storage files under each directory are relatively stable, the directory names and the file names have the functions of physical labels or time labels, the historical data are directly and accurately positioned according to the names when being inquired, and the data search range in the inquiry process is narrowed.
If the data amount of all the points in one day is required to be stored, data compression is not performed, and the file size is related to the number of points and the sampling time. The sampling time stamp is sufficient to sense the dynamic changes of the process data, which is about 100 milliseconds to 10 minutes, so that the amount of stored data is large. The data compression has different targets, algorithms and patents. The cost of the existing storage space is greatly reduced, but in consideration of the processing efficiency and the requirement of intelligent analysis of later-stage big data, the invention discloses a compression implementation technical route of an industrial cloud data set facing to an operating timestamp, integrates a plurality of simple and clear compression methods, and is easy to realize universality and cleartext. Fig. 2 is a flowchart of a cloud data set storage model compression process for a runtime stamp, where a sampling timestamp is set to 1 second, a snapshot timestamp is set to 60 seconds, and a data range covered by a storage file is assumed to be 1 day, and the implementation steps are as follows:
Step 1: recording the full set data of the first sampling time stamp, and synchronizing the full set data to the reference array;
Step 2: reading the corpus data of the next sampling timestamp;
And step 3: judging whether the timestamp is an integral multiple of the snapshot timestamp, in this example, the whole minute, if so, recording all data in a snapshot manner, recording the key value of the polymorphic data set of the running timestamp to a default mapping key value, namely, only recording the running timestamp and the measured data of the whole set, synchronously referring to the array, and performing step 5;
And 4, step 4: comparing the full set data with the reference array in sequence, recording mapping key value pairs of analog quantity and deflection switching value of a threshold range of a change over-compression condition, synchronizing data values of corresponding positions of the reference array, if the full set data do not meet the compression condition, equivalently recording in a snapshot mode, but requiring no default point mapping key value, and if the full set data meet the compression condition, defaulting current operation timestamp entries;
And 5: judging whether the time stamp is an integral multiple of the time stamp of the storage file, in this example, an integral day, if so, generating the storage file; if not, performing the step 2.
It should be noted that, in the above implementation steps, the two adjacent pieces of runtime stamp data are compared in sequence, provided that the corresponding version numbers of the storage point tables are the same. If the version numbers of the point tables are different, the same mapping key value may correspond to two different points of the subsystem, and the compression cannot be performed by adopting the implementation steps. For the case of point table update, storage in a manner of a full set data snapshot is considered. When some data are compressed when historical data in a certain period are inquired, the analog quantity is restored according to the previous stored data, and the switching value is the same as the previous stored data.
It is clear to a person skilled in the art that the invention can be implemented by means of software. Based on such understanding, the definition and contribution of the industrial cloud data distributed storage model facing the runtime stamp in the invention to the technical idea and the plain steps can be embodied in various software products. Therefore, the appended claims are not to be interpreted as limiting, and any modifications, equivalents, and improvements made to the spirit, principles, descriptions, models, and methods of the present invention are intended to be included within the scope of the present invention.
Claims (10)
1. An industrial cloud data storage model facing to an operation timestamp is characterized by comprising a client, an application server, a micro-service and a data cluster, wherein the application server responds to a real-time/historical data storage or query request from the client, receives field industrial control system data from a first layer of an enterprise information architecture, packages data of different operation timestamps for communication transmission, and calls a corresponding micro-service program to process the packaged data according to the request of the client; the micro service program is divided into two types according to the service, one is a storage micro service, the other is a query micro service, and each type is continuously subdivided according to different specific implementation functions; the data cluster is responsible for managing, monitoring, storing point information and data information following a cloud data storage model.
2. The industrial cloud Data storage system oriented to the runtime stamp as claimed in claim 1, wherein the storage model surrounds two common basic features of industrial operation in terms of a temporal scale, that is, a time length of Data storage and a time interval of Data storage, that is, a dynamically changing sampling density satisfying a lifetime length, and is located on a second layer of an enterprise information architecture, and two types of distributed physical storage of industrial cloud Data are constructed in a micro-service form of a subsystem node, that is, a Point record (Point) type, a Data record (Data) type, Point information and Data information corresponding to a storage record unit set, a subsystem, and the like, the storage Point information covers a subsystem node number of 1000 to 100000 bits, and the storage Data information covers a subsystem node lifetime to a permanent state. The invention breaks through the constraint of a storage model facing to the traditional technology of the database from the actual requirement and the basic characteristics of industrial operation, definitely provides an industrial cloud data distributed storage model facing to the operation timestamp, and requires to carry out design with wide applicability and clear format.
3. The runtime stamp oriented industrial cloud data storage model of claim 1, wherein the model creates a storage point information database table based on an open source HBase database, and the storage point table includes attributes such as sequence number, point identification number, point type, point name, alias, description, upper and lower limits, engineering units, display digits, association system, compression function, compression characteristics, and the like. Because the storage point information needs to perform the conditions of adding, deleting, modifying the data point attribute information, modifying the attribute and the like, the patent requires the design version control to synchronize the data information and directly associate the running time stamp with the version number, thereby ensuring that the actually stored data corresponds to the data point during data query. It is generally recommended that a new storage point table be created in runtime stamp naming.
4. The industrial cloud data storage model oriented to the runtime stamps as claimed in claim 2, wherein the dedicated micro service program creates a storage data information file, stores a real-time data set with the runtime stamps, the runtime stamps are embodied in the time length of data storage and the time interval of data storage, and the design deployment of the data set is expanded around the core to form a runtime stamp pedigree of the data storage model.
5. the industrial cloud data storage model oriented to the runtime stamp as claimed in claim 3, wherein the model is based on a self-designed and compiled special micro service program software similar to the NoSQ L characteristic, stores the real-time data set with the runtime stamp, is in a key-value pair "K-V" mode, and has double key-value pairs, the first is to directly use the runtime stamp as a key of the key-value pair, and use the polymorphic data set as a value of the key-value pair, the second is the polymorphic data set, the storage point is mapped to the key of the polymorphic data set key-value pair, and the measurement data is a value of the polymorphic data set key-value pair.
6. The industrial cloud data storage model facing the runtime stamp as claimed in claims 3, 4 and 5, wherein the real-time data set key value pair with the runtime stamp is stored, and the unification of the industrial operation data compression result on the big data cloud computing architecture platform can be realized by following a simple and clear compression method. For the compression result, if data encryption is not carried out, the data in the file can be directly read out, and the wide applicability and the format clearness storage mode are realized.
7. The runtime-stamp-oriented industrial cloud data storage model compression method of claim 6, wherein a runtime-stamp-record internal compression method is proposed: each sampling time stamp of the data storage model is a compressed data storage time stamp, and the sampling time stamp can be set to 1 second. At this time, only data conforming to the compression storage condition is stored, including storage in which the analog quantity change exceeds the compression range, storage in which the switching value change is stored, and the like. In compact storage, the corresponding mapping key-value pairs must be maintained in the polymorphic dataset.
8. The industrial cloud data storage model compression method oriented to the runtime stamp as claimed in claim 6, wherein a runtime stamp abandonment compression method is proposed: at the time of storing the compressed data timestamp, if any data which meets the compressed storage condition is not stored at the time, including the range that all analog quantity changes and is not over-compressed, all switching values are not shifted, and the like, the sampling timestamp is abandoned.
9. The running-timestamp-oriented industrial cloud data storage model compression method of claim 6, wherein a running-timestamp snapshot compression method is proposed: the integral multiple of the sampling time stamp of the data storage model is the snapshot time stamp of the full set of data, for example, the sampling time stamp is 1 second, and the snapshot time stamp can be set to be 60 seconds or 300 seconds. At this time, all data including an analog quantity, a switching quantity, and the like are stored in a snapshot manner. And when the snapshot is stored, default mapping key values are used for compression.
10. The runtime-stamp-oriented industrial cloud data storage model compression method of claim 6, wherein a runtime-stamp event compression method is proposed: at the time of storing the compressed data, the event at that time is stored in each sampling time stamp, and character-type storage is adopted. The event store may include code for the occurrence of an event, event descriptions, code for triggering an alarm, alarm descriptions, hyperlinks, multimedia resource entries, and the like. The integrated storage is realized, the additional creation of event records, alarm records, multimedia records and the like is not needed, and the data compression of various event record files, file directories and file guidance is realized.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010296258.6A CN111427964A (en) | 2020-04-15 | 2020-04-15 | Industrial cloud data storage model for running timestamp |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010296258.6A CN111427964A (en) | 2020-04-15 | 2020-04-15 | Industrial cloud data storage model for running timestamp |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111427964A true CN111427964A (en) | 2020-07-17 |
Family
ID=71556873
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010296258.6A Pending CN111427964A (en) | 2020-04-15 | 2020-04-15 | Industrial cloud data storage model for running timestamp |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111427964A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112632127A (en) * | 2020-12-29 | 2021-04-09 | 国华卫星数据科技有限公司 | Data processing method for real-time data acquisition and time sequence of equipment operation |
| CN112881052A (en) * | 2021-01-14 | 2021-06-01 | 深圳市杉川机器人有限公司 | Method and device for constructing working scene of mobile robot |
| CN117896035A (en) * | 2024-03-14 | 2024-04-16 | 杭州义益钛迪信息技术有限公司 | Data acquisition method and equipment of edge controller |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107229639A (en) * | 2016-03-24 | 2017-10-03 | 上海宝信软件股份有限公司 | The storage system of distributing real-time data bank |
| CN108133043A (en) * | 2018-01-12 | 2018-06-08 | 福建星瑞格软件有限公司 | A kind of server running log structured storage method based on big data |
| CN109815026A (en) * | 2018-12-18 | 2019-05-28 | 国电南京自动化股份有限公司 | Electric power time series database based on distributed component |
| CN110196885A (en) * | 2019-06-13 | 2019-09-03 | 东方电子股份有限公司 | A kind of cloud distributed real-time database system |
| CN110809017A (en) * | 2019-08-16 | 2020-02-18 | 云南电网有限责任公司玉溪供电局 | Data analysis application platform system based on cloud platform and micro-service framework |
-
2020
- 2020-04-15 CN CN202010296258.6A patent/CN111427964A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107229639A (en) * | 2016-03-24 | 2017-10-03 | 上海宝信软件股份有限公司 | The storage system of distributing real-time data bank |
| CN108133043A (en) * | 2018-01-12 | 2018-06-08 | 福建星瑞格软件有限公司 | A kind of server running log structured storage method based on big data |
| CN109815026A (en) * | 2018-12-18 | 2019-05-28 | 国电南京自动化股份有限公司 | Electric power time series database based on distributed component |
| CN110196885A (en) * | 2019-06-13 | 2019-09-03 | 东方电子股份有限公司 | A kind of cloud distributed real-time database system |
| CN110809017A (en) * | 2019-08-16 | 2020-02-18 | 云南电网有限责任公司玉溪供电局 | Data analysis application platform system based on cloud platform and micro-service framework |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112632127A (en) * | 2020-12-29 | 2021-04-09 | 国华卫星数据科技有限公司 | Data processing method for real-time data acquisition and time sequence of equipment operation |
| CN112881052A (en) * | 2021-01-14 | 2021-06-01 | 深圳市杉川机器人有限公司 | Method and device for constructing working scene of mobile robot |
| CN112881052B (en) * | 2021-01-14 | 2024-02-20 | 深圳市杉川机器人有限公司 | Method and device for constructing working scene of mobile robot |
| CN117896035A (en) * | 2024-03-14 | 2024-04-16 | 杭州义益钛迪信息技术有限公司 | Data acquisition method and equipment of edge controller |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111723160B (en) | Multi-source heterogeneous incremental data synchronization method and system | |
| CN111427964A (en) | Industrial cloud data storage model for running timestamp | |
| CN102737127B (en) | Massive data storage method | |
| CN107423422B (en) | Spatial data distributed storage and search method and system based on grid | |
| CN110196885B (en) | Cloud distributed real-time database system | |
| CN114116716A (en) | Hierarchical data retrieval method, device and equipment | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN102779138B (en) | The hard disk access method of real time data | |
| CN106682077A (en) | Method for storing massive time series data on basis of Hadoop technologies | |
| CN108763234A (en) | A kind of real time data synchronization method and system | |
| CN112328702B (en) | Data synchronization method and system | |
| CN110659283A (en) | Data label processing method and device, computer equipment and storage medium | |
| CN114647716A (en) | Generalization data warehouse | |
| CN109828966A (en) | Gradual heavy hash method, device, computer equipment and storage medium | |
| CN108228725A (en) | GIS application systems based on distributed data base | |
| US11210212B2 (en) | Conflict resolution and garbage collection in distributed databases | |
| Wang et al. | Apache IoTDB: A time series database for IoT applications | |
| WO2023142610A1 (en) | Data processing method and apparatus | |
| CN115905313A (en) | MySQL big table association query system and method | |
| CN105761164A (en) | Hadoop-based electric power system accident recall method | |
| CN113515518A (en) | Data storage method and device, computer equipment and storage medium | |
| CN116644039B (en) | Automatic acquisition and analysis method for online capacity operation log based on big data | |
| CN116010677B (en) | Spatial index method and device and electronic equipment thereof | |
| CN116860732A (en) | Control method and device for data life cycle, electronic equipment and medium | |
| McGlothlin et al. | Scalable queries for large datasets using cloud computing: A case study |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 211111 Jiangsu city of Nanjing province Jiangning economic and Technological Development Zone, Mo Zhou Road No. 9 Applicant after: Nuclear new cloud technology (Nanjing) Co.,Ltd. Address before: 211111 Jiangsu city of Nanjing province Jiangning economic and Technological Development Zone, Mo Zhou Road No. 9 Applicant before: NANJING HEXIN DIGITAL TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information |