DE102007040675A1 - System and method for the lossless processing of process values of a technical plant or a technical process - Google Patents

Abstract

The invention relates to a system and method for the lossless processing of process values, which are integrated in a process control system of a technical plant or a technical process, characterized in that the process values are detected as raw data by means of a detection unit. Archived data are generated from the acquired raw data after assigning a physical unit and a measuring range to the respective process value. The archivable data located in a predetermined tolerance band are selected by means of an archiving function and stored in a raw data format in a history server. The selected data present in the raw data format is continuously compressed by means of a coding module provided by the history server for its storage in the history server. Following a request from the process control system, the compressed data is decompressed by means of a decoding module provided by the history server, and the decompressed data provided by the decoding module as well as other available raw data format data are transmitted to a presentation module for further processing.

Description

The The invention relates to a method for lossless processing of process values, also referred to as process measured values or process data, a technical process or a technical installation which is used especially in process automation. Farther The invention relates to a system for carrying out the method.

to Long-term archiving of process values, also as process measured values or process data, the data is called a series of measurements, too referred to as measured value histories or process value histories, in one stored or stored so-called history server. The archiving The data is very memory intensive. For reading the stored Data is therefore a mass storage with a very large storage area required and during further processing of the data read out must be a large number to be moved by data. The database to be saved is set this is the product of the number of signals whose expected rate of change and their recording time together.

Of the required by the operator of the technical system data from the Process to be stored for archiving is in The past few years have risen sharply and there is a desire for the data on the one hand to save losses and on the other hand with a preferably read high speed again. These wishes can be combined with a easy storage in uncompressed raw format. the But now there is the disadvantage that with this principle the raw value storage of the available mass storage not optimally used. The raw data stock to be stored is currently in the Range from 0.5 to about 5 terabytes, with the trend being based on the data to be stored is still rising.

Even though relatively cheap Mass storage for storing the costs arising from the process or the system Amounts of data available are the use of compression methods for storing large amounts of data sought in process automation, using a compression method desirable that would be the goals for the lossless storage of the data and especially this data with the highest possible Speed read again, optimally united. There is one less optimal compression factor acceptable.

to Data compression of the process or plant Datasets are becoming current lossless or lossy methods used.

at the known lossy compression method, which BoxCar / Backslope, transformation methods count, stand after decompression the pre-compressed data with more or less pronounced deviations in terms of the measured value and the time stamp available again.

The BoxCar / Backslope, Swinging Door or wavelet transformation methods are for example in " http://med.ee.nd.edu/MED7/med99/papers/MED101.pdf "(Automatic Tuning of Window Size in the Box Car Backslope Data Compression Algorithm)," http://training.osisoft.com/NR/rdonlyres/5547CC68-65AD-4E55-A365-B30C1FCF74F9/0/SwingingDoorCompression.doc "(Swinging Door Compression), and" http://de.wikipedia.org/wiki/Wavelet "(Wavelets and transformations) described.

These Procedures provide a good compression rate but are not lossless. Furthermore slows down due to the amount of data to be processed the reading process considerably opposite reading the data stored in non-compressed form.

Also known from the lossless compression methods, such as the LZW, LZ77 or LZ78 method (here a sequence of characters of the original text is replaced by a sequence of characters of another alphabet), for example, in http://de.wikipedia.org/wiki/LZ77-Datenkompression and http://de.wikipedia.org/wiki/LZ78 In the case of the raw data provided and to be stored and to be stored from the process or the installation, only unsatisfactory compression rates are provided and, in addition, they are too slow during unpacking for many applications in process automation.

at the so-called lossy compression method is against it the compression of the accumulated data with a loss of information connected, because the read or decompressed data from the original ones Differentiate data.

Just in the environment of process or plant information systems, which the archived data often as a billing-relevant basis, for example So if in the office environment preferably accurate measurements from the process environment as a basis for balancing serve, no lossy value may be used here, use or related to this data in retrospect analyzes exactly To drive to the sampling times of the process data, the advance lossy methods described are not suitable.

One Another disadvantage of the lossy method is based on that the decompression of the data, also known as unpacking, is very computationally intensive. For the large amounts of data to be processed acts Therefore, a compression is more of a hindrance, because the decompression the data just takes a long time.

Of the Invention is therefore the task underlying, a method for the lossless processing of process values, also referred to as process measurements or process data, a technical Process or a technical installation and a corresponding System for execution specify the method, whereby the aforementioned disadvantages of the prior overcome the technique become. With the method according to the invention for data compression in particular the unpacking of the compressed Data speeds up. Furthermore it is lossless unlike most of the methods used.

These Task is achieved by a method having the features specified in claim 1. advantageous Embodiments and improvements of the method according to the invention as well as a system for execution of the method are in further claims and in the description specified.

The The invention relates to a method which always has the fact of being lossy acquisition of digitized process values of a technical facility exploits to in a downstream special lossless Compression method to an optimal mix in terms of compression rate and unpacking speed of process data in a process control system a technical installation or a technical process.

According to the invention those provided by the technical facility or the technical process Process values recorded as raw data by means of a detection unit, and the acquired raw data becomes a physical unit, respectively and a measuring range assigned. The archivable generated in this way Raw data is checked whether they are within a specified tolerance band. through An archiving function will be within the specified Tolerance band located archivable data selected and in their original, ie not yet compressed data format, also called raw data format, as selected Data stored in a history server.

Of the History Server is a special database with real-time functions for the collection of process or measurement data for long-term storage and to archive the collected data, with the collected Data at fixed intervals stored on a hard disk of a log memory.

The in the raw data format selected data are using a time-delayed provided by the history server Codiermoduls their storage in the history server compressed, taking from the compressed data is generated a compressed file.

Around the data of the compressed file to the process control system for further Processing, for example, to display the course of a process value via a given time range, are the Data of the compressed file after a request by means of a decoding module provided by the history server. The request to decompress the compressed file and to Provide more available Data in raw data format is automatically or by the process control system event driven provided.

The decompressed data provided by the decoder module and others while the creation of the compressed file from the process or from the Plant read data in the raw data format, preferably a presentation module, transmitted for further processing.

With the method according to the invention In particular, the long-term archiving of process value histories, which are formed from the continuously recorded process values, improved so that the process values are not only compressed lossless but also one as possible good packing density of the compressed data is achieved. In contrast All loss-related procedures are all billing-relevant Data recoverable without loss of original information.

The Storage of raw data in its original uncompressed Data format in the history server is temporally split in each case independent Files per metric, typically in so-called two-day files one without compression of the raw data.

in the The following is the storage of raw data in the history server means described special techniques.

In a history server are for thousands of different measuring points, also called signals, each store the historical measured value series. By doing this historical measured value series of a measuring point like individual partial databases will be treated effectively storage of the historical Measured value series in a standard file system are enabled as follows: Per signal part database or measured value series of a signal) a separate folder used in the file system. The signal sub-databases are so over Any number of disks can be redistributed, resulting in one more higher Performance and additional Redundancy aspects be achieved.

Farther is the introduction of a short-term write / read cache in RAM for each signal. Of the Short-term write / read cache is already "visible" to the internal SQL machine, which causes the values for client requests delay available are.

By the introduction A cache flush process becomes a so-called "crisp-friendly" dump of spa time Controlled write / read caches. This is done by the cache flush process the complete writing back of the cache contents into a main memory. After that is the now empty signal cache again to capture the next Values available.

Of the Cache flush process continues to control an individual dynamic Cache size depending from the signal speed. For example, the so-called fast signals are assigned many values per unit of time. this Signals become larger short-term caches allocated. Slow signals, on the other hand, gradually give their share Cache memory to fast signals.

Of the thus time- and degree-controlled Cache flushing works so that as few data accesses are necessary, or the data access distributed as evenly as possible he follows. This is achieved in an advantageous manner, that too in standard file systems, such as the New Technology File System (NTFS), a optimal performance when reading and writing the data given is.

The executed by means of the coding module Compression method works independently in a concurrent process and coded delayed for the actual archiving the raw data, that is the process values are first stored in raw data format and gradually with a configurable delay in this concurrent Process is converted into a compressed format, taking from the compressed Data is generated a compressed file.

Of the The resulting benefit is due to the fact that the temporal delay the compression of the raw data is controlled so that, for example, via a longer Period worked with raw values, because typically in this Period increased requests for access, for example at the creation of daily reports and trends, and here without decoding and so that the storage medium can be accessed quickly.

In Advantageously, the time delay of the compression of the Raw data individually for individual process values or groups or classes of process values adjustable. For example, become a class of billing-relevant Data about held in raw data format for a few years before compression this data is started.

From the above knowledge, another concurrent process (Modul KompressDelayKonfigurator) is now assigned with the automatic determination of the compression delays and is based on the following principle:
The history server internally provides access statistics for each signal. These access statistics include, for example, who accessed the individual time series with which history periods in relation to the current time. On the basis of this data, the module KompressDelayKonfigurator determines an optimal delay time.

Another advantage of the invention is based on a fast-running unpacking algorithm of compressed data showing the inverse of the in 2 described packing algorithm and the in 3 described unpacking algorithm, and whereby the speed when unpacking the compressed data is significantly reduced without restricting the function of the history server.

The System for execution the method according to the invention is to be taken from claim 7.

The inventive system for the lossless processing of process values is in a process control system integrated into a technical system or a technical process and comprises a detection device, which is the one from the process or the process values provided as raw data. The detection unit interacts with a processing unit, from the acquired raw data after assignment of a physical Unit and a measuring range for the respective process value can be archived Data generated.

The Processing unit interacts with an archiving function, which checks, whether the archivable data exceeds a specified tolerance band. The archiving function checks Thus, every incoming process value, whether this exceeds the specified tolerance band. Continue to dial the archiving function is within the tolerance band archivable data and stores the selected data in a raw data format without compression, that is, everyone incoming process value within the specified tolerance band is included as raw value, for example in the 2-day files in raw data format a timestamp, an IEEE double-format value, and a status in 16-bit integer format (file type that stores integer values) stored in a history server.

Of the History server has a coding module for compressing the selected data on. The coding module continuously compresses the data in raw data format present selected Data delayed for their storage in the history server.

Of the decoupled and individually delayed expiring coding process in the history server now compresses gradually the data stored in the raw data format.

To an automatic or event - dependent request, the compressed data directly to a decoding module for decompression supplied and the decompressed data as well as other available data in raw data format become a presentation module, such as a module for trend display for further processing and / or display.

The inventive system and methods are preferably for the long-term archiving of measurement series used in automation technology.

On the basis of in the 1 and 2 illustrated embodiment, the invention and advantageous refinements and improvements of the invention will be explained and described in detail.

It demonstrate:

1 an embodiment of the system according to the invention for the lossless processing of process values of a technical installation or a technical process,

2 an exemplary procedure for the lossless compression of process values of a technical installation or a technical process,

3 an exemplary procedure for decompression of the compressed file, and

4 an exemplary history of selected process or signal values and corresponding compression results for different time ranges.

In 1 an embodiment of the system according to the invention for the lossless processing of process values is shown, which is integrated in a process control system of a technical plant or a technical process

The system according to the invention comprises a detection device 10 , which is preferably designed as input / output unit with a resolution of, for example, 12 bits, that is, from a gemes voltage is generated a bit pattern corresponding to a number range from 0 to 4096. The detection device 10 thus records the process values provided by the process or plant as digital data. By means of a processing unit 20 that with the capture unit 10 archived data are generated by converting the acquired process values as raw data into a so-called engineering value. For this purpose, one physical unit (eg ° C) and one measuring range (eg 0 to 600 ° C) are assigned to the process values and the one from the input / output unit 10 The detected numerical value is correspondingly transformed, for example, from the detected value 2048 in the present example to 300 ° C., where the number 300 as an IEEE double value is available to the system.

An integrated archiving function in the system according to the invention 30 checks each one from the capture unit 10 raw value provided and transformed into an engineering value, if it exceeds a specified tolerance band, and then stores this raw value in 2-day raw data format files, the raw value having a timestamp, an IEEE double format value, and 16-bit integer status.

Table 1 below shows the structure of the raw data file, where the number of bytes corresponds to the product of the number of process values with 18 bytes per measured value. Each file contains the values of a maximum of two days in the example below. Table 1 time stamp data data type Timestamp (IEEE Double, 64 Bit) Value (IEEE Double, 64-bit) Status (16 bits) Timestamp (IEEE Double, 64 Bit) Value (IEEE Double, 64-bit) Status (16 bits) Timestamp (IEEE Double, 64 Bit) Value (IEEE Double, 64-bit) Status (16 bits) Timestamp (IEEE Double, 64 Bit) Value (IEEE Double, 64-bit) Status (16 bits) Timestamp (IEEE Double, 64 Bit) Value (IEEE Double, 64-bit) Status (16 bits)

The storage of the data provided by the technical process or the technical plant as raw data is split in separate files per measuring point in a history server 10 without compression of the data.

In the history server 100 is a coding module 40 which decouples and individually compresses the raw data stored in the raw data format bit by bit, whereby the raw data are converted into a compressed format in parallel and independently of their actual archiving.

The coding module 40 acts with a decoding module 50 together, which also in the history server 100 is integrated. The decoding module 50 is intended, by the coding module 40 Unpack compressed data after a request from the process control system and the unpacked data and other available data in raw data format a presentation module 60 which, for example, depicts a trend or a report for further processing.

Thus, when reading the data from the coding module 40 For example, for the presentation of a long-term trend or a report, the compressed or packed data via the decoding module 50 transformed and like the raw data still available and together to the trend or the report 60 cleverly.

2 shows an exemplary process flow for the lossless compression of process values of a technical plant or a technical process.

outgoing from the structure described above or the content of the raw files and the source chain of the data will be described below Coding methods for compressing the raw data.

The first step 1 from the detection unit 10 Raw data read in which are saved as process values in IEEE double format are based on the data from the acquisition unit 10 and the processing unit 20 executed detection chain still on digital values with a resolution of, for example, 12 bits. Consequently, it is to be expected that at 12 bits a maximum of 4096 ver different values occur in a file. The only exceptions are the calculated values formed in process processing by arithmetic operations. These show an increased range of values, but only by a factor of 3. A tolerance band during the detection is usually switched so that one or two further bits are filtered, whereby the expected amount of data or value decreases accordingly.

Thus, according to the invention in a second step 2 In a CreateValueDictionary, the raw values from the file are prepared in a first directory so that only the directory indexes are saved.

In a third step 3 a second directory of all occurring value differences is created from two raw values each. Since the values between two raw values typically do not change abruptly, there is a high probability that the second directory contains even fewer values than the first directory. Both directories are now compared with each other and the directory that has the fewest values is selected and provided as a measurement dictionary (ValueDictionary) for further processing.

In a fourth step 4 A similar procedure is followed in a CreateTimestampDictionary for the time resolution. When storing the raw values from the process, the time resolution is typically exactly stamped to milliseconds. Furthermore occur in a file approximately the same time differences. Thus, in the CreateTimestampDictionary, all timestamps are first converted to milliseconds and then the corresponding differences, hereinafter referred to as time differences, are formed. From this, a directory of the time differences, the timestamp dictionary (TimestampDictionary), is created.

When packing the timestamp directory in a fifth step 5 the effect described below comes into play. When reading in fast signals, the corresponding values are typically provided relatively continuously, for example, a new value is read in within a time range of 0.5 to 0.8 seconds. Consequently, it can be assumed that most of the time differences lie within a close difference window of time, and thus almost all millisecond figures are available in the timestamp directory in this area. Therefore, before the bit-coding of the value indices starts, the timestamp directory is sorted in ascending order and the corresponding time differences are formed. This creates so-called long one-chains of ascending values, which in turn are compressed quickly with a run-length coding.

The Run Length Encoding is a very simple lossless compression algorithm for the in digital form and is particularly well suited Repeats or sequences of equal values shortened. If there is a repetition, the number of repetitions will be and the repeated value saved.

In An embodiment of the invention beforehand the time units coded as differences. The slower a signal is, so the fewer changes has the signal, the larger and different are the time differences between the values.

As a result, the timestamp directory grows though there are fewer numbers or data for a corresponding signal. Due to the increased need of the time stamp directory together with the indices to be coded on data for the time units, the time units are immediately coded as time differences. Thus, the processing of a directory is advantageously saved by determining the required bits for the coding from the maximum time difference. The coding module 40 In this case, select the directory with the more compact format.

In a sixth step 6 During the time coding it is checked whether the longitudinal transformation of the bytes may be more effective: Since all times are ascending times, long strings of similar bytes also result here, which are very well compressible by means of the run length coding described above. This procedure is particularly advantageous when there are very irregular and few data per file.

at The status information (16-bit integer) is also usually the run-length coding described above carried out.

In a further step 7 are next to the already created measurement values directory and he set timestamp directory in the compressed file which encodes indexes for the values. Here, it has been shown that encoding according to the commonly used methods according to Shannon-Fano or Huffman in comparison to a rigidly determined bit field size provides no better and sometimes even slightly worse results.

The Shannon-Fano codification and Huffman coding are based on one Type of entropy coding is the one each character of a Text assigns a different length sequence of bits. Because a certain Minimum number of bits is necessary to keep all characters from each other To differentiate, the number of bits allocated to the characters can be will not be infinitely small.

The now used constant coding scheme in which always the same many bits for An index also has the advantage that the coding scheme is essential is unpacked faster because no binary tree has to be traversed, to form the corresponding index.

The so in a final step 8th The compressed file generated has the structure shown in Table 2: Table 2 Difference start values and pointers to all structure blocks List of time differences List of value differences Bit-coded time indices Bit-coded time indices run-length-coded status information

The so achieved compression reduces the raw data of about 100% about 18 ... 32%, which corresponds to a factor of 3 to 5, with the the amount of raw data is reduced. Considering the by the Tolerance band already achieved compression, which approx Factor of 2 to 6 when reducing the amount of raw data, gives a maximum total compression of about thirty times (about 5 · 6 = 30) of the original existing raw data amount, with exemplary average achievable Values are at a factor of 8.

A significant advantage of the invention is based on the fact that although the upstream tolerance band represents a lossy compression, the billing-relevant data are clearly recoverable, since the above-mentioned method steps 2 to 7 only after the selection of the archivable data located within the specified tolerance band. The determination of the billing-relevant data at runtime, which is typically executed in a system information system, thus has exactly the same result as the values determined in the compressed history.

In order to make the data of the compressed file available to the asset information system for further processing, for example for displaying the course of a process value over a predetermined time range, the data of the compressed file is retrieved from the asset information system by means of the request from the history server 100 provided decoding module 50 after the in 3 Decompressed procedure shown.

3 shows an exemplary procedure for decompressing the compressed file, wherein in a first step 11 the generated compressed file from the decoding module 50 is read.

In a subsequent step 12 The timestamp directory (TimestampDictionary) and the measurement value dictionary (ValueDictionary) are generated, as are the value indices and timestamp indices from the history server 100 read and generate values and timestamps. After decoding the run-length coded status information, the data is restored in its raw data format.

In a last step 13 become the so from the decoder module 50 provided decompressed data in its raw data format and other read during the generation of the compressed file data in raw data format a presentation module 60 transmitted for further processing.

When reading from the decoder module 50 provided decompressed data from the compressed area, for example, for the representation of a long-term trend, thus become the packed data via the decoding module 50 converted and with the still available raw data together to the trend 60 cleverly.

4 shows the course of selected process or signal values and corresponding compression results for one day, for a time range of 30 minutes and for a time range of one minute using the example of a typical fast pressure signal DS (Ta = 250 ms) and a conductivity measurement of a salinity LF of a desalination plant, which represents an exemplary medium-fast signal (Ta = 1 sec).

The results of the compression of in 4 The signal curves shown in Table 3 below: Table 3 Curve RawTripel dt.Dict v.Dict dv.Dict dt.bits v.bits dv.bits % dt-v % dt-dv % S & F dt-dv % Win Zip DS 178304 3539 2408 515 13 12 10 22.4 19.7 49.7 20.6 LF 16704 7195 2081 76 13 12 7 22.3 27.8 27.3 52.8

Raw-Tripel:

Anzahl der Zeit/Wert/Status-Tripel, die in 2-Tagesdateien enthalten sind und die Datenbasis der Signalverläufe DS, LF darstellen

dt.Dict:

alle enthaltenen Zeitdifferenzen

v.Dict:

alle enthaltenen Werte

dv.Dict:

alle enthaltenen Wertdifferenzen

dt.bits:

zur Zeit-Kodierung benötigte Bits für einen Index

v.bits:

zur Wert-Kodierung benötigte Bits für einen Index

dv.bits:

zur Differenzwert-Kodierung benötigte Bits für einen Index

%dt-v:

erreichte Kompressionsrate in %, incl. Time/Value-Dictionary

%dt-dv:

erreichte Kompressionsrate in %, wenn Differenzwerte kodiert werden

%WinZip:

als Vergleich eine mit WinZip erzielte Kompressionsrate in %

The abbreviations used in Table 3 are shown below:

Raw triples:

Number of time / value / status triples, which are contained in 2-day files and represent the database of the DS, LF waveforms

dt.Dict:

all included time differences

v.Dict:

all contained values

dv.Dict:

all contained value differences

dt.bits:

bits required for time coding for an index

v.bits:

for value encoding required bits for an index

dv.bits:

bits needed for differential value coding for an index

% Dt v:

achieved compression rate in%, incl. Time / Value-Dictionary

% Dt-dv:

achieved compression ratio in% when encoding difference values

% WinZip:

as comparison a compression rate achieved with WinZip in%

Claims (13)

Method for the lossless processing of process values, which is integrated in a process control system of a technical plant or a technical process, characterized in that - the process values are recorded as raw data by means of a detection unit, - from the acquired raw data after assignment of a physical unit and a measuring range to archivable data are generated, the archivable data located in a predetermined tolerance band are selected by means of an archiving function, the selected data are stored in a raw data format in a history server, the time-delayed data selected in the raw data format are provided by means of a coding module provided by the history server are continuously compressed in their storage in the history server, - after a request from the process control system, the compressed data is made available by means of a history server decompression module decompressed, and - the decompressed data provided by the decoding module as well as other available data are transmitted in raw data format to a presentation module for further processing. Method according to claim 1, characterized in that that the request to decompress the compressed data and to provide further available data in raw data format automatic or event-dependent provided by the process control system. A method according to claim 1 or 2, characterized in that the data in the raw data format split in time in separate files per measurement point in the history server 10 be saved without compression. Method according to one of the preceding claims, characterized characterized in that the compression of the present in the raw data format Data with a time delay individually for individual process values or groups or classes of process values is set. Method according to one of the preceding claims, characterized characterized in that the process values are first stored in the raw data format be and time-shifted gradually with a configurable delay be converted into a compressed format. Method according to one of the preceding claims, characterized characterized in that the decompressed data as well as the others available Data in raw data format a module for trend representation or a Module for mapping a report. System for lossless processing of process values, which in a process control system of a technical plant or integrated into a technical process, characterized in that that - one Detection device detects the process values as raw data, - the registration unit cooperates with a processing unit, which from the detected Raw data after assignment of a physical unit and a measuring range generates archivable data for the respective process value, - the processing unit interacts with an archiving function, which checks whether the archivable data exceed a specified tolerance band, the archivable data within the tolerance band selects and the archivable in a raw data format in a history server stores, - of the History server has a coding module, which in the raw data format present selected data Time Lag continuously compressed for their storage in the history server, - the coding module interacts with a decoding module, which from the coding module compressed data after a request from the process control system decompressed, and the decompressed data and other available data in raw data format a presentation module for further processing provides. System according to claim 7, characterized in that the process control system the request for decompressing the compressed Data and providing other available data in raw data format automatic or event-dependent provides. System according to claim 7 or 8, characterized in that the data in the raw data format temporally split in separate files per measuring point in a history server 10 can be stored without compression. System according to one of claims 7 to 9, characterized that the compression of the raw data format data with a time delay individually for individual process values or groups or classes of process values is adjustable. System according to one of claims 7 to 10, characterized that the history server has a database for collecting process data or measurement data, messages and reports, for long-term storage and for archiving the data collected, the collected Data and completed reports can be stored at fixed intervals are. System according to one of claims 7 to 11, characterized that the history server intended for that is to save the process values first in raw data format and with a configurable delay convert the stored data into a compressed format. System according to one of claims 7 to 12, characterized that the presentation module the decompressed data and the other available Data in raw data format as a trend or report.