CN116700630A - Organic-inorganic compound fertilizer production data optimized storage method based on Internet of things - Google Patents
Organic-inorganic compound fertilizer production data optimized storage method based on Internet of things Download PDFInfo
- Publication number
- CN116700630A CN116700630A CN202310969163.XA CN202310969163A CN116700630A CN 116700630 A CN116700630 A CN 116700630A CN 202310969163 A CN202310969163 A CN 202310969163A CN 116700630 A CN116700630 A CN 116700630A
- Authority
- CN
- China
- Prior art keywords
- data
- segment
- segments
- index
- data segment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003860 storage Methods 0.000 title claims abstract description 45
- 238000004519 manufacturing process Methods 0.000 title claims abstract description 42
- 238000000034 method Methods 0.000 title claims abstract description 41
- 239000003337 fertilizer Substances 0.000 title claims abstract description 40
- 229910010272 inorganic material Inorganic materials 0.000 title claims abstract description 33
- 238000007906 compression Methods 0.000 claims abstract description 19
- 230000006835 compression Effects 0.000 claims abstract description 19
- 230000002159 abnormal effect Effects 0.000 claims description 28
- 238000010606 normalization Methods 0.000 claims description 26
- 150000002484 inorganic compounds Chemical class 0.000 claims description 9
- 238000005259 measurement Methods 0.000 claims description 9
- 150000002894 organic compounds Chemical class 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000012216 screening Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 3
- 238000012935 Averaging Methods 0.000 claims description 2
- 238000013500 data storage Methods 0.000 abstract description 3
- 238000013144 data compression Methods 0.000 abstract 1
- 238000007781 pre-processing Methods 0.000 abstract 1
- 238000012544 monitoring process Methods 0.000 description 10
- 150000001875 compounds Chemical class 0.000 description 7
- 230000006837 decompression Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 2
- 238000003908 quality control method Methods 0.000 description 2
- 239000002994 raw material Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
- G16Y10/00—Economic sectors
- G16Y10/25—Manufacturing
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
- G16Y20/00—Information sensed or collected by the things
- G16Y20/20—Information sensed or collected by the things relating to the thing itself
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention relates to the technical field of data storage, in particular to an organic-inorganic compound fertilizer production data optimized storage method based on the Internet of things, which comprises the following steps: acquiring acquisition data, and preprocessing to obtain a data source to be processed; calculating the fluctuation degree of a data source to be processed; dividing the data segments according to the fluctuation degree; obtaining a storage index for a data segment with large fluctuation degree by utilizing a fluctuation segment similarity index and the structural characteristics of the data per se; compressing by using the storage index; the compressed data is stored. According to the invention, targeted compression is carried out according to indexes and characteristics of different data segments, data is decomposed into similar data segments for compression storage, the data compression rate is improved while the data characteristics are maintained, and the data redundancy is reduced.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to an organic-inorganic compound fertilizer production data optimized storage method based on the Internet of things.
Background
In the production process of the compound fertilizer, the internet of things technology can monitor parameters such as raw material input flow, temperature, humidity, pressure, flow and the like of a production site in real time, realize accurate control, thereby improving quality and quality of the compound fertilizer, has large data redundancy when monitoring data are complicated and stored, increases pressure of a storage system, and can influence data processing efficiency due to excessive data quantity when monitoring data such as temperature, humidity and pressure are analyzed. In the process of storing the monitoring data of the compound fertilizer production site of the Internet of things, the discrete data of different data of the monitoring results such as the recorded temperature, pressure, humidity, flow and the like are complicated, and when the run-length code compression is directly used, the redundancy of the data of the storage bits is large.
Disclosure of Invention
The invention provides an organic-inorganic compound fertilizer production data optimized storage method based on the Internet of things, which aims to solve the existing problems.
The method for optimally storing the production data of the organic and inorganic compound fertilizer based on the Internet of things adopts the following technical scheme:
the embodiment of the invention provides an organic-inorganic compound fertilizer production data optimized storage method based on the Internet of things, which comprises the following steps:
acquiring a plurality of indexes acquired on an organic-inorganic compound fertilizer production site, and screening and replacing the data of the plurality of indexes according to the data values of the indexes to obtain a data source to be processed of each index;
in the data source to be processed of each index, obtaining a fluctuation factor of each data point according to the adjacent difference of the data points and dividing the data points into a plurality of data segments according to the fluctuation factors; calculating the fluctuation degree of each data segment according to the fluctuation factor of each data point, and obtaining the storage degree between any two data segments according to the fluctuation degree of any two data segments; obtaining a similar segment group of each data segment according to the storage degree between any two data segments, classifying each data segment and all the similar segment groups into a set, and marking the set as a homomorphic segment group;
and in all the data segments of each homomorphic segment group, dividing each data segment into a intercepting sub-sequence and a sub-independent sequence, compressing the data according to the intercepting sub-sequence and the sub-independent sequence to obtain compressed data, and storing the compressed data in an Internet of things database to finish the optimized storage of the organic and inorganic compound fertilizer production data.
Preferably, the step of filtering and replacing the data of the multiple indexes according to the size of the data value of the index to obtain the data source to be processed of each index comprises the following specific steps:
acquiring time sequence data of a plurality of indexes acquired by an organic-inorganic compound fertilizer production site, calculating absolute values of differences of numerical values of each data point and the next data point in time sequence for all data points of each index, and recording the absolute values as difference values of each data point; calculating the absolute value of the difference value between each data point and the difference value of the next data point in time sequence, and recording the absolute value as the adjacent abnormal value of each data point; recording the ratio of the difference value of each data point to the adjacent abnormal value as the abnormal difference value of each data point; comparing the abnormal difference value of each data point with a preset abnormal threshold value, recording the data points with abnormal difference values larger than the preset abnormal threshold value as abnormal data points, and replacing all abnormal data points on each index with the average value of the data points of the abnormal data points at two adjacent acquisition moments in time sequence to obtain the data source to be processed of each index.
Preferably, in the to-be-processed data source of each index, the fluctuation factor of each data point is obtained according to the adjacent difference of the data points, and the data points are divided into a plurality of data segments according to the fluctuation factors, including the following specific steps:
in the data source to be processed of each index, calculating the derivative of each data point on the data source to be processed, and calculating each data point to obtain the fluctuation factor of each data point, wherein the specific calculation formula is as follows:
wherein ,represent the firstThe fluctuation factor of the data points,andrespectively represent the firstData point, thData point and the firstThe value of the data point is calculated,andrespectively represent the firstData point and the firstDerivative of data points on the data source to be processed;
processing the fluctuation factor of each data point in the data source to be processed of each index by a linear normalization means to obtain the normalized index of each data pointThe method comprises the steps of carrying out a first treatment on the surface of the From the slaveStarting traversing each data point in the data source to be processed of each index, and comparing the first data pointNormalization index of data point and size of normalization threshold, whenWhen the normalization index of the data points is greater than the normalization threshold value, the first data point isData point and the firstData points are classified into a similar section, and the comparison is carried outNormalization index and normalization threshold size of data point until data point is traversedData points at the timeIf the normalized index of (2) is less than the normalized threshold, traversing excluding the data pointsAll data points included are divided into a data segment, and the data points are divided intoAnd traversing the rest data points again and comparing until all the data points in the data source to be processed of each index are traversed to obtain a plurality of data segments in the data source to be processed of each index.
Preferably, the calculating the fluctuation degree of each data segment according to the fluctuation factor of each data point comprises the following specific steps:
the number of data points in each data segment is counted, the fluctuation degree of each data segment is calculated, and a specific calculation formula is as follows:
wherein ,represent the firstThe degree of fluctuation of the individual data segments,represent the firstThe first data segmentThe fluctuation factor of the data points,represent the firstNumber of data points in each data segment.
Preferably, the obtaining the storage degree between any two data segments according to the fluctuation degree of any two data segments includes the following specific steps:
counting the number of data points contained in each data segment of the data source to be processed of all indexes, recording the number as the length of each data segment, arranging all the data segments from short to long according to the length of each data segment to obtain a data segment set, acquiring a distance factor of each data segment and another data segment in the data segment set, comparing the fluctuation degree of each two data segments, taking the ratio as a storage degree factor of a molecular data segment and a denominator data segment, and recording the multiplied result of the distance factor and the storage degree factor as the storage degree between any two data segments.
Preferably, the step of obtaining the distance factor between each data segment and another data segment includes the following specific steps:
in the data segment set, any one data segment is marked as a target data segment, the number of the data segments between the target data segment and any one data segment except the target data segment is counted, and the data segment is marked as a rough interval distance between the target data segment and another data segment; counting the number of data segments with the same length as the target data segment in the data segments between the target data segment and any one data segment except the target data segment, recording as the repetition distance between the target data segment and another data segment, subtracting the repetition distance from the rough interval distance to obtain the interval distance between the target data segment and the another data segment, obtaining the interval distance between any two data segments, and carrying out linear normalization on the interval distances between all data segments to obtain the distance factor between each data segment and the another data segment.
Preferably, the obtaining the similar segment group of each data segment according to the storage degree between any two data segments includes the following specific steps:
subtracting the storage degree between any two data segments and taking the absolute value to obtain the measurement index of any two data segments, comparing the measurement index of any two data segments with the size of the similarity threshold, and if the measurement index of any two data segments is smaller than the similarity threshold, respectively marking the two data segments as respective similarity segment groups of the two data segments to obtain the similarity segment group of each data segment.
Preferably, the compressing the data according to the similar segment group to obtain compressed data includes the following specific steps:
obtaining a plurality of homomorphic segment groups, obtaining truncated subsequences of all data segments, and averaging the truncated subsequences of all data segments to obtain a local compressed sequence; obtaining a local compressed sequence and a sub-independent sequence of each data segment in each homomorphic segment group; and further compressing the data segments in the data segment set to obtain compressed data.
Preferably, the step of further compressing the data segment in the data segment set to obtain compressed data includes the following specific steps:
and performing run-length coding compression on all the local compression sequences, all the sub-independent sequences and all the data segments outside the homomorphic segment groups to obtain compressed data.
Preferably, the dividing each data segment into an intercepting subsequence and a sub-independent sequence specifically refers to:
and acquiring the shortest data segment in all data segments of each homomorphic segment group, wherein the length of the shortest data segment is marked as A, intercepting a subsequence which is the forefront in time sequence and has the length of A on each data segment, marking the subsequence as the intercepting subsequence of each data segment, and marking the subsequences except the intercepting subsequence on each data segment as the sub-independent sequence of each data segment.
The technical scheme of the invention has the beneficial effects that: aiming at the technical problem that the redundancy of the data stored by directly using the run-length encoded compressed data in the compound fertilizer production field monitoring data storage of the Internet of things is large, the invention analyzes different fluctuation conditions of important areas according to different index requirements and the fluctuation and similarity relation of data segments, compresses the data with different difference levels to different degrees, reduces the data redundancy while retaining the data details, and ensures that the quality of the compressed data is higher.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flow chart of steps of an optimized storage method of organic and inorganic compound fertilizer production data based on the internet of things.
Detailed Description
In order to further explain the technical means and effects adopted by the invention to achieve the preset aim, the following is a detailed description of the specific implementation, structure, characteristics and effects of the method for optimizing and storing the production data of the organic and inorganic compound fertilizer based on the internet of things according to the invention by combining the accompanying drawings and the preferred embodiment. In the following description, different "one embodiment" or "another embodiment" means that the embodiments are not necessarily the same. Furthermore, the particular features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
The invention provides a specific scheme of an organic-inorganic compound fertilizer production data optimized storage method based on the Internet of things, which is specifically described below with reference to the accompanying drawings.
Referring to fig. 1, a step flowchart of an optimized storage method for production data of an organic-inorganic compound fertilizer based on internet of things according to an embodiment of the present invention is shown, where the method includes the following steps:
s001: and acquiring a plurality of index data acquired on the organic-inorganic compound fertilizer production site, and screening and replacing the index data according to the arrangement of the data values in time sequence to obtain a data source to be processed of each index.
Specifically, time sequence data of a plurality of indexes including raw material input flow, temperature, humidity, pressure and flow of a production site are acquired from the production site of the organic-inorganic compound fertilizer, for all data points of each index in time sequence, the absolute value of the difference value of the numerical value of each data point and the data point after each data point in time sequence is calculated and recorded as the difference value of each data point; calculating the absolute value of the difference value between each data point and the difference value of the next data point in time sequence, and recording the absolute value as the adjacent abnormal value of each data point; recording the ratio of the difference value of each data point to the adjacent abnormal value as the abnormal difference value of each data point; comparing the anomaly difference value of each data point with an anomaly threshold valueComparing, the abnormal difference value is larger than the abnormal threshold valueData points of (a)And replacing all abnormal data points on each index with the average value of the data points of the abnormal data points at two adjacent acquisition moments in time sequence to obtain the data source to be processed of each index, and acquiring the initial acquisition moment of the data source to be processed of each index on the time sequence.
The embodiment uses the abnormal threshold valueThe present embodiment is described by way of example, and is not particularly limited.
So far, the data source to be processed of each index is obtained.
S002: and in the data source to be processed of each index, carrying out numerical arrangement characteristic calculation on each data point to obtain a fluctuation factor of each data point, and traversing the data points according to the fluctuation factors to obtain a plurality of data segments.
It should be noted that, the overall fluctuation range of humidity and temperature monitoring data in the monitoring data of compound fertilizer production is small, the temperature at adjacent collection time is not changed drastically, and no obvious change occurs in time sequence, so that a great amount of redundancy exists in the recorded monitoring data, the result after similar numerical value replacement is not greatly different from the source data in numerical value, wherein the fluctuation of flow data and flow velocity data at certain time is great, the change between the data is obvious, different data in fluctuation segments may have certain similarity, the complex fluctuation range of the change condition is long, the quality of the stored data is high, and different compression degrees are determined to compress according to the similarity of the data fluctuation segments among indexes and the fluctuation range of the data. The acquired monitoring data of each index is a plurality of groups of discrete data with uneven fluctuation degree in each section of the index and the like, so that the fluctuation range is small, the fluctuation condition of a curve with long duration is simpler, the fluctuation degree of the data is small, the smaller the requirement for the detail information of the curve is, the larger the fluctuation degree of the data is, the more complex the condition of the curve with large fluctuation range and small duration is, and the more the detail of the curve is required to be stored in a lossless manner to control the quality of the compound fertilizer production process. In the production process of the compound fertilizer, the fluctuation degree of each point of the data is different. For this, the data needs to be processed in a segmented manner according to the relations of the derivative of each index acquisition data point, the fluctuation amplitude and the like, and the greater the fluctuation degree of the data is, the smaller the duration of the fluctuation amplitude is, the more the storage detail information is monitored.
Specifically, in the data source to be processed of each index, calculating the derivative of each data point on the data source to be processed, and calculating each data point to obtain the fluctuation factor of each data point, wherein a specific calculation formula is as follows:
wherein ,represent the firstThe fluctuation factor of the data points,andrespectively represent the firstData point, thData point and the firstThe value of the data point is calculated,andrespectively represent the firstData point and the firstDerivative of data points on the data source to be processed. The larger the sum of absolute values of the values of two adjacent data points is, the larger the fluctuation amplitude of the data point is, and the larger the sum of absolute values of derivatives of two adjacent data points is, the faster the fluctuation speed changes, and the more the detailed information is required to be monitored and stored, so that the product quality control is realized.
It should be further noted that, because the fluctuation degree describes the fluctuation condition of the data points in the same index, the larger the fluctuation amplitude of the data points, the faster the fluctuation speed changes, and the more detail information needs to be stored so as to facilitate accurate quality control, each data point in the data source is calculated, and the data with small fluctuation factors in time sequence are selected in each index to divide the ZP data segments with adjacent values.
Wherein the present embodiment normalizes the thresholdThe present embodiment is described by way of example, and is not particularly limited.
Further, the fluctuation factor of each data point in the data source to be processed of each index is processed by a linear normalization means to obtain the normalized index of each data pointThe method comprises the steps of carrying out a first treatment on the surface of the From the slaveStarting traversing each data point in the data source to be processed of each index, and comparing the first data pointNormalization index of data point and size of normalization threshold, whenWhen the normalization index of the data points is greater than the normalization threshold value, the first data point isData point and the firstData points are classified into a similar section, and the comparison is carried outNormalization index and normalization threshold size of data point until data point is traversedData points at the timeIf the normalized index of (2) is less than the normalized threshold, traversing excluding the data pointsAll data points included are divided into a data segment, and the data points are divided intoAnd traversing the rest data points again and comparing until all the data points in the data source to be processed of each index are traversed to obtain a plurality of data segments in the data source to be processed of each index. Wherein the data pointsOnly represents the data point when the judging condition is met in the traversal process, and does not refer to a specific data point. And (3) carrying out integer numbering from 1 according to the acquisition sequence of the data segments, and recording the number of each data segment and the index corresponding to the data segment.
Thus, a number of data segments are obtained.
S003: calculating the fluctuation degree of each data segment by combining the fluctuation factor of each data point, and obtaining the storage degree between any two data segments according to the fluctuation degree of any two data segments; and obtaining the similar segment group of each data segment according to the storage degree between any two data segments.
It should be noted that, due to different characteristics of data structures, each index of the monitoring production process of the internet of things, for example, more information can be allowed to be lost if temperature data changes slowly, details of data segments with large fluctuation degrees such as flow speed and flow rate need to be stored, whether detail information is stored or not needs to be determined according to the fluctuation degree of data points in the data fluctuation segments for the fluctuation data segments, and the data segments with larger fluctuation in time sequence need to be stored for the detail information.
Specifically, the number of data points in each data segment is counted, and the fluctuation degree of each data segment is calculated, and a specific calculation formula is as follows:
wherein ,represent the firstThe degree of fluctuation of the individual data segments,represent the firstThe first data segmentThe fluctuation factor of the data points,represent the firstNumber of data points in each data segment. As the fluctuation factor of the data points in each data segment is greater, the degree of fluctuation of the entirety of the data segment that they combine is greater.
It should be further noted that, the fluctuation degree of each data segment is calculated, the data segments are classified according to the fluctuation degree of each data segment, the average value run compression is performed on the data segments with small fluctuation degree, the detail storage necessity of the data segments with large fluctuation degree is high, and further judgment needs to be performed by combining the similarity between the data segments.
Further, counting the number of data points contained in each data segment of the data source to be processed of all indexes, recording the number as the length of each data segment, and arranging all the data segments from short to long according to the length of the data segment to obtain a data segment set;
in the data segment set, any one data segment is marked as a target data segment, the number of the data segments between the target data segment and any one data segment except the target data segment is counted, and the data segment is marked as a rough interval distance between the target data segment and another data segment; counting the number of data segments with the same length as the target data segment in the data segments between the target data segment and any one data segment except the target data segment, recording as the repetition distance between the target data segment and another data segment, subtracting the repetition distance from the rough interval distance to obtain the interval distance between the target data segment and the another data segment, obtaining the interval distance between any two data segments, and carrying out linear normalization on the interval distances between all data segments to obtain a distance factor between any two data segments; the fluctuation degree of all the data segments in the data segment set is calculated, the storage degree between any two data segments in the data segment set is calculated, and a specific calculation formula is as follows:
wherein ,representing the first of a set of data segmentsData segment numberThe extent of storage of the individual data segments,andrespectively represent the data segment setsFirst, theData segment numberThe degree of fluctuation of the individual data segments,representing the first of a set of data segmentsData segment numberDistance factor of individual data segments. When the similarity of the fluctuations of two data segments in the data segment set is larger, the storage degree of the two data segments is closer to 1, which means that the compression degree of the two data segments in the storage process should be the same, and the two data segments are embodied in similar positions in the run-length matrix. The degree of storage between the fluctuation segments represents the degree of fluctuation and the degree of similarity between the data segments, and the greater the parameter value is, the more the compression degree of the corresponding two data segments should be consistent when stored.
Further, subtracting the storage degree between any two data segments and taking the absolute value to obtain the measurement index of any two data segments, and comparing the measurement index of any two data segments with the similarity threshold valueIf the measurement index of any two data segments is smaller than the similarity threshold, the two data segments are called as the similarity segment groups, and all the similarity segment groups of each data segment are obtained.
Wherein the present embodiment uses a similar thresholdThe present embodiment is described by way of example, and is not particularly limited.
To this end, all the groups of similar segments for each data segment are obtained.
S004: and integrating the data sources to be processed of each index in time sequence to obtain a replacement sequence, compressing the data according to the replacement sequence and the similar segment group, and storing the compressed data in an Internet of things database to finish the optimized storage of the organic and inorganic compound fertilizer production data.
It should be noted that, for a data segment with a large fluctuation degree, after the data segment with no correlation with other indexes and large fluctuation is compressed, lossless data is to be obtained during decompression; however, the data segment with large fluctuation and correlation with other indexes is compressed, the decompression process is a series of processes of stretching, compressing, translating and recovering at corresponding time according to the data replacement segment, and certain detail loss is caused to the data information in the process. Meanwhile, the fluctuation degree of the data segments in a similar segment group and the similarity of the data are relatively high, and when the data of a plurality of indexes are analyzed, the general trend and the fluctuation degree of each index in the production process of the organic-inorganic compound fertilizer are similar, but the specific numerical values of each index cannot be completely equal, so that the numerical value of each index needs to be integrated according to the distribution on a time sequence to obtain an integrated compression standard. When the value of any one index exceeds or falls below the standard value but the fluctuation characteristic of the index is similar to that of another data segment, the value of the index is replaced and then compressed. And different data are compressed to different degrees according to different index requirements by combining discrete characteristics in the data segment with larger fluctuation degree and considering the similarity between the fluctuation data segments, so that the data redundancy is better reduced and the data detail is reserved.
Specifically, all similar segment groups of each data segment in the data segment set are obtained, each data segment and all similar segment groups thereof are classified into one set, and the set is marked as a homomorphic segment group to obtain a plurality of homomorphic segment groups; the method comprises the steps of obtaining the shortest data segment in all data segments of each homomorphic segment group, wherein the length of the shortest data segment is recorded as A, intercepting a subsequence which is the forefront in time sequence and has the length of A on each data segment, recording the subsequence as an intercepting subsequence of each data segment, and recording subsequences except the intercepting subsequence on each data segment as a sub-independent sequence of each data segment; dividing each data segment into a truncated subsequence and a sub-independent sequence; and acquiring the intercepting subsequences of all the data segments, and solving the average value of the intercepting subsequences of all the data segments to obtain a local compressed sequence. And the local compressed sequence and all sub-independent sequences of each homomorphic segment group are numbered.
So far, in each homomorphic segment group, a local compressed sequence and a sub-independent sequence of each data segment are obtained; and performing run-length coding compression on all local compression sequences, all sub-independent sequences and all data segments outside the homomorphic segment groups in all homomorphic segment groups to obtain compressed data, and storing the compressed data into an Internet of things database to finish the optimized storage of the organic and inorganic compound fertilizer production data. The run-length encoding compression is a known technique, and the present embodiment is not described herein.
The decompression method of the compression method provided in the embodiment is as follows: and obtaining all local compression sequences, all sub-independent sequences and all data segments outside the homomorphic segment groups according to a decompression means of run-length coding compression, obtaining original data segments originally in each homomorphic segment group according to labels of all local compression sequences and all sub-independent sequences, obtaining all data segments in an original data source according to the original data segments and all data segments outside the homomorphic segment groups, and restoring the data segments to be processed of each index according to the number of each data segment and the index corresponding to the data segment to complete the decompression process.
Thus, the optimized storage of the organic-inorganic compound fertilizer production data based on the Internet of things is completed.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.
Claims (10)
1. The method for optimally storing the production data of the organic and inorganic compound fertilizer based on the Internet of things is characterized by comprising the following steps of:
acquiring a plurality of indexes acquired on an organic-inorganic compound fertilizer production site, and screening and replacing the data of the plurality of indexes according to the data values of the indexes to obtain a data source to be processed of each index;
in the data source to be processed of each index, obtaining a fluctuation factor of each data point according to the adjacent difference of the data points and dividing the data points into a plurality of data segments according to the fluctuation factors; calculating the fluctuation degree of each data segment according to the fluctuation factor of each data point, and obtaining the storage degree between any two data segments according to the fluctuation degree of any two data segments; obtaining a similar segment group of each data segment according to the storage degree between any two data segments, classifying each data segment and all the similar segment groups into a set, and marking the set as a homomorphic segment group;
and in all the data segments of each homomorphic segment group, dividing each data segment into a intercepting sub-sequence and a sub-independent sequence, compressing the data according to the intercepting sub-sequence and the sub-independent sequence to obtain compressed data, and storing the compressed data in an Internet of things database to finish the optimized storage of the organic and inorganic compound fertilizer production data.
2. The method for optimizing and storing the production data of the organic and inorganic compound fertilizer based on the Internet of things according to claim 1, wherein the method for screening and replacing the data of a plurality of indexes according to the size of the data values of the indexes to obtain the data source to be processed of each index comprises the following specific steps:
acquiring time sequence data of a plurality of indexes acquired by an organic-inorganic compound fertilizer production site, calculating absolute values of differences of numerical values of each data point and the next data point in time sequence for all data points of each index, and recording the absolute values as difference values of each data point; calculating the absolute value of the difference value between each data point and the difference value of the next data point in time sequence, and recording the absolute value as the adjacent abnormal value of each data point; recording the ratio of the difference value of each data point to the adjacent abnormal value as the abnormal difference value of each data point; comparing the abnormal difference value of each data point with a preset abnormal threshold value, recording the data points with abnormal difference values larger than the preset abnormal threshold value as abnormal data points, and replacing all abnormal data points on each index with the average value of the data points of the abnormal data points at two adjacent acquisition moments in time sequence to obtain the data source to be processed of each index.
3. The method for optimizing and storing the production data of the organic-inorganic compound fertilizer based on the internet of things according to claim 1, wherein in the to-be-processed data source of each index, the fluctuation factor of each data point is obtained according to the adjacent difference of the data points, and the data points are divided into a plurality of data segments according to the fluctuation factors, and the method comprises the following specific steps:
in the data source to be processed of each index, calculating the derivative of each data point on the data source to be processed, and calculating each data point to obtain the fluctuation factor of each data point, wherein the specific calculation formula is as follows:
wherein ,indicate->Fluctuation factor of data point-> and />Respectively represent +.>Data point, th->Data points and->Numerical value of data point,/> and />Respectively represent +.>Data point and the firstDerivative of data points on the data source to be processed;
processing the fluctuation factor of each data point in the data source to be processed of each index by a linear normalization means to obtain the normalized index of each data pointThe method comprises the steps of carrying out a first treatment on the surface of the From->Starting traversing each data point in the data source to be processed of each index, comparing +.>Normalization index of data point and size of normalization threshold, when +.>When the normalization index of the data points is greater than the normalization threshold, the +.>Data points and->Data points are classified into a similar segment, and the +.>Normalization index of data point and size of normalization threshold value until traversing to data point +.>Data point->When the normalization indicator of (2) is smaller than the normalization threshold, the traversed data point is not included +.>All data points are divided into a data section and the data points are +.>And traversing the rest data points again and comparing until all the data points in the data source to be processed of each index are traversed to obtain a plurality of data segments in the data source to be processed of each index.
4. The method for optimizing and storing the production data of the organic-inorganic compound fertilizer based on the internet of things according to claim 1, wherein the step of calculating the fluctuation degree of each data segment according to the fluctuation factor of each data point comprises the following specific steps:
the number of data points in each data segment is counted, the fluctuation degree of each data segment is calculated, and a specific calculation formula is as follows:
wherein ,indicate->Degree of fluctuation of individual data segments,/->Indicate->First->Fluctuation factor of data point->Indicate->Number of data points in each data segment.
5. The method for optimizing and storing the production data of the organic-inorganic compound fertilizer based on the internet of things according to claim 1, wherein the method for obtaining the storage degree between any two data segments according to the fluctuation degree of any two data segments comprises the following specific steps:
counting the number of data points contained in each data segment of the data source to be processed of all indexes, recording the number as the length of each data segment, arranging all the data segments from short to long according to the length of each data segment to obtain a data segment set, acquiring a distance factor of each data segment and another data segment in the data segment set, comparing the fluctuation degree of each two data segments, taking the ratio as a storage degree factor of a molecular data segment and a denominator data segment, and recording the multiplied result of the distance factor and the storage degree factor as the storage degree between any two data segments.
6. The method for optimizing and storing the production data of the organic-inorganic compound fertilizer based on the internet of things according to claim 5, wherein the step of obtaining the distance factor between each data segment and the other data segment comprises the following specific steps:
in the data segment set, any one data segment is marked as a target data segment, the number of the data segments between the target data segment and any one data segment except the target data segment is counted, and the data segment is marked as a rough interval distance between the target data segment and another data segment; counting the number of data segments with the same length as the target data segment in the data segments between the target data segment and any one data segment except the target data segment, recording as the repetition distance between the target data segment and another data segment, subtracting the repetition distance from the rough interval distance to obtain the interval distance between the target data segment and the another data segment, obtaining the interval distance between any two data segments, and carrying out linear normalization on the interval distances between all data segments to obtain the distance factor between each data segment and the another data segment.
7. The method for optimizing and storing the production data of the organic-inorganic compound fertilizer based on the internet of things according to claim 1, wherein the method for obtaining the similar segment group of each data segment according to the storage degree between any two data segments comprises the following specific steps:
subtracting the storage degree between any two data segments and taking the absolute value to obtain the measurement index of any two data segments, comparing the measurement index of any two data segments with the size of the similarity threshold, and if the measurement index of any two data segments is smaller than the similarity threshold, respectively marking the two data segments as respective similarity segment groups of the two data segments to obtain the similarity segment group of each data segment.
8. The method for optimally storing the production data of the organic-inorganic compound fertilizer based on the Internet of things according to claim 1, wherein the method for compressing the data according to the intercepting subsequence and the sub-independent sequence to obtain compressed data comprises the following specific steps:
obtaining a plurality of homomorphic segment groups, obtaining truncated subsequences of all data segments, and averaging the truncated subsequences of all data segments to obtain a local compressed sequence; obtaining a local compressed sequence and a sub-independent sequence of each data segment in each homomorphic segment group; and further compressing the data segments in the data segment set to obtain compressed data.
9. The method for optimizing and storing the production data of the organic-inorganic compound fertilizer based on the internet of things according to claim 8, wherein the method for further compressing the data segments in the data segment set to obtain compressed data comprises the following specific steps:
and performing run-length coding compression on all the local compression sequences, all the sub-independent sequences and all the data segments outside the homomorphic segment groups to obtain compressed data.
10. The method for optimizing and storing the production data of the organic-inorganic compound fertilizer based on the internet of things according to claim 1, wherein each data segment is divided into a truncated subsequence and a sub-independent sequence, specifically comprising the following steps:
and acquiring the shortest data segment in all data segments of each homomorphic segment group, wherein the length of the shortest data segment is marked as A, intercepting a subsequence which is the forefront in time sequence and has the length of A on each data segment, marking the subsequence as the intercepting subsequence of each data segment, and marking the subsequences except the intercepting subsequence on each data segment as the sub-independent sequence of each data segment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310969163.XA CN116700630B (en) | 2023-08-03 | 2023-08-03 | Organic-inorganic compound fertilizer production data optimized storage method based on Internet of things |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310969163.XA CN116700630B (en) | 2023-08-03 | 2023-08-03 | Organic-inorganic compound fertilizer production data optimized storage method based on Internet of things |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116700630A true CN116700630A (en) | 2023-09-05 |
| CN116700630B CN116700630B (en) | 2023-11-10 |
Family
ID=87841821
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310969163.XA Active CN116700630B (en) | 2023-08-03 | 2023-08-03 | Organic-inorganic compound fertilizer production data optimized storage method based on Internet of things |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116700630B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117421686A (en) * | 2023-12-18 | 2024-01-19 | 山东金诺种业有限公司 | Water and fertilizer integrated irrigation dosage data collection method |
| CN117459072A (en) * | 2023-12-22 | 2024-01-26 | 深圳市消防救援支队 | Data processing method for performance test of self-oxygen generating device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030097356A1 (en) * | 2001-10-17 | 2003-05-22 | Seok-Lyong Lee | Apparatus and method for similarity searches using hyper-rectangle based multidimensional data segmentation |
| FR3049623A1 (en) * | 2016-04-04 | 2017-10-06 | Valcap Valence Capteur | METHOD AND COMPUTER DEVICE FOR CONTINUOUS MONITORING OF A NETWORK OF WASTEWATER PIPELINES |
| CN114885234A (en) * | 2022-07-11 | 2022-08-09 | 山东美丽乡村云计算有限公司 | Scenic spot service equipment anomaly detection method based on Internet of things |
| CN114970630A (en) * | 2022-06-01 | 2022-08-30 | 中国人民解放军63796部队 | Model training method for multi-scale segmentation of time sequence data |
| CN115359807A (en) * | 2022-10-21 | 2022-11-18 | 金叶仪器(山东)有限公司 | Noise online monitoring system for urban noise pollution |
| CN116095182A (en) * | 2023-01-10 | 2023-05-09 | 广东电网有限责任公司江门供电局 | Data transmission method for GIL pipe gallery distributed sensor |
| CN116455941A (en) * | 2023-04-28 | 2023-07-18 | 中国通信建设集团设计院有限公司 | Indoor environment multi-source data transmission method and system based on Internet of things |
-
2023
- 2023-08-03 CN CN202310969163.XA patent/CN116700630B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030097356A1 (en) * | 2001-10-17 | 2003-05-22 | Seok-Lyong Lee | Apparatus and method for similarity searches using hyper-rectangle based multidimensional data segmentation |
| FR3049623A1 (en) * | 2016-04-04 | 2017-10-06 | Valcap Valence Capteur | METHOD AND COMPUTER DEVICE FOR CONTINUOUS MONITORING OF A NETWORK OF WASTEWATER PIPELINES |
| CN114970630A (en) * | 2022-06-01 | 2022-08-30 | 中国人民解放军63796部队 | Model training method for multi-scale segmentation of time sequence data |
| CN114885234A (en) * | 2022-07-11 | 2022-08-09 | 山东美丽乡村云计算有限公司 | Scenic spot service equipment anomaly detection method based on Internet of things |
| CN115359807A (en) * | 2022-10-21 | 2022-11-18 | 金叶仪器(山东)有限公司 | Noise online monitoring system for urban noise pollution |
| CN116095182A (en) * | 2023-01-10 | 2023-05-09 | 广东电网有限责任公司江门供电局 | Data transmission method for GIL pipe gallery distributed sensor |
| CN116455941A (en) * | 2023-04-28 | 2023-07-18 | 中国通信建设集团设计院有限公司 | Indoor environment multi-source data transmission method and system based on Internet of things |
Non-Patent Citations (1)
| Title |
|---|
| 孙竞: "支持分布式存储删冗的相似文件元数据集合索引", 《计 算 机 研 究 与 发 展》, vol. 50, no. 1, pages 197 - 205 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117421686A (en) * | 2023-12-18 | 2024-01-19 | 山东金诺种业有限公司 | Water and fertilizer integrated irrigation dosage data collection method |
| CN117421686B (en) * | 2023-12-18 | 2024-03-05 | 山东金诺种业有限公司 | Water and fertilizer integrated irrigation dosage data collection method |
| CN117459072A (en) * | 2023-12-22 | 2024-01-26 | 深圳市消防救援支队 | Data processing method for performance test of self-oxygen generating device |
| CN117459072B (en) * | 2023-12-22 | 2024-03-29 | 深圳市消防救援支队 | Data processing method for performance test of self-oxygen generating device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116700630B (en) | 2023-11-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116700630B (en) | Organic-inorganic compound fertilizer production data optimized storage method based on Internet of things | |
| CN110018670B (en) | Industrial process abnormal working condition prediction method based on dynamic association rule mining | |
| CN116938256B (en) | Rotary furnace operation parameter intelligent management method based on big data | |
| CN109740648B (en) | Method and device for identifying abnormal data of power load and computer equipment | |
| CN107562865A (en) | Multivariate time series association rule mining method based on Eclat | |
| CN116228176B (en) | Sewage treatment data efficient management system based on data processing | |
| CN116303374B (en) | Multi-dimensional report data optimization compression method based on SQL database | |
| CN115329910B (en) | Intelligent processing method for enterprise production emission data | |
| CN116414076B (en) | Intelligent monitoring system for recovered alcohol production data | |
| CN116032294B (en) | Intelligent processing method for atmosphere monitoring data | |
| CN1783092A (en) | Data analysis device and data analysis method | |
| CN116910285B (en) | Intelligent traffic data optimized storage method based on Internet of things | |
| CN117041359A (en) | Efficient compression transmission method for information data | |
| CN117376430B (en) | Industrial data rapid transmission method and system based on DCS | |
| CN116915259B (en) | Bin allocation data optimized storage method and system based on internet of things | |
| CN117540238B (en) | Data security management method for industrial digital information acquisition device | |
| CN112765562A (en) | Time series data trend feature extraction method based on dynamic grid division | |
| CN117155407A (en) | Intelligent mirror cabinet disinfection log data optimal storage method | |
| CN117579080B (en) | Medical care remote monitoring system based on 5G communication | |
| CN115964347B (en) | Intelligent storage method for data of market supervision and monitoring center | |
| CN117076408B (en) | Temperature monitoring big data transmission method | |
| CN115269679A (en) | Multidimensional time series overall complexity evaluation method | |
| CN113780354A (en) | Telemetry data anomaly identification method and device for dispatching automation master station system | |
| CN117473351B (en) | Power supply information remote transmission system based on Internet of things | |
| JP2023513203A (en) | An Improved Quality Value Compression Framework for New Context-Based Aligned Sequencing Data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |