CN117176176B - Data analysis processing method based on big data - Google Patents
Data analysis processing method based on big data Download PDFInfo
- Publication number
- CN117176176B CN117176176B CN202311441402.0A CN202311441402A CN117176176B CN 117176176 B CN117176176 B CN 117176176B CN 202311441402 A CN202311441402 A CN 202311441402A CN 117176176 B CN117176176 B CN 117176176B
- Authority
- CN
- China
- Prior art keywords
- data
- abnormal
- vibration
- range
- trend
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007405 data analysis Methods 0.000 title claims abstract description 12
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 230000002159 abnormal effect Effects 0.000 claims abstract description 182
- 238000007906 compression Methods 0.000 claims abstract description 74
- 230000006835 compression Effects 0.000 claims abstract description 71
- 238000012545 processing Methods 0.000 claims abstract description 15
- 238000000034 method Methods 0.000 claims description 33
- 238000004364 calculation method Methods 0.000 claims description 17
- 230000005856 abnormality Effects 0.000 claims description 10
- 238000010606 normalization Methods 0.000 claims description 7
- 238000009499 grossing Methods 0.000 description 12
- 238000010223 real-time analysis Methods 0.000 description 8
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005251 gamma ray Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Landscapes
- Measurement Of Mechanical Vibrations Or Ultrasonic Waves (AREA)
Abstract
The invention relates to the technical field of data processing, and provides a data analysis processing method based on big data, which comprises the following steps: collecting vibration data of equipment operation to obtain equipment operation big data to be compressed; acquiring a plurality of inflection point data of equipment operation big data through a revolving door trend algorithm, and acquiring the abnormal possibility of each inflection point data and acquiring a plurality of abnormal data according to vibration data and trends before and after the inflection point data; according to the trend of the abnormal data and the front and rear vibration data, obtaining the trend consistency of each neighborhood range and obtaining the minimum consistent range; combining vibration data before the abnormal data to obtain a smooth range and smooth data of each abnormal data; and replacing the abnormal data by the smooth data to obtain the replaced equipment operation big data and compressing the revolving door trend algorithm. The invention aims to solve the problem of lower compression efficiency caused by the influence of abnormal data when equipment operates big data for storage.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data analysis processing method based on big data.
Background
Big data is used as a data set with huge scale, various types and high processing speed, and is usually used for monitoring the real-time running state of various devices so as to monitor the abnormal running and faults of the devices in real time; however, the large data of the equipment is large in data size and high in real-time requirement, so that how to effectively store the large data of the large size is always an important problem.
The existing technology for carrying out real-time compression on big data operated by equipment is usually a revolving door trend algorithm, the algorithm is a lossy compression algorithm with controllable precision, the big data of the lost part information with stable change can be compressed in real time, but the revolving door trend algorithm is extremely sensitive to noise in the data, and in the compression process, the trend judgment of the revolving door trend algorithm is influenced if the noise data is encountered, so that the overall compression rate is lowered, and the storage cost is increased; abnormal data can be inevitably existed in the equipment operation big data along with the equipment operation, the abnormal data can still participate in storage after the equipment is processed, the outlier characteristics of the abnormal data can affect trend judgment of a revolving door trend algorithm like noise data, and further the compression efficiency of the equipment operation big data is affected, so that the real-time analysis of the equipment operation big data is interfered.
Disclosure of Invention
The invention provides a data analysis processing method based on big data, which aims to solve the problem of lower compression efficiency caused by the influence of abnormal data when the existing equipment operates and stores the big data, and adopts the following technical scheme:
one embodiment of the invention provides a data analysis processing method based on big data, which comprises the following steps:
collecting vibration data of equipment operation to obtain equipment operation big data to be compressed;
acquiring a plurality of inflection point data of equipment operation big data through a revolving door trend algorithm, and acquiring the abnormal possibility of each inflection point data and acquiring a plurality of abnormal data according to vibration data and trends before and after the inflection point data;
according to the trend of the abnormal data and the front and rear vibration data, obtaining the trend consistency of each neighborhood range and obtaining the minimum consistent range; combining vibration data before the abnormal data to obtain a smooth range and smooth data of each abnormal data;
and replacing the abnormal data by the smooth data to obtain the replaced equipment operation big data and compressing the revolving door trend algorithm.
Further, the method for obtaining the equipment operation big data to be compressed comprises the following specific steps:
all vibration data of the equipment are collected to form a vibration data sequence, and the vibration data is encoded to obtain an adjusted vibration data sequence which is used as equipment operation big data to be compressed.
Further, the device runs a plurality of inflection point data of the big data, and the specific acquisition method comprises the following steps:
compressing the equipment operation big data through a revolving door trend algorithm, obtaining a plurality of inflection points in the equipment operation big data through revolving door compression, and recording vibration data corresponding to the inflection points as inflection point data.
Further, the specific method for obtaining the abnormal probability of each inflection point data and obtaining a plurality of abnormal data includes:
when any compression period is compressed to nth vibration data in the period, the vibration data is inflection point data, and the calculation method of the abnormal possibility of the inflection point data comprises the following steps:
;
wherein P is n Represents an abnormal coefficient, gamma, when the nth vibration data in the compression cycle is used as inflection point data n,n-1 Indicating the trend of the inflection point data and the immediately preceding vibration data,representing the overall trend of the compression cycle, A n A data value representing the inflection point data, +.>Representing the mean value of n-1 vibration data in total from the vibration data next to the inflection point data; absolute value is calculated by the expression;
and acquiring the abnormal possibility and abnormal data of each inflection point data according to the abnormal coefficient of the inflection point data.
Further, the specific method for obtaining the likelihood of abnormality and the abnormal data of each inflection point data is as follows:
obtaining an abnormal coefficient of each inflection point data, and carrying out linear normalization on all abnormal coefficients, wherein an obtained result is recorded as the abnormal possibility of each inflection point data;
and recording inflection point data with the possibility of abnormality greater than an abnormality threshold value as abnormal data to obtain a plurality of abnormal data in the equipment operation big data.
Further, the specific method for obtaining the trend consistency of each neighborhood range and obtaining the minimum consistency range includes:
compression to period for any one compression periodWhen the ith vibration data is in the interior, the vibration data is abnormal data, and a plurality of neighborhood ranges of the abnormal data are obtained, wherein trend consistency QY of the neighborhood range m m The calculation method of (1) is as follows:
;
wherein, gamma n-m,n-1 Representing the overall trend between the mth vibration data and the immediately preceding vibration data before the abnormal data, gamma n+1,n+m Representing the overall trend between the next vibration data and the m-th vibration data adjacent to the abnormal data, || represents absolute value, exp () represents an exponential function based on natural constant;
and obtaining trend consistency of each neighborhood range for the abnormal data, taking the neighborhood range corresponding to the maximum value in the trend consistency as the minimum consistency range of the abnormal data, and taking the minimum neighborhood range in the corresponding multiple neighborhood ranges as the minimum consistency range if the maximum value corresponds to the multiple neighborhood ranges.
Further, the method for obtaining the smoothing range and the smoothed data of each abnormal data includes the following specific steps:
when any compression period is compressed to the ith vibration data in the period, the vibration data is abnormal data, and the calculation method of the smooth range L of the abnormal data is as follows:
；
；
wherein m is 0 Represents the minimum coincidence range of the abnormal data, delta represents the integral outlier degree of the abnormal data, A i A data value representing the abnormal data, A j Representing the abnormal data by m 0 Data value of j-th vibration data among the vibration data []Representation rounding, || represents absolute value, exp () representsAn exponential function based on a natural constant;
and acquiring a smooth range of each piece of abnormal data, and acquiring the smooth data of each piece of abnormal data according to the vibration data in the smooth range before the abnormal data.
Further, the method for obtaining the smoothed data of each abnormal data includes the following specific steps:
obtaining the error degree of each abnormal data in the smooth range according to the vibration data in the smooth range before the abnormal data; when any compression cycle is compressed to the ith vibration data in the cycle, the vibration data is abnormal data, and the smooth data B of the abnormal data i The calculation method of (1) is as follows:
；
wherein L represents the smooth range of the abnormal data, A l A data value representing the first vibration data of L pieces of vibration data before the abnormal data,representing the overall trend of L vibration data before the abnormal data, C i Indicating the degree of error in the smoothed range of the abnormal data.
Further, the error degree in the smooth range of each abnormal data is obtained by the specific method:
；
wherein C is i Indicating the degree of error in the smoothed range of the abnormal data, A l A data value representing the first vibration data of L pieces of vibration data before the abnormal data,representing the overall trend of L vibration data before the abnormal data, A l+1 Represents the 1 st vibration data in the L pieces of vibration data before the abnormal data, ||represents the calculationAbsolute value.
Further, the method for obtaining the replaced equipment operation big data and compressing the revolving door trend algorithm comprises the following specific steps:
replacing each abnormal data by the corresponding smooth data to obtain replaced equipment operation big data; the compression of the revolving door trend algorithm is carried out again on the large data operated by the replaced equipment, and when any one piece of replaced smooth data is compressed to the smooth data, if the trend of the smooth data and the adjacent previous vibration data is in a threshold range, the backward compression of the current compression period is continued; if the trend of the smooth data and the adjacent previous vibration data is not in the threshold value range, the previous vibration data is used as the compression end data of the previous compression period, and the smooth data is used as the new compression period to compress the revolving door trend algorithm;
and compressing the rotating door trend algorithm to the replaced equipment operation big data to obtain compressed equipment operation big data.
The beneficial effects of the invention are as follows: according to the method, the equipment operation big data is compressed and stored in real time, real-time analysis processing of the equipment operation data based on the big data is realized, the equipment operation big data is compressed by using a revolving door trend algorithm, abnormal data in the equipment operation big data is smoothed, the compression efficiency of the equipment operation big data is improved, and further the real-time analysis processing of the equipment operation big data is guaranteed; the method comprises the steps of firstly carrying out primary revolving door compression to obtain inflection point data, analyzing and quantizing trends and data values of the inflection point data and adjacent front-rear vibration data to obtain abnormal possibility of the inflection point data, screening out abnormal data, and ensuring that the inflection point data which normally reflects the change of the running state of equipment is not smoothed to cause inaccurate compression results; and obtaining a minimum consistent range according to the trend change of the vibration data before and after the abnormal data, and adjusting the minimum consistent range according to the integral outlier degree of the vibration data in the abnormal data compared with the previous minimum consistent range to obtain a smooth range, so that the larger the deviation is, the larger the smooth range corresponding to the abnormal data is, the accuracy of the smooth data is ensured, the smooth data is obtained according to the vibration data in the previous smooth range for the abnormal data, the smooth data is utilized to replace the abnormal data, and then the rotating door compression is carried out on the replaced equipment operation big data, thereby avoiding the influence on the compression efficiency of the rotating door trend algorithm caused by the early end of the compression period of the abnormal data.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.
Fig. 1 is a flow chart of a data analysis processing method based on big data according to an embodiment of the invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, a flowchart of a data analysis processing method based on big data according to an embodiment of the invention is shown, and the method includes the following steps:
and S001, collecting vibration data of equipment operation to obtain equipment operation big data to be compressed.
The purpose of the embodiment is to analyze and process big data generated by equipment operation, so that the big data of the equipment operation needs to be acquired and compressed for storage, thereby providing a basis for real-time analysis; in the running process of the equipment, various index data of the equipment need to be monitored through various sensors, such as vibration data through an acceleration sensor, temperature data through a temperature sensor, current, voltage, equipment load and other data are monitored through a universal meter, and the vibration data are taken as an example for description; all vibration data from the beginning of the use of the acquisition equipment to the current moment are collected, the sampling time interval is set to be 1 second in the embodiment, and all vibration data are arranged according to time sequence to form a vibration data sequence; the data types acquired by the data acquisition devices of different models are different, and the revolving door trend algorithm compresses the vibration data based on the data changes, so that the embodiment encodes the vibration data through an information encoding technology, each vibration data in the vibration data sequence is unified into a decimal number form, and the adjusted vibration data sequence is used as large data of the device to be compressed.
So far, the equipment operation big data to be compressed is obtained.
Step S002, a plurality of inflection point data of equipment operation big data are obtained through a revolving door trend algorithm, and the abnormal possibility of each inflection point data is obtained and a plurality of abnormal data are obtained according to the vibration data and the trend before and after the inflection point data.
When the compression algorithm of the revolving door trend is used for compressing the big data of the equipment, trend judgment is carried out backwards gradually along the time sequence from the first vibration data, when a certain trend value exceeds a threshold value, the compression of the vibration data is ended, and the compression of the vibration data is carried out iteratively from the next vibration data exceeding the threshold value. However, when outliers, namely abnormal data, occur in the equipment operation big data, the whole threshold is ended prematurely, so that a compression interval is ended in advance, and the whole compression rate of the equipment operation big data is lowered; therefore, the rotating door trend compression algorithm is utilized to normally compress the equipment operation big data, when outlier data larger than a threshold value appears, namely inflection point data is obtained, whether the inflection point data is abnormal data or not is analyzed by judging the change trend of the already compressed vibration data of a compression section where the inflection point data is located and the change trend of the subsequent vibration data of the inflection point data, and then a basis is provided for judging whether the inflection point data needs to be smoothed or the compression of a corresponding compression period or not.
Specifically, the big data of the equipment operation is compressed through a revolving door trend algorithm, and a threshold value implementer sets the big data of the equipment operation according to the data type and the monitoring index in the big data of the equipment operation, and the embodiment is not described in detail; and (3) obtaining a plurality of inflection points in the equipment operation big data through revolving door compression, and recording vibration data corresponding to the inflection points as inflection point data, wherein the inflection point data is the first data in each compression period in a plurality of compression periods obtained after compression, and the trend of the inflection point data and the adjacent previous vibration data is greatly changed and exceeds a threshold value, so that the inflection point is formed.
Further, when any compression cycle is compressed to the nth vibration data in the cycle, the vibration data is inflection point data, and the calculating method of the abnormal possibility is as follows:
；
wherein P is n Represents an abnormal coefficient, gamma, when the nth vibration data in the compression cycle is used as inflection point data n,n-1 Representing the trend of the inflection point data and the adjacent previous vibration data, namely the slope between the adjacent two vibration data;representing the overall trend of the compression period, namely, the difference value between the adjacent previous vibration data and the first vibration data of the compression period, and comparing the obtained ratio with the time difference value, namely, calculating the overall slope between the head vibration data and the tail vibration data of the compression period; a is that n A data value representing the inflection point data, +.>Representing the mean value of n-1 vibration data in total from the vibration data next to the inflection point data; absolute value is calculated by the expression; because each inflection point data is obtained after the trend exceeds the threshold value in the compression process of a certain compression period, the difference between the corresponding trend of the inflection point data and the overall trend of the compression period just ended is analyzed firstThe larger the difference is, the larger the change of adjacent vibration data is, and the larger the probability of occurrence of abnormality is; meanwhile, the change of the vibration data can be the change of the running state of the equipment, namely the subsequent vibration data is changed, the outlier degree of the mean value of the inflection point data and the subsequent vibration data is analyzed, if the outlier degree is smaller, the probability of abnormality of the inflection point data is smaller, and the probability of the change of the vibration data caused by the change of the equipment is larger; if the outlier degree is larger, the inflection point data has larger trend difference from the previous vibration data, and meanwhile, the outlier degree of the inflection point data and the subsequent vibration data is larger, so that the inflection point data is more likely to be abnormal data; obtaining the abnormal coefficient of each inflection point data according to the method, and carrying out linear normalization on all abnormal coefficients, wherein the obtained result is recorded as the abnormal possibility of each inflection point data.
Further, an abnormal threshold is preset, the abnormal threshold is described by 0.7, inflection point data with the possibility of abnormality greater than the abnormal threshold is recorded as abnormal data, and a plurality of abnormal data in the equipment operation big data are obtained; the inflection point data with the possibility of abnormality less than or equal to the abnormality threshold value is still used as normal inflection point data without special processing.
The method comprises the steps of obtaining a plurality of inflection point data of equipment operation big data compressed by a turnstile trend algorithm, analyzing according to the abnormal possibility of the inflection point data, obtaining a plurality of abnormal data, and providing a basis for subsequent smooth analysis of the abnormal data.
Step S003, according to the trend of the abnormal data and the front and rear vibration data, obtaining the trend consistency of each neighborhood range and obtaining the minimum consistent range; and combining vibration data before the abnormal data to obtain the smooth range and the smooth data of each abnormal data.
After obtaining the abnormal data and the normal inflection point data, the normal inflection point data is the normal inflection point in the compression process of the revolving door, namely the vibration data overall change caused by the change of the running state of the equipment; the abnormal data is caused by various factors in the acquisition process, the equipment is already finished in abnormal processing, but the abnormal data is still stored as vibration data in equipment operation big data, and the vibration data can cause the compression period to be finished in advance so as to influence the compression effect, so that the abnormal data needs to be smoothed; the difference of the smoothing ranges adopted in the smoothing process can lead to the change of the compression result, the calculated amount is increased due to the excessively large smoothing range, and the inaccuracy of the smoothing result is caused by the excessively small smoothing range; therefore, the smooth range is required to be quantized by combining the trend of the abnormal data and the front and rear vibration data, so that the smooth data is obtained for the abnormal data, and a basis is provided for compressing the large data of the follow-up equipment by reusing the revolving door trend algorithm.
Specifically, when any compression cycle compresses to the ith vibration data in the cycle, the vibration data is abnormal data, a variable neighborhood range is preset, the minimum neighborhood range in this embodiment starts from 2, and the maximum neighborhood range is denoted as M, where M<i and the neighborhood range is an integer, i.e. the neighborhood range traverses forward without exceeding the compression period, the trend consistency QY of the neighborhood range m m The calculation method of (1) is as follows:
;
wherein, gamma n-m,n-1 Representing the overall trend between the mth vibration data and the adjacent previous vibration data before the abnormal data, namely taking the ratio of the difference value of the two vibration data to the time difference value as the overall trend of the two vibration data; gamma ray n+1,n+m Representing the overall trend between the next vibration data and the next mth vibration data adjacent to the abnormal data, wherein the overall trend is obtained according to the method; the absolute value is represented by the I, the exponential function taking a natural constant as a base is represented by exp (), the inverse proportion relation and normalization processing are presented by adopting an exp (-x) model in the embodiment, wherein x represents the input of the model, and an implementer can set the inverse proportion function and the normalization function according to actual conditions; the more similar the overall trend of the plurality of vibration data before the abnormal data is to the overall trend of the plurality of vibration data after the abnormal data, the closer the ratio is to 1, the greater the trend consistency is; obtaining trend consistency of each neighborhood range for the abnormal data according to the method, and obtaining trend oneAnd taking the neighborhood range corresponding to the maximum value in the consistency as the minimum consistent range of the abnormal data, and taking the minimum neighborhood range in the corresponding multiple neighborhood ranges as the minimum consistent range if the maximum value corresponds to the multiple neighborhood ranges.
Further, after the minimum consistent range is obtained, the calculating method of the smoothing range L of the abnormal data is as follows:
;
;
wherein m is 0 Represents the minimum coincidence range of the abnormal data, delta represents the integral outlier degree of the abnormal data, A i A data value representing the abnormal data, A j Representing the abnormal data by m 0 Data value of j-th vibration data among the vibration data []The expression rounding and rounding, ||represents the absolute value, exp () represents an exponential function based on a natural constant, and the embodiment adopts an exp (-x) model to present an inverse proportion relation and normalization processing, wherein x represents the input of the model, and an implementer can set the inverse proportion function and the normalization function according to actual situations; after the minimum consistent range is obtained, the minimum consistent range is required to be adjusted by the integral outlier degree, the integral outlier degree is quantified by the vibration data in the previous minimum consistent range and the difference of two mean values obtained by whether the abnormal data is contained or not, the larger the difference is, the larger the outlier degree of the abnormal data and the vibration data in the previous minimum consistent range is, the larger the integral outlier degree is, the larger the minimum consistent range is required to be enlarged, and the larger smooth range is required to be obtained, so that the smooth effect is ensured, and more vibration data participate in smoothing.
Further, for the abnormal data, the smooth data B of the abnormal data is obtained i The calculation method of (1) is as follows:
;
;
wherein L represents the smooth range of the abnormal data, A l A data value representing the first vibration data of L pieces of vibration data before the abnormal data,representing the overall trend of L vibration data before the abnormal data, wherein the overall trend is obtained by adopting the calculation method of the overall trend of the adjacent previous vibration data and the previous L data of the abnormal data i Indicating the degree of error in the smoothed range of the abnormal data, A l+1 The first +1st vibration data in the L vibration data before the abnormal data is represented, and I represents the absolute value; calculating the overall trend of the vibration data in the previous smoothing range of the abnormal data, predicting the data value of the next adjacent vibration data for each vibration data in the previous smoothing range according to the overall trend, and acquiring the average value to serve as the reference value of the abnormal data smoothing data; meanwhile, the allowable error degree of the smooth range is quantized according to the difference between the predicted data value and the data value of the next adjacent vibration data, and the average value of the ratios of all the differences and the corresponding vibration data is calculated to be used as the error degree; and obtaining a maximum value and a minimum value after smoothing according to the error degree and the reference value, and obtaining smoothed data by averaging the two values.
Further, according to the method, a plurality of neighborhood ranges are obtained for each piece of abnormal data, trend consistency of each neighborhood range is calculated, and the minimum consistent range, the smooth range and the smooth data of each piece of abnormal data are obtained.
After a plurality of neighborhood ranges of each abnormal data are obtained and the minimum consistent range is obtained, the smooth range of each abnormal data is obtained according to the combination of the minimum consistent range and the vibration data, and the smooth data of each abnormal data are finally obtained through the overall trend of the vibration data in the smooth range.
And S004, replacing the abnormal data by the smooth data to obtain the replaced equipment operation big data and compressing the revolving door trend algorithm.
After the smooth data of each abnormal data are obtained, replacing each abnormal data in the equipment operation big data through the corresponding smooth data to obtain replaced equipment operation big data; the compression of the revolving door trend algorithm is carried out again on the large data operated by the replaced equipment, the threshold value is kept unchanged, and when any one piece of replaced smooth data is compressed to the smooth data, if the trend (slope) of the smooth data and the adjacent previous vibration data is in the threshold value range, the backward compression of the current compression period is continued; if the trend (slope) of the smooth data and the adjacent previous vibration data is not in the threshold value range, the previous vibration data is used as compression end data of the previous compression period, and the smooth data is used as a new compression period to compress a revolving door trend algorithm; the method comprises the steps of compressing a revolving door trend algorithm according to the replaced equipment operation big data to obtain compressed equipment operation big data, storing the compressed equipment operation big data, providing a basis for subsequent real-time analysis of equipment operation, and realizing real-time analysis and processing of equipment operation based on the big data.
So far, the real-time compression storage of the big data of the equipment operation provides a basis for the real-time analysis of the equipment operation, and the real-time analysis processing of the data of the equipment operation based on the big data is realized.
It should be noted that, in this embodiment, there are several calculations of the trend between the vibration data, and since the trend calculation adopts a slope calculation or is an existing calculation method in the revolving door trend algorithm, the trend calculation is not described in detail in this embodiment, and is directly referred to and participates in the calculation.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.
Claims (8)
1. The data analysis processing method based on big data is characterized by comprising the following steps:
collecting vibration data of equipment operation to obtain equipment operation big data to be compressed;
acquiring a plurality of inflection point data of equipment operation big data through a revolving door trend algorithm, and acquiring the abnormal possibility of each inflection point data and acquiring a plurality of abnormal data according to vibration data and trends before and after the inflection point data;
according to the trend of the abnormal data and the front and rear vibration data, obtaining the trend consistency of each neighborhood range and obtaining the minimum consistent range; combining vibration data before the abnormal data to obtain a smooth range and smooth data of each abnormal data;
replacing the abnormal data by the smooth data to obtain the replaced equipment operation big data and compressing a revolving door trend algorithm;
the specific method for acquiring the abnormal possibility of each inflection point data and obtaining a plurality of abnormal data comprises the following steps:
when any compression period is compressed to nth vibration data in the period, the vibration data is inflection point data, and the calculation method of the abnormal possibility of the inflection point data comprises the following steps:
；
wherein P is n Represents an abnormal coefficient, gamma, when the nth vibration data in the compression cycle is used as inflection point data n,n-1 Indicating the trend of the inflection point data and the immediately preceding vibration data,representing the overall trend of the compression cycle, A n A data value representing the inflection point data, +.>Representing the mean value of n-1 vibration data in total from the vibration data next to the inflection point data;absolute value is calculated by the expression;
acquiring the abnormal possibility and abnormal data of each inflection point data according to the abnormal coefficient of the inflection point data;
the abnormal possibility and abnormal data of each inflection point data are obtained by the specific method:
obtaining an abnormal coefficient of each inflection point data, and carrying out linear normalization on all abnormal coefficients, wherein an obtained result is recorded as the abnormal possibility of each inflection point data;
and recording inflection point data with the possibility of abnormality greater than an abnormality threshold value as abnormal data to obtain a plurality of abnormal data in the equipment operation big data.
2. The method for analyzing and processing data based on big data according to claim 1, wherein the method for obtaining the equipment operation big data to be compressed comprises the following specific steps:
all vibration data of the equipment are collected to form a vibration data sequence, and the vibration data is encoded to obtain an adjusted vibration data sequence which is used as equipment operation big data to be compressed.
3. The big data-based data analysis processing method according to claim 1, wherein the device runs a plurality of inflection point data of the big data, and the specific acquisition method is as follows:
compressing the equipment operation big data through a revolving door trend algorithm, obtaining a plurality of inflection points in the equipment operation big data through revolving door compression, and recording vibration data corresponding to the inflection points as inflection point data.
4. The big data based data analysis processing method according to claim 1, wherein the obtaining the trend consistency of each neighborhood range and obtaining the minimum consistency range comprises the following specific steps:
when any compression period is compressed to ith vibration data in the period, the vibration data is abnormal data, and a plurality of neighborhood ranges of the abnormal data are obtained, wherein the trend of the neighborhood range mPotential uniformity QY m The calculation method of (1) is as follows:
;
wherein, gamma n-m,n-1 Representing the overall trend between the mth vibration data and the immediately preceding vibration data before the abnormal data, gamma n+1,n+m Representing the overall trend between the next vibration data and the m-th vibration data adjacent to the abnormal data, || represents absolute value, exp () represents an exponential function based on natural constant;
and obtaining trend consistency of each neighborhood range for the abnormal data, taking the neighborhood range corresponding to the maximum value in the trend consistency as the minimum consistency range of the abnormal data, and taking the minimum neighborhood range in the corresponding multiple neighborhood ranges as the minimum consistency range if the maximum value corresponds to the multiple neighborhood ranges.
5. The method for analyzing and processing data based on big data according to claim 1, wherein the step of obtaining the smoothed range and the smoothed data of each abnormal data comprises the following specific steps:
when any compression period is compressed to the ith vibration data in the period, the vibration data is abnormal data, and the calculation method of the smooth range L of the abnormal data is as follows:
;
;
wherein m is 0 Represents the minimum coincidence range of the abnormal data, delta represents the integral outlier degree of the abnormal data, A i A data value representing the abnormal data, A j Representing the abnormal data by m 0 Data value of j-th vibration data among the vibration data []Representation rounding and rounding, ||tableAbsolute values are shown, exp () represents an exponential function based on a natural constant;
and acquiring a smooth range of each piece of abnormal data, and acquiring the smooth data of each piece of abnormal data according to the vibration data in the smooth range before the abnormal data.
6. The method for analyzing and processing data based on big data according to claim 5, wherein the step of obtaining the smoothed data of each abnormal data comprises the following specific steps:
obtaining the error degree of each abnormal data in the smooth range according to the vibration data in the smooth range before the abnormal data; when any compression cycle is compressed to the ith vibration data in the cycle, the vibration data is abnormal data, and the smooth data B of the abnormal data i The calculation method of (1) is as follows:
;
wherein L represents the smooth range of the abnormal data, A l A data value representing the first vibration data of L pieces of vibration data before the abnormal data,representing the overall trend of L vibration data before the abnormal data, C i Indicating the degree of error in the smoothed range of the abnormal data.
7. The method for analyzing and processing data based on big data according to claim 6, wherein the error degree in the smooth range of each abnormal data is obtained by:
;
wherein C is i Indicating the degree of error in the smoothed range of the abnormal data, A l Representing the total L vibration numbers before the abnormal dataAccording to the data value of the first vibration data,representing the overall trend of L vibration data before the abnormal data, A l+1 The first +1st vibration data among the L vibration data before the abnormal data is represented, and || represents an absolute value.
8. The big data-based data analysis processing method according to claim 1, wherein the obtained replaced equipment runs big data and compresses a revolving door trend algorithm, and the specific method comprises the following steps:
replacing each abnormal data by the corresponding smooth data to obtain replaced equipment operation big data; for the replaced device
Carrying out compression of a revolving door trend algorithm again on the running big data, and continuing backward compression of the current compression period when any one of the replaced smooth data is compressed to the smooth data if the trend of the smooth data and the adjacent previous vibration data is in a threshold range; if the trend of the smooth data and the adjacent previous vibration data is not in the threshold value range, the previous vibration data is used as the compression end data of the previous compression period, and the smooth data is used as the new compression period to compress the revolving door trend algorithm;
and compressing the rotating door trend algorithm to the replaced equipment operation big data to obtain compressed equipment operation big data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311441402.0A CN117176176B (en) | 2023-11-01 | 2023-11-01 | Data analysis processing method based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311441402.0A CN117176176B (en) | 2023-11-01 | 2023-11-01 | Data analysis processing method based on big data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117176176A CN117176176A (en) | 2023-12-05 |
| CN117176176B true CN117176176B (en) | 2024-01-23 |
Family
ID=88937849
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311441402.0A Active CN117176176B (en) | 2023-11-01 | 2023-11-01 | Data analysis processing method based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117176176B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106649026A (en) * | 2016-09-26 | 2017-05-10 | 国家电网公司北京电力医院 | Monitoring data compression method applicable to operation and maintenance automation system |
| CN115425985A (en) * | 2022-08-10 | 2022-12-02 | 国网宁夏电力有限公司超高压公司 | Real-time data compression storage method, medium and system |
| CN116823492A (en) * | 2023-05-05 | 2023-09-29 | 陕西长瑞安驰信息技术集团有限公司 | Data storage method and system |
-
2023
- 2023-11-01 CN CN202311441402.0A patent/CN117176176B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106649026A (en) * | 2016-09-26 | 2017-05-10 | 国家电网公司北京电力医院 | Monitoring data compression method applicable to operation and maintenance automation system |
| CN115425985A (en) * | 2022-08-10 | 2022-12-02 | 国网宁夏电力有限公司超高压公司 | Real-time data compression storage method, medium and system |
| CN116823492A (en) * | 2023-05-05 | 2023-09-29 | 陕西长瑞安驰信息技术集团有限公司 | Data storage method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117176176A (en) | 2023-12-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115801901B (en) | Enterprise production emission data compression processing method | |
| CN115987295B (en) | Crop monitoring data efficient processing method based on Internet of things | |
| CN110705181A (en) | Rolling bearing residual life prediction method based on convolution length-time memory cyclic neural network | |
| CN115766607A (en) | Data compression transmission method based on 5G Internet of things | |
| Chandak et al. | LFZip: Lossy compression of multivariate floating-point time series data via improved prediction | |
| CN115902647B (en) | Intelligent battery state monitoring method | |
| CN110995273B (en) | Data compression method, device, equipment and medium for power database | |
| CN117131035B (en) | Intelligent storage method and system for chemical dye production data | |
| CN110581834A (en) | communication capability opening abnormity detection method and device | |
| CN117390591B (en) | Operation monitoring method and system for coal conveying belt sampling machine based on electric parameter analysis | |
| CN111612216A (en) | Customized product quality control prediction method based on GM grey model and Bootstrap theory | |
| CN114969060A (en) | Industrial equipment time sequence data compression storage method and device | |
| CN117176176B (en) | Data analysis processing method based on big data | |
| CN117459418A (en) | Real-time data acquisition and storage method and system | |
| CN113869526A (en) | Data processing model performance improving method and device, storage medium and electronic equipment | |
| CN116631563B (en) | Big data storage and intelligent matching method for pharmaceutical industry | |
| KR102469238B1 (en) | Real-time neural spike detection | |
| CN112329272B (en) | Residual life fitting prediction method integrating photovoltaic module degradation and life data | |
| CN116743180A (en) | Intelligent storage method for energy storage power supply data | |
| CN117095828A (en) | Blood glucose value prediction and alarm method based on follow-up record of diabetics | |
| CN116561927A (en) | Digital twin-driven small sample rotary machine residual life prediction method and system | |
| CN116384574A (en) | W-LSTM-based non-parameter load interval prediction method | |
| CN117113107B (en) | Rotor die casting abnormal error data processing method | |
| CN112798949A (en) | Pumped storage unit generator temperature early warning method and system | |
| CN112085926A (en) | River water pollution early warning method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |