CN116561084B - Flexible engineering platform data intelligent storage method and system - Google Patents
Flexible engineering platform data intelligent storage method and system Download PDFInfo
- Publication number
- CN116561084B CN116561084B CN202310825990.1A CN202310825990A CN116561084B CN 116561084 B CN116561084 B CN 116561084B CN 202310825990 A CN202310825990 A CN 202310825990A CN 116561084 B CN116561084 B CN 116561084B
- Authority
- CN
- China
- Prior art keywords
- data
- bit number
- layer
- optimal
- value range
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000007906 compression Methods 0.000 claims abstract description 27
- 230000006835 compression Effects 0.000 claims abstract description 27
- 238000013500 data storage Methods 0.000 claims abstract description 6
- 238000013144 data compression Methods 0.000 claims abstract description 4
- 230000000694 effects Effects 0.000 abstract description 11
- 229920005994 diacetyl cellulose Polymers 0.000 abstract description 4
- 238000004364 calculation method Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 4
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000007115 recruitment Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention relates to the technical field of data compression and storage, in particular to a flexible engineering platform data intelligent storage method and system, which comprises the following steps: obtaining binary data, obtaining a data length mark graph and a data probability density graph, obtaining a first layer bit number value range, obtaining the preference degree of the first layer bit number value range according to the data occurrence frequency of the data length, obtaining a predicted layer depth according to the data occurrence frequency of the data length in a sub-range of the optimal first layer bit number value range, obtaining a reduced optimal first layer bit number value range according to the predicted layer depth and a preset layer depth threshold, obtaining the optimal first layer bit number according to the number of codes required to be increased under any bit number value in the reduced optimal first layer bit number value range, further obtaining the optimal bit number of each layer, and finally performing data layered compression. The invention enables the compression effect to be optimal by adaptively selecting the optimal bit number of each layer of the DACs algorithm.
Description
Technical Field
The invention relates to the technical field of data compression and storage, in particular to a flexible engineering platform data intelligent storage method and system.
Background
With the progress of the age, a number of flexible recruitment platforms are now presented. An employer issues tasks on the platform and employees accept the tasks. The data information contains a large amount of data information such as skill level, order number, order amount and the like, and the data needs to be compressed and stored for storage. If the stored data is to be queried, all the data needs to be decompressed, so that a large amount of time is wasted. Therefore, the compressed data needs to be positioned, and only the data to be queried needs to be decompressed. The DACs algorithm in the existing compression algorithm can compress data by using variable length codes and can be positioned.
The DACs algorithm adopts a fixed bit number to carry out bit layering (the conventional value is 2) on the code, and the compression effect is different for different data. If the number of bits of each layer is to be set according to the data length, the number of bits is to be set manually according to the empirical value, so that the compression effect cannot be guaranteed to be optimal, and therefore, the adaptive number of bits is to be calculated according to the length characteristics of binary data, thereby improving the efficiency of the compression effect.
Disclosure of Invention
The invention provides a flexible engineering platform data intelligent storage method and system, which are used for solving the existing problems.
The invention discloses a flexible engineering platform data intelligent storage method and a system, which adopt the following technical scheme:
the embodiment of the invention provides a flexible engineering platform data intelligent storage method, which comprises the following steps:
acquiring binary data, and acquiring the data length and the data occurrence frequency of the binary data;
obtaining a plurality of first layer bit number value ranges according to the data length, the data occurrence frequency and the interval data length average value;
obtaining the preference degree of the first layer bit number value range according to the occurrence frequency of the data length in the first layer bit number value range, and taking the first layer bit number value range corresponding to the maximum value of the preference degree as the optimal first layer bit number value range;
obtaining a plurality of sub-ranges of the optimal first layer bit number value range, obtaining a predicted layer depth according to the occurrence frequency of data with the data length in the sub-ranges, and obtaining the reduced optimal first layer bit number value range according to the predicted layer depth and a preset layer depth threshold;
obtaining the optimal first layer bit number according to the coding number to be increased under any bit number value in the reduced optimal first layer bit number value range, and obtaining the optimal bit number of each layer;
and carrying out data layering compression according to the optimal bit number of each layer.
Further, the obtaining the value range of the first layer bit number according to the data length, the data occurrence frequency and the interval data length average value includes the following specific steps:
wherein ,indicate->The frequency of occurrence of data of the data length of the binary data; />Is shown in the intervalMiddle->A data length of the binary data; />Expressed in interval +.>A data length average value of all binary data in the database; />Representation interval->The number of binary data; />Is a super parameter; />Representing the minimum value of the first layer bit number value range; />Indicate->A data length; />Representing the maximum value of the first layer bit number value range; />Representing the difference between the maximum and minimum of the first layer bit number range, will +.>As->A first layer bit number value range of the binary data; a first layer bit number value range of each binary data is acquired.
Further, the obtaining the preference degree of the first layer bit number value range according to the data occurrence frequency of the data length in the first layer bit number value range includes the following specific steps:
acquiring any first layer bit number value range, acquiring the occurrence frequencies of all data lengths in the first layer bit number value range, and taking the sum of the occurrence frequencies of all data lengths in the range as the preference degree of the first layer bit number value range; and obtaining the preference degree of each first layer bit number value range.
Further, the predicting layer depth is obtained according to the occurrence frequency of the data with the data length in the sub-range, and the method comprises the following specific steps:
wherein ,representing a predicted layer depth; />Representing the depth at which the layer is located; />A data length maximum value representing all binary data; />Representation ofMaximum value of data length in any one of the sub-ranges, < >>Representing the minimum value of the data length in this sub-range; />Representing the data length in this sub-range +.>Frequency of occurrence of data of (2); />Representing the data length in this sub-range +.>Is a data length value of (a).
Further, the obtaining the reduced optimal first layer bit number value range according to the predicted layer depth and the preset layer depth threshold value includes the following specific steps:
obtaining the predicted layer depth of all sub-ranges in the optimal first layer bit number value range to obtain a plurality of satisfied valuesJudging a sub-range of a condition, selecting the sub-range with the smallest predicted layer depth as the reduced optimal first layer bit number value range, and selecting the sub-range with the smallest predicted layer depth as the reduced optimal first layer bit number value range>Representing predicted layer depth,/->Representing a preset layer depth threshold.
Further, the method for obtaining the optimal first layer bit number according to the coding number to be increased under any bit number value in the reduced optimal first layer bit number value range includes the following specific steps:
wherein ,indicate selection of +.>The number of codes to be increased when the number of bits is valued; />Indicate selection of +.>When the number of bits is taken, the +.>The number of bits of the binary data to be zero padding; u represents that the data length is smaller than +.>The number of binary data of the value of the number of bits; />Indicating that the data length is greater than +.>The number of binary data of the value of the number of bits;
traversing all the bit number values in the reduced optimal first layer bit number value range, and taking the bit number value with the minimum coding number required to be increased as the optimal first layer bit number.
Further, the specific method for obtaining the optimal bit number of each layer is as follows:
and removing the optimal first layer bit number from all binary data, acquiring the optimal second layer bit number according to the method for acquiring the optimal first layer bit number in the rest binary data according to the length of the removed data, and continuously removing to sequentially acquire the optimal bit number of all layers.
The embodiment of the invention provides a flexible engineering platform data intelligent storage system, which comprises the following modules:
and a data acquisition module: the method comprises the steps of obtaining binary data, and obtaining the data length and the data occurrence frequency of the binary data;
bit number value range module: the method comprises the steps of obtaining a plurality of first layer bit number value ranges according to data length, data occurrence frequency and interval data length average value;
the optimal value range module: obtaining the preference degree of the first layer bit number value range according to the occurrence frequency of the data length in the first layer bit number value range, and taking the first layer bit number value range corresponding to the maximum value of the preference degree as the optimal first layer bit number value range;
an optimal value range reduction module: the method comprises the steps of obtaining a plurality of sub-ranges of an optimal first-layer bit number value range, obtaining a predicted layer depth according to the occurrence frequency of data with data length in the sub-ranges, and obtaining a reduced optimal first-layer bit number value range according to the predicted layer depth and a preset layer depth threshold;
an optimal hierarchical bit number module: the method comprises the steps of obtaining the optimal first layer bit number according to the coding number to be increased under any bit number value in the reduced optimal first layer bit number value range, and obtaining the optimal bit number of each layer;
and the data layering compression module is used for: for performing data hierarchical compression according to the optimal number of bits per layer.
The technical scheme of the invention has the beneficial effects that: according to the invention, flexible labor platform data are converted into binary data, self-adaptive bit number calculation is performed according to the length characteristics of the binary data, and then compression is completed through DACs algorithm to improve compression effect; the method comprises the steps of firstly selecting a first layer of bit number through the data length of binary data, acquiring an optimal first layer of bit number range through the occurrence frequency of data with different data lengths, so that the sum of the occurrence frequency of the data with the data length in the optimal first layer of bit number range is maximum, namely the optimal first layer of bit number range contains as much binary data as possible, performing range reduction through bit layer depth change caused by different sub-ranges in the range, finally acquiring the optimal first layer of bit number according to the coding number which is increased as required, and sequentially acquiring the optimal bit number of each layer; the number of bits of each layer can be ensured to be as small as possible, the compression effect of the flexible working platform data is ensured, and the intelligent storage efficiency of the flexible working platform data is improved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of steps of a flexible tooling platform data intelligent storage method of the invention;
FIG. 2 is a system frame diagram of a flexible worker platform data intelligent storage system of the present invention;
FIG. 3 is a diagram of a data length marker of a flexible tooling platform data intelligent storage method according to the present invention;
FIG. 4 is a graph of probability density of data for a flexible worker platform data intelligent storage method of the present invention.
Detailed Description
In order to further describe the technical means and effects adopted by the invention to achieve the preset aim, the following is a detailed description of specific implementation, structure, characteristics and effects of the flexible platform data intelligent storage method and system according to the invention in combination with the accompanying drawings and the preferred embodiment. In the following description, different "one embodiment" or "another embodiment" means that the embodiments are not necessarily the same. Furthermore, the particular features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
The invention provides a flexible working platform data intelligent storage method and a system specific scheme by combining a drawing.
Referring to fig. 1, a flowchart illustrating steps of a flexible platform data intelligent storage method according to an embodiment of the present invention is shown, where the method includes the following steps:
step S001, acquiring data, and converting the data into binary data.
The purpose of this embodiment is to store various data in a flexible tooling platform, so various data in the platform needs to be acquired first; the flexible working platform is characterized in that various data of the flexible working platform are numerical data, such as remuneration, working time, skill level, employer score, order number and the like, the numerical data are collected, and meanwhile, the obtained numerical data are decimal data; to facilitate compressed storage, all numerical data is converted to binary data.
So far, the data of the flexible working platform are collected and converted into binary data.
And S002, constructing a data length mark graph according to the data length and the data position, and constructing a data probability density graph according to the data length and the data occurrence frequency.
It should be noted that, for binary data of different lengths, their optimal hierarchical bit numbers are also different. The number of bits selected is smaller, for longer data, the bit layer is deeper, the identifier is increased, and the compression ratio cannot be determined to be necessarily smaller; the number of bits selected is large, for shorter data, zero padding is needed before binary data, and although the number of obtained bit layers is smaller, namely the bit layers are shallower, more zero padding is needed, so that meaningless data are more, and the compression efficiency is difficult to ensure; therefore, it is important to select a proper number of hierarchical bits; meanwhile, the optimal solution can be found out by traversing all possible bit numbers, but the calculated amount is too large, a great amount of time is wasted, and the burden of a computer is increased; therefore, the discrete degree is calculated, the value range of the bit number is reduced, and the calculation time is shortened.
It should be further noted that, due to the variation of the data length of each data, the number of bits selected is different, and the influence on the compression result is large. If the bit number of each layer is smaller, the longer data is affected more; if the number of bits is chosen larger, the shorter data is affected more. If the influence of the selection of the number of bits on the data compression is to be reduced, the selection of the number of bits is more desirable to select the distribution range of most of the data length, so that the sum of the complement length of the whole data and the number of data to be identified in the next layer is smaller.
Specifically, all data lengths are acquired, and a data length mark graph is constructed according to the data lengths and the data positions. Referring to fig. 3, a corresponding data length mark diagram is shown, wherein the data length refers to the data length of each binary data, the data position refers to the position of each binary data in the whole data, and the data length of the binary data is marked in a rectangular coordinate system to obtain the data length mark diagram.
It should be further noted that all data lengths are staggered, and selecting any number of bits will have different effects on other lengths of data. It is therefore necessary to choose an appropriate number of bits to minimize this effect, i.e. the operations of "zero padding" and "identification" are required. The optimal solution can be obtained by traversing all the bit numbers, but the calculation amount is too large. Therefore, the embodiment reduces the range of the bit number value by analyzing the discrete degree of the curve, and reduces the calculated amount; when determining the value range of the bit number, the area where most of the data length is located needs to be selected, so that the operations of zero padding and identification are as few as possible, and the compression rate can be further improved.
Specifically, the data length mark graph is converted into a data probability density graph according to the occurrence frequency of the data length, please refer to fig. 4, which shows the data probability density graph, wherein the data length refers to the data length of binary data, and the data occurrence frequency refers to the occurrence frequency of any data length.
Thus, a data length mark graph and a data probability density graph are obtained.
And step S003, obtaining a first layer bit number value range according to the data length, the data occurrence frequency and the interval data length average value.
It should be noted that, if the data length with the highest occurrence frequency of the direct data access is the first layer bit number, the layer depth may be uncontrollable. Therefore, the approximate value range of the first layer bit number is determined only according to the data occurrence frequency of the data length.
Specifically, by the firstFor example, the first layer bit number is set to be +.>The method comprises the steps of obtaining a first layer bit number value range according to the data length, the data occurrence frequency and the interval data length average value, wherein the specific method comprises the following steps of:
wherein ,indicate->The frequency of occurrence of data of the data length of the binary data; />Is shown in the intervalMiddle->Number of binary dataAccording to the length; />Expressed in interval +.>A data length average value of all binary data in the database; />Representation interval->The number of binary data, the present embodiment adopts +.>Calculating; />For super parameters, the present embodiment uses +.>Calculating; />Representing the minimum value of the first layer bit number value range; />Indicate->A data length; />Representing the maximum value of the first layer bit number value range; />Representing the difference between the maximum value and the minimum value of the first layer bit number value range, namely the first layer bit number value range;expressed in interval +.>The larger the data length fluctuation, the larger the value; the smaller the data length fluctuation, the smaller the value; at the same time due to->The calculated value is not within the possible range of the number of bits, so that a super parameter needs to be added to map the value within the possible range of the number of bits; the size of the value range is related to the fluctuation condition of the occurrence frequency and the data length of the data, and the larger the occurrence frequency is, the smaller the fluctuation is, the larger the value range is; the smaller the occurrence frequency, the larger the fluctuation, and the smaller the value range.
Further, the range of the first layer bit number corresponding to all binary data is calculated, and the rightmost end of the binary data is causedIn the case of deficiency, the calculation is performed directly on the remaining binary data.
Thus, the value range of the first layer bit number corresponding to all binary data is obtained.
Step S004, obtaining the preference degree of the first layer bit number value range according to the data occurrence frequency of the data length in the first layer bit number value range.
It should be noted that, the optimal first layer bit number value range needs to be found out from all the first layer bit number value ranges. In all the first layer bit number value ranges, the greater the frequency of occurrence of the data length corresponding to the data in the first layer bit number value range, the better the interval.
Specifically, taking any first layer bit number value range as an example, a specific calculation method of the preference degree of the first layer bit number value range is as follows:
wherein ,indicating the preference of the range of the first layer bit number,/, for>Indicating the first bit number of the first layer>Frequency of occurrence of data of individual data length, +.>Represents the maximum value of the first layer bit number value range,representing the minimum value of the first layer bit number value range; and acquiring the preference degree of each first layer bit number value range, and taking the first layer bit number value range with the largest preference degree as the optimal first layer bit number value range.
So far, the optimal value range of the first layer bit number is obtained.
Step S005, obtaining a plurality of sub-ranges of the optimal first layer bit number value range, obtaining a predicted layer depth according to the occurrence frequency of the data with the data length in the sub-ranges, and obtaining the reduced optimal first layer bit number value range according to the predicted layer depth and a preset layer depth threshold.
It should be noted that, the above steps calculate the optimal first layer bit number value range. If the value is too small in the value range, the layer depth is too deep, and the decryption cost is too high. And predicting the number of bits in the value range by limiting the maximum number of the bits, and further reducing the value range of the number of bits.
It should be further noted that, in the above step, the range of the optimal first layer bit number is determined according to the degree of dispersion of the data, so that the bit layer depth is limited to further reduce the range of the first layer bit number in order to prevent the range of the bit number from being too absolute. The bit layer depth affects the decoding time and compression rate, and after limiting the bit layer depth, a part of compression rate is lost, so that a large amount of compression time is increased.
Specifically, step S004 obtains an optimal first-layer bit number range, and traverses sub-ranges of selecting different optimal first-layer bit number ranges to obtain sub-ranges of a plurality of optimal first-layer bit number ranges, where the sub-ranges refer to ranges smaller than the optimal first-layer bit number range, that is, the sub-ranges are included in the optimal first-layer bit number range.
Further, according to the occurrence frequency of the data length in the sub-range of the optimal first layer bit number value range, the predicted layer depth is obtained, and according to the predicted layer depth and the preset layer depth threshold value, the reduced optimal first layer bit number value range is obtained, and the specific calculation method is as follows:
setting a preset depth thresholdIt should be noted that, in this embodiment, the preset layer depth threshold is described as 10, and other values may be set during implementation;
wherein ,representing a predicted layer depth; />Indicating the depth at which the layer is located, i.e. the value for the first layer is 1; />A data length maximum value representing all binary data; />Representation ofMaximum value of data length in any one of the sub-ranges, < >>Representing the minimum value of the data length in this sub-range; />Representing the data length in this sub-range +.>Frequency of occurrence of data of (2); />Representing the data length in this sub-range +.>A data length value of (2); />Representing a preset layer depth threshold;is indicated at->And (3) obtaining a data length average value according to the occurrence frequency of the data by all the data lengths in the interval, wherein the smaller the predicted layer depth is, the larger the data length average value is.
Further, calculating the predicted layer depth of all sub-ranges in the optimal first layer bit number value range to obtain a plurality of values meeting the judgment conditions) Selecting the sub-range with the smallest predicted layer depth as the reduced optimal first layer bit number value range.
And obtaining the reduced optimal first layer bit number value range according to the predicted layer depth and the preset layer depth threshold.
Step S006, obtaining the optimal first layer bit number according to the coding number to be increased under any bit number value in the reduced optimal first layer bit number value range, and further obtaining the optimal bit number of each layer.
It should be noted that, in the reduced optimal first layer bit number value range, the first layer bit number is calculated by going through the traversal, which reduces the extremely large calculation amount. Within the range of values, different values may have different effects on data of different lengths. A smaller value can make a bit layer with a larger length deeper, and the deeper the layer depth is, the more identifiers are needed; a larger value would require a much larger "zero-padding" operation for smaller length data. Therefore, it is necessary to take a proper value to minimize the sum of the number of bits of the identifier and the number of bits of the "zero padding".
Specifically, according to the number of codes to be increased in any bit number within the reduced optimal first layer bit number value range, the optimal first layer bit number is obtained, wherein the bit number is the data length of binary data, and the first layer bit number within the reduced optimal first layer bit number value range is obtainedThe specific calculation method for the coding number which needs to be increased under the value of the number of bits is as follows:
wherein ,indicate selection of +.>The number of codes to be increased when the number of bits is valued; />Indicate selection of +.>When the number of bits is taken, the +.>Two advancesThe number of bits of the data to be zero-padded, i.e. +.>A difference value obtained by subtracting the data length of the binary data from the value of the number of bits; u represents that the data length is smaller than +.>The number of binary data of the value of the number of bits;indicating that the data length is greater than +.>The number of binary data of the number of bits, i.e. the number of identifiers needs to be increased for this layer; the smaller the sum of the number of bits that need to be "zero-filled" and the number of identifier bits that need to be added at this layer, the better the compression ratio at that number of bits, the better the value of that number of bits.
Further, traversing all bit number values in the reduced optimal first layer bit number value range, and taking the bit number value with the minimum coding number to be added as the optimal first layer bit number; when calculating the optimal number of bits for the subsequent layer, it is necessary to remove the optimal first layer number of bits from all binary data, that is, the data length minus the optimal first layer number of bits, where the data length is smaller than or equal to the binary data of the optimal first layer number of bits, and since the binary data is already divided into the first layer, the binary data does not participate in the subsequent layering, and the remaining binary data obtains the optimal second layer number of bits according to the removed data length and the method, and then continues to remove the optimal number of bits of all layers sequentially.
So far, the optimal number of bits per layer is solved by calculating the minimum value of the sum of the number of bits that need to be "zero-filled" and the number of identifier bits that need to be added for this layer.
Step S007, performing data hierarchical compression according to the optimal number of bits of each layer.
Step S006 calculates the optimal number of bits for each layer, and layers the entire data according to the optimal number of bits for each layer. And carrying out layered compression on the data according to different bit numbers of different layers.
The first layer can accurately position the data, and the identifier can completely extract the distribution of the data in different bit layers to form the original finished data.
When a user wants to inquire certain data, the user can directly position the position of the compressed file according to the position of the data, only decompress the compressed content of the inquired data, and save decompression time.
Through the steps, the flexible intelligent data storage of the engineering platform is completed.
Another embodiment of the present invention provides a flexible industrial platform data intelligent storage system, as shown in fig. 2, which includes the following modules:
and a data acquisition module: the method comprises the steps of obtaining binary data, and obtaining the data length and the data occurrence frequency of the binary data;
bit number value range module: the method comprises the steps of obtaining a plurality of first layer bit number value ranges according to data length, data occurrence frequency and interval data length average value;
the optimal value range module: the method comprises the steps of obtaining the preference degree of a first-layer bit number value range according to the occurrence frequency of data of the data length in the first-layer bit number value range, and taking the first-layer bit number value range corresponding to the maximum value of the preference degree as an optimal first-layer bit number value range;
an optimal value range reduction module: the method comprises the steps of obtaining a plurality of sub-ranges of an optimal first-layer bit number value range, obtaining a predicted layer depth according to the occurrence frequency of data with data length in the sub-ranges, and obtaining a reduced optimal first-layer bit number value range according to the predicted layer depth and a preset layer depth threshold;
an optimal hierarchical bit number module: the method comprises the steps of obtaining the optimal first layer bit number according to the coding number to be increased under any bit number value in the reduced optimal first layer bit number value range, and obtaining the optimal bit number of each layer;
and the data layering compression module is used for: for performing data hierarchical compression according to the optimal number of bits per layer.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.
Claims (6)
1. The intelligent storage method for the flexible working platform data is characterized by comprising the following steps of:
acquiring binary data, and acquiring the data length and the data occurrence frequency of the binary data;
obtaining a plurality of first layer bit number value ranges according to the data length, the data occurrence frequency and the interval data length average value;
obtaining the preference degree of the first layer bit number value range according to the occurrence frequency of the data length in the first layer bit number value range, and taking the first layer bit number value range corresponding to the maximum value of the preference degree as the optimal first layer bit number value range;
obtaining a plurality of sub-ranges of the optimal first layer bit number value range, obtaining a predicted layer depth according to the occurrence frequency of data with the data length in the sub-ranges, and obtaining the reduced optimal first layer bit number value range according to the predicted layer depth and a preset layer depth threshold;
obtaining the optimal first layer bit number according to the coding number to be increased under any bit number value in the reduced optimal first layer bit number value range, and obtaining the optimal bit number of each layer;
carrying out data layering compression according to the optimal bit number of each layer;
the method for obtaining the value range of the first layer bit number according to the data length, the data occurrence frequency and the interval data length average value comprises the following specific steps:
wherein ,indicate->The frequency of occurrence of data of the data length of the binary data; />Expressed in interval +.>Middle->A data length of the binary data; />Expressed in interval +.>A data length average value of all binary data in the database; />Representation interval->The number of binary data; />Is a super parameter; />Representing the minimum value of the first layer bit number value range; />Indicate->A data length; />Representing the maximum value of the first layer bit number value range;representing the difference between the maximum and minimum of the first layer bit number range, will +.>As->A first layer bit number value range of the binary data; acquiring a first layer bit number value range of each binary data;
the predicted layer depth is obtained according to the occurrence frequency of the data with the data length in the sub-range, and the method comprises the following specific steps:
wherein ,representing a predicted layer depth; />Representing the depth at which the layer is located; />A data length maximum value representing all binary data; />Representing the maximum data length in any one sub-rangeValue of->Representing the minimum value of the data length in this sub-range; />Representing the data length in this sub-range +.>Frequency of occurrence of data of (2); />Representing the data length in this sub-range +.>Is a data length value of (a).
2. The intelligent flexible platform data storage method according to claim 1, wherein the obtaining the preference degree of the first layer bit number value range according to the occurrence frequency of the data with the data length in the first layer bit number value range comprises the following specific steps:
acquiring any first layer bit number value range, acquiring the occurrence frequencies of all data lengths in the first layer bit number value range, and taking the sum of the occurrence frequencies of all data lengths in the range as the preference degree of the first layer bit number value range; and obtaining the preference degree of each first layer bit number value range.
3. The intelligent flexible platform data storage method according to claim 1, wherein the obtaining the reduced optimal first layer bit number value range according to the predicted layer depth and the preset layer depth threshold value comprises the following specific steps:
obtaining the predicted layer depth of all sub-ranges in the optimal first layer bit number value range to obtain a plurality of satisfied valuesJudging the sub-range of the condition, selecting the sub-range with the minimum predicted layer depth as the reduced optimal first layer bit number value range,representing predicted layer depth,/->Representing a preset layer depth threshold.
4. The intelligent flexible platform data storage method according to claim 1, wherein the obtaining the optimal first layer bit number according to the number of codes to be increased in any bit number value in the reduced optimal first layer bit number value range comprises the following specific steps:
wherein ,indicate selection of +.>The number of codes to be increased when the number of bits is valued; />Indicate selection of +.>When the number of bits is taken, the +.>The number of bits of the binary data to be zero padding; u represents that the data length is smaller than +.>The number of binary data of the value of the number of bits; />Indicating that the data length is greater than +.>The number of binary data of the value of the number of bits;
traversing all the bit number values in the reduced optimal first layer bit number value range, and taking the bit number value with the minimum coding number required to be increased as the optimal first layer bit number.
5. The intelligent storage method of flexible tooling platform data according to claim 1, wherein the specific acquisition method of the optimal bit number of each layer is as follows:
and removing the optimal first layer bit number from all binary data, acquiring the optimal second layer bit number according to the method for acquiring the optimal first layer bit number in the rest binary data according to the length of the removed data, and continuously removing to sequentially acquire the optimal bit number of all layers.
6. The flexible work platform data intelligent storage system is characterized by comprising the following modules:
and a data acquisition module: the method comprises the steps of obtaining binary data, and obtaining the data length and the data occurrence frequency of the binary data;
bit number value range module: the method comprises the steps of obtaining a plurality of first layer bit number value ranges according to data length, data occurrence frequency and interval data length average value;
the optimal value range module: obtaining the preference degree of the first layer bit number value range according to the occurrence frequency of the data length in the first layer bit number value range, and taking the first layer bit number value range corresponding to the maximum value of the preference degree as the optimal first layer bit number value range;
an optimal value range reduction module: the method comprises the steps of obtaining a plurality of sub-ranges of an optimal first-layer bit number value range, obtaining a predicted layer depth according to the occurrence frequency of data with data length in the sub-ranges, and obtaining a reduced optimal first-layer bit number value range according to the predicted layer depth and a preset layer depth threshold;
an optimal hierarchical bit number module: the method comprises the steps of obtaining the optimal first layer bit number according to the coding number to be increased under any bit number value in the reduced optimal first layer bit number value range, and obtaining the optimal bit number of each layer;
and the data layering compression module is used for: the data compression method comprises the steps of performing data layering compression according to the optimal bit number of each layer;
the method for obtaining the value range of the first layer bit number according to the data length, the data occurrence frequency and the interval data length average value comprises the following specific steps:
wherein ,indicate->The frequency of occurrence of data of the data length of the binary data; />Expressed in interval +.>Middle->A data length of the binary data; />Expressed in interval +.>A data length average value of all binary data in the database; />Representation interval->The number of binary data; />Is a super parameter; />Representing the minimum value of the first layer bit number value range; />Indicate->A data length; />Representing the maximum value of the first layer bit number value range;representing the difference between the maximum and minimum of the first layer bit number range, will +.>As->A first layer bit number value range of the binary data; acquiring a first layer bit number value range of each binary data;
the predicted layer depth is obtained according to the occurrence frequency of the data with the data length in the sub-range, and the method comprises the following specific steps:
wherein ,representing a predicted layer depth; />Representing the depth at which the layer is located; />A data length maximum value representing all binary data; />Represents the maximum value of the data length in any one of the sub-ranges, < >>Representing the minimum value of the data length in this sub-range; />Representing the data length in this sub-range +.>Frequency of occurrence of data of (2); />Representing the data length in this sub-range +.>Is a data length value of (a).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310825990.1A CN116561084B (en) | 2023-07-07 | 2023-07-07 | Flexible engineering platform data intelligent storage method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310825990.1A CN116561084B (en) | 2023-07-07 | 2023-07-07 | Flexible engineering platform data intelligent storage method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116561084A CN116561084A (en) | 2023-08-08 |
| CN116561084B true CN116561084B (en) | 2023-09-19 |
Family
ID=87496835
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310825990.1A Active CN116561084B (en) | 2023-07-07 | 2023-07-07 | Flexible engineering platform data intelligent storage method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116561084B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115543946A (en) * | 2022-12-02 | 2022-12-30 | 陕西湘秦衡兴科技集团股份有限公司 | Financial big data optimized storage method |
| CN116303374A (en) * | 2023-05-22 | 2023-06-23 | 深圳市维度数据科技股份有限公司 | Multi-dimensional report data optimization compression method based on SQL database |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7512180B2 (en) * | 2003-06-25 | 2009-03-31 | Microsoft Corporation | Hierarchical data compression system and method for coding video data |
| US9350384B2 (en) * | 2014-09-30 | 2016-05-24 | International Business Machines Corporation | Hierarchical data compression and computation |
-
2023
- 2023-07-07 CN CN202310825990.1A patent/CN116561084B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115543946A (en) * | 2022-12-02 | 2022-12-30 | 陕西湘秦衡兴科技集团股份有限公司 | Financial big data optimized storage method |
| CN116303374A (en) * | 2023-05-22 | 2023-06-23 | 深圳市维度数据科技股份有限公司 | Multi-dimensional report data optimization compression method based on SQL database |
Non-Patent Citations (1)
| Title |
|---|
| 倒排索引压缩算法研究综述;姜琨 等;《小型微型计算机系统》;第41卷(第4期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116561084A (en) | 2023-08-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116303374B (en) | Multi-dimensional report data optimization compression method based on SQL database | |
| KR100969764B1 (en) | Method for coding and decoding 3d data implemented as mesh model | |
| CN106846425A (en) | A kind of dispersion point cloud compression method based on Octree | |
| CN108229681A (en) | A kind of neural network model compression method, system, device and readable storage medium storing program for executing | |
| CN109859281B (en) | Compression coding method of sparse neural network | |
| CN107395211B (en) | Data processing method and device based on convolutional neural network model | |
| CN115514375B (en) | Cache data compression method | |
| US11928599B2 (en) | Method and device for model compression of neural network | |
| CN111199740B (en) | Unloading method for accelerating automatic voice recognition task based on edge calculation | |
| CN111240746B (en) | Floating point data inverse quantization and quantization method and equipment | |
| CN116561084B (en) | Flexible engineering platform data intelligent storage method and system | |
| CN116861271B (en) | Data analysis processing method based on big data | |
| CN115882867B (en) | Data compression storage method based on big data | |
| CN113630125A (en) | Data compression method, data encoding method, data decompression method, data encoding device, data decompression device, electronic equipment and storage medium | |
| CN103746701A (en) | Rapid encoding option selecting method applied to Rice lossless data compression | |
| CN105099460A (en) | Dictionary compression method, dictionary decompression method, and dictionary construction method | |
| CN110349228B (en) | Triangular mesh compression method for data-driven least square prediction | |
| CN117155408B (en) | Efficient transmission method for production data | |
| US20150078674A1 (en) | Component sorting based encoding for 3d mesh compression | |
| CN114125070A (en) | Communication method, system, electronic device and storage medium for quantization compression | |
| CN114065923A (en) | Compression method, system and accelerating device of convolutional neural network | |
| CN113810058A (en) | Data compression method, data decompression method, device and electronic equipment | |
| CN116865768B (en) | PLC equipment data optimization storage method | |
| CN117560016B (en) | College recruitment information management method based on big data | |
| CN115866264B (en) | Equipment operation data compression storage method for intelligent factory MES system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |