CN110716688A - Big data information compression method - Google Patents
Big data information compression method Download PDFInfo
- Publication number
- CN110716688A CN110716688A CN201810755357.9A CN201810755357A CN110716688A CN 110716688 A CN110716688 A CN 110716688A CN 201810755357 A CN201810755357 A CN 201810755357A CN 110716688 A CN110716688 A CN 110716688A
- Authority
- CN
- China
- Prior art keywords
- compressed
- data information
- module
- big data
- compression
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
Abstract
The invention discloses a big data information compression method, which comprises the following steps: the large data information module to be compressed is matched and marked with a preset threshold value, the large data information module to be compressed is matched with the preset threshold value, the capacity of the preset threshold value is smaller than that of the large data information module to be compressed, the large data information module to be compressed is input and stored in a first storage, the large data information module to be compressed is matched with the preset threshold value, and the large data information module to be compressed, the capacity of which is lower than or equal to the preset threshold value, is stored in a second storage. Setting a selection module among the first memory, the second memory and the N storages, and quickly calling the required big data information to be compressed from the first memory or the second memory and the N storages according to the capacity of the big data information module to be compressed by a user based on the estimated compression time, and starting a compression program to perform compression processing; step four, the operation of compressing processing is carried out by the compression program, which specifically comprises the following steps: dividing the selected big data information module to be compressed into a first data sub-stream and N data sub-streams, sequentially recording the first data sub-stream to first data compression equipment, and sequentially recording the N data sub-streams to Nth data compression equipment. The technical scheme of the invention can overcome the defects of low reliability, poor safety, complex information operation difficulty and the like in the prior art, and has the advantages of high reliability, good safety and small information operation difficulty.
Description
Technical Field
The invention relates to the technical field of electronic products, in particular to a big data information compression method.
Background
The electronic technology is a new technology developed at the end of the nineteenth century and at the beginning of the twentieth century, is developed most rapidly in the twentieth century and is applied most widely, and becomes an important mark for the development of modern science and technology.
The first generation of electronic products have electronic tubes as the core. The first semiconductor triode comes out in the world in the end of the forty years, has the characteristics of small size, light weight, electricity saving, long service life and the like, is rapidly applied by various countries, and replaces an electron tube in a large range. In the last fifties, the world appeared the first integrated circuit, which integrated many electronic components such as transistors on a silicon chip, and made the electronic products more compact. The integrated circuit is rapidly developed from a small-scale integrated circuit to a large-scale integrated circuit and a very large-scale integrated circuit, so that the electronic product is developed towards the directions of high efficiency, low consumption, high precision, high stability and intellectualization.
The electronic product brings great convenience to life and work of people, but the lack of safety of the electronic product becomes the most important factor for the leakage of life and work information of people.
In the process of implementing the invention, the inventor finds that the prior art at least has the defects of low reliability, poor safety, high difficulty in information encryption, complex operation flow, simple function, incapability of meeting the industry development requirement and the like.
Disclosure of Invention
The invention aims to provide a big data information compression method aiming at the problems so as to realize the advantages of high reliability, good safety and simple operation.
In order to achieve the purpose, the invention adopts the technical scheme that: a big data information compression method comprises the following steps:
the method comprises a big data information compression module and a big data information module to be compressed, wherein the big data information compression module is matched and marked with a preset threshold value, and the method comprises the following specific steps:
step one, the big data information module to be compressed is matched with a preset threshold value, the capacity of the preset threshold value is smaller than that of the big data information module to be compressed, and the big data information module to be compressed is input and stored in a first memory.
And step two, the big data information module to be compressed is matched with a preset threshold value, and the big data information module to be compressed with the capacity lower than or equal to the preset threshold value is stored in a second memory.
Setting a selection module among the first memory, the second memory and the N storages, and quickly calling the required big data information to be compressed from the first memory or the second memory and the N storages according to the capacity of the big data information module to be compressed by a user based on the estimated compression time, and starting a compression program to perform compression processing;
step four, the operation of compressing processing is carried out by the compression program, which specifically comprises the following steps:
dividing the selected big data information module to be compressed into a first data sub-stream and N data sub-streams,
sequentially recording the first data sub-stream onto a first data compression device,
sequentially recording the N data sub-streams onto an Nth data compression device.
The big data information module to be compressed is internally provided with an automatic format conversion module, a capacity matching module and a data security detection module.
The module operation flow is that before the information enters the module of the compressed big data information to be compressed, the data security detection module can carry out security detection analysis on the information, the information enters the format conversion mode after the security detection, and the information enters the capacity matching module after the format conversion.
The big data information compression method of each embodiment of the invention comprises the following steps: the method comprises a big data information compression module and a big data information module to be compressed, wherein the big data information compression module is matched and marked with a preset threshold value, and the method comprises the following specific steps:
step one, the big data information module to be compressed is matched with a preset threshold value, the capacity of the preset threshold value is smaller than that of the big data information module to be compressed, and the big data information module to be compressed is input and stored in a first memory.
And step two, the big data information module to be compressed is matched with a preset threshold value, and the big data information module to be compressed with the capacity lower than or equal to the preset threshold value is stored in a second memory.
Setting a selection module among the first memory, the second memory and the N storages, and quickly calling the required big data information to be compressed from the first memory or the second memory and the N storages according to the capacity of the big data information module to be compressed by a user based on the estimated compression time, and starting a compression program to perform compression processing;
step four, the operation of compressing processing is carried out by the compression program, which specifically comprises the following steps:
dividing the selected big data information module to be compressed into a first data sub-stream and N data sub-streams,
sequentially recording the first data sub-stream onto a first data compression device,
sequentially recording the N data sub-streams onto an Nth data compression device.
The big data information module to be compressed is internally provided with an automatic format conversion module, a capacity matching module and a data security detection module.
The module operation flow is that before the information enters the module of the compressed big data information to be compressed, the data security detection module can carry out security detection analysis on the information, the information enters the format conversion mode after the security detection, and the information enters the capacity matching module after the format conversion.
Therefore, the defects of complex operation, poor safety and high information encryption difficulty in the prior art can be overcome, and the advantages of high reliability, good safety and low information operation difficulty are realized.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
The technical solution of the present invention is further described in detail by the following examples.
The following description of the preferred embodiments of the present invention is provided for the purpose of illustration and description, and is in no way intended to limit the invention.
According to an embodiment of the present invention, a big data information compression method is provided, including: the method comprises a big data information compression module and a big data information module to be compressed, wherein the big data information compression module is matched and marked with a preset threshold value, and the method comprises the following specific steps:
step one, the big data information module to be compressed is matched with a preset threshold value, the capacity of the preset threshold value is smaller than that of the big data information module to be compressed, and the big data information module to be compressed is input and stored in a first memory.
And step two, the big data information module to be compressed is matched with a preset threshold value, and the big data information module to be compressed with the capacity lower than or equal to the preset threshold value is stored in a second memory.
Setting a selection module among the first memory, the second memory and the N storages, and quickly calling the required big data information to be compressed from the first memory or the second memory and the N storages according to the capacity of the big data information module to be compressed by a user based on the estimated compression time, and starting a compression program to perform compression processing;
step four, the operation of compressing processing is carried out by the compression program, which specifically comprises the following steps:
dividing the selected big data information module to be compressed into a first data sub-stream and N data sub-streams,
sequentially recording the first data sub-stream onto a first data compression device,
sequentially recording the N data sub-streams onto an Nth data compression device.
The big data information module to be compressed is internally provided with an automatic format conversion module, a capacity matching module and a data security detection module.
The module operation flow is that before the information enters the module of the compressed big data information to be compressed, the data security detection module can carry out security detection analysis on the information, the information enters the format conversion mode after the security detection, and the information enters the capacity matching module after the format conversion.
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (3)
1. A big data information compression method comprises a big data information module to be compressed, and the big data information module to be compressed is matched and marked with a preset threshold value, and the method specifically comprises the following steps; step one, the big data information module to be compressed is matched with a preset threshold, the capacity of the preset threshold is smaller than that of the big data information module to be compressed, and the big data information module to be compressed is input and stored in a first memory: step two, the big data information module to be compressed is matched with a preset threshold value, and the big data information module to be compressed with the capacity lower than or equal to the preset threshold value is stored in a second memory: setting a selection module among the first memory, the second memory and the N storages, and quickly calling the required big data information to be compressed from the first memory or the second memory and the N storages according to the capacity of the big data information module to be compressed by a user based on the estimated compression time, and starting a compression program to perform compression processing: step four, the operation of compressing processing is carried out by the compression program, which specifically comprises the following steps: dividing a module for selecting big data information to be compressed into a first data sub-stream and N data sub-streams, sequentially recording the first data sub-stream to first data compression equipment, and sequentially recording the N data sub-streams to Nth data compression equipment.
2. The method of claim 1, further comprising: the big data information module to be compressed is internally provided with an automatic format conversion module, a capacity matching module and a data security detection module.
3. The method of claim 2, further comprising: the module operation flow is that before the information enters the module of the compressed big data information to be compressed, the data security detection module can carry out security detection analysis on the information, the information enters the format conversion mode after the security detection, and the information enters the capacity matching module after the format conversion.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810755357.9A CN110716688A (en) | 2018-07-11 | 2018-07-11 | Big data information compression method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810755357.9A CN110716688A (en) | 2018-07-11 | 2018-07-11 | Big data information compression method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110716688A true CN110716688A (en) | 2020-01-21 |
Family
ID=69208137
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810755357.9A Pending CN110716688A (en) | 2018-07-11 | 2018-07-11 | Big data information compression method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110716688A (en) |
-
2018
- 2018-07-11 CN CN201810755357.9A patent/CN110716688A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102073563A (en) | Flash memory equipment, and method and system for managing storage blocks in same | |
| US7970215B2 (en) | Automatic generation of compact code tables | |
| CN100525450C (en) | Method and device for realizing Hoffman decodeng | |
| CN102833492B (en) | A kind of video scene dividing method based on color similarity | |
| CN104837031A (en) | Method for high-speed self-adaptive video keyframe extraction | |
| CN105824574A (en) | Memory data storage method | |
| CN102179133A (en) | Method for recovering ammonia from ammonia-containing gas based on ionic liquid | |
| CN103227660A (en) | Real-time software frame synchronizing method | |
| CN110716688A (en) | Big data information compression method | |
| CN103793437A (en) | Wafer test data processing method and system | |
| CN106199323A (en) | A kind of virtual terminals of intelligent substation connects correctness method of discrimination | |
| CN101998122B (en) | Method and device for decoding normal form Hoffman hardware in JPEG (Joint Photographic Expert Group) image | |
| CN105161135A (en) | Electronic product test information storage method | |
| CN104021100A (en) | Method for processing data received by serial port of single-chip microcomputer | |
| CN105183545A (en) | Big data information compression method | |
| CN104700351B (en) | A kind of high-precision image processing system based on Image Coding processing | |
| CN102567209B (en) | Flash memory chip data analyzing method and flash memory chip data analyzing device | |
| EP1096369A3 (en) | A counter readout control apparatus and control method therefor | |
| US9793896B1 (en) | Semiconductor device | |
| CN105049178A (en) | Multi-user information screening method | |
| CN104994391A (en) | High-efficiency VP9 entropy decoding prob data obtaining method and equipment | |
| CN109933695B (en) | Chip processing method and device for hierarchical UDF | |
| CN105187734A (en) | Loading method for dynamic image and static image | |
| CN105072481A (en) | Video data loading and playing method | |
| US8009770B2 (en) | Decoder for 4QAM-NR and method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200121 |