CN109213737A - A kind of data compression method and apparatus - Google Patents
A kind of data compression method and apparatus Download PDFInfo
- Publication number
- CN109213737A CN109213737A CN201811082900.XA CN201811082900A CN109213737A CN 109213737 A CN109213737 A CN 109213737A CN 201811082900 A CN201811082900 A CN 201811082900A CN 109213737 A CN109213737 A CN 109213737A
- Authority
- CN
- China
- Prior art keywords
- compressed
- data
- compression
- fpga
- storage region
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The application provides a kind of data compression method and apparatus, comprising: receives data compression request, obtains the storage address of data to be compressed；The data to be compressed stored in the storage address are moved to default storage region, compression processing is carried out to the compressed data in order to which FPGA reads the data defect to be compressed from the default storage region, obtains compressed data；Receive the compressed data that the FPGA is sent.It can be seen that, in the embodiment of the present application, it is to be moved to default storage region using by data to be compressed, data to be compressed are read from default storage region by FPGA, and compressed data is treated by FPGA and is compressed, since the rate that the FPGA speed ratio CPU compressed to data compresses data is fast, using scheme provided by the embodiments of the present application, it can be promoted and treat the efficiency that compressed data is compressed.
Description
Technical field
This application involves data processing fields, more particularly to a kind of data compression method and apparatus.
Background technique
In cloud computing era, the storage of mass data and data are analyzed and bring great challenge to data center.In order to meet
Cloud computing era comes into being to mass data and storage and data analysis requirements, distributed system architecture Hadoop.Hadoop
Mainly calculated with Hadoop distributed file system (Hadoop Distributed File System, HDFS) and MapReduce
Method is core.Wherein, HDFS can be used for storing mass data, and MapReduce algorithm can be used for carrying out data analysis.
When HDFS storing data, in order to save memory space, data can be compressed by compression algorithm.Currently,
Mainly data are compressed using central processing unit (Central Processing Unit, CPU).But utilize CPU pairs
The rate that data are compressed is slow, and compression efficiency is low.
Accordingly, it is desirable to provide a kind of scheme, can solve the above problem.
Summary of the invention
Technical problems to be solved in this application are the rates that CPU compresses data, provide a kind of data compression side
Method and device.
In a first aspect, the embodiment of the present application provides a kind of data compression method, comprising:
Data compression request is received, the storage address of data to be compressed is obtained；
The data to be compressed stored in the storage address are moved to default storage region, in order to field programmable gate
Array FPGA reads the data to be compressed from the default storage region, and carries out compression processing to the data to be compressed,
Obtain compressed data；
Receive the compressed data that the FPGA is sent.
Optionally, the default storage region is preparatory to consult with the FPGA.
Optionally, the method also includes:
Obtain the compressed format of the data to be compressed；
Correspondingly, described be moved to default storage region for the data to be compressed stored in the storage address, in order to
On-site programmable gate array FPGA reads the data to be compressed from the default storage region, and to the data to be compressed into
Row compression processing, obtains compressed data, comprising:
The data to be compressed and the compressed format are moved to default storage region, in order to which FPGA is preset from described
Storage region reads the data to be compressed and the compressed format, and according to the compressed format to the data to be compressed into
Row compression processing, obtains compressed data.
Optionally, the compressed format includes:
Gzip or Zlib.
Optionally, the method also includes:
Compression head information and/or compression tail information are added for the compressed data；
Wherein, the compression head information and the compression tail information are determined according to the compressed format.
Second aspect, the embodiment of the present application provide a kind of data compression device, comprising:
First receiving unit, compression request, obtains the storage address of data to be compressed for receiving data；
Mobile unit, for the data to be compressed stored in the storage address to be moved to default storage region, so as to
Programmable gate array FPGA reads the data to be compressed from the default storage region on site, and to the data to be compressed
Compression processing is carried out, compressed data is obtained；
Second receiving unit, the compressed data sent for receiving the FPGA.
Optionally, the default storage region is preparatory to consult with the FPGA.
Optionally, described device further include:
Acquiring unit, for obtaining the compressed format of the data to be compressed；
Correspondingly, the mobile unit, is specifically used for:
The data to be compressed and the compressed format are moved to default storage region, in order to which FPGA is preset from described
Storage region reads the data to be compressed and the compressed format, and according to the compressed format to the data to be compressed into
Row compression processing, obtains compressed data.
Optionally, the compressed format includes:
Gzip or Zlib.
Optionally, described device further include:
Adding unit, for compressing head information and/or compression tail information for compressed data addition；
Wherein, the compression head information and the compression tail information are determined according to the compressed format.
Compared with prior art, the embodiment of the present application has the advantage that
Data compression method and apparatus provided by the embodiments of the present application, comprising: receive data compression request, obtain to be compressed
The storage address of data；The data to be compressed stored in the storage address are moved to default storage region, in order to FPGA
The data defect to be compressed is read from the default storage region, compression processing is carried out to the compressed data, obtain compression number
According to；Receive the compressed data that the FPGA is sent.It can be seen that being to use to move data to be compressed in the embodiment of the present application
It moves to default storage region, reads data to be compressed from default storage region by FPGA, and compressed data is treated by FPGA and is carried out
Compression, since the rate that the FPGA speed ratio CPU compressed to data compresses data is fast, utilize the application
The scheme that embodiment provides, can be promoted and treat the efficiency that compressed data is compressed.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The some embodiments recorded in application, for those of ordinary skill in the art, without creative efforts,
It is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of flow diagram of data compression method provided by the embodiments of the present application；
Fig. 2 is a kind of structural schematic diagram of data compression device provided by the embodiments of the present application.
Specific embodiment
In order to make those skilled in the art more fully understand application scheme, below in conjunction in the embodiment of the present application
Attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described embodiment is only this
Apply for a part of the embodiment, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art exist
Every other embodiment obtained under the premise of creative work is not made, shall fall in the protection scope of this application.
Present inventor has found after study, in the prior art, empty in order to save storage when HDFS storing data
Between, data can be compressed by compression algorithm.Currently, mainly utilizing central processing unit (Central
Processing Unit, CPU) data are compressed.But, compression slow using the rate that CPU compresses data
Low efficiency.
To solve the above-mentioned problems, in the embodiment of the present application, data compression method and dress provided by the embodiments of the present application
It sets, comprising: receive data compression request, obtain the storage address of data to be compressed；It will be stored in the storage address wait press
Contracting data are moved to default storage region, in order to which FPGA reads the data defect pair to be compressed from the default storage region
The compressed data carries out compression processing, obtains compressed data；Receive the compressed data that the FPGA is sent.It can be seen that
It in the embodiment of the present application, is read by FPGA from default storage region using data to be compressed are moved to default storage region
Data to be compressed, and compressed data is treated by FPGA and is compressed, the speed ratio CPU logarithm that data are compressed due to FPGA
It is fast according to the rate compressed, therefore, using scheme provided by the embodiments of the present application, it can be promoted and treat compressed data and pressed
The efficiency of contracting.
With reference to the accompanying drawing, the various non-limiting embodiments of the application are described in detail.
Illustrative methods
Referring to Fig. 1, which is a kind of flow diagram of data compression method provided by the embodiments of the present application.
In the present embodiment, the method can for example pass through following steps: S101-S103 is realized.
S101: data compression request is received, the storage address of data to be compressed is obtained.
It should be noted that referred in the embodiment of the present application data compression request, for request treat compressed data into
Row processing.
It should be noted that the reception data compression request referred in the embodiment of the present application, such as can be reception upper layer
The data compression request that application program is sent.
It should be noted that being built-in with compression library, which may be implemented to treat using CPU in Hadoop framework
Compressed data carries out compression function.In the embodiment of the present application, it is compressed due to not recycling CPU to treat compressed data, because
This, in the embodiment of the present application, the content for providing compression interface in the compression library can no longer retain.For example, hadoop-
In zlibcompressor.c file under common engineering, the function such as LOAD_ of data compression interface is provided for CPU
DYNAMIC_SYMBOL function can no longer retain.
It should be noted that in the embodiment of the present application, the storage address of the data to be compressed, such as can refer to institute
State storage address of the data to be compressed in HDFS distributed memory system.
Do not limit the specific implementation for obtaining the storage address of data to be compressed specifically in the embodiment of the present application, as
The storage address of a kind of example, the data to be compressed can be stored in stream structural body, therefore, can be by this
Stream structural body obtains the storage address of data to be compressed.As another example, can be taken in the data compression request
Storage address with the data to be compressed, so as to be parsed to data compression request, to obtain to be compressed
The storage address of data.
S102: the data to be compressed stored in the storage address are moved to default storage region, on site may be used
It programs gate array FPGA and reads the data to be compressed from the default storage region, and the data to be compressed are compressed
Processing, obtains compressed data.
S103: the compressed data that the FPGA is sent is received.
In the embodiment of the present application, the default storage region can be consulted with the FPGA in advance.Namely
It says, accessible default storage region of FPGA, to read data to be compressed from the default storage region.
It should be noted that in the embodiment of the present application, between FPGA the and HDFS distributed file system, have with
The compatible communication interface of the local library of HDFS distributed file system, that is to say, that the FPGA can use the communication and connect
Mouth reads data to be compressed from the default storage region.
It should be noted that the embodiment of the present application does not limit what the FPGA compressed the data to be compressed specifically
Specific implementation.
After carrying out compression processing to the data to be compressed to FPGA, the compression number of the FPGA transmission ready to receive
According to, and the compressed data is stored in HDFS distributed file system.
Data compression method provided by the embodiments of the present application, comprising: receive data compression request, obtain data to be compressed
Storage address；The data to be compressed stored in the storage address are moved to default storage region, in order to which FPGA is from described
Default storage region reads the data defect to be compressed and carries out compression processing to the compressed data, obtains compressed data；It connects
Receive the compressed data that the FPGA is sent.It can be seen that being pre- using data to be compressed are moved in the embodiment of the present application
If storage region, data to be compressed are read from default storage region by FPGA, and compressed data is treated by FPGA and is compressed, by
It is fast in the rate that the speed ratio CPU that FPGA compresses data compresses data, therefore, mentioned using the embodiment of the present application
The scheme of confession can be promoted and treat the efficiency that compressed data is compressed.
In view of the compressed format for different data to be compressed may be different, corresponding data compression logic etc. also has can
It can be different.In an example of the embodiment of the present application, the compressed format of the data to be compressed can also be obtained.
It should be noted that the embodiment of the present application does not limit the realization for obtaining the compressed format of the data to be compressed specifically
Mode can pass through the configuration file of the MapReduce algorithm in the Hadoop framework in one possible implementation
The middle compressed format for obtaining the data to be compressed.Specifically, the compressed format also can store mentioned above
In stream structural body, so as to obtain the compressed format by the stream structural body.
If obtaining the compressed format, step S102 mentioned above can be by described wait press in specific implementation
Contracting data and the compressed format are moved to default storage region, in order to FPGA from the default storage region read it is described to
Compressed data and the compressed format, and compression processing is carried out to the data to be compressed according to the compressed format, it is pressed
Contracting data.
That is, data to be compressed and compressed format are moved to the accessible default storage region of FPGA, this
Sample one, FPGA can read the data to be compressed and the compressed format from the default storage region, and according to institute
It states compressed format and compression processing is carried out to the data to be compressed, obtain compressed data.
It should be noted that the embodiment of the present application does not limit the compressed format specifically, in a kind of possible implementation
In, the compressed format for example may include Gzip or Zlib.
It can also be compressed data addition compression head information and/or pressure in an example of the embodiment of the present application
Contracting tail information and/or check code.
It should be noted that the compression head information referred in the embodiment of the present application, refers to the frame before compressed data
Head information, the compression tail information refer to the trailer information after compressed data.
The embodiment of the present application does not limit the compression head information and compression tail information specifically, in one kind of the embodiment of the present application
In implementation, the compression head information and compression tail information be can be according to compressed format determination.In a kind of realization
In mode, it is described compression tail information for example may include the compressed data length and verification and (such as CRC check and
Adler verification and).
It should be noted that can be the compression number when receiving data compression request in the embodiment of the present application
Head information is compressed according to addition.It can be after receiving the compressed data that the FPGA is sent, for compressed data addition pressure
Contracting tail information.
As above, the storage address and compressed format of the data to be compressed be can store in stream structural body.It needs
It is noted that other information relevant to the data to be compressed can also be preserved in the stream structural body, such as
The information such as verification and information.It should be noted that in the embodiment of the present application, carrying out compression end treating compressed data
Afterwards, the data stored in the stream structural body can be emptied, the stream structural body can also be discharged.
Example devices
Referring to fig. 2, which is a kind of structural schematic diagram of data compression device provided by the embodiments of the present application.
Data compression device 200 provided by the embodiments of the present application, such as can specifically include: the first receiving unit 210 moves
Moving cell 220 and the second receiving unit 230.
First receiving unit 210, compression request, obtains the storage address of data to be compressed for receiving data；
Mobile unit 220, for the data to be compressed stored in the storage address to be moved to default storage region, with
The data to be compressed are read from the default storage region convenient for on-site programmable gate array FPGA, and to the number to be compressed
According to compression processing is carried out, compressed data is obtained；
Second receiving unit 230, the compressed data sent for receiving the FPGA.
Optionally, the default storage region is preparatory to consult with the FPGA.
Optionally, described device 200 further include:
Acquiring unit, for obtaining the compressed format of the data to be compressed；
Correspondingly, the mobile unit 220, is specifically used for:
The data to be compressed and the compressed format are moved to default storage region, in order to which FPGA is preset from described
Storage region reads the data to be compressed and the compressed format, and according to the compressed format to the data to be compressed into
Row compression processing, obtains compressed data.
Optionally, the compressed format includes:
Gzip or Zlib.
Optionally, described device 200 further include:
Adding unit, for compressing head information and/or compression tail information for compressed data addition；
Wherein, the compression head information and the compression tail information are determined according to the compressed format.
Since described device 200 is device corresponding with the data compression method of above method embodiment offer, the dress
The realization of 200 each unit is set, is same design with the method for above method embodiment, accordingly, with respect to described device 200
Each unit specific implementation, can refer to above method embodiment description, details are not described herein again.
Data compression device provided by the embodiments of the present application, comprising: the first receiving unit, compression is asked for receiving data
It asks, obtains the storage address of data to be compressed；Mobile unit, the data to be compressed for will store in the storage address are mobile
To default storage region, in order to which on-site programmable gate array FPGA reads the number to be compressed from the default storage region
According to, and compression processing is carried out to the data to be compressed, obtain compressed data；Second receiving unit, for receiving the FPGA
The compressed data of transmission.It can be seen that being to be moved to default memory block using by data to be compressed in the embodiment of the present application
Domain reads data to be compressed from default storage region by FPGA, and treats compressed data by FPGA and compressed, due to FPGA pairs
The rate that the speed ratio CPU that data are compressed compresses data is fast, therefore, utilizes side provided by the embodiments of the present application
Case can be promoted and treat the efficiency that compressed data is compressed.
Those skilled in the art after considering the specification and implementing the invention disclosed here, will readily occur to its of the application
Its embodiment.This application is intended to cover any variations, uses, or adaptations of the application, these modifications, purposes or
Person's adaptive change follows the general principle of the application and including the undocumented common knowledge in the art of the disclosure
Or conventional techniques.The description and examples are only to be considered as illustrative, and the true scope and spirit of the application are by following
Claim is pointed out.
It should be understood that the application is not limited to the precise structure that has been described above and shown in the drawings, and
And various modifications and changes may be made without departing from the scope thereof.Scope of the present application is only limited by the accompanying claims
The foregoing is merely the preferred embodiments of the application, not to limit the application, it is all in spirit herein and
Within principle, any modification, equivalent replacement, improvement and so on be should be included within the scope of protection of this application.
Claims (10)
1. a kind of data compression method characterized by comprising
Data compression request is received, the storage address of data to be compressed is obtained；
The data to be compressed stored in the storage address are moved to default storage region, in order to field programmable gate array
FPGA reads the data to be compressed from the default storage region, and carries out compression processing to the data to be compressed, obtains
Compressed data；
Receive the compressed data that the FPGA is sent.
2. the method according to claim 1, wherein the default storage region is preparatory and the FPGA negotiates
Alright.
3. the method according to claim 1, wherein the method also includes:
Obtain the compressed format of the data to be compressed；
Correspondingly, described be moved to default storage region for the data to be compressed stored in the storage address, so as on site
Programmable gate array FPGA reads the data to be compressed from the default storage region, and presses the data to be compressed
Contracting processing, obtains compressed data, comprising:
The data to be compressed and the compressed format are moved to default storage region, in order to which FPGA is from the default storage
The data to be compressed and the compressed format are read in region, and are pressed according to the compressed format the data to be compressed
Contracting processing, obtains compressed data.
4. according to the method described in claim 3, it is characterized in that, the compressed format includes:
Gzip or Zlib.
5. the method according to claim 3 or 4, which is characterized in that the method also includes:
Compression head information and/or compression tail information are added for the compressed data；
Wherein, the compression head information and the compression tail information are determined according to the compressed format.
6. a kind of data compression device characterized by comprising
First receiving unit, compression request, obtains the storage address of data to be compressed for receiving data；
Mobile unit, for the data to be compressed stored in the storage address to be moved to default storage region, in order to existing
Field programmable gate array FPGA reads the data to be compressed from the default storage region, and carries out to the data to be compressed
Compression processing obtains compressed data；
Second receiving unit, the compressed data sent for receiving the FPGA.
7. device according to claim 6, which is characterized in that the default storage region is preparatory and the FPGA negotiates
Alright.
8. device according to claim 6, which is characterized in that described device further include:
Acquiring unit, for obtaining the compressed format of the data to be compressed；
Correspondingly, the mobile unit, is specifically used for:
The data to be compressed and the compressed format are moved to default storage region, in order to which FPGA is from the default storage
The data to be compressed and the compressed format are read in region, and are pressed according to the compressed format the data to be compressed
Contracting processing, obtains compressed data.
9. device according to claim 8, which is characterized in that the compressed format includes:
Gzip or Zlib.
10. device according to claim 8 or claim 9, which is characterized in that described device further include:
Adding unit, for compressing head information and/or compression tail information for compressed data addition；
Wherein, the compression head information and the compression tail information are determined according to the compressed format.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811082900.XA CN109213737A (en) | 2018-09-17 | 2018-09-17 | A kind of data compression method and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811082900.XA CN109213737A (en) | 2018-09-17 | 2018-09-17 | A kind of data compression method and apparatus |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109213737A true CN109213737A (en) | 2019-01-15 |
Family
ID=64984321
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811082900.XA Pending CN109213737A (en) | 2018-09-17 | 2018-09-17 | A kind of data compression method and apparatus |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109213737A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021026761A1 (en) * | 2019-08-13 | 2021-02-18 | 深圳大学 | Compression and decompression method and apparatus, electronic device, and storage medium |
| WO2021248869A1 (en) * | 2020-06-10 | 2021-12-16 | 京信网络系统股份有限公司 | Packet processing method and apparatus, and communication device and communication system |
| CN117331514A (en) * | 2023-12-01 | 2024-01-02 | 超越科技股份有限公司 | Solid-state disk data compression system and method based on region division |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130318084A1 (en) * | 2012-05-22 | 2013-11-28 | Xockets IP, LLC | Processing structured and unstructured data using offload processors |
| CN104685377A (en) * | 2012-12-13 | 2015-06-03 | 雪佛龙美国公司 | Memory-bound surface-related multiple prediction method for large datasets |
| CN108141471A (en) * | 2016-04-07 | 2018-06-08 | 华为技术有限公司 | The method, apparatus and equipment of compressed data |
-
2018
- 2018-09-17 CN CN201811082900.XA patent/CN109213737A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130318084A1 (en) * | 2012-05-22 | 2013-11-28 | Xockets IP, LLC | Processing structured and unstructured data using offload processors |
| CN104685377A (en) * | 2012-12-13 | 2015-06-03 | 雪佛龙美国公司 | Memory-bound surface-related multiple prediction method for large datasets |
| CN108141471A (en) * | 2016-04-07 | 2018-06-08 | 华为技术有限公司 | The method, apparatus and equipment of compressed data |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021026761A1 (en) * | 2019-08-13 | 2021-02-18 | 深圳大学 | Compression and decompression method and apparatus, electronic device, and storage medium |
| WO2021248869A1 (en) * | 2020-06-10 | 2021-12-16 | 京信网络系统股份有限公司 | Packet processing method and apparatus, and communication device and communication system |
| CN117331514A (en) * | 2023-12-01 | 2024-01-02 | 超越科技股份有限公司 | Solid-state disk data compression system and method based on region division |
| CN117331514B (en) * | 2023-12-01 | 2024-02-23 | 超越科技股份有限公司 | Solid-state disk data compression system and method based on region division |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105204781B (en) | Compression method, device and equipment | |
| US20140215170A1 (en) | Block Compression in a Key/Value Store | |
| CN109213737A (en) | A kind of data compression method and apparatus | |
| CN113296718B (en) | Data processing method and device | |
| US9836248B2 (en) | In-memory data compression complementary to host data compression | |
| US20160277544A1 (en) | Protocol data unit interface | |
| CN111177025B (en) | Data storage method and device and terminal equipment | |
| CN111435943B (en) | Data processing method, device, system and storage medium | |
| US20180293987A1 (en) | Speech recognition method, device and system based on artificial intelligence | |
| CN109756568A (en) | Processing method, equipment and the computer readable storage medium of file | |
| CN109743757B (en) | Data processing method and device, wireless module and Internet of things equipment | |
| CN113138969A (en) | Data transmission method and device, electronic equipment and computer readable storage medium | |
| CN105262491A (en) | Data compression method, data compression system, data decompression method, data decompression system and communication equipment | |
| CN109688606A (en) | Data processing method, device, computer equipment and storage medium | |
| CN117271014A (en) | Data processing method and client based on web worker | |
| CN110019347A (en) | A kind of data processing method, device and the terminal device of block chain | |
| CN103558995A (en) | Memory control chip and disk message transmission method | |
| WO2023124428A1 (en) | Chip, accelerator card, electronic device and data processing method | |
| CN110309138B (en) | Data merging method, merger based on FPGA and database system | |
| US20210141723A1 (en) | Memory usage in managed runtime applications | |
| CN104484174A (en) | Processing method and processing device for compressed file with RAR (Roshal A Rchive) format | |
| CN104516740A (en) | Function calling method and device and mobile terminal | |
| CN110998524A (en) | Method for processing configuration file, processing unit, touch chip, device and medium | |
| CN114996134A (en) | Containerized deployment method, electronic equipment and storage medium | |
| CN111477229B (en) | Voice recognition request processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190115 |