CN108242931B - Data compression providing method - Google Patents
Data compression providing method Download PDFInfo
- Publication number
- CN108242931B CN108242931B CN201611208590.2A CN201611208590A CN108242931B CN 108242931 B CN108242931 B CN 108242931B CN 201611208590 A CN201611208590 A CN 201611208590A CN 108242931 B CN108242931 B CN 108242931B
- Authority
- CN
- China
- Prior art keywords
- data
- receiving end
- common
- available storage
- common receiving
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- H03M7/3059—Digital compression and data reduction techniques where the original information is represented by a subset or similar information, e.g. lossy compression
- H03M7/3064—Segmenting
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- H03M7/60—General implementation details not specific to a particular type of compression
- H03M7/6058—Saving memory space in the encoder or decoder
Abstract
The invention relates to a data compression providing method, which comprises the steps that a data source obtains data with corresponding quality grade based on the selection of a control platform, and the data with the quality grade is provided to a corresponding common receiving end after being segmented and compressed; the common receiving end receives the data blocks provided by the data source and stores the received data blocks in a local storage space in sequence; the method can consider the available condition of storage/calculation resources of the data receiving end when starting to compress the data, and perform the block compression of the data under the condition of acquiring the receiving capacity of the data receiving end, the compression of the same data is completed in a shorter time by using a limited available memory, and the memory occupation of a program is reduced under the condition of not reducing the performance, so that the data compression can be performed in a shorter time by using fewer machines.
Description
[ field of technology ]
The invention belongs to the field of data compression, and particularly relates to a data compression providing method.
[ background Art ]
Information compression techniques refer to techniques that use fewer bits (bits) or bytes (bytes) to represent information. The compression method can detect and use structures present in the information to create a more compact representation so that the original information can be reconstructed into a compressed version. In an information delivery environment, information compression is becoming increasingly important. Information compression techniques may reduce costs associated with bandwidth requirements, input/output lead time, and communication latency, in other words, less time may be used to transfer compressed information. Therefore, there is a need for an information compression method and system for efficiently compressing information.
The data compression providing method in the prior art can not compress data based on the available condition of storage resources of a data receiving end, and the compression efficiency is improved on the basis of sacrificing the program memory. Based on the above problems, a new data compression providing method is needed, which can consider the available condition of storage resources of a data receiver when starting to perform data compression, perform block compression of data under the condition of acquiring the receiving capability of a data receiving end, complete compression of equivalent data in a shorter time by using a limited available memory, and reduce the memory occupation of a program under the condition of not reducing the performance, so that data compression can be performed in a shorter time by using fewer machines.
[ invention data ]
In order to solve the problems in the prior art, the technical scheme adopted by the invention is as follows: a data compression providing method, comprising the steps of:
step 1: selecting a common receiving end or a non-common receiving end to provide data for a client user, and selecting one or more common receiving ends to load data based on data priority;
step 2: the data source acquires data of corresponding quality grades based on the selection of the control platform, blocks and compresses the data of the quality grades and provides the data of the quality grades to corresponding common receiving ends;
step 3: the common receiving end receives the data blocks provided by the data source and stores the received data blocks in a local storage space in sequence;
further, the priority of the data reflects the strong willingness of the data source to provide the data to the receiving end, and the stronger the willingness to provide the data is, the higher the priority is.
Further, the priority is set by the control platform or the data side.
Further, the blocking of the data needs to consider the situation of available storage resources at the receiving end.
Further, the available storage resource condition includes an available receiving buffer space size of the receiving end.
Further, the available storage resource condition includes an available storage port condition of the receiving end.
Further, when the available storage resources of the receiving end are more, the data are split into larger blocks, and otherwise, the data are split into smaller blocks.
Further, the available storage resource condition is provided by the receiving end, and the available storage resource condition is changed in real time.
Further, the available storage resource condition provided by the receiving end is the storage resource condition provided by the data end when the data is allocated to the data end, and is not the actual all available storage resource condition.
Further, the blocking of the data needs to consider the situation of available computing resources at the receiving end.
The beneficial effects of the invention include: the method and the device can consider the available condition of storage/calculation resources of the data receiving end when starting to compress the data, perform the block compression of the data under the condition of acquiring the receiving capacity of the data receiving end, complete the compression of the same data in a short time by using the limited available memory, and reduce the memory occupation of the program under the condition of not reducing the performance, so that the data compression can be performed in a short time by using fewer machines.
[ description of the drawings ]
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate and together with the description serve to explain the invention, if necessary:
FIG. 1 is a flow chart of a data compression providing method of the present invention;
fig. 2 is a block diagram of a data compression providing system of the present invention.
[ detailed description ] of the invention
The present invention will now be described in detail with reference to the drawings and the specific embodiments thereof, wherein the exemplary embodiments and the description are for the purpose of illustrating the invention only and are not to be construed as limiting the invention.
Referring to fig. 1, the basic steps of a data compression providing method of the present invention are shown:
step 1: selecting a common receiving end or a non-common receiving end to provide data for a client user, and selecting one or more common receiving ends to load data based on data priority;
step 2: the data source acquires data of corresponding quality grades based on the selection of the control platform, blocks and compresses the data of the quality grades and provides the data of the quality grades to corresponding common receiving ends;
step 3: the common receiving end receives the data blocks provided by the data source and stores the received data blocks in a local storage space in sequence;
referring to fig. 2, a data compression providing system to which the present invention is applied includes one or more data sources, one or more control platforms, one or more receiving terminals, and one or more clients.
Based on the above system, a data compression providing method of the present invention is described in detail below:
(1) The client sends a data request for specified data through a data request interface, wherein the request carries a data identifier; the data request is sent to a control platform; the data request carries the demand parameters (Q, S) of the requested data; wherein Q is data quality, S is arrival speed; wherein the data quality defines a presentation quality of the requested data, generally, the higher the data quality, the larger the size of the data; and the arrival rate defines the loading rate of the data to the client; there may be a discrepancy between the data quality and the arrival rate, which may need to be sacrificed when the data quality is high;
preferably: the data request interface is presented through an application program, a webpage and a specific function of the client, and the presented data is provided by a data source;
(2) The control platform sends a data query request to the common receiving end groups corresponding to the client, each common receiving end group searches the data requested by the client from the local catalogue according to the data identification, if the requested data is found, the data quality of the requested data is returned as a response, otherwise, the data is not responded;
the control platform stores a corresponding table of the client and the common receiving end group; the common set of receivers for each client may be the same or different (different includes completely different and partially different); the comparison table is updated by the control platform in real time; the common receiving terminals in the common receiving terminal group are selected from receiving terminals with communication overhead between the common receiving terminals and the client terminals smaller than a first threshold; the common receiving end makes a response once, so that the hit rate is increased by 1; reducing the hit rate by a first preset value every one time period, and deleting the receiving end with the hit rate smaller than the second preset value from the common receiving end group corresponding to the client if the hit rate is smaller than the second preset value; if the communication overhead between a receiving end and the client is smaller than a first threshold, adding the communication overhead into a common receiving end group corresponding to the client, and setting the hit rate to an initial value; preferably: the initial value is greater than a second preset value; the first preset value and the second preset value are set by a control platform;
(3) Selecting a common receiving end or a non-common receiving end to provide data for a client user based on the response of the common receiving end, and selecting one or more common receiving ends to load data based on the data priority; the common receiving end and the unusual receiving end are opposite and dynamically changed, one receiving end is common to one client and may not be common to the other client, and at the moment, the receiving end is common and may not be common after one update;
if the control platform receives the response of the common receiving end, selecting one common receiving end to provide data according to the response common receiving end Li, the returned data quality QLi and the demand parameters (Q, S); specific: calculating a balance coefficient Bi of each commonly used receiving end of response according to a formula (1), wherein Ci is communication overhead between the receiving end Li and the client; the communication overhead value is a dynamic value, and needs to be updated periodically in real time, w1 is an adjustment value, and the adjustment value is a preset value set by a control platform; selecting a common receiving end with the minimum Bi as the selected common receiving end; the data quality provided by the common receiving end is closest to the arrival speed and the requirement of the client;
preferably: when a common receiving end can provide data of a plurality of quality grades, calculating the balance coefficient value for each quality grade; wherein the quality grade is divided into n grades, Q1-Qn respectively;
if the control platform receives no response of any common receiving end, selecting one or more common receiving ends to load data according to the data priority level HT, and selecting one non-common receiving end to provide data; wherein the priority level is m levels; the priority reflects the preference degree of the content by the user, and the higher the preference degree by the user is, the higher the priority is, and the lower the priority is otherwise; preferably, the priority reflects the intensity of the willingness of the data end to provide the data to the client;
a very common receiving end is selected, specifically: acquiring a group of very-used receiving terminals with the communication overhead Ci smaller than a second threshold value, wherein the group of very-used receiving terminals can be provided with the data quality grade being larger than or equal to Q, and if the number of elements in the group is smaller than 1, notifying a user to wait; at this time, the user needs to wait for the data of the common receiving terminal to be loaded and then provide the data; otherwise, selecting the very-used receiving end with the smallest communication overhead Ci from the very-used receiving end group as the selected very-used receiving end; the unusual receiving terminal provides data for the client, specifically: acquiring a data Block with a size of SB, wherein SB= (w2×Q)/MIN { Ci }, w2 is an adjustment factor, and under the condition of large communication overhead, firstly transmitting a smaller data Block; if the data quality which can be provided by the unusual receiving terminal is greater than Q, firstly carrying out lossy compression on the data Block, compressing the data Block into a data Block Q_Block with the quality grade Q, then carrying out lossless compression on the lossy compressed data Block Q_Block into a data Block C_Q_Block, and then supplying the data Block C_Q_Block to a client;
according to the HT grade of data priority (HT is HT 1-HTm, wherein HT1 priority is lowest, HTm priority is highest), selecting one or more commonly used receiving terminals for data loading, specifically:
(A) If data priority p1×htm < ht= < HTm; firstly, selecting a common receiving end with the smallest communication overhead Ci to load data with the quality grade Q, then acquiring available storage space of each common receiving end in the rest common receiving end groups, arranging the available storage space from large to small, selecting the common receiving end of n-1 before arrangement to load data with n-1 quality grades except the quality grade Q respectively, and loading data with one quality grade by each common receiving end; wherein P1 is a first ratio value; preferably, p1=90%;
(B) If data priority p2×htm < ht= < p1×htm; firstly, selecting a common receiving end with the smallest communication overhead Ci to load data with the quality grade Q, then acquiring available storage space of each common receiving end in the rest common receiving end groups, arranging the available storage space from large to small, respectively loading data with the quality grade (n-1) mod 2 except the quality grade Q on the common receiving end (n-1) mod 2 before selecting the arrangement, and loading data with the quality grade on each common receiving end;
wherein P2 is a second ratio value; preferably, p1=80%;
(C) If data priority p3×htm < ht= < p2×htm; firstly, selecting a common receiving end with the smallest communication overhead Ci to load data with the quality grade Q, and then acquiring the data of the common receiving end with the largest remaining available storage space for loading the quality grade Qn, and acquiring the data of the common receiving end with the smallest available storage space for loading the quality grade Q1; wherein P3 is a third ratio value; preferably, p1=50%;
(D) If data priority ht= < p3×htm; selecting a common receiving end with the smallest communication overhead Ci to load data with the quality grade Q, wherein P3 is a third ratio value; preferably, p1=50%;
(4) The data source acquires data of corresponding quality grades based on the selection of the control platform, blocks and compresses the data of the quality grades and provides the data of the quality grades to corresponding common receiving ends; the method comprises the following steps: for each common receiving end to which data needs to be provided, the data source firstly creates a providing process, the process obtains the available receiving buffer size Siz_Buf of the common receiving end, and calculates the data block size Siz_ BlK according to a formula (2), wherein CPmin_Qk is the minimum compression rate corresponding to quality grade Qk data, PC is the redundancy percentage, and PC is a preset value corresponding to a compression algorithm;
Siz_BlK＝Siz_Buf/(CPmin_Qk×(1+PC)) (2)
the process divides data into a plurality of blocks with the size Siz_ BlK from beginning to end, and the data with the size less than 1 block is divided into 1 block; applying for a space with the size of Siz_Buf in a sending buffer space of a data source, acquiring each block one by a process, compressing the block, storing the compressed block into the sending buffer space with the size of Siz_Buf, and then sending the compressed block to a corresponding common receiving end; after the transmission is completed, releasing the space with the size of Siz_Buf, enabling the process to enter a dormant state, and after receiving a message of finishing the reception of a common receiving end, waking up the process and continuing the acquisition, compression and transmission of the next data block until all the data blocks are processed;
because the data source faces all receiving ends, the sending buffer space is very limited, and the sending buffer space can be shared in both time dimension and space dimension in a process asynchronous awakening mode, so that the space availability is increased;
preferably: the quality grade and the corresponding minimum compression rate are stored in a comparison table, and the comparison table is stored in the local storage of the data source; preferred pc=10%;
(5) The common receiving end is used for calculating the size Siz_Buf of the available receiving buffer, receiving the data blocks provided by the data source, and sequentially storing the received data blocks into the local storage space; the method comprises the following steps: the common receiving end calculates the size Siz_Buf of the available receiving buffer based on the formula (3); wherein Qk is the data quality to be loaded allocated by the control platform, buf_all is the current total available receiving buffer size of the common receiving end, par_cur is the number of parallel loads currently performed, and par_all is the number of maximum parallel loads supported by the common receiving end;
Siz_Buf＝Buf_All×(Qk/Qk+Q1)/(Par_all－Par_Cur) (3)
a common receiving end creates a receiving process, the receiving process applies for a space with the size of Siz_Buf in a receiving buffer space, a received data block is stored in the space, and the received data block is stored in a local buffer space in a related manner according to the logic sequence of data and the data identifier after the receiving is finished; after the current data block is received, the common receiving terminal sends a data block receiving completion message to the data source so as to continue the provision of the subsequent data blocks; the space with the size Siz_Buf is reserved and not released all the time in the data loading process until all the data blocks corresponding to the data are loaded, and the process releases the receiving buffer space of the application and ends the receiving process;
the common receiving end needs to leave a small amount of space, and can allow the data loading with the lowest service quality besides exceeding the supported maximum parallelism; loading efficiency beyond the maximum parallelism is greatly reduced, so that the service quality is relatively low;
(6) A common receiving end for receiving data with the quality grade Q provides data for a client; specific: when the data block received by the common receiving end of the data with the quality level Q is the data which is not provided for the client end by the selected common receiving end, stopping the data provision of the common receiving end, and carrying out the data provision by the common receiving end instead; that is to say: when the loading speed of the common receiving end catches up with the providing speed of the non-common receiving end, switching the providing receiving end;
(7) The client acquires the data blocks provided by the common receiving end or the non-common receiving end, decompresses the received data blocks, associates the data blocks with the data identifiers according to the logic sequence of the data, and stores the data blocks in the local storage space; the received data blocks do not need to be all aligned, and the data blocks conforming to the presentation sequence can be presented to a user according to the presentation sequence as long as the data blocks are received completely;
the method also comprises the steps of actively pushing the hot spot data:
(8) The data source provides the new hot spot data to each receiving end; specific: the data source acquires a receiving end group with the activity degree higher than a first activity value, acquires the size of an available storage space of each receiving end in the receiving end group, and provides data of a grade with corresponding quality for the receiving end based on the size of the storage space; preferably: providing data with higher quality level for a receiving end with larger available storage space;
the data compression providing method can consider the available condition of storage/calculation resources of the data receiving end when starting to compress the data, and perform the block compression of the data under the condition of acquiring the receiving capacity of the data receiving end, and complete the compression of the same data in a short time by using a limited memory, and reduce the memory occupation of a program under the condition of not reducing the performance, so that the data compression can be performed in a short time by using fewer machines.
The foregoing description is only of the preferred embodiments of the invention, and all changes and modifications that come within the meaning and range of equivalency of the structures, features and principles of the invention are therefore intended to be embraced therein.
Claims (2)
1. A data compression providing method, comprising the steps of:
step 1: selecting a common receiving end or a non-common receiving end to provide data for a client user, and selecting one or more common receiving ends to load data based on data priority;
step 2: the data source acquires data of corresponding quality grades based on the selection of the control platform, blocks and compresses the data of the quality grades and provides the data of the quality grades to corresponding common receiving ends;
step 3: the common receiving end receives the data blocks provided by the data source and stores the received data blocks in a local storage space in sequence;
the priority of the data reflects the intensity of the willingness of the data source to provide the data to the receiving end, and the stronger the willingness to provide the data is, the higher the priority is;
the priority is set by a control platform or a data end;
the situation of available storage resources of a receiving end needs to be considered for partitioning the data;
the available storage resource condition comprises the size of an available receiving buffer space of a receiving end;
the available storage resource conditions comprise the conditions of available storage ports of a receiving end;
when the available storage resources of the receiving end are more, the data are segmented into larger blocks, and otherwise, the data are segmented into smaller blocks;
the available storage resource condition is provided by the receiving end, and the available storage resource condition is changed in real time;
the available storage resource condition provided by the receiving end is the storage resource condition which is allocated to the data end and provided by the data, and is not the actual all available storage resource condition.
2. The data compression providing method according to claim 1, wherein the partitioning of the data takes into account available computing resource conditions at the receiving end.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611208590.2A CN108242931B (en) | 2016-12-23 | 2016-12-23 | Data compression providing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611208590.2A CN108242931B (en) | 2016-12-23 | 2016-12-23 | Data compression providing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108242931A CN108242931A (en) | 2018-07-03 |
| CN108242931B true CN108242931B (en) | 2023-04-28 |
Family
ID=62704221
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611208590.2A Active CN108242931B (en) | 2016-12-23 | 2016-12-23 | Data compression providing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108242931B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110209640A (en) * | 2019-06-06 | 2019-09-06 | 四川长虹电器股份有限公司 | The method of switching at runtime lz4 compression algorithm type under cell phone system operating status |
| CN111277274A (en) * | 2020-01-13 | 2020-06-12 | 平安国际智慧城市科技股份有限公司 | Data compression method, device, equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101820426A (en) * | 2010-04-22 | 2010-09-01 | 华中科技大学 | Data compression method in on-line backup service software |
| WO2016095149A1 (en) * | 2014-12-18 | 2016-06-23 | 华为技术有限公司 | Data compression and storage method and device, and distributed file system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7930436B1 (en) * | 2009-03-09 | 2011-04-19 | Znosko Dmitry Y | System and method for dynamically adjusting data compression parameters |
| CN101527654B (en) * | 2009-04-20 | 2012-01-25 | 中兴通讯股份有限公司 | Data transmission method and system in network management system |
| CN105791245B (en) * | 2014-12-26 | 2019-09-03 | 北京航天测控技术有限公司 | A kind of data sending, receiving method and device |
| CN105045873B (en) * | 2015-07-17 | 2018-02-09 | 深圳市腾讯计算机系统有限公司 | A kind of method for pushing of data file, apparatus and system |
-
2016
- 2016-12-23 CN CN201611208590.2A patent/CN108242931B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101820426A (en) * | 2010-04-22 | 2010-09-01 | 华中科技大学 | Data compression method in on-line backup service software |
| WO2016095149A1 (en) * | 2014-12-18 | 2016-06-23 | 华为技术有限公司 | Data compression and storage method and device, and distributed file system |
| CN106170968A (en) * | 2014-12-18 | 2016-11-30 | 华为技术有限公司 | A kind of data compression storage method, device, and distributed file system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108242931A (en) | 2018-07-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9172771B1 (en) | System and methods for compressing data based on data link characteristics | |
| WO2016095149A1 (en) | Data compression and storage method and device, and distributed file system | |
| CN103516744A (en) | A data processing method, an application server and an application server cluster | |
| CN106790552B (en) | A kind of content providing system based on content distributing network | |
| EP2266265A1 (en) | System and method for memory allocation in embedded or wireless communication systems | |
| CN108242931B (en) | Data compression providing method | |
| CN112888005B (en) | MEC-oriented distributed service scheduling method | |
| CN112650721A (en) | File storage method, device, system and equipment | |
| CN101305577A (en) | Device and method correlating message transmission | |
| CN115858184B (en) | RDMA memory management method, device, equipment and medium | |
| US10560515B2 (en) | Method and apparatus for transmitting and receiving information between servers in contents transmission network system | |
| CN105282045B (en) | A kind of distributed computing and storage method based on consistency hash algorithm | |
| CN113138969A (en) | Data transmission method and device, electronic equipment and computer readable storage medium | |
| CN114253698A (en) | Resource scheduling method, system, electronic device and storage medium | |
| CN109617960B (en) | Attribution separation-based web AR data presentation method | |
| CN112911708A (en) | Resource allocation method, server and storage medium | |
| CN115168302A (en) | Business data export method and device and electronic equipment | |
| US20120137017A1 (en) | System and method for controlling server usage in peer-to-peer (p2p) based streaming service | |
| CN116389454A (en) | Data downloading system | |
| US10728291B1 (en) | Persistent duplex connections and communication protocol for content distribution | |
| CN111587417A (en) | Method, device, computer program and computer program product for data storage | |
| CN111193607B (en) | Method, device and storage medium for transmission resource management | |
| CN110266814B (en) | Transmission method and transmission device | |
| CN109766363B (en) | Streaming data processing method, system, electronic device and storage medium | |
| CN108243147B (en) | Viewing data loading method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 101399 No. 2 East Airport Road, Shunyi Airport Economic Core Area, Beijing (1st, 5th and 7th floors of Industrial Park 1A-4) Applicant after: Zhongke Star Map Co.,Ltd. Address before: 101399 Building 1A-4, National Geographic Information Technology Industrial Park, Guomen Business District, Shunyi District, Beijing Applicant before: GEOVIS TECHNOLOGY (BEIJING) Co.,Ltd. |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |