CN117708069A

CN117708069A - Data compression method and electronic equipment

Info

Publication number: CN117708069A
Application number: CN202310920380.XA
Authority: CN
Inventors: 王启
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2023-07-24
Filing date: 2023-07-24
Publication date: 2024-03-15

Abstract

The embodiment of the application is applicable to the technical field of data processing, and provides a data compression method and electronic equipment, which comprise the steps of obtaining a file type of data to be compressed and a first data volume of the data to be compressed, further determining a first numerical value of the data to be compressed according to the file type and the first data volume, and then carrying out data compression on the data to be compressed based on the first numerical value to obtain compressed data, wherein clustering numerical values refer to the number of data blocks in each cluster when the data are clustered, and because the value of the clustering numerical value (the first numerical value) is determined according to the first data volume of the data to be compressed of the file type, the value of the clustering numerical value is more matched with the data to be compressed, so that when the value of the clustering numerical value is the first numerical value, the data compression rate of each cluster of the data to be compressed and the duty ratio of the data not to be compressed can be balanced, and the overall compression rate of the data to be compressed can be improved.

Description

Data compression method and electronic equipment

Technical Field

The present application relates to the field of data processing, and more particularly, to a data compression method and an electronic device.

Background

At present, a large number of applications are installed in an electronic device, and file data generated during the running process of the applications is often required to be stored in the process of using the applications. When the file data is stored in the memory, the electronic device compresses the file data, so that the storage space in the electronic device is saved.

The electronic device performs a compression process on file data, and typically performs data compression according to a clustering compression method. For example, a storage space for storing compressed file data in an electronic device is divided into a plurality of clusters, each of which is divided into a plurality of data blocks (e.g., 8 data blocks). Wherein the electronic device may manage a plurality of data blocks by node, the number of data blocks corresponding to each node is typically a fixed value (e.g., 1018). It will be appreciated that the number of data blocks in each cluster is a fixed value N, and the data blocks corresponding to each node are also fixed values i, in one possible case i is not divisible by N, for example, such that the data blocks in the same cluster belong to different nodes, respectively, i.e. there is a case where the data blocks in the cluster cross the node. Typically, for ease of management, the data in clusters across nodes is not compressed. In a specific data compression process, if the data amount in one cluster is large, the situation of compressible repeated data in the same cluster is large, the compression rate in the same cluster is high, but the data amount in a cluster crossing a node is also large at the same time, and the data cannot be compressed. In combination, the overall compression rate of the data to be compressed is affected.

Based on this, how to increase the overall compression rate of the data to be compressed becomes a problem to be solved.

Disclosure of Invention

The application provides a data compression method which can improve the overall compression rate of data to be compressed.

In a first aspect, a data compression method is provided, including:

acquiring a file type of data to be compressed and a first data volume of the data to be compressed;

determining a first value of data to be compressed according to the file type and the first data volume, wherein the first value refers to a value of a clustering value when the data to be compressed is subjected to clustering compression, and the clustering value refers to the number of data blocks in each cluster when the data is clustered;

and carrying out data compression on the data to be compressed based on the first numerical value to obtain compressed data.

The data compression method includes the steps of obtaining a file type of data to be compressed and a first data amount of the data to be compressed, determining a first value of the data to be compressed according to the file type and the first data amount, and performing data compression on the data to be compressed based on the first value to obtain compressed data, wherein the clustering value refers to the number of data blocks in each cluster when the data are clustered, and the value of the clustering value (the first value) is determined according to the first data amount of the data to be compressed of the file type and is a value of the clustering value which is matched with the first data amount of the data to be compressed and the file type, so that when the value of the clustering value is the first value, the data compression rate of each cluster of the data to be compressed and the duty ratio of the data not to be compressed in the data to be compressed can be balanced, and the overall compression rate of the data to be compressed is improved.

With reference to the first aspect, in some implementations of the first aspect, determining the first value of the data to be compressed according to the file type and the first data value includes: acquiring a first parameter set according to the file type of the data to be compressed, wherein the first parameter set comprises a plurality of first parameters, and the first parameters are used for indicating the data compression rate corresponding to one value of the clustering numerical value; determining a second parameter set according to the first data volume, wherein the second parameter set comprises a plurality of second parameters, and the second parameters are used for indicating the data volume of uncompressed data corresponding to one value of the clustering value when the data volume of the data to be compressed is the first data volume; a first value is determined from the first parameter set and the second parameter set.

With reference to the first aspect, in certain implementation manners of the first aspect, determining the first value according to the first parameter set and the second parameter set includes:

determining a first value according to a first parameter set, a second parameter set, and a first formula, the first formula comprising:

wherein,representing a first set of parameters, m representing the number of bytes of stored data in a data block,s represents a first data amount, N represents the number of data blocks in each cluster, +. >Representing a third parameter corresponding to different file types, the third parameter being a parameter that maximizes the covariance between the theoretical compression rate and the actual compression rate, wherein +.>Representing a second set of parameters.

With reference to the first aspect, in certain implementation manners of the first aspect, determining the first numerical value according to the first parameter set, the second parameter set, and the first formula includes: and iterating in a first formula by adopting each parameter in the first parameter set and each parameter in the second parameter set to obtain a first numerical value corresponding to the maximum value of the plurality of F.

With reference to the first aspect, in certain implementations of the first aspect, the number of bytes of stored data m in one data block is 4096.

When the number of bytes of the stored data in one data block is 4096, the first formula may be:

with reference to the first aspect, in certain embodiments of the first aspect, the clustering value comprises 2 ⁿ Wherein n is a positive integer.

With reference to the first aspect, in certain embodiments of the first aspect, the clustering value ranges from [4,8, 16, 32].

According to the data compression method provided by the embodiment of the application, the value range of the clustering values is [4,8, 16 and 32], so that the value number of the clustering values is limited, and further the iteration times can be reduced in the process of obtaining the first value based on iteration of the clustering values, and further the efficiency of obtaining the first data is improved, namely the data compression efficiency is improved.

With reference to the first aspect, in some implementations of the first aspect, data compression is performed on data to be compressed based on a first value, to obtain compressed data, including: determining a number of data blocks in each cluster based on the first value; and carrying out de-duplication processing on the data in the data blocks in each cluster to obtain compressed data.

In a second aspect, there is provided a data compression apparatus comprising means for performing any of the methods of the first aspect. The device can be a server, terminal equipment or a chip in the terminal equipment. The apparatus may include an input unit and a processing unit.

When the apparatus is a terminal device, the processing unit may be a processor, and the input unit may be a communication interface; the terminal device may further comprise a memory for storing computer program code which, when executed by the processor, causes the terminal device to perform any of the methods of the first aspect.

When the device is a chip in the terminal device, the processing unit may be a processing unit inside the chip, and the input unit may be an output interface, a pin, a circuit, or the like; the chip may also include memory, which may be memory within the chip (e.g., registers, caches, etc.), or memory external to the chip (e.g., read-only memory, random access memory, etc.); the memory is for storing computer program code which, when executed by the processor, causes the chip to perform any of the methods of the first aspect.

In one possible implementation, the memory is used to store computer program code; a processor executing the computer program code stored in the memory, the processor, when executed, configured to perform: acquiring a file type of data to be compressed and a first data volume of the data to be compressed; determining a first value of data to be compressed according to the file type and the first data volume, wherein the clustering value refers to the number of data blocks in each cluster when the data are clustered; and carrying out data compression on the data to be compressed based on the first numerical value to obtain compressed data.

In a third aspect, there is provided a computer readable storage medium storing computer program code which, when run by data compression means, causes the data compression means to perform any one of the data compression methods of the first aspect.

In a fourth aspect, there is provided a computer program product comprising: computer program code which, when run by data compression means, causes the data compression means to perform any of the means methods of the first aspect.

The data compression method and the electronic device provided by the embodiment of the application comprise the steps of obtaining a file type of data to be compressed and a first data volume of the data to be compressed, further determining a first value of the data to be compressed according to the file type and the first data volume, and then carrying out data compression on the data to be compressed based on the first value to obtain compressed data, wherein the clustering value refers to the number of data blocks in each cluster when the data are clustered, and the value (the first value) of the clustering value is determined according to the first data volume of the data to be compressed and is a value of the clustering value which is matched with the data to be compressed, so that when the value of the clustering value is the first value, the data compression rate of each cluster of the data to be compressed and the ratio of non-compressed data in the data to be compressed can be balanced, and the overall compression rate of the data to be compressed is improved.

Drawings

FIG. 1 is a schematic illustration of a clustered compression;

FIG. 2 is a schematic diagram of a node managing a plurality of clusters;

FIG. 3 is a schematic diagram of a hardware system suitable for use with the electronic device of the present application;

fig. 4 is a schematic flow chart of a data compression method according to an embodiment of the present application;

Fig. 5 (a) is a schematic diagram of a compression ratio corresponding to a file type when the values of the cluster values are different according to the embodiment of the present application;

fig. 5 (b) is a schematic diagram of a compression ratio corresponding to a file type when the values of the cluster values are different according to the embodiment of the present application;

fig. 5 (c) is a schematic diagram of a compression ratio corresponding to a file type when the values of the cluster values are different according to the embodiment of the present application;

fig. 5 (d) is a schematic diagram of a compression ratio corresponding to a file type when the values of the cluster values are different according to the embodiment of the present application;

FIG. 6 is a schematic diagram of a data compression device provided herein;

fig. 7 is a schematic diagram of an electronic device for data compression provided herein.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Wherein, in the description of the embodiments of the present application, "/" means or is meant unless otherwise indicated, for example, a/B may represent a or B; "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In addition, in the description of the embodiments of the present application, "plurality" means two or more than two.

The terms "first," "second," "third," and the like, are used below for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first", "a second", or a third "may explicitly or implicitly include one or more such feature.

The electronic device performs a compression process on file data, and typically performs data compression according to a clustering compression method. For example, as shown in (a) of fig. 1, a storage space for storing compressed file data in an electronic device is divided into m clusters of clusters each including N data blocks. It should be appreciated that the number of data blocks in each cluster is determined based on the cluster value stored in advance in the electronic device, and that the number of data blocks N in each cluster is typically 2, i.e. the value of the cluster value N is 2 ⁿ N is a positive integer. For example, as shown in (a) of fig. 1, n has a value of 2, i.e., the number of data blocks in each Cluster is 4. Data compression according to the Cluster compression method generally refers to performing data deduplication processing on each Cluster to obtain compressed data. Illustratively, before the compression of the cluster 1, as shown in (a) of fig. 1, data is stored in the data blocks 11, 12, 13 and 14 in the cluster 1, and after the compression of the cluster 1, data is still stored in the data blocks 11 and 12, and the data in the data blocks 13 and 14 is emptied. With the cluster 1 before compression, the data in the data blocks 13 and 14 are released, saving space in the cluster 1.

It should be appreciated that the electronic device may manage multiple data blocks through the node. The number of data blocks for each node is typically a fixed value (e.g., 1018). It will be appreciated that the number of data blocks in each cluster is a fixed value N, and the data blocks corresponding to each node are also fixed values i, where i cannot be divided by N in an integer manner, so that the data blocks in the same cluster belong to different nodes, respectively, i.e. there is a case where the data blocks in the cluster cross the node. Illustratively, as shown in FIG. 2, node1 manages data block m1 and data block m2 in cluster m, and node2 manages other data blocks in cluster m including data block mN, that is, cluster m is a cross-node cluster. For ease of management, the data in the cluster across the node is not compressed.

In a specific data compression process, if the number of data blocks in one cluster is large, the probability of occurrence of compressible repeated data in the same cluster is large, which is equivalent to that of the cluster. However, a larger number of data blocks in one cluster may result in a larger amount of data in the cluster across nodes, which in turn results in a larger amount of uncompressed data in the data to be compressed. For example, the default cluster value in the file system is 8, that is, each cluster includes 8 data blocks, and typically, the number of bytes of storable data in one data block is 4096, that is, each cluster may store 8×4096=32768 bytes of data. In the case where the data amount of the data to be stored is large, for example, the data amount of the data to be stored is greater than 1MB, the probability of occurrence of compressible repeated data in each cluster is large, which corresponds to a high compression rate of the cluster. However, a larger number of data blocks in one cluster may result in a larger amount of data in the cluster across nodes, which in turn results in a larger amount of uncompressed data in the data to be compressed. In the case where the data amount of the data to be stored is small, for example, less than 32KB, the data amount of the data to be stored is less than the data amount of one cluster, and therefore, the file system does not compress the data to be stored either, resulting in that the small file cannot save the storage space by compressing the data. In combination, the overall compression rate of the data to be compressed is affected by the number of data blocks in each cluster.

In view of this, an embodiment of the present application provides a data compression method, including: the method comprises the steps of obtaining a file type of data to be compressed and a first data volume of the data to be compressed, determining a first value of the data to be compressed according to the file type and the first data volume, and performing data compression on the data to be compressed based on the first value to obtain compressed data, wherein a clustering value refers to the number of data blocks in each cluster when the data is clustered, and the value (the first value) of the clustering value is determined according to the first data volume of the data to be compressed of the file type and is a clustering value which is matched with the data to be compressed, so that when the value of the clustering value is the first value, the data compression rate of each cluster of the data to be compressed and the duty ratio of the data not to be compressed in the data to be compressed can be balanced, and the overall compression rate of the data to be compressed is improved.

The data compression method provided by the embodiment of the application can be applied to electronic equipment. Optionally, the electronic device includes a terminal device, which may also be referred to as a terminal (terminal), a User Equipment (UE), a Mobile Station (MS), a Mobile Terminal (MT), and so on. The terminal device may be a mobile phone, a smart television, a wearable device, a tablet (Pad), a computer with wireless transceiving function, a Virtual Reality (VR) terminal device, an augmented reality (augmented reality, AR) terminal device, a wireless terminal in industrial control (industrial control), a wireless terminal in unmanned driving (self-driving), a wireless terminal in teleoperation (remote medical surgery), a wireless terminal in smart grid (smart grid), a wireless terminal in transportation safety (transportation safety), a wireless terminal in smart city (smart city), a wireless terminal in smart home (smart home), or the like. The embodiment of the application does not limit the specific technology and the specific equipment form adopted by the terminal equipment.

By way of example, fig. 3 shows a schematic structural diagram of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, keys 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identity module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer components than shown, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, such as: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural network processor (neural-networkprocessing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller may be a neural hub and a command center of the electronic device 100, among others. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universalasynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processorinterface, MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, and/or a universal serial bus (universal serial bus, USB) interface, among others.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to enable expansion of the memory capabilities of the electronic device 100. The external memory card communicates with the processor 110 through an external memory interface 120 to implement data storage functions. For example, files such as music, video, etc. are stored in an external memory card.

The internal memory 121 may be used to store computer executable program code including instructions. The processor 110 executes various functional applications of the electronic device 100 and data processing by executing instructions stored in the internal memory 121. The internal memory 121 may include a storage program area and a storage data area. The storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system, etc. The storage data area may store data created during use of the electronic device 100 (e.g., audio data, phonebook, etc.), and so on. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (universal flash storage, UFS), and the like.

It should be noted that any of the electronic devices mentioned in the embodiments of the present application may include more or fewer modules in the electronic device 100.

The software system of the electronic device 100 may employ a layered architecture, an event driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. In this embodiment, taking an Android system with a layered architecture as an example, a software structure of the electronic device 100 is illustrated.

The layered architecture of the electronic device 100 divides the software into several layers, each with a distinct role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, from top to bottom, an application layer, an application framework layer, an Zhuoyun row (Android run) and system libraries, and a kernel layer, respectively.

The application layer may include a series of application packages.

The application package may include camera, gallery, calendar, talk, map, navigation, WLAN, bluetooth, music, video, short message, etc. applications.

The application framework layer provides an application programming interface (application programminginterface, API) and programming framework for application programs of the application layer. The application framework layer includes a number of predefined functions.

Android run time includes a core library and virtual machines. Android run time is responsible for scheduling and management of the Android system.

The system library may include a plurality of functional modules. For example: surface manager (surface manager), media library (MediaLibraries), three-dimensional graphics processing library (e.g., openGL ES), 2D graphics engine (e.g., SGL), etc.

The kernel layer is a layer between hardware and software. The kernel layer at least comprises a display driver, a camera driver, an audio driver, a sensor driver, a Wi-Fi driver and the like.

It should be noted that, the electronic device according to the embodiment of the present application may include more or fewer modules in the electronic device.

The data compression method provided in the embodiment of the present application is described in detail below with reference to fig. 4.

Fig. 4 is a flow chart of a data compression method according to an embodiment of the present application, as shown in fig. 4, where the method includes:

s101, acquiring a file type of data to be compressed and a first data volume of the data to be compressed.

It should be understood that the data to be compressed may refer to various types of data generated by an application program in the electronic device during operation, and in order to reduce the occupation of the memory of the electronic device by the data, the electronic device will generally compress the data and then store the compressed data in a corresponding storage address. It should be understood that the different data has its corresponding file type, which may include, by way of example, an odex format, a vdex format, an SO format, an APK format, an MP4 format, a jpg format, and a db format.

The method comprises the steps of generating a file in an odex format, wherein the file in the odex format is an executable file extracted from an application program APK in an android system, namely, the class. Dex file in the APK is generated into a file through dex optimization, and the file is usually stored independently, and the class. Dex file in the original APK file can be reserved or deleted.

The vdex format is an execution file format of an Android application program, and aims to improve the starting time and the memory use efficiency of the application program on an Android device. It is designed on the basis of the DEX file (Dalvik Executable) and contains pre-computed parsing data to alleviate the operations required by the application to perform class loading at run-time. The vdex file is typically used with the odex file for optimizing the launch and performance of applications on Android devices.

The APK format file is an application package file format used by the Android operating system. Code of an Android application program is required to run on an Android device, the code is compiled first and is packed into a file which can be identified by an Android system and can be run, and the file format which can be identified and run by the Android system is an APK format.

The SO format is ELF file format, which is the unified format of executable files, shared library files and target files under Linux.

The MP4 format is a commonly used video format with file extensions of mainly. MP4 or. Avi. The MP4 video format has the main advantages of high encoding efficiency, capability of processing a large amount of data in a relatively short time, and capability of providing higher quality video images.

The jpg format is a common image file format, and is a standard for continuous tone static image compression, and the image compression corresponding to the jpg format file is lossy compression, so that the data image is compressed in a small storage space and the image data is damaged.

The db format is a database file format, which is a file format used by software to store data, and may be equivalent to a database.

It will be appreciated that the file of the data to be stored may include information indicative of the file type, which the electronic device may determine the file type of the data to be stored by reading.

Similarly, the file of the data to be stored may also include information indicating the size of the file, and the electronic device may obtain the data amount of the data to be stored, that is, the first data amount, by reading the information.

S102, determining a first numerical value of data to be compressed according to the file type and the first data volume.

The first value refers to a value of a clustering value when clustering compression is performed on data to be compressed, and the clustering value refers to the number of data blocks in each cluster when the data is clustered.

It should be understood that the number of data blocks included in each cluster is the same, for example, as shown in fig. 1 (a) or fig. 1 (b), the number of data blocks in cluster 1 is 4, and the number of data blocks in cluster 2 through cluster m is 4. That is, for data to be compressed as shown in fig. 1 (a) or fig. 1 (b), the corresponding first value is 4.

As can be seen from the description of fig. 2, the value of the cluster value affects the overall compression rate of the data to be compressed, because the value of the cluster value affects the amount of uncompressed data in the data to be compressed and the repetition rate of the data in the same cluster in the data to be compressed. While different file types may also affect the compression rate of data in the same cluster. Therefore, the first numerical value which is more suitable for the current data to be compressed can be determined according to the file type and the first data quantity of the data to be compressed, so that when the value of the cluster numerical value is the first numerical value, the ratio of the uncompressed data in the data to be compressed is the smallest, and the overall compression rate of the data to be compressed is further improved.

Illustratively, S102 "determining the first value of the data to be compressed according to the file type and the first data amount" one possible implementation includes:

step 1, acquiring a first parameter set according to the file type of data to be compressed, wherein the first parameter set comprises a plurality of first parameters, and the first parameters are used for indicating the data compression rate corresponding to one value of the clustering numerical value.

The clustering value refers to the number of data blocks in each cluster when the data is clustered. Unlike the first value, the first value is a specific number of data blocks in each cluster when cluster compression is performed on data to be stored. The clustering value refers to a range of the number of data blocks in each cluster when data is clustered. The first value is a specific value, the clustering value is a value range of values, and the value range comprises a plurality of values, namely, the first value is a value of the clustering value, and when the value of the clustering value is the first value, the ratio of uncompressed data in the data to be compressed is minimum.

Illustratively, the cluster value may include a value of 2 ⁿ Wherein n is a positive integer. That is, the value of the cluster value may be an integer power of 2.

It will be appreciated that taking different values for the cluster values affects the overall compression rate of the data to be compressed. As shown in fig. 5 (a), 5 (b), 5 (c) and 5 (d), for example, when the clustering values of the data to be compressed of different file types take different values, the overall compression rate of the data to be compressed is changed. Wherein fig. 5 (a) refers to the overall compression rate of the APK format data to be compressed when the cluster values take different values, fig. 5 (b) refers to the overall compression rate of the db format data to be compressed when the cluster values take different values, fig. 5 (c) refers to the overall compression rate of the SO format data to be compressed when the cluster values take different values, and fig. 5 (d) refers to the overall compression rate of the mp4 format data to be compressed when the cluster values take different values. It can be seen that when the cluster value takes a value of 32 or more, the overall compression rate of the data to be compressed is lowered.

Alternatively, the cluster values range from [4,8, 16, 32].

The first parameter set may include a plurality of first parameters, where each first parameter is a data compression rate of data to be stored corresponding to a value of the cluster numerical value and the same file type. It should be appreciated that the first parameter may refer to the user deriving the data compression rate from an actual test. For example, a file type may be marked as T _a Wherein a = 1,2,3 … …; that is, when a is 1, the file type of the data to be stored is the file type corresponding to the value 1, and when a is 2, the file type of the data to be stored is the file type corresponding to the value 2, … …. It should be understood that the correspondence between the values and the file types is pre-stored in the electronic device, so that the file types of the data to be stored can be determined according to different values of a. In addition, as can be seen from the above description, the cluster values may include 2 ⁿ Wherein n is a positive integer. That is, the value N of the cluster value may include [2,4,8, 16, 32, 64, … ]]。

Thus, the file may be of type T _a The data compression rate of the data to be stored is recorded as when the value of the clustering value is NIt will be appreciated that->May be a value obtained by the user from an actual test. After obtaining- >Afterwards, +.>Pre-stored in the electronic device for recall by the electronic device.

And 2, determining a second parameter set according to the first data volume, wherein the second parameter set comprises a plurality of second parameters, and the second parameters are used for indicating the data volume of uncompressed data corresponding to one value of the clustering value when the data volume of the data to be compressed is the first data volume.

In compressing data to be stored, the data amount of the data to be stored is not usually a positive integer multiple of the number in one cluster, and thus there is a part of data that cannot occupy the storage space of one cluster, which may be referred to as end-of-file data. It will be appreciated that the data storage is sequentially stored, and the data in the cluster across the nodes is the end of file data. That is, in the process of data compression, the end-of-file data is not compressed.

For example, the size of the data to be stored occupies 63 data blocks, the value of the cluster value is 32, that is, the data blocks in one cluster are 32, so that the tail data of the data to be stored also needs to occupy 31 data blocks for storage, but does not occupy all the data blocks in one cluster completely, and the data cannot be compressed. Thus, nearly half of the data in the data to be stored cannot be compressed, and the overall compression rate of the data to be stored is affected due to the large duty ratio of the uncompressed data.

The data size of the end data of the file can be determined byThe expression, wherein m represents the byte number of the data stored in a data block, S represents the data volume of the data to be stored, namely the first data volume, N is the value of the clustering value, and +.>Representing the upward rounding of the value obtained for S/m, mod is the modulo operator, ++>The remainder obtained by dividing the value obtained by rounding up the value obtained by S/m by N is represented. Because the value of N is different, therefore,can represent the corresponding file end data when the clustering value takes different values, namelyWhen the data volume of the data to be compressed is the first data volume, the data volumes of the uncompressed data corresponding to different clustering values are equivalent to the second parameter set.

Optionally, the number of bytes of stored data in one data block is 4096.

When the byte number of the stored data in one data block is 4096, the number of the end data of the file can be determined byWherein S represents the data amount of the data to be stored, i.e. the first data amount, N is the value of the cluster value, < >>Representing the upward rounding of the value obtained for S/4096, mod is the modulo operator,the remainder obtained by dividing the value obtained by rounding up the value obtained by S/4096 by N is represented.

And 3, determining a first numerical value according to the first parameter set and the second parameter set.

Optionally, "determining the first value from the first set of parameters and the second set of parameters" may include: determining a first value according to a first parameter set, a second parameter set, and a first formula, the first formula comprising:

wherein,represents a first set of parameters, m represents the number of bytes of data stored in one data block, S represents a first amount of data, N represents the number of data blocks in each cluster,/->Representing different textsA third parameter corresponding to the type of the part, the third parameter being a parameter that maximizes a covariance between the theoretical compression rate and the actual compression rate, wherein +.>Representing a second set of parameters.

Wherein,may refer to that the file type of the data to be stored is T _a And a parameter that maximizes the covariance between the theoretical compression rate and the actual compression rate. />May be a value obtained by the user from an actual test. />May be pre-stored in the electronic device.

It will be appreciated that the number of components,may be a value obtained by the user from an actual test. />Can be pre-stored in the electronic device,/for example>May be pre-stored in the electronic device.

It should be appreciated that when the number of bytes of data stored in one data block is 4096, the first formula may be:

When the first value needs to be determined according to the first parameter set, the second parameter set and the first formula, the pre-stored value can be calledAnd->Substituting the first numerical value into a first formula to obtain a first numerical value.

For example, each parameter in the first parameter set and each parameter in the second parameter set may be iterated in the first formula, so as to obtain N corresponding to the maximum value in the plurality of F as the first numerical value.

Exemplary cluster values include [4,8, 16, 32]Compression ratio when the clustering value is 4And taking the value 4 of the clustering value into a first formula to obtain F ₄ The method comprises the steps of carrying out a first treatment on the surface of the Compression ratio when the clustering value is 8And taking the value 8 of the clustering value into a first formula to obtain F ₈ The method comprises the steps of carrying out a first treatment on the surface of the Compression ratio at 16 value of clustering valueAnd taking the value 16 of the clustering value into a first formula to obtain F ₁₆ The method comprises the steps of carrying out a first treatment on the surface of the The compression ratio at the clustering value of 32 is +.>And taking the value 32 of the clustering value into a first formula to obtain F ₃₂ The method comprises the steps of carrying out a first treatment on the surface of the Then from F ₄ 、F ₈ 、F ₁₆ And F ₃₂ Selecting the value of the clustering value corresponding to the maximum value as the first value.

For example F ₁₆ Is F ₄ 、F ₈ 、F ₁₆ And F ₃₂ The largest value of the data blocks is 16, namely the first value is 16, when the data to be compressed is compressed.

And S103, carrying out data compression on the data to be compressed based on the first numerical value to obtain compressed data.

In general, when data is compressed, three compression parameters, that is, a compression algorithm type, a value of a clustering value, and a compression level, need to be read from a file system, and then data compression is performed based on the three parameters. However, in the data compression method provided in the embodiment of the present application, the value of the clustering value is not a preset value, but is obtained according to the file type and the first data amount. Therefore, when data compression is performed on the data to be compressed, two compression parameters, namely a compression algorithm type and a compression level, can be read from the file system, then a first value obtained according to the file type and the first data volume is used as a value of the clustering value, and then the data to be compressed is subjected to data compression, so that the compressed data is obtained.

The compression algorithm types may include LZO algorithm, LZ4 algorithm, ZSTD algorithm, and lzonle algorithm, among others.

The LZO algorithm refers to a Lempel-Ziv-Obenumer algorithm, which is a lossless algorithm, and the reference implementation program is thread safe.

The LZ4 algorithm is a fast compression algorithm suitable for compression and decompression applications requiring high performance. The LZ4 algorithm has the main advantages of high compression and decompression efficiency, high compression speed, particularly high decompression speed, and small occupation of a central processing unit (Central Processing Unit, CPU).

The ZSTD algorithm is a real-time coding algorithm, can compress data more quickly and more efficiently when processing big data, and can effectively improve the data compression rate and simultaneously consider the compression speed compared with other algorithms.

The lzonle algorithm is an improved algorithm to the LZO algorithm, wherein RLE refers to Run-length coding, and the lzonle algorithm can provide better compression performance on the basis of keeping the compression ratio of the LZO algorithm.

Optionally, performing data compression on the data to be compressed based on the first value, where obtaining the compressed data may include:

and step 1, determining the number of data blocks in each cluster based on the first numerical value.

And 2, performing de-duplication processing on the data of the data blocks in each cluster to obtain compressed data.

The process of de-duplicating the data of the data block in each cluster is similar to the process shown in fig. 1 (a) and fig. 1 (b), and will not be repeated here.

The data compression method includes the steps of obtaining a file type of data to be compressed and a first data amount of the data to be compressed, determining a first value of the data to be compressed according to the file type and the first data amount, and performing data compression on the data to be compressed based on the first value to obtain compressed data, wherein the clustering value refers to the number of data blocks in each cluster when the data are clustered, and the value (the first value) of the clustering value is determined according to the first data amount of the data to be compressed of the file type and is a value of the clustering value which is matched with the data to be compressed, so that when the value of the clustering value is the first value, the data compression rate of each cluster of the data to be compressed and the duty ratio of the data not to be compressed in the data to be compressed can be balanced, and the overall compression rate of the data to be compressed is improved.

It should be understood that, although the steps in the flowcharts in the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least a portion of the steps in the flowcharts may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order in which the sub-steps or stages are performed is not necessarily sequential, and may be performed in turn or alternately with at least a portion of the sub-steps or stages of other steps or other steps.

It will be appreciated that in order to achieve the above-described functionality, the electronic device comprises corresponding hardware and/or software modules that perform the respective functionality. The steps of an algorithm for each example described in connection with the embodiments disclosed herein may be embodied in hardware or a combination of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may implement the described functionality using different approaches for each particular application in conjunction with the embodiments, but such implementation is not to be considered as outside the scope of this application.

The embodiment of the application may divide the functional modules of the electronic device according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one module. It should be noted that, in the embodiment of the present application, the division of the modules is schematic, which is merely a logic function division, and other division manners may be implemented in actual implementation. It should be noted that, in the embodiment of the present application, the names of the modules are schematic, and the names of the modules are not limited in actual implementation.

Fig. 6 is a schematic structural diagram of a data compression device according to an embodiment of the present application.

It should be understood that the data compression apparatus 600 may perform the data compression method shown in fig. 4; the data compression apparatus 600 includes: an acquisition unit 610 and a processing unit 620.

The acquiring unit 610 is configured to acquire a file type of data to be compressed and a first data amount of the data to be compressed;

the processing unit 620 is configured to determine a first value of data to be compressed according to the file type and the first data amount, where the clustering value refers to the number of data blocks in each cluster when the data is clustered;

the processing unit 620 is configured to perform data compression on the data to be compressed based on the first value, to obtain compressed data.

In one embodiment, the processing unit 620 is specifically configured to obtain a first parameter set according to a file type of the data to be compressed, where the first parameter set includes a plurality of first parameters, and the first parameters are used to indicate a data compression rate corresponding to one value of the cluster value; determining a second parameter set according to the first data volume, wherein the second parameter set comprises a plurality of second parameters, and the second parameters are used for indicating the data volume of uncompressed data corresponding to one value of the clustering value when the data volume of the data to be compressed is the first data volume; a first value is determined from the first parameter set and the second parameter set.

In one embodiment, the processing unit 620 is specifically configured to determine the first numerical value according to the first parameter set, the second parameter set, and a first formula, where the first formula includes:

wherein,represents a first set of parameters, m represents the number of bytes of data stored in one data block, S represents a first amount of data, N represents the number of data blocks in each cluster,/->Representing a third parameter corresponding to different file types, the third parameter being a parameter that maximizes the covariance between the theoretical compression rate and the actual compression rate, wherein +.>Representing a second set of parameters.

In one embodiment, the processing unit 620 is specifically configured to iterate in the first formula by using each parameter in the first parameter set and each parameter in the second parameter set, so as to obtain N corresponding to the maximum value in the plurality of F as the first numerical value.

In one embodiment, the number of bytes of stored data m in a data block is 4096.

In one embodiment, the cluster value comprises a value of 2 ⁿ Wherein n is a positive integer.

In one embodiment, the cluster values range from [4,8, 16, 32].

In one embodiment, the processing unit 620 is specifically configured to determine the number of data blocks in each cluster based on the first value; and carrying out de-duplication processing on the data in the data blocks in each cluster to obtain compressed data.

The data compression device provided in this embodiment is configured to perform the data compression method in the foregoing embodiment, and the technical principles and technical effects are similar and are not described herein again.

The data compression device 600 is embodied as a functional unit. The term "unit" herein may be implemented in software and/or hardware, without specific limitation.

For example, a "unit" may be a software program, a hardware circuit or a combination of both that implements the functions described above. The hardware circuitry may include application specific integrated circuits (application specific integrated circuit, ASICs), electronic circuits, processors (e.g., shared, proprietary, or group processors, etc.) and memory for executing one or more software or firmware programs, merged logic circuits, and/or other suitable components that support the described functions.

Thus, the elements of the examples described in the embodiments of the present application can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

Fig. 7 shows a schematic structural diagram of an electronic device provided in the present application. The dashed line in fig. 7 indicates that the unit or the module is optional. The electronic device 700 may be used to implement the data compression method described in the method embodiments above.

The electronic device 700 includes one or more processors 701, which one or more processors 701 may support the electronic device 700 to implement the data compression method in the method embodiments. The processor 701 may be a general-purpose processor or a special-purpose processor. For example, the processor 701 may be a central processing unit (central processing unit, CPU), digital signal processor (digital signalprocessor, DSP), application specific integrated circuit (application specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA), or other programmable logic device such as discrete gates, transistor logic, or discrete hardware components.

The processor 701 may be used to control the electronic device 700, execute a software program, and process data of the software program. The electronic device 700 may further comprise a communication unit 705 for enabling input (reception) and output (transmission) of signals.

For example, the electronic device 700 may be a chip, the communication unit 705 may be an input and/or output circuit of the chip, or the communication unit 705 may be a communication interface of the chip, which may be an integral part of a terminal device or other electronic device.

For another example, the electronic device 700 may be a terminal device, the communication unit 705 may be a transceiver of the terminal device, or the communication unit 705 may be a transceiver circuit of the terminal device.

The electronic device 700 may include one or more memories 702 having a program 704 stored thereon, the program 704 being executable by the processor 701 to generate instructions 703 such that the processor 701 performs the impedance matching method described in the above method embodiments according to the instructions 703.

Optionally, the memory 702 may also have data stored therein. Alternatively, processor 701 may also read data stored in memory 702, which may be stored at the same memory address as program 704, or which may be stored at a different memory address than program 704.

The processor 701 and the memory 702 may be provided separately or may be integrated together; for example, integrated on a System On Chip (SOC) of the terminal device.

Illustratively, the memory 702 may be used to store a related program 704 of the data compression method provided in the embodiments of the present application, and the processor 701 may be used to invoke the related program 704 of the data compression method stored in the memory 702 when performing data compression, to execute the data compression method of the embodiments of the present application; comprising the following steps: acquiring a file type of data to be compressed and a first data volume of the data to be compressed; determining a first value of data to be compressed according to the file type and the first data volume, wherein the clustering value refers to the number of data blocks in each cluster when the data are clustered; and carrying out data compression on the data to be compressed based on the first numerical value to obtain compressed data.

The present application also provides a computer program product which, when executed by the processor 701, implements a data compression method as described in any of the method embodiments of the present application.

The computer program product may be stored in the memory 702, for example, the program 704, and the program 704 is finally converted into an executable object file capable of being executed by the processor 701 through preprocessing, compiling, assembling, and linking.

The present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a computer, implements a data compression method according to any of the method embodiments of the present application. The computer program may be a high-level language program or an executable object program.

Such as memory 702. The memory 702 may be volatile memory or nonvolatile memory, or the memory 702 may include both volatile and nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory. The volatile memory may be random access memory (random accessmemory, RAM) which acts as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous DRAM (SLDRAM), and direct memory bus RAM (DR RAM).

In the present application, "at least one" means one or more, and "a plurality" means two or more. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b, or c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or plural.

It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of the units is only one logic function division, and other division modes can be adopted in actual implementation; for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.

The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of data compression, comprising:

determining a first value of the data to be compressed according to the file type and the first data volume, wherein the first value refers to a value of a clustering value when the data to be compressed is clustered, and the clustering value refers to the number of data blocks in each cluster when the data is clustered;

2. The method of claim 1, wherein determining the first value of the data to be compressed based on the file type and the first data value comprises:

acquiring a first parameter set according to the file type of the data to be compressed, wherein the first parameter set comprises a plurality of first parameters, and the first parameters are used for indicating the data compression rate corresponding to one value of the clustering numerical value;

determining a second parameter set according to the first data volume, wherein the second parameter set comprises a plurality of second parameters, and the second parameters are used for indicating the data volume of uncompressed data corresponding to one value of the clustering value when the data volume of the data to be compressed is the first data volume;

the first value is determined from the first set of parameters and the second set of parameters.

3. The method of claim 2, wherein said determining said first value from said first set of parameters and said second set of parameters comprises:

determining the first value according to the first parameter set, the second parameter set and a first formula, the first formula comprising:

wherein, Representing the first parameter set, m representing the number of bytes of data stored in one data block, S representing the first data amount, N representing the number of data blocks in each cluster, B _Ta Representing a third parameter corresponding to different file types, wherein the third parameter refers to a parameter which enables the theoretical compression rate to be equal to the actual compression rateA maximum parameter of inter-covariance, wherein +.>Representing the second set of parameters.

4. A method according to claim 3, wherein said determining said first value from said first set of parameters, said second set of parameters and a first formula comprises:

and iterating in the first formula by adopting each parameter in the first parameter set and each parameter in the second parameter set to obtain that N corresponding to the maximum value in a plurality of F is the first numerical value.

5. The method of claim 3 or 4, wherein m is 4096.

6. The method of any one of claims 2 to 4, wherein the clustering value comprises a value of 2 ⁿ Wherein n is a positive integer.

7. The method of claim 6, wherein the clustering value ranges from [4,8, 16, 32].

8. The method according to any one of claims 1 to 7, wherein the data compressing the data to be compressed based on the first value to obtain compressed data includes:

Determining a number of data blocks in each cluster based on the first value;

and carrying out de-duplication processing on the data in the data blocks in each cluster to obtain compressed data.

9. An electronic device comprising means for performing the method of any one of claims 1 to 8.

10. An electronic device, comprising:

one or more processors;

a memory;

and one or more computer programs, wherein the one or more computer programs are stored on the memory, which when executed by the one or more processors, cause the electronic device to perform the method of any of claims 1-8.

11. A chip system, characterized in that the chip system comprises a processor for calling and running a computer program from a memory, so that an electronic device on which the chip system is installed performs the method according to any of claims 1 to 8.

12. A computer readable storage medium comprising a computer program, characterized in that the computer program, when run on an electronic device, causes the electronic device to perform the method of any one of claims 1 to 8.